Automatic Defects Segmentation and Identification by Deep Learning Algorithm with Pulsed Thermography: Synthetic and Experimental Data

Infrared thermography is used for evaluating composite materials due to the properties of low cost, fast inspection of large surfaces. The application of deep neural networks tends to be a prominent direction in the Infrared Non-Destructive Testing. During the training of the neural network, the Achilles heel is the database. The collection of huge amounts of training data is the high expense task. In Non-Destructive Testing with deep learning, the synthetic data contributing to training in infrared thermography remains unexplored. In this paper, synthetic data from the standard Finite Element Models is combined with experimental data to build repositories with Mask-RCNN to achieve defect segmentation


Introduction
Quality evaluation is playing a fundamental role for modern industrial production and manufacturing processing.The demand for inspection of materials with respect to the possible presence of defects, damage and flaws has increased due to the wide use of composite materials and metals in industry.The method used to inspect defects via manual, visual evaluation involving humans can be hampered by the human fatigue.In order to meet the need of high-quality production and maintain the stringent high-quality level, an advanced visual inspection system is becoming more and more essential for structural health monitoring and production lines application.Automated quality control [1] can be applied in the industrial field to facilitate the consistent and efficient inspection.The high inspection rate and the inspection procedure without human involvement are the main advantages of automated inspection system [2].
Non-Destructive Testing (NDT) [3] encompasses a group of inspection techniques that are intended to assess the integrity of an object's interest without causing any kind of modification or permanent damage.Infrared Testing (IRT) is an NDT technique in which the differences in thermo-physical properties between the inspected object material and possible surface and/or subsurface defects are exploited through the use of an infrared camera.However, automatically identifying defects in materials via infrared thermography still remains a fresh and challenging task in the image processing domains.
In this research, an automatic instance segmentation and identification algorithm in deep learning (Mask-RCNN) [4] is introduced for defect segmentation in an automatic infrared detection system.The defect detection procedure can be regarded as either an object detection task [5] or object segmentation task [6].In the object detection task, the objective is to fit the bounding box localized around each defect in the image.In the object segmentation task, the objective is based on pixel level classification to distinguish each pixel if it is detected.Compared with the earlier segmentation strategies [7][8], the Instance segmentation associated each pixel of an image with an instance label [9].It can predict a whole segmentation mask for each of those objects and predict which pixel in the input image corresponds to each object instance.It also reduces the restriction concerning the position of defects rather than predicting a group of bounding boxes for the defects.
Meanwhile, like any other learning approach, deep learning's Achilles heel is the training dataset.It must be accurate enough and contain enough images and exceptions (e.g.occlusions) so as to allow the algorithm to learn reliable features of interest.If the dataset does not contain enough images and cases of possible situations or if the data contains too much noise or artifacts, the algorithm may not be accurate enough or even may learn to identify the wrong features.In recent years, the exploitation of synthetic data [10] during the training and validation of deep neural networks has emerged as a popular topic.In this work, synthetic data generated with Finite Element Models (FEM) is used during the training process to greatly reduce the high expense from real experiments in infrared thermography.The synthetic data generated 2 by the simulator is based on the same parameters with real specimens in non-realistic ways being able to force the neural network to learn the essential features of the object of interest [11].The generated features from synthetic data can cover or overlap of the amount of data distribution that are not sufficiently represented in the original raw dataset [12].The results also show the possibility of using inexpensive synthetic data for the deep neural network training but avoiding the necessity of collecting large amounts of hand-annotated raw data.
Therefore, instance segmentation of defects for infrared thermography achieved by a deep spatial characteristic model (Mask-RCNN) will train with a small amount of experimental data with synthetic data in this research.The remainder of the paper is structured as follows: section 2 provides the pulsed thermography and related experimental set-up indication.Section 3 gives a detailed explanation of the Mask-RCNN defects detection system and the synthetic data generation pipeline.Section 4 describes the main features and dataset for evaluation.Section 5 provides the experimental results and implementation stages.Section 6 concludes this paper.

Thermophysical Consideration
In pulsed thermography (PT), the surface of the inspected specimen is exposed to a heat pulse using an energy source such as photographic flashes as indicated in Figure 1.A heat pulse can be represented as the combination of several periodic waves having different frequencies and amplitudes.After the thermal front touches the surface of the specimen, a thermal front travels from the surface through the specimen.As the time elapses, the normal path of the thermal front changes due to the presence of the subsurface discontinuity, and lead to the thermal contrast at the specimen's surface, which can be monitored with an IR camera.The one-dimensional solution of the Fourier equation of conduction for a Dirac heat pulse propagating through a semi-infinite isotropic solid has the form [13]: where ܳ ሾ‫݉/ܬ‬ ଶ ሿis the energy absorbed from the surface, the initial temperature is the ܶ ሾ‫ܭ‬ሿ.‫‬ ሾ݇݃/݉3ሿ its density.The specific heat at constant pressure is ܿሾ‫ܭ݃݇/ܬ‬ሿ .‫ܭ‬ሾܹ/݉‫ܭ‬ሿ is the thermal conductivity that indicated the energy transformation rate by the materials.ߙ [݉ ଶ /s] is the thermal diffusivity to measure the material ability to conduct heat ܼ ሾ݉݉ሿ is the depth of the defect.At the surface (z=00 mm), the temperature evolution can be written as follows [14]: From eq. ( 2), it can be concluded that the temperature evolution at the surface following a Dirac heat pulse follows a monotonous decrease with a slope of -1/2 for areas without defects, whilst areas with defects will diverge more or less from this behavior depending on their actual thermo-physical properties.

Automatic defect segmentation strategy
As indicated in Figure 2, a detection system training with synthetic and experimental data has been proposed to segment and identify defects in thermal images.This design of detection system is based on the Mask R-CNN network.The infrared thermal sequences are acquired by the pulsed thermography (PT) system.Then, both the raw and synthetic data are pre-processed with the bilateral filtering and the thresholding technique.Therefore, two different characteristic datasets (raw data; synthetic data) are generated respectively, to be trained with Mask-RCNN in different combination cases.In the final step, anormal areas, defects, delamination are identified through the deep region convolutional neural network and then visualized with the bounding box or instance segmentation map.

Mask-RCNN
Convolutional Neural Networks are utilized as a supervised feature extractor to analyse the defect localization and segmentation from the materials.The Region based-CNN (R-CNN) [15] move towards the object bounding box detection method based on each ROI.Faster-RCNN [5] have further advanced the network via learning the attention mechanism with a Region Proposal Network (RPN).Mask R-CNN [4] extends from Faster-RCNN through a mask branch construction.It is an instance segmentation algorithm, which recognizes object boundaries at the pixel level and designs pixel-wise segmentation with alignment of pixel-to-pixel among the input and output.Mask R-CNN adopts the same twostage with Faster-RCNN architecture.During the first stages, the input images were scanned and the proposals were generated via the Region Proposal Network (RPN).Then, specifically, in the second stage of detection, Mask-RCNN parallel adds a binary mask regarding each of the ROI.Mask R-CNN is intended to address the instance segmentation task and aim at adjusting the plentiful hyper-parameters from the neural network.This model predicts the certain number of bounding boxes for the defects and meanwhile segments each defect region within bounding boxes.The architecture is shown in Fig. 3

Synthetic Data Generation Pipeline
Synthetic data is manufactured information rather than data collected from real word events [10].The main application involves using synthetic data as a training dataset.Through synthetic data, we can acquire basically unlimited training data based on modifying and creating new variables and features on individual data cases.In synthetic data generation for instance labeling (Mask-RCNN, etc.), one major benefit of using synthetic data is having excellent labels due to the place of objective precisely are set by a human, not only the case in bounding boxes but also each pixel instance.
Finite Element Modelling (FEM; COMSOL, etc.) has become an economical platform to evaluate the thermal response of pulsed thermography.It is also intended for predicting and stimulating the results of Thermal NDT (nondestructive testing) experiments and allows the solving of a three-dimensional heat conduction problem of heating that contains subsurface defects.In Fig. 4, a synthetic thermal image (a) shows the pulsed thermography result of a plexiglass synthetic sample from FEM stimulation at t=106.5 s after heating, which indicates defects more distinctly compared to the corresponding experimental frame at t= 106.5 s indicated in (b).The dimensions are 30cm×30cm.In Fig. 4(c), 25 artificial defects were inserted at increasing depths in either circles or square shapes as indicated in the figure.This Plexiglass sample was designed with back-drilled defects.A good correspondence temperature of the region is observed which verifies that synthetic modelling in FEM is in outstanding accordance with the real experiment [16].The detailed implementation procedure for using deep neural networks with FEM is described in Fig. 5.The training of synthetic data generated by FEM, provides clean and automated supervision data compared to the other applications [17][18], where most other physical software is highly non-linear and has uncertainty issue with respect to quantification.In addition, the learning process can be abundantly simplified due to the thermal principle of generated data which are governed by the same type of equations.Specific feature positioning and extraction for invisible defect regions can be referenced from the synthetic data.

Automatic Pre-processing stage
The main purposes of the preprocessing stage are to make defects more distinguishable from the cluttered background of the thermal sequence and to normalize the different noising conditions.The bilateral filtering and thresholding technique) were adapted from [19] for the preprocessed stage in this work so as to be the reference database.In step 1, the bilateral filtering removes the possible existing noise in the thermal image, while it preserves the edges of possible objects.Secondly, two thermal criteria are applied in a thermal image sequence acquired from Pulsed Thermography: 1. 1st thermal criterion: automatically determine the time instant just before the appearance of the first thermal footprint belonging to a defect (reference point); 2. 2nd thermal criterion: automatically find the intersection point between the overlap of each of the thermal image histograms after the reference point, and the thermal image histogram corresponding to the reference point.In the end, the pixel values above the corresponding intersection point are equal to this value.Pixel values corresponding to the thermal images after the reference point are multiplied by a scale in (4).The final result of this method is believed to be fitted with the convolutional neural network stage thereafter.8, which includes the shape of circle, multilateral; square, rectangle; multi-angle of defect instances.In order to label each defect region on the images, we introduced the Labelme Image Annotator [20] to give a simple and standard manual annotation for all of the images.

Evaluation metrics
Average precision (AP) is introduced to evaluate the performance of the detection model, which is analyzed by (5)(6).
where TP is the true positive rate.FP is the false positive rate which indicated that defects are the non-defects regions which are falsely detected as a defect.FN is the false negative rate which means a certain number of the defects which failed to be detected during this experiment.

Main Results analysis and training
The operating system is set as: Ubuntu 14.04; CPU: i7-5930k; Memory: 64GB.Each training processing was conducted on GPU (NVIDIA GeForce GTX 1080Ti) and required approximately 30 min.Some main hyper parameters and training parameters are set as below: 1.The Network training used Resnet101 as backbone; 2. The learning momentum is 0.9 and learning rate is 0.0003; 3. The first 30 epochs of were trained on network heads, then all network layers were trained for 1030 epochs, the model weight (in .h5format) was used of COCO (could be replaced with original weight model file).This could be improved if a similar binary segmentation weight model was used; 4. The weight decay is 0.0001 and mini mask size is 56*56.As shown in Fig. 9 above, the segmentation results of the Mask-RCNN model are indicated.The segmented results in the middle column were the validated results from the model trained on raw images after pre-processed (Database A, C).The segmented results in the right column were the validated results from the model trained on raw images merged with synthetic images after pre-processed (Database B, D).Meanwhile, in the first three rows, the results from Plexiglas specimens (Validation Database E) are shown.The last two rows show the results for CFRP specimens (Validation Database F).The segmented defects from obtained images clearly show the improvement when trained on raw images merged with synthetic images after pre-processed in Fig. 9.
In addition, we can see the segmented results from the last column in Fig. 9.When the model is trained with raw images (in the middle column), then the model is able to detect 12 defects out of 25 defects in the whole specimen with limited raw images during the validation.Each detected defect is covered by a solid mask.It is notable that the model produces one false positive defect case by highlighting one non-existing defect on the image and 13 false negative defects.Further, in right column, when the training of the databases was increased with synthetic data, the trained model is able to detect 17 defects in total.However, four false positive and false negative cases still appear on the image.During the training of database (B, D), Mask-RCNN has seen limited pre-processed raw thermal images (100 images) merging with pre-processed synthetic data (beyond pretraining of the early layers on COCO [22]).The mAP values in the mixed database (B, D) are higher than databases (A, C) (B (76.40)>A (68.66);D (72.68) >C (70.29)).From the plot of the PR curves, database B achieves higher precision than database A consistently for all of the recall values in the Mask-RCNN model.On the other hand, database D consistently obtained lower precision than database C between the recall values from 0.5 to 0.55.This helps to explain that the Plexiglas databases obtained the higher mAP difference than CFRP between the training of raw images databases and the training from databases merging with synthetic data.
In each ROI, this experiment obtained better results with the performance metric of precision and recall values compared with another cited deep learning IRT project [21].Especially, the Plexiglas obtained higher performance of mAP from Mask-RCNN when it merged with synthetic data for training compared with the highest mAP value obtained from previous research.This compelling performance also illustrates the ability of synthetic data to bridge the reality gap and demonstrates that merging with synthetic data for training can improve the accuracy in both types of material (CFRP; Plexiglas).As a result, the training model merged with synthetic images could be able to outperform databases with pure raw thermal databases in Deep learning IRT projects.

c) Discussion
In this project, we used Resnet101 as a feature extractor with weights initialized from COCO [22].Since the weights from COCO obtained the features trained by a group of datasets that contain basic lines and shapes of the objectives (circle; square; rectangle), it can be beneficial to the defect detection to a certain degree even without training with thermal images.So, the selection of beneficial initial weights from the pretrained models could be a key training strategy to apply for the future direction of research in Deep learning IRT.
Synthetic data can more easily to generate a dataset with a large amount of variety.Small raw training databases merging with synthetic data can help to detect most of the defects successfully.However, based on the results, there is still some place for improvement.The defects from synthetic data (COMSOL) may still have blurry boundaries and unclear shapes that influence the training results from learning.
Besides, Mask-R-CNN is being robust to the limited size of synthetic data and more likely overfit unless the data increases the augmentation (rotating; flipping images or random gaussian noise). 10.21611/qirt.2020.012

Conclusion and future work
In summary, we present a method where the deep learning architecture is combined with a small amount of synthetic thermal data for defect segmentation.The proposed architecture is based on a Mask-RCNN that has been applied in natural image segmentation.We collected our thermal database from the experimental results and synthetic data in the Finite Element Model (COMSOL).The network yields a better performance when it is fused with synthetic data for training.Meanwhile, different types of composite materials (CFRP; Plexiglas) and defects will be validated via proposed methods to reveal the performance.The proposed algorithm-Mask-RCNN without modification in line with the main objective in this work is to assess whether synthetic data can improve the detection accuracy of defects on the CFRP or Plexiglas composite samples.Quantitative analysis through diameter/depth ratio in defect detection with modified Mask-RCNN network will be carried out in the future work.

Fig. 7 .
Fig. 7. Scheme of preprocessing stage 10.21611/qirt.2020.012Four groups of training databases are created based on the images we collected in the previous steps.The description is explained as below.1.Database A, C: (Original database)100 raw thermal images from thermal sequences with corresponding instants; 2.Database B, D: (Mixed database) 100 raw thermal images with 100 new synthetic images; both selected from the same corresponding instants in the thermal sequences; All thermal images described in four training databases (Database A; B; C; D) are preprocessed by the method in section 3.3 in order to enhance the defect contrast.In the end, two independent pure raw thermal databases without preprocessed (each 50 images) (E; F) were validated by the trained model (Mask-RCNN) to obtain the actual results, which have different shapes and depths of defects compared with the sequences during training.For the sake of consistency, the pure raw thermal Plexiglas database E was validated on the learning model trained on Plexiglas (A, B) independently.Then, the pure raw thermal CFRP database F was validated on the learning model trained on CFRP (C, D).The defect labels and instance are shown in Fig.

Fig. 9 .
Fig. 9.The best obtained validation results of Mask-RCNN segmentation on different training database.From left to right: original images, training on the pre-processed raw images database, training on the mixed database (preprocessed data from synthetic and raw images).From the first three rows to the last two rows: Plexiglas, CFRP

Fig. 10 .
Fig. 10.The Average Learning loss for two types of specimens: Plexiglas (a)(b); CFRP(c)(d) The learning loss indicated how well the Learning model performed.In this experiment, Fig.10 (a) and (b) showed the acquired average loss curve of the training and validation on Database A, B. The loss has a certain range of oscillation during the first 800 epochs in both Fig (a) and (b), then decreases less rapidly after 800 epochs until it flattens out.The cost curve is decreasing as the batch number increases and converges approximately from 0.25.Correspondingly, Figure (c) and (b) indicated the learning curve with training and validation on the Database C, D. The learning curve of the training and validation for Database D became more stable compared with the curves in Fig.10 (c).Although the validation loss from Figure (c) is instantly unstable around the point of 380th epoch, the learning curves of training and validation (Database C, D) eventually converges around 0.2 around the 1000th epoch.Meanwhile, the model accuracy of the validation from Database A, B, C, D in this experiment is respectively 0.990, 0.996,0.98738,0,98738.We obtained the model accuracy from four different training databases: B (0.996)>A (0.990)>C (0.987) =D (0.987).As a result, the obtained model accuracy from four groups of databases is impressive and shows the good performance during the training and validation when the Mask-RCNN model is used for defect segmentation of Plexiglas and CFRP specimen in this project.

Fig. 11 .
Fig. 11.Different detection results with 4 groups of Datasets (2 types of materials) Precision-recall curves for the Mask-RCNN architectures trained on 4 different groups of databases are shown in Fig. 11.This figure indicated two PR curves for training sets of Plexiglas specimens (pre-processed raw images database (A); synthetic images merging with raw images database from pre-processed (B)); an independent pure raw images validation set without preprocessed (E)) and two PR curves for training sets of CFRP specimens (pre-processed raw images database (C); synthetic images merging with raw images database from pre-processed (D); an independent pure raw images validation set without preprocessed (F)).The mean average precision (mAP) values in the database A, B, C, D respectively reach 68.66; 76,40; 70.29; 72.68 as shown in Fig,11.During the training of database (B, D), Mask-RCNN has seen limited pre-processed raw thermal images (100 images) merging with pre-processed synthetic data (beyond pretraining of the early layers on COCO[22]).The mAP values in the mixed database (B, D) are higher than databases (A, C) (B (76.40)>A (68.66);D (72.68) >C (70.29)).From the plot of the PR curves, database B achieves higher precision than database A consistently for all of the recall values in the Mask-RCNN model.On the other hand, database D consistently obtained lower precision than database C between the recall values from 0.5 to 0.55.This helps to explain that the Plexiglas databases obtained the higher mAP difference than CFRP between the training of raw images databases and the training from databases merging with synthetic data.In each ROI, this experiment obtained better results with the performance metric of precision and recall values compared with another cited deep learning IRT project[21].Especially, the Plexiglas obtained higher performance of mAP from Mask-RCNN when it merged with synthetic data for training compared with the highest mAP value obtained from previous research.This compelling performance also illustrates the ability of synthetic data to bridge the reality gap and demonstrates that merging with synthetic data for training can improve the accuracy in both types of material (CFRP; Plexiglas).As a result, the training model merged with synthetic images could be able to outperform databases with pure raw thermal databases in Deep learning IRT projects.