Classification of Microscopic Laser Engraving Surface Defect Images Based on Transfer Learning Method

Round 1

Reviewer 1 Report

The paper looks interesting and quite well prepared although there is no breakthrough idea but a standard application of recent machine learning advances.

As a general comment, I must say that the structure of the paper does not reflect a usual research paper but it is possible to readjust the content in order to achieve a more comfortable structure for the reader.

Indeed, subsections 2.1 and 2.2 should be considered in a "Related work" section. The real proposed method starts in section 2.3 only. I suggest thus introducing section 2 "Relate works" putting as content subsections 2.1 and 2.2. Subsection 2.3 can be renamed Section 3. 

In the "Related Works" or even in the "Introduction" I suggest also mentioning other imaging technologies that can be helpful in the non-destructive control of electronic components besides the use of standard RGB images. I refer in specific to infrared imaging and multimodal technologies in which deep learning might have an impact. Here are some suggested recent references to broaden the state of the art:

• Hussain, Bilal, et al. "Thermal vulnerability detection in integrated electronic and photonic circuits using infrared thermography." Applied Optics 59.17 (2020): E97-E106.
• ​Lu, Hangwei, et al. "FICS-PCB: A Multi-Modal Image Dataset for Automated Printed Circuit Board Visual Inspection." IACR Cryptol. ePrint Arch. 2020 (2020): 366.
• Mallaiyan Sathiaseelan, Mukhil Azhagan, et al. "Why Is Deep Learning Challenging for Printed Circuit Board (PCB) Component Recognition and How Can We Address It?." Cryptography 5.1 (2021): 9.

Besides this enrichment, I suggest putting at the end of the new Section 2 "Related works" the contributions of the paper, to be reported as clear and easy understandable claims showing the value and originality of your work and the advances you proposed beyond the related works you have cited. It is mandatory to convey a clear but strong message to show that the paper deserves to be first published and then read by the audience.

Similarly in the discussion section, there are some tables that should be reported instead in the experimental results section. Please, anticipate the results and then discuss and compare them in section 4.

The direction for future work are not convincing. Elaborate more on this and name some future challenges you would like to address if any.

Are the dataset used and the software openly available? If not, why? 

Is there any image processing used e.g. in preprocessing microscope images before classification by the CNNs?

The quality of figures is somewhat scarce, especially because some of them are too small.  However, if a pdf is procured it might be suitable for online reading.

 

I collect some minor fixes in the list below.

• Line 1-14 revise English
• line 21 "by experiencer" -> "by experts" or "by experienced staff/users"
• line 57 "he first binary". Please revise. "He first binarizes"??
• line 92. "machine learning methods" -> "Machine learning methods". Capitalize.
• Line 101. SVMs were extended to non-linear spaces very soon in 1996. I would just cite extended SVM with the work by Vapnik and colleagues.
• Line 114 "deep learning methods"-> "Deep learning methods"
• Line 148 "cors-validation were"-> "cross-validation was"
• Line 164-165. Revise English and rephrase (too many fragments starting with "to": to simulate, to collect, to establish".
• Caption to figure 3. Elaborate more on the way images were collected (lenses, illumination, image size in pixel and real-world units)
• Line 171. "Imagenet .... included 1.2M images". Elaborate more on what is imagenet(i.e. an imaged database) and made explicit that some of your models came pre-trained on that database.

Author Response

Dear Reviewer:

Thank you for your letter and for the reviewers’ comments concerning our manuscript entitled “Classification of Microscopic Laser Engraving Surface Defect Image Based on Transfer Learning Method” (ID:1335512). Those comments are all valuable and very helpful for revising and improving our paper, as well as the important guiding significance to our researches. We have studied comments carefully and have made corrections which we hope to meet with approval. Revised portion are highlighted in red in the paper. The main corrections in the paper and the responds to the reviewer’s comments are as flowing:

 

Reviewer #1:

1. Response to comment: Indeed, subsections 2.1 and 2.2 should be considered in a "Related work" section. The real proposed method starts in section 2.3 only. I suggest thus introducing section 2 "Relate works" putting as content subsections 2.1 and 2.2. Subsection 2.3 can be renamed Section 3. 

We would like to thank Reviewer for the constructive comments. Considering the Reviewer’s suggestion, we modified the structure of the article. We put Section 2.1 and section 2.2 in the section 2 “Related Works”, and then renamed section 2.3 as Section 3 which describes “Proposed Method”. 

2. Response to comment: In the "Related Works" or even in the "Introduction" I suggest also mentioning other imaging technologies that can be helpful in the non-destructive control of electronic components besides the use of standard RGB images. I refer in specific to infrared imaging and multimodal technologies in which deep learning might have an impact. Here are some suggested recent references to broaden the state of the art:

• Hussain, Bilal, et al. "Thermal vulnerability detection in integrated electronic and photonic circuits using infrared thermography." Applied Optics 59.17 (2020): E97-E106.
• Lu, Hangwei, et al. "FICS-PCB: A Multi-Modal Image Dataset for Automated Printed Circuit Board Visual Inspection." IACR Cryptol. ePrint Arch. 2020 (2020): 366.
• Mallaiyan Sathiaseelan, Mukhil Azhagan, et al. "Why Is Deep Learning Challenging for Printed Circuit Board (PCB) Component Recognition and How Can We Address It?." Cryptography 5.1 (2021): 9.

Thank you for pointing this out. We referred to these articles to introduce different imaging technologies and broaden the state of the art at the end of the “introduction” section (Line 84-92) as follows:

Bilal [1] presented an infrared assisted thermal vulnerability detection technique, which applied affine transform to multimodal image fusion. It can accurately locate the hot center and obtain the spatial information of the component. The technology pro-vided a reliable automated identification of hot spots in the system. Lu [2] proposed a publicly available multimodal printed circuit board dataset, FICS-PCB, for automated visual inspection of printed circuit boards. Mukhil [3] proposed an electronic component localization and detection network to detect the defects of PCB components, and finally achieved a good performance of 87.2% accuracy and 98.9% recall.

In this laser engraving surface defect image classification project, metallurgical microscope is most commonly used in actual industrial scenes to collect image dataset. For this particular industrial scenario, infrared imaging and multi-modal technology are not common in actual industrial applications. However, the suggestion of the reviewer gives us new inspiration in data collection. In the future, it is promising to collect data set using infrared imaging and multimodal technologies, which can enrich the diversity of the data and may improve the generalization and accuracy of the model.

3. Response to comment: Besides this enrichment, I suggest putting at the end of the new Section 2 "Related works" the contributions of the paper, to be reported as clear and easy understandable claims showing the value and originality of your work and the advances you proposed beyond the related works you have cited. It is mandatory to convey a clear but strong message to show that the paper deserves to be first published and then read by the audience.

Thanks for the comment. We added a paragraph to describe the contributions of this research at the end of the new section 2 “Related works” (Line 146-167).

The contribution of our work can be summarized in two aspects. Firstly, we innovatively use laser engraving microscope images to surface defect classification. We used the metallographic microscope to collect the image of the radium carving position in the mobile phone shell and labelled by quality inspection workers. We took pictures in different light fields to imitate actual production environment and different types of laser engraving surface. In this specific industrial application field, it is very meaningful to propose an accurately labeled and high-quality image data set that can be used for deep learning model training.

Secondly, we designed a high accuracy microscopic laser engraving surface defect detection algorithm through two comparison experiments using transfer learning methods, which greatly saves the cost of labeling and training time without compromising the performance of the model. We discussed the detection performance of various state-of-the-art CNN architectures (i, e, VGG19, ResNet50, DenseNet121 and InceptionV3 networks) under two different fine-tuning models. The results showed that deep fine-tuning models had better effect compared with shallow fine-tuning except VGG19. Then we made a comparison of the accuracy of deep learning methods and two machine learning detection methods. The best accuracy of 96.72% was achieved under deep learning methods. Besides, the proposed algorithm has been applied to production detection equipment and achieved good application results.

 

The remaining paper is organized as follows: Section 2 introduces the application of machine learning and deep learning in defect detection. Section 3 presents various convolutional network structures and experimental method used in this paper. Section 4 summarizes the experiments and discusses the results. Section 5 concludes our work and prospects the future research.

 

4. Response to comment: Similarly in the discussion section, there are some tables that should be reported instead in the experimental results section. Please, anticipate the results and then discuss and compare them in section 4.

It is really constructive as reviewer suggested that modify the structure of the article, we moved the related tables (table2 and table 3) from the “discussion” section to section 4 (i,e, the experimental results section).

5. Response to comment: The direction for future work are not convincing. Elaborate more on this and name some future challenges you would like to address if any.

Considering the Reviewer’s suggestion, we have elaborated this part and added possible future challenges in this field we are interested. (Line 335-342).

In this paper, due to the huge workload of collecting microscopic laser engraving surface images and labeling, we only performed binary classifications of whether the microscopic images were defective, and did not train multi-classification model to determine which type of defect the defective images have. In the future research, we will continue to collect and annotate images, and conduct multi-classification model training. At the same time, we will conduct experiments using few-shot learning and active learning technologies to reduce the cost of labeling while ensuring the accuracy of classification.

6. Response to comment: Are the dataset used and the software openly available? If not, why?

The dataset and the software in this paper were jointly developed by us and some cooperating companies, therefore they are available only on request due to privacy restrictions. The researchers who are interested in the dataset and the software are expected to write an application email to the corresponding author and sign an agreement that stipulates the terms of use.

7. Response to comment: Is there any image processing used e.g. in preprocessing microscope images before classification by the CNNs?

In this work, we used nearest neighbor interpolation transformation to adjust the image size to 224x224 before classification by the CNNs (line 195-196). The purpose of resizing the images is to save computation resources and training time, and to ensure CNNs make the maximum use of all the information of the image at the same time.

We also adopted data augmentation technique to preprocess microscope images before inputting them to the models, because the number of samples collected is small and the training of deep learning model requires a large amount of data, the data augmentation technology can efficiently expand the dataset size. Common data augmentation techniques include random noise, clipping, histogram equalization, random rotation and so on. Histogram equalization technology can enhance the effective information hidden in the image low contrast. In actual industrial production, the conductive bits on the mobile phone shell are often randomly placed in the image acquisition system, so the model needs to have rotation invariance. Therefore, histogram equalization and random rotation techniques were proposed to enhance the laser engraving microscope images.

8. Response to comment: The quality of figures is somewhat scarce, especially because some of them are too small. However, if a pdf is procured it might be suitable for online reading.

Thank you for the reminder. We increased the image resolution to 300 dpi and magnified the images in the manuscript.

9. Response to comment: Caption to figure 3. Elaborate more on the way images were collected (lenses, illumination, image size in pixel and real-world units)

Thank you for pointing this out. We added caption to Fig. 3.

We added descriptions (line 181-192) about the images collection methods as follows: The metallographic microscopy system used in this project to collect the microscopic images of laser engraving on the mobile phone shell is Aosvi professional metallographic microscope. The industrial camera is Aosvi M140 camera, which supports USB2.0 interface and has the highest resolution of 4096×3288, real-world units is 5.73um×4.60um. The resolution can make the surface of the laser engraving microscopic image clearly visible and achieve the required effect of classification. Metallographic microscope adopts falling illumination system, light source is 6V/20W halogen lamp, brightness is adjustable. According to the practical application of complex environment, such as different characteristics of light source, we use yellow and green different filter for experiment. In the process of laser engraving, regular small space units are formed on the metal surface due to high-energy laser irradiation., the diameter of these units is very small, so the metallographic microscope magnification for 200 times, meeting the demand of classification.

 

Author Response File: Author Response.pdf

Reviewer 2 Report

Microscopic laser engraving surface defect detection is important in industrial quality inspection field.

The disadvantage of traditional classification methods is that they are focused on the terms of feature extraction and classification independently.

The solution of this problem is the use of deep convolutional networks (dCNN), which integrate feature extraction and classification into self-learning. The disadvantage is that the dCCN require large datasets.

The training datasets for microscopic laser engraving image classification are small, therefore, the authors have used pre-trained dCNN models and applied fine-tuning strategies.

Through a large number of experiments, the authors have demonstrated the application of neural network to accurately classify the defect of laser engraving surface based on transfer learning. In addition, we evaluated the classification effect of four models on transfer learning.

 

I have some reviewer notes:

The paper is well structured. All of the results are presented correctly.

In Table 3. If it is possible, give more criteria about the work of classifiers. Only “accuracy” is not enough to know how the classifiers work.

It will be good to compare your results with those from other authors (in the DISCUSSION part).

Remove the reference source from the CONCLUSION part. You can include them in the DISCUSSION part.

Author Response

Point 1: Response to comment: In Table 3. If it is possible, give more criteria about the work of classifiers. Only “accuracy” is not enough to know how the classifiers work.


 

Response 1: We thank the reviewer for the constructive comment. In Table 3, we only compare the detection effects of different methods, mainly for illustration the better effect of the DenseNet121 network. A more comprehensive comparisons of classification performance of different networks, such as F1-Score, Recall, Precision and Accuracy are given in detail in Table 2.

 

Point 2: Response to comment: It will be good to compare your results with those from other authors (in the DISCUSSION part).

 

Response 2: Thank you for the comment. Since using laser engraving microscope images to perform surface defect detection task is a relatively new industrial application, we did not find relative literature to make comparisons. We are also aware of the gaps in the comparisons with the other papers, therefore, we added the comparison of the change of fine-tuning method, as shown in table 2, and the other network architectures in industrial application using machine learning and deep learning methods, as shown in table 3.

 

Point 3: Response to comment: Remove the reference source from the CONCLUSION part. You can include them in the DISCUSSION part.

 

Response 3: Considering the Reviewer’s suggestion, we removed the reference source [43-45] from the CONCLUSION part. Because the influence of the similarity of data sets on transfer learning has been explained in the front section, we do not cite these literatures again. 

Author Response File: Author Response.pdf

Round 2

Reviewer 1 Report

Thank you for having revised the manuscript and for having analytically replied to my comments. I do understand the problem of dataset sharing and your answer, although being negative, is satisfactory.

Author Response

Thank you for your comment.