Isolation Forests and Deep Autoencoders for Industrial Screw Tightening Anomaly Detection

Round 1

Reviewer 1 Report

The paper can be accepted in its current form.

Author Response

---- Notes for Reviewer #1 ----
We are grateful for your comments and suggestions (here in []).

[The paper can be accepted in its current form.]

We thank the positive feedback.

Reviewer 2 Report

In this work, the detection of abnormal screw tightening processes is studied. Two unsupervised machine learning algorithms (iForest and a deep learning autoencoder) are compared with a local outlier factor method and a supervised Random Forest firstly, then iForest and AE are further compared by adopting a more recent and larger dataset. Besides, an interactive visualization tool for human operators is developed to provide explainable artificial intelligence knowledge. As there is few research on the abnormal screw tightening processes using machine learning methods, this paper is overall interesting. 

As a research paper, it can be improved, such as 

1. Authors mainly introduced the process, but not the scientific aspect of the problem. Is it a pattern recognition problem? Is it similar to other industrial application? Where is the most difficult part in this problem?
2. Authors listed many works in this area, but didn't give any comments on their works? At least authors shall mention their contribution and limitation, and how their works related to proposed work in this manuscript. 
3. In table 1 authors gave four types of data available in the studied process, but only picked angle-torque data for their study. Authors shall provide the reason to give up other data. Is it due to the limitation of method or enough information included in angle-torque data?
4. Angle-torque data for each operation will change along time scale. Did author considered the trajectory info in their study? It is possible that few study has been conducted on  industrial screw tightening, but many similar studies are available in literature. Authors can find batch process monitoring, which is very similar to studied process. A block diagram shall be included to illustrate proposed method. 
5. Line 266: Would it be better to adjust the parameters of each model to the optimal in order to provide a fair comparison?
6. Line 290: There’s no symbol description for equation 1.

Author Response

---- Notes for Reviewer #2 ----

We are grateful for your comments and suggestions (here in []).

[In this work, the detection of abnormal screw tightening processes is studied. Two unsupervised machine learning algorithms (iForest and a deep learning autoencoder) are compared with a local outlier factor method and a supervised Random Forest firstly, then iForest and AE are further compared by adopting a more recent and larger dataset. Besides, an interactive visualization tool for human operators is developed to provide explainable artificial intelligence knowledge. As there is few research on the abnormal screw tightening processes using machine learning methods, this paper is overall interesting.]

We appreciate the positive feedback.

[As a research paper, it can be improved, such as 1. Authors mainly introduced the process, but not the scientific aspect of the problem. Is it a pattern recognition problem? Is it similar to other industrial application? Where is the most difficult part in this problem?]

To better clarify this issue, we have improved the Introduction (Section 1), please check the new green colored text.

[2. Authors listed many works in this area, but didn't give any comments on their works? At least authors shall mention their contribution and limitation, and how their works related to proposed work in this manuscript.]

We have improved the text of Section 2 to better reflect this comment (please check the new green colored text).

[3. In table 1 authors gave four types of data available in the studied process, but only picked angle-torque data for their study. Authors shall provide the reason to give up other data. Is it due to the limitation of method or enough information included in angle-torque data?]

Following this remark, we now explain in Section 3.2 why we have only adopted the angle-torque data.

[4. Angle-torque data for each operation will change along time scale. Did author considered the trajectory info in their study? It is possible that few study has been conducted on  industrial screw tightening, but many similar studies are available in literature. Authors can find batch process monitoring, which is very similar to studied process. A block diagram shall be included to illustrate proposed method.]

We thank the reviewer for this interesting point, which is also clarified in Sections 3.1 and 3.2. Please check the new green colored text and also the new Figure 2.

[5. Line 266: Would it be better to adjust the parameters of each model to the optimal in order to provide a fair comparison?]

As explained in the text, the default parameterization allows a more fair comparison, since we adopted it whenever possible, thus the methods were compared as much as possible under the same conditions. A more detailed discussion is provided here:
LOF - obtained a substantially worst results in Table 4, in such a magnitude that any tuning would not produce competitive results.
RF - it consists of a supervised learning method, thus contrary to IForest or AE, it requires labeled data, which is expensive in this domain. This particular method was only added as an interesting baseline to compare the proposed IForest and AE. Still, when adopting the default parameters, RF obtained high quality results (AUC of 99% when the maximum possible is 100%!).
IForest - using the default parameters, this method obtained the best one-class Table 4 results (AUC of 99% when the maximum possible is 100%!). This model was selected for the experiments with more recent and larger data.
AE - it is consists in the only method that does not have default parameters and actually requires a parameter setup (as any deep learning model). We note that, as explained in Section 3.3, the AE was tuned by performing preliminary experiments using older screw tightening data. After this tuning, the AE setup was fixed and then tested using the newer datasets, leading to the results obtained in Tables 4 and 5. 

[6. Line 290: There’s no symbol description for equation 1.]

We now explain the symbols from equation 1 in Section 3.3.

Reviewer 3 Report

This paper presented an automated screw tightening inspection system by adopting machine learning (ML) algorithms. However, there are some issues that should be addressed as listed below.

 

Abstract:

1. in line 7, isolation forest was (iForest), while lines 61, 88, and other places were (IForest).

 

Introduction:

2. In line 61, the same schemes were used in the previous of their work. It is recommended to involve other ML schemes, along with extending the data application.

 

Related Work:

3. Line 92, it might be the AE needs more computational limitations than the existed scheme such as SVM.

 

Machine Learning Models:

4. Since the LOF scheme was not used in the paper, it might be removed. Line 251 to 258.
5. The same for the RF method. Line 258 to 261.
6. In figure 4. The test set should be unseen through all iterations.

 

Results:

7. In Table 5, it is recommended to show the EER along with AUC. If the detection system is biased on one class, the EER will detect and report the robustness of the ML method.

Author Response

---- Notes for Reviewer #3 ----
We are grateful for your comments and suggestions (here in []).

[Abstract: 1. in line 7, isolation forest was (iForest), while lines 61, 88, and other places were (IForest).]

We have fixed this typo and changed all term instances to IForest.

[Introduction: 2. In line 61, the same schemes were used in the previous of their work. It is recommended to involve other ML schemes, along with extending the data application.]

We appreciate the suggestion given by the reviewer. The submitted manuscript  corresponds to an invited extended paper that was previously published at the ICCSA 2021 conference. The invitation for the special issue "Selected Papers from ICCSA 2021" of the Computers (ISSN 2073-431X) journal required that at least 50% extension of new results (e.g., in the form of technical extensions, more in-depth evaluations, or additional use cases).
When compared with the ICCSA 2021 conference paper, the submitted manuscript includes a substantial portion of new text (much more than 50%) and also two major additions:
1) we have analyzed more recent and larger industrial data, related with three distinct assembled products; and
2) we propose and demonstrate an interactive visualization tool that provides explainable artificial intelligence (XAI) knowledge for 19 the human operators.

[Related Work: 3. Line 92, it might be the AE needs more computational limitations than the existed scheme such as SVM.]

We have updated the text to address this comment. Please check the new green colored text in Section 2.

[Machine Learning Models: 4. Since the LOF scheme was not used in the paper, it might be removed. Line 251 to 258. 5. The same for the RF method. Line 258 to 261.]

The submitted manuscript corresponds to an invited extended paper that was previously published at the ICCSA 2021 conference. Therefore, all previous experiments were included in this study (e.g., Table 4 includes LOF and RF results) to provide the reader insights of the full methods that were explored and compared in this R&D project.  

[6. In figure 4. The test set should be unseen through all iterations.]

Figure 4 represents the rolling window evaluation procedure. This procedure consists of several model training and testing iterations over time, thus realistically mimicking what would occur in a real environment if the models were deployed and monitored over time. For each iteration, the model is trained using a training set with a fixed window (W) size and then evaluated by using a test set of size T. The test set is used  as "unseen" data. To better clarify this issue, we have added new green text in Section 3.4.

[Results: In Table 5, it is recommended to show the EER along with AUC. If the detection system is biased on one class, the EER will detect and report the robustness of the ML method.]

We thank the reviewer for this comment. We have updated table 5 to include the EER values. The EER measure is also now introduced in Section 3.4 (please check the new green colored text).

Round 2

Reviewer 2 Report

Questions in previous review are properly answered.