Viewpoint Robustness of Automated Facial Action Unit Detection Systems
Round 1
Reviewer 1 Report
The paper presents the comparison of three different software solutions for automated facial action/emotion detection. The performance is measured by observing the facial images taken from various angles.
The structure of the paper is good. The methodology of research and the experiment is straightforward and sound. The results are properly described.
In my opinion, the number of subjects (six males and six females) in the experiment can present a problem. The algorithms for extracting facial features are notoriously biased. The authors properly try to avoid it by using proper training sets. However, the algorithms can also be subjective to the facial particularities of an individual testing person, which can skew the statistical analysis with a small number of participants. I am not sure this is also true for the result presented in the paper. It can be tested by systematically removing or adding observations for a subject in the calculation and observing if the results are significantly different.
Another minor issue is a typographic glitch due to the long URL address on page 5, line 157. Maybe this can be corrected by the editor, or the URL can be put into the references.
Author Response
We are grateful for the reviewers’ extremely helpful comments. We have addressed all of the issues highlighted, and believe that the manuscript has been improved considerably as a result of these changes.
Point 1
In my opinion, the number of subjects (six males and six females) in the experiment can present a problem. The algorithms for extracting facial features are notoriously biased. The authors properly try to avoid it by using proper training sets. However, the algorithms can also be subjective to the facial particularities of an individual testing person, which can skew the statistical analysis with a small number of participants. I am not sure this is also true for the result presented in the paper. It can be tested by systematically removing or adding observations for a subject in the calculation and observing if the results are significantly different.
Response
This is an important point. As the reviewer noted, it is possible that the facial particularities of an individual testing person affected algorithm performance. Therefore, we systematically performed calculations removing each subject. The resultant findings were consistent with the results reported in the main paper (ANOVA: rotation effects Fs > 7.60, ps < 0.000; machine effects Fs > 15.74, ps < 0.000; interaction effects Fs > 6.86, ps < 0.000). We have added the corresponding results (Page 4, Lines 173–177).
Point 2
Another minor issue is a typographic glitch due to the long URL address on page 5, line 157. Maybe this can be corrected by the editor, or the URL can be put into the references.
Response
Thank you for the comment. As suggested, we created a shortened URL (Page 5, Line 183). We will confirm more details at the proofreading stage.
Reviewer 2 Report
In this paper, the authors propose to compare the performance of different automatic facial expression detection systems by varying the viewpoint angles.
First, the manuscript must be revised in its entirety as there are many syntactic and grammatical errors.
The paper has also some inconsistencies that should be clarified, including:
- How were the states of anger, disgust, fear, happiness, sadness, and surprise solicited?
- “The data were binary (presence or absence of an expression).”: I do not see the neutral expression in the above-mentioned states? What is meant by the “absence of an expression”?
- Has the evaluation of the detection algorithms been done using these states or the actions units?
Previous studies done on automatic facial expression detection were rather scarce in the paper. A whole section should have been devoted to a more extensive review of current related work.
In the introduction it is stated that “OpenFace is the best choice among the many automated facial action detection systems available”: Why? In terms of what characteristics?
Appendix A should be included in the paper.
Author Response
We are grateful for the reviewers’ extremely helpful comments. We have addressed all of the issues highlighted, and believe that the manuscript has been improved considerably as a result of these changes.
Point 1
First, the manuscript must be revised in its entirety as there are many syntactic and grammatical errors.
Response
Thank you for the feedback. The English in the revised document has been checked by at least two professional editors, both native speakers of English. For a certificate, please see http://www.textcheck.com/certificate/OkLrn9
Point 2
The paper has also some inconsistencies that should be clarified, including:
- How were the states of anger, disgust, fear, happiness, sadness, and surprise solicited?
- “The data were binary (presence or absence of an expression).”: I do not see the neutral expression in the above-mentioned states? What is meant by the “absence of an expression”?
- Has the evaluation of the detection algorithms been done using these states or the actions units?
Response
Thank you for bringing this to our attention. We have modified these descriptions in the revised manuscript (Page 2, Lines 87–95; Pages 3, Lines 104–105; Page 4, Lines 137–138).
Point 3
Previous studies done on automatic facial expression detection were rather scarce in the paper. A whole section should have been devoted to a more extensive review of current related work.
Response
We have carefully checked the manuscript in response to this comment. We have added a new section that reviews the automated facial detection systems and organizes a review of papers that introduce many aspects of automated facial detection systems that are beyond the current scope. Additionally, we have provided the bridge from the facial expression analysis algorithms that have been developed so far to a facial expression system that can be accessed by novice users, which was the motivation for this research (Page 1, Lines 32–40).
Point 4
In the introduction it is stated that “OpenFace is the best choice among the many automated facial action detection systems available”: Why? In terms of what characteristics?
Response
As suggested, we have modified the exaggerated expression to produce a more reasonable description (i.e., OpenFace is easy to use and performed well compared to other available automated facial action detection systems). [Pages 1–2, 41–52].
Point 5
Appendix A should be included in the paper.
Response
Data in Appendix A are presented separately from the main results because the number of available data computed by AFAR toolbox differed from that used by the others. We have emphasized this point in the revised manuscript [Page 2, Lines 73–78]. Thank you for noticing it.
Reviewer 3 Report
The paper shows an interesting study, the experiments conducted were appropriate and the results were consistent. I probably missed some more motivation and background information to make the document a bit more self-contained.
Author Response
We are grateful for the reviewers’ extremely helpful comments. We have addressed all of the issues highlighted, and believe that the manuscript has been improved considerably as a result of these changes.
Point 1
I probably missed some more motivation and background information to make the document a bit more self-contained.
Response
Thank you for this comment, according to which we have carefully checked the manuscript. We have added a new section that reviews the automated facial detection systems and provides a guide to the review papers, which introduce many aspects of automated facial detection systems that are beyond the scope of this paper. Additionally, we have provided a bridge between the facial expression analysis algorithms that have been developed so far and a facial expression system that can be accessed by novice users, which was the motivation for this research (Page 1, Lines 32–40).
Round 2
Reviewer 2 Report
The paper could be accepted in this last version
