Multivariate Timing and Granger Causality Analysis of Spontaneous Facial Mimicry in Response to Android Dynamic Facial Expressions

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

1. The demographic characteristics of the sample, such as age, gender, and cultural background, need to be clearly defined, as this can affect the universality and reproducibility of the results.

2. Add necessary unit annotations for Figures 2, 3, and 4.

3. The abstract and introduction sections need to clarify the research question and highlight the innovative aspects of methodology.

4. We need to conduct a detailed analysis and discussion on the limitations and future directions of this study. What are the limitations of the experimental setup (such as static background, single robot expression)? In addition, specific improvement suggestions and application prospects need to be proposed.

5. Research requires in-depth analysis and supplementation to explore the fundamental differences between "human-machine" imitation and "human human" imitation from theoretical perspectives such as social cognition, mentalization, and emotional resonance.

6. The conclusion section needs to clearly propose specific research implications and application pathways (such as robot design, clinical intervention, educational applications, etc.).

7. A detailed analysis diagram of the theoretical framework proposed in this article needs to be drawn.

Author Response

We have sent the revised manuscript to the English proofreading service before resubmission. Please see the link below for a certificate:

http://www.textcheck.com/certificate/Ckqiqt

 

Comment 1: The demographic characteristics of the sample, such as age, gender, and cultural background, need to be clearly defined, as this can affect the universality and reproducibility of the results.

Response 1: We have now ensured that the demographic characteristics of the sample are comprehensively reported. Particularly, we have added that the participants were native Japanese.

 

Comment 2: Add necessary unit annotations for Figures 2, 3, and 4.

Response 2: We have added “Frame” as the unit for “Lag” to the x-axes of the new Figures 4 and 5. The correlation coefficients and beta coefficients on the y-axes do not have units.

 

Comment 3: The abstract and introduction sections need to clarify the research question and highlight the innovative aspects of methodology.

Response 3: We have added sentences to the Abstract and Introduction that highlight our innovative approach. We used multivariate analysis and Granger causality to investigate spontaneous facial mimicry. In the Abstract (lines 21–23): “Multilevel vector autoregression incorporated the android and participant AUs and quantified the temporal evolution of the Granger causality for the first time.” In the Introduction (lines 141–143): To the best of our knowledge, Granger causality has not yet been used to test spontaneous facial mimicry.

 

Comment 4: We need to conduct a detailed analysis and discussion on the limitations and future directions of this study. What are the limitations of the experimental setup (such as static background, single robot expression)? In addition, specific improvement suggestions and application prospects need to be proposed.

Response 4: We have added a dedicated paragraph on the limitations and future directions at the end of the Discussion (lines 493–523).

 

Comment 5: Research requires in-depth analysis and supplementation to explore the fundamental differences between "human-machine" imitation and "human human" imitation from theoretical perspectives such as social cognition, mentalization, and emotional resonance.

Response 5: We have expanded the paragraph in the Discussion that deals with the socio-cognitive mirror mechanisms (lines 455–471). We now also cover the issues of mentalization and emotional resonance (lines 471–481).

 

Comment 6: The conclusion section needs to clearly propose specific research implications and application pathways (such as robot design, clinical intervention, educational applications, etc.).

We have expanded the Conclusion to cover the implications of our research and future research directions (lines 536–544).

 

Comment 7: A detailed analysis diagram of the theoretical framework proposed in this article needs to be drawn.

We have added an analysis-and-theoretical-framework diagram (the new Figure 3).

Author Response File: Author Response.pdf

Reviewer 2 Report

Comments and Suggestions for Authors

This study analyzed human facial mimicry of an android's expressions using FACS. Cross-correlation showed participants mimicked frowns (AU04) after 0.97s and smiles (AU12) after 0.63s. Granger causality confirmed android AU04 and AU25 (lip part) significantly predicted corresponding human AUs, providing evidence for spontaneous mimicry and validating android expressions' social function. But there are many questions that the authors need to clarify as follows:

1. The study reports mixed results for the positive condition (smiling). While the cross-correlation for AU12 was significant with a standard t-test, it was non-significant with a robust t-test, and the subsequent Granger causality analysis for AU12 was also non-significant. Could the authors please clarify how they interpret the overall evidence for spontaneous mimicry of a smile, given these inconsistencies and the reliance on AU25 for the Granger causality finding? How do the authors address concerns about the robustness of the effect for the key mimicry AU (AU12)?
2. The authors used PCA to identify AU25 and AU26 as significant contributors to the android's expressions and included them in the multilevel VAR models. However, the Granger causality analysis for the positive condition revealed a significant effect from android AU25 to participant AU25, rather than the hypothesized AU12. Was there a risk of circularity in this approach? Specifically, does testing for temporal effects on AUs that were selected based on their contribution to the android's expressions increase the chance of finding a significant relationship, and how was this possibility addressed?
3. The multilevel VAR models were estimated using a single fixed lag (29 frames for negative, 19 frames for positive) determined from the group-level peak cross-correlation of AU04 and AU12, respectively. Does this approach assume that all participants have the same response latency? Could the use of a single fixed lag potentially obscure or misrepresent individual differences in mimicry timing, and how might this affect the interpretation of the Granger causality results?
4. Please refer to the papers as follows:

Intelligent disassembly scenario understanding for human behavior and intention recognition towards self-perception human-robot collaboration system

Knowledge graph-driven process reasoning of human-robot collaborative disassembly strategy for end-of-life products

Author Response

Comment 1: The study reports mixed results for the positive condition (smiling). While the cross-correlation for AU12 was significant with a standard t-test, it was non-significant with a robust t-test, and the subsequent Granger causality analysis for AU12 was also non-significant. Could the authors please clarify how they interpret the overall evidence for spontaneous mimicry of a smile, given these inconsistencies and the reliance on AU25 for the Granger causality finding? How do the authors address concerns about the robustness of the effect for the key mimicry AU (AU12)?

Response 1: The electromyography remains the gold standard for detection of spontaneous facial mimicry. In our previous analysis (Yang et al., 2025), we reported that participants’ responses to the zygomaticus major and AU12 were significant, thereby confirming the presence of zygomaticus and AU12 mimicry. This information is now provided in the Introduction (lines 51–58). In addition, despite the fact that the distribution of correlation coefficients violated the normality assumption, paired t-tests with more than 25 observations are considered robust to such violations (Canavos, 1988). This information is provided in the Methods (lines 260–261). Although we report both the standard and Yuen’s robust t-test results, we consider that the standard t-test results are totally valid. The robust t-test might have suffered from a certain loss of power and degrees of freedom caused by data trimming, but we nevertheless report the results. We have also added a comment stating that a larger sample size should allow researchers to avoid this limitation (lines 510–520). In this study, we focused on the temporal latency and temporal precedence effects of such a process. On further investigation, we found that our original assumption that the temporal precedence effect could be observed at the peak cross-correlation was incorrect. We therefore revised our analysis to seek the cross-correlations and Granger causality from the 5^th lag – it is not really neurobiologically viable to explore earlier lags – to the 40^th lag. We found a series of significant AU12 cross-correlations, and also significant android AU12-to-participant AU12 temporal precedence effects. We offer a possible explanation why the timing of the Granger causality did not correspond to the peak cross-correlation: the observed participant AU time series was influenced by different android AUs across various time windows. The AU12 response is likely a consequence of both non-linear and temporal integrative effects. Together with the expanded analysis, the new information is now treated in the Discussion (lines 438–454).

 

 

Comment 2: The authors used PCA to identify AU25 and AU26 as significant contributors to the android's expressions and included them in the multilevel VAR models. However, the Granger causality analysis for the positive condition revealed a significant effect from android AU25 to participant AU25, rather than the hypothesized AU12. Was there a risk of circularity in this approach? Specifically, does testing for temporal effects on AUs that were selected based on their contribution to the android's expressions increase the chance of finding a significant relationship, and how was this possibility addressed?

Response 2: The rationale for selecting Nikola’s most contributory AUs was related to the definition of behavioral matching and mimicry. If Nikola activated an AU during a facial expression and the participant followed suit, we could be certain that the participant engaged in spontaneous facial mimicry. If Nikola did not activate an AU, and the participant also did not, then it would be unclear whether the participant “behaved” after Nikola’s non-expression, or if the participant simply remained static. However, we included three AUs per agent in the mlVAR models, because six time series is the maximum that could be estimated with correlated temporal random effects in the linear mixed effect model implementation in the multilevel VAR (mlVAR) package of R. This information is now added to the Introduction (lines 118–126).

As we mentioned in reply to your previous comment, on detailed investigation, the original assumption that the temporal precedence effect could be observed at the peak cross-correlation was incorrect, and we acknowledge your concern about circularity. We have now also included cross-correlation results for both AU25 and AU26, and we tested both cross-correlation and Granger causality statistics across lags 5–40. The false discovery rate (FDR) has also been reported. We found that the temporal evolution (time series) of Granger causality was informative and worth further investigation.

 

Comment 3: The multilevel VAR models were estimated using a single fixed lag (29 frames for negative, 19 frames for positive) determined from the group-level peak cross-correlation of AU04 and AU12, respectively. Does this approach assume that all participants have the same response latency? Could the use of a single fixed lag potentially obscure or misrepresent individual differences in mimicry timing, and how might this affect the interpretation of the Granger causality results?

Response 3: There are two aspects to this comment: Group-level statistics or inferences test a group distribution, and also factors in any individual variances. High individual variation is reflected in a high standard deviation or error, associated with insignificant group-level statistics. However, we acknowledge your concern that the peak in the group-averaged time series might not have corresponded to the time when mimicry in fact occurred uniformly across all participants. We have now derived cross-correlation coefficients and Granger causality statistics not only at the peak but also from lags 5 to 40, and we also report the FDRs. In the revised manuscript, the analysis of Granger causality is no longer consequent on the cross-correlation results. We also acknowledge that it is important to investigate individual differences that might modulate the socio-cognitive process further. Relevant text has been added to the limitations (lines 520–523) and Conclusion (lines 536–540).

 

Comment 4: Please refer to the papers as follows:

Intelligent disassembly scenario understanding for human behavior and intention recognition towards self-perception human-robot collaboration system

 

Knowledge graph-driven process reasoning of human-robot collaborative disassembly strategy for end-of-life products

Response 4: The citations have been added to the bibliography.

 

Canavos, G. C. (1988). The sensitivity of the one-sample and two-sample student t statistics. Computational Statistics & Data Analysis, 6(1), 39–46. https://doi.org/10.1016/0167-9473(88)90061-8

Yang, D., Sato, W., Hsu, C.-T., Minato, T., & Nishida, S. (2025). Visually detectable facial mimicry in response to android facial expressions. Scientific Reports, 15(1), 41376. https://doi.org/10.1038/s41598-025-25394-6

Yang, D., Sato, W., Liu, Q., Minato, T., Namba, S., & Nishida, S. (2022). Optimizing Facial Expressions of an Android Robot Effectively: A Bayesian Optimization Approach. 2022 IEEE-RAS 21st International Conference on Humanoid Robots (Humanoids), 542–549. https://doi.org/10.1109/Humanoids53995.2022.10000154

 

Author Response File: Author Response.pdf

Reviewer 3 Report

Comments and Suggestions for Authors

This paper studies the temporal dynamics of spontaneous facial mimicry when humans observe an android (Nikola) producing dynamic facial expressions. The authors extract FACS action units from synchronized videos of the android and participants using an automated AU pipeline, estimate response delays via cross-correlation, and then test directed temporal dependencies with a multilevel VAR / Granger-style analysis over selected AU time series. The work is clearly structured, uses a fully time-resolved approach rather than static endpoints, and attempts to connect simple lag estimation with a principled multivariate temporal model. However, the paper also has weaknesses:

1. Latency values: should these numbers match across the paper? In the Abstract you report 0.976 s for AU04 and 0.667 s for AU25, while in the Results and figure captions the corresponding delays are 0.967 s (29 frames at 30 fps) and 0.633 s (19 frames at 30 fps). Is this rounding, a different definition of “delay,” or an inconsistency that needs correction?
2. Smile mimicry via AU12: should the main claim about smile mimicry be revised given that android AU12 to participant AU12 is not significant in the mlVAR (p = 0.105)? Also, in cross-correlation the AU12 “significance” appears to depend on the statistical test choice (standard versus robust). How should the reader interpret the evidence for AU12-driven mimicry?
3. Since the positive-condition lag is selected from the AU12 cross-correlation peak (19 frames), why is the significant directed effect reported in mlVAR AU25 to AU25 at that lag? Should the lag-selection procedure be aligned with the AU that drives the reported effect, or should additional lags be tested with appropriate control?
4. You run 30 trials per condition but analyze only 15 “prototypical” trials. Could you clarify the exact selection criteria and report whether the main findings hold when using the full set of trials, including the Bayesian-optimized expressions, or at least provide a sensitivity analysis?

The paper’s direction is reasonable, but the current version has internal inconsistencies in reported key numbers and a narrative that does not cleanly match the statistical evidence across analyses. The conclusions would be more convincing after resolving the latency discrepancies, clarifying which AUs are truly supported by the multivariate model, and demonstrating robustness to trial selection.

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Round 2

Reviewer 1 Report

Comments and Suggestions for Authors

   

Reviewer 2 Report

Comments and Suggestions for Authors

it can be accepted now

Reviewer 3 Report

Comments and Suggestions for Authors

The authors have addressed the review comments in a satisfactory and constructive way. The revised version is clearer, and several important points have been clarified, particularly in the description of the analyses and in the interpretation of the findings. While some responses are stronger than others and a few limitations remain, I believe the main concerns have been sufficiently resolved at this stage. Overall, the manuscript has improved after revision and is now suitable for publication in its current form.