Towards an Emotion-Aware Metaverse: A Human-Centric Shipboard Fire Drill Simulator

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

This paper focuses on a fire training simulation in immersive VR, a timely and important topic. The rationale of the study is well established. The literature review is sufficiently wide but not thorough enough at certain places, see below for details. However, it features some notable methodological weaknesses that need to be addressed:

No RQs were formulated.

The testing evaluation measures only completion time, not the performance or effectiveness of the training application.

The value of the biometric emotional data tracking analysis of a low number of participants should be explained.

A section of the study’s limitation should be added.

Additional issues:

L142: The section 2.2.1 “Metaverse Applications for Maritime Industries” needs to be enhanced with additional notable studies in the field e.g. [1,2]. Moreover, research on VR fire training applications is also relevant to supplement existing evidence e.g. [3,4].

L199: “Emotion-aware applications on the Metaverse” Please check one recent systematic literature review that could be useful here [5].

L366: The description of the VR environment does not explain or justify why this immersive learning application constitutes “Metaverse”.

L425: From the experiment description it becomes evident that each participant experienced the VR app in a single-user mode. If this is true, the “Metaverse” aspect should be corrected in the manuscript to the more scientifically accurate concept “immersive VR”.

L528: Please include indicative snapshots of how both conditions (text and visual cues) look like. Moreover, please explain the rationale for designing the cues in VR. Why were they necessary? Did you take findings from prior research e.g. [6–8] in consideration?

 

References (mentioned above indicatively)

1. doi: 10.54941/ahfe1001513
2. doi: 10.1007/978-3-030-50896-8_40
3. doi:10.3390/educsci12040281
4. doi:10.1016/j.compedu.2021.104145
5. doi:10.70401/ec.2025.0004
6. doi: 10.1007/978-3-031-80475-5_23
7. doi: 10.1109/ISMAR62088.2024.00030
8. doi:10.1007/s11423-023-10313-1

Author Response

The authors would like to thank the reviewers for their valuable comments, which have certainly helped us to improve the manuscript. Please find attached our detailed responses to the comments. In order to ease the labour of the reviewers we have colored in red the differences with the previous version of the article.

Author Response File: Author Response.pdf

Reviewer 2 Report

Comments and Suggestions for Authors

i am grateful for the opportunity to review your manuscript. there is much to commend, such as the elaboration shared about the method, the grounding of the latter within the literature, the transparent and detailed nature of the reporting of results. in fact, i find the manuscript to be especially strong in its reporting of results, because different aspects of the emotion detection measures are reported and interpreted at a granular level and also at a composite level.

i strongly believe your manuscript can stand on its own if it were to position itself as a study in emotion-awareness in a human-centric shipboard fire drill simulator.

i invite you to consider the perspective that the discussion within the first third of the manuscript of multiple metaverses and meta-galaxies is a distraction and one that is ultimately not explored meaningfully within the actual simulator because - by your own admission - you "selected" the Quest Pro, and - again by your own admission - the Quest Pro is but one component of Meta's ecosystem and thus Figure 2, and section 2.1 really adds minimal value to the overall strength of the study you report.

to be candid, i find sections 2.4 and 2.5 unnecessarily confusing, not only because in 2.4 you refer to applications with a small 'a' (applications of emotion-awareness in XR), while in 2.5 you use the capital 'A' (applications as in platforms and hardware / software combos, of which you list (for example) TactSuit and Teslaglove).

another critique i level in this unnecessarily meandering section is that within 2.5.2 (which is purportedly on software, as per the sub-title), you do mention the TactSuit and the Teslaglove, and you indeed refer to the latter explicitly as a "device" (the word "device" / "devices" is used a few times in this sub-section which is purportedly on software).

we agree that for emotion-aware XR, hardware and software act in combination, often through wider platforms. there is no need to split-hairs within this, especially because the hairs you attempt to split are so fine that you yourselves get caught in them (as evidenced by the multiple instances of 'devices' within a sub-section purportedly on 'software').

Author Response

The authors would like to thank the reviewers for their valuable comments, which have certainly helped us to improve the manuscript. Please find attached our detailed responses to the comments. In order to ease the labour of the reviewers we have colored in red the differences with the previous version of the article.

Author Response File: Author Response.pdf

Reviewer 3 Report

Comments and Suggestions for Authors

The article "Towards an Emotion-Aware Metaverse: A Human-Centric Shipboard Fire Drill Simulator" presents a virtual reality application for maritime safety training, featuring the novel incorporation of real-time emotion detection technologies via eye-tracking and facial expression analysis. The objective is to evaluate the participants' technical proficiency, emotional engagement, and stress resilience during exercises. The study offers a robust methodological framework, with an initial experimental version of the simulator succeeded by a refined version informed by user input. The two experimental phases encompassed seventeen participants, all naval engineering students. The findings analysis used quantitative metrics (including completion durations and the number of things viewed) and qualitative data about emotional expression.

The scientific contribution is novel and aligns with contemporary research in the industrial metaverse and immersive training domain of technologies like Meta Quest Pro, and the Facial Action Coding System is well-justified. This research highlights the potential for enhanced learning experiences through immersive environments, demonstrating how advanced technologies can facilitate deeper engagement and understanding among students. By integrating emotional responses into the analysis, the study not only broadens the scope of traditional educational methodologies but also paves the way for future explorations in the intersection of technology and learning. Certain sections of the text might be enhanced to augment the communication efficacy and thoroughness of the analysis. 

The abstract is well-structured yet overly detailed. The introduction of the primary numerical data might be enhanced by explicitly presenting the reduction in task completion time between the initial and subsequent versions of the simulator, including an apparent reference to the percentage of improvement seen (ranging from 14% to 32.72%). Would render the efficacy of the experimental intervention readily apparent. 

Another aspect that warrants consideration is the management of the identified emotions. The established approach classified emotions into three categories (good, negative, absent), but occasionally seems overly simplistic. It would be beneficial to elucidate why, in several instances, the anticipated emotions, such as fear or surprise, were not identified, or why specific individuals exhibited euphoric responses during the fire containment. An expanded discourse on potential explanations, such as the whimsical impression of the simulator or the extent of graphic immersion, might help understand the data. 

The segment addressing the sequence of items analyzed using eye tracking is technically sound; however, it is somewhat opaque to those without expertise in bioinformatics techniques. The elucidation of the LCS and Sliding Window techniques ought to be simplified or supplemented by a tangible visual example, perhaps contrasting two sequences of seen entities, to enhance the intuitiveness of the "similarity score." 

The paper presents intriguing insights despite the presumably restricted sample size. It would be beneficial to enhance the concluding section on future views by suggesting specific techniques to augment the emotional realism of the simulator. The authors acknowledge the potential for including auditory or haptic input but fail to delineate the technical implementation or testing methods for future research. 

The work is well-written, well-structured, and offers a significant scientific contribution. The language is explicit and formal, with few stylistic elements requiring modification. The essay is acceptable pending minor adjustments to enhance communication effectiveness and deepen the interpretation of some outcomes. Incorporating a summary table of the primary emotional indicators for each tester and level, or providing further analysis of the practical implications of the results, would enhance the overall validity of the contribution.

Author Response

The authors would like to thank the reviewers for their valuable comments, which have certainly helped us to improve the manuscript. Please find attached our detailed responses to the comments. In order to ease the labour of the reviewers we have colored in red the differences with the previous version of the article.

Author Response File: Author Response.pdf

Reviewer 4 Report

Comments and Suggestions for Authors

This contribution presents the design, implementation, and evaluation of an emotion-aware Virtual Reality (VR) fire drill simulator for maritime safety training, which integrates emotion detection technologies, such as eye tracking and facial expression analysis, to assess trainees' emotional responses and decision-making under stress during shipboard fire emergencies. Authors at the end point out that the presented application/simulation provides guidelines for developing next-generation emotion-aware Metaverse applications, emphasising the importance of user-centric design, emotional realism, and adaptability.

 

General comments

 

The authors tackle an interesting and current topic in the field of training with the help of VR simulations. To this, they add an interesting possibility, which is the observation of emotions in various simulations. This is an interesting possibility, namely observing emotions, which is an easy problem, but it is always followed by a "difficult" problem, which is analysing these responses and assuming actions based on that. However, the authors completely avoid this area and end the paper somewhat boldly: "In conclusion, this article provided, through the description of a fire drill simulator, detailed guidelines for future researchers and developers on how to create the next generation of emotion-aware Metaverse applications". The second problem, in my opinion, is that the inclusion of emotions in the simulation of shipboard fire emergencies requires a fundamentally unemotional response and a high level of experience, and I can argue that a fire in a simulation is not comparable to a fire in reality.

What I also miss on a general level is that we don't know the structure of the participants in the experiment, whether they are "experts" in charge of extinguishing on the ship, or random participants who are with or without experience. Whenever we run an experiment, we try to get an approximately homogeneous group, because otherwise the interpretation of the results is questionable, all the more so because you have an extremely small sample. Certainly, however, the article needs significant additional clarification and editing of the entire article for publication. First of all, I suggest that you start from the whole, present the whole system at the beginning and then focus on explaining the individual elements. The reader is just reading the article "in order", so I did too, so I've made additional comments on the fly (but this comment may have been answered somewhere below). Try typing your answers in place of these comments. There is a huge amount of information in the article that is not crucial and lacks information that would be necessary or at least welcome. I would like to make some additional, concrete comments below.

 

 Particular comments

 

1. At the beginning, define the correlations between the Metaverse - XR and emotions in your specific case.
2. Define and make sense of the correlations between the effectiveness of your simulation and how it is affected by the results of the emotion pursuit.
3. I think sections 2.1 and 2.2 are too extensive; try to define/adopt this presentation for your specific point. All of these "additional common knowledge facts" disrupt the red thread of your story.
4. If you are giving examples, focus more on similar cases (as you call them, industrial applications) and not on cases that have a completely different background, such as health care cases, where everything is extremely emotional in addition to professionalism.
5. Why did you include emotions in your simulation, and how and what has improved in the simulation as a result of this?
6. Line 217-128 - you mention AI in the virtual environment for the first time (why here?), AI is the basis of all such applications (AI - XR - Metaverse; or AI-XR-API). In doing so, additional explanations are needed as to how this is related to your example, what is the point of these connections (how improving the quality of training, or...?)
7. Now let's get to the point, according to my opinion, about your architecture. Chapter 2 gives all the options (a general description of generally well-known information), which could somehow be compressed and told in a more concise, but above all more concrete, way related to your work/application. Describe - What did you choose and why did you decide to use it?
8. In my opinion, the fundamental part of the whole paper is also Chapter 3, where you try to present your case, the architecture of your application. To quote the last paragraph of the introduction 3.1:

"With the incorporation of emotional recognition, the system not only assesses procedural competencies but also emotional resilience, thus providing personalised feedback and enhancing preparedness for high-pressure situations" This sentence captures the essence of your contribution; try to stick to it.

The whole of Chapter 3 would need further explanations in the mentioned direction.

9. Section 3.2 System architecture is essential for the interpretation of your application. This should be a fundamental part of the article (and you described it in 6 lines), and one image that says almost nothing in this sense. Since I assume that you are experts in the field of computer science, express your system in Figure 1 in the form of an algorithm/flow chart, where you clearly define the course of the architecture of your application. Try to track your record:

 "Line 283-284 "With the incorporation of emotional recognition, the system not only assesses procedural competencies but also emotional resilience, thus providing personalized feedback and enhancing preparedness for high pressure situations."

10. In Sections 3.3 and 4.2 - Where and how is emotion analysis included in your scenarios?
11. Line 320 - effectiveness of games- simulations are serious activities that must be based on researched cases (not games that can also be unrealistic, imaginary) and enhancing problem-solving skills and facilitating learning emotions? Don't use game terms.

12. Give Chapter 4 a little more detail with the facts of your experiment. The whole chapter is a bit confusing; try to follow the logic already mentioned: from the whole to the individual parts.  Figure 3 needs further explanation. The image must be self-confessing. Define Level 1-4 a bit on it, e.g. enter the data from Figure 4 into Level 1 (then Figure 4 is unnecessary), etc.

13. Line - 390. "expressions to determine through a rule-based logic the detected emotions" - Further explanation of what it means in your case, Rule Based Logic, how to analyse these responses?
14. Line 394 - "mechanism allows for determining when the trainee is gazing at an object, and associates such an action with the detected emotions". Explanation needed!
15. Line 398 - Two sets of experiments were conducted in order to evaluate two different versions of the application - An additional explanation is needed! These are not two sets but two phases. In the first phase, you analyse the beta version, you get the results (what are these results), which you include in the next phase (how did you improve the next version?)
16. Check the results obtained, I perceive illogical deviations in the interpretation, the reasonableness of Table 1 and Table 2 in such a form (it would be interesting to carry out a qualitative analysis with an appropriate psychological questionnaire that would further explain the results from these tables (e.g. you can use at least a 5-point Lycard scale), analyze the experiance with the results obtained, experiance of fire Drills VR an VG, how the first ones affect the performance in your experiment (VG) and relate this to the results obtained. Figure 10, if you link to Table 1 and your claim: "experience in video gaming was the fastest to complete the tasks" - how can you analyse such an inhomogeneous group? Additional explanations are needed. In that regard, check out any further explanations of the results.
17. And a few additional questions that come out of the Conclusion

a." This article introduced an XR fire drill simulator designed for maritime safety training that is able to monitor the interactions of the users, what they stare at, and to detect their emotions." - And what is it for?

14. "reducing task completion times between 14.18% and 32.72%", - Reasons why and how arguments relate to emotions, how does this relate to other research theses/problems?
15. "In conclusion, this article provided, through the description of a fire drill simulator, detailed guidelines for future researchers and developers on how to create the next generation of emotion-aware Metaverse applications"???

I disagree, this argument should be watered down considerably - as a first step on this path, for example!?

Comments on the Quality of English Language

It could be improved; generally, it is OK, but I am not a language expert.

Author Response

The authors would like to thank the reviewers for their valuable comments, which have certainly helped us to improve the manuscript. Please find attached our detailed responses to the comments. In order to ease the labour of the reviewers we have colored in red the differences with the previous version of the article.

Author Response File: Author Response.pdf

Round 2

Reviewer 1 Report

Comments and Suggestions for Authors

Authors addressed satisfactorily most issues detected and improved the overall quality of their work. However, the following points remain unsolved:

RQs should be numbered.

Moreover, RQs should be answered clearly in respective subsections in the Results section.

More importantly, regarding the conceptual framing of the study, authors have adopted the term Metaverse and justified their choice. Two issues can be raised here:

Regarding the thesis that multiple Metaverses exist, this reviewer disagrees and would argue that there is just one Metaverse just like there is one Internet not many Internets. However, there are countless intranets and other networks. In the same fashion, there are many virtual worlds in Metaverse.  Notably, as some researchers do use interchangeably the terms metaverse and virtual world, the plural form can be accepted based on sufficient evidence from literature which must be added by authors.

Second, a single-user immersive VR experience that is persistent is not a virtual world or (part of the) Metaverse. In this case, the term Metaverse must be replaced with “immersive VR”. If the users’ experiences exist in rooms or in discrete spaces in a larger virtual environment, then this constitutes indeed a social virtual world. If this is the case, please explain how the different instances are interconnected. Can one user visit the ship of the other user? How does this teleportation system work?

Author Response

The authors would like to thank the reviewers for their valuable comments, which have certainly helped us to improve the manuscript. Please find attached our detailed responses to the comments. In order to ease the labour of the reviewers we have colored in red the differences with the previous version of the article.

Author Response File: Author Response.pdf

Reviewer 2 Report

Comments and Suggestions for Authors

thank you for taking the time to understand the intent and the spirit through which the suggestions from the preceding round were offered.

i accept your responses, and note that you have likewise taken made attempts to refine the manuscript at appropriate places, such as in Sections 2.1 and 3.1 and the former section 5. in addition, i acknowledge your re-structuring of the manuscript.

i am happy to endorse this iteration of the manuscript for subsequent stages of the publication process.

Author Response

The authors would like to thank the reviewers for their valuable comments, which have certainly helped us to improve the manuscript. Please find attached our detailed responses to the comments.  

Author Response File: Author Response.pdf

Reviewer 4 Report

Comments and Suggestions for Authors

I got a really thorough and comprehensive explanation of all the dilemmas posed and answers to mor or less all the questions. I now fully agree with the authors, "that the article represents a first step toward the development of emotion-aware Metaverse applications, offering initial insights through the description of a fire drill simulator that could guide future researchers and developers."  The paper thus represents a research application that, if they want to develop an industrial application, will need a lot of additional work. At this stage, however, I consider the article to be of good quality and appropriate, perhaps with minor editorial corrections for publication.

Author Response

The authors would like to thank the reviewers for their valuable comments, which have certainly helped us to improve the manuscript. Please find attached our detailed responses to the comments.  

Author Response File: Author Response.pdf

Round 3

Reviewer 1 Report

Comments and Suggestions for Authors

Authors addressed all issues satisfactorily.