Multimodal Technologies and Interaction

28 pages, 3350 KiB

Open AccessArticle

Designing Gamified Intergenerational Reverse Mentorship Based on Cognitive Aging Theory

by Sijie Sun

Multimodal Technol. Interact. 2025, 9(6), 64; https://doi.org/10.3390/mti9060064 - 19 Jun 2025

Viewed by 307

With the intensification of population aging, elderly individuals face significant barriers in learning digital skills, leading to a widening digital divide. Based on Cognitive Aging Theory, this study proposes and validates an original paradigm called Structured Gamified Intergenerational Digital Reverse Mentorship to enhance [...] Read more.

With the intensification of population aging, elderly individuals face significant barriers in learning digital skills, leading to a widening digital divide. Based on Cognitive Aging Theory, this study proposes and validates an original paradigm called Structured Gamified Intergenerational Digital Reverse Mentorship to enhance digital literacy among the elderly. Cognitive Aging Theory suggests that due to declines in memory, attention, and executive function, older adults encounter challenges when learning new technologies, while gamified learning combined with intergenerational interaction can help reduce cognitive load and increase learning motivation. This study designed a collaborative gamified digital reverse mentorship application, “Digital Bridge”, and employed a randomized controlled trial method, assigning 90 participants aged 60 and above into three groups: the traditional digital mentorship group (Group A), the independent gamified learning group (Group B), and the collaborative gamified digital mentorship group (Group C). Each intervention session lasted 30 min and was conducted in a controlled environment. Experimental results showed that Group C significantly outperformed Groups A and B in digital skill acquisition, user experience, and learning motivation (p < 0.001), indicating that the combination of gamified learning and intergenerational interaction effectively enhances learning interest, reduces learning anxiety, and improves skill transferability. This study provides a new approach to elderly digital literacy education and offers theoretical and practical support for the design of future age-friendly digital learning tools. Full article

(This article belongs to the Special Issue Video Games: Learning, Emotions, and Motivation)

► Show Figures

Figure 1

21 pages, 2985 KiB

Open AccessArticle

Non-Invasive Fatigue Detection and Human–Machine Interaction Using LSTM and Multimodal AI: A Case Study

by Muon Ha, Yulia Shichkina and Xuan-Hien Nguyen

Multimodal Technol. Interact. 2025, 9(6), 63; https://doi.org/10.3390/mti9060063 - 13 Jun 2025

Viewed by 377

Abstract

Fatigue in high-stress work environments poses significant risks to employee performance and safety. This study introduces a non-invasive fatigue detection system utilizing facial parameters processed via a Long Short-Term Memory (LSTM) neural network, coupled with a human–machine interaction interface via a Telegram chatbot. [...] Read more.

Fatigue in high-stress work environments poses significant risks to employee performance and safety. This study introduces a non-invasive fatigue detection system utilizing facial parameters processed via a Long Short-Term Memory (LSTM) neural network, coupled with a human–machine interaction interface via a Telegram chatbot. The system analyzes eye blink patterns and facial expression changes captured through a webcam, achieving an accuracy of 92.35% on the UTA-RLDD dataset. An interactive feedback mechanism allows users to verify predictions, enhancing system adaptability. We further propose a multimodal AI framework to integrate physiological and environmental data, laying the groundwork for broader applications. This approach provides an effective solution for early fatigue detection and adaptive collaboration between humans and machines in real-time settings. Full article

(This article belongs to the Special Issue Multimodal User Interfaces and Experiences: Challenges, Applications, and Perspectives—2nd Edition)

► Show Figures

Figure 1

23 pages, 3946 KiB

Open AccessArticle

The Impact of Color Blindness on Player Engagement and Emotional Experiences: A Multimodal Study in a Game-Based Environment

by Merve Tillem and Ahmet Gün

Multimodal Technol. Interact. 2025, 9(6), 62; https://doi.org/10.3390/mti9060062 - 13 Jun 2025

Viewed by 366

Abstract

Color blindness can create challenges in recognizing visual cues, potentially affecting players’ performance, emotional involvement, and overall gaming experience. This study examines the impact of color blindness on player engagement and emotional experiences in digital games. The research aims to analyze how color-blind [...] Read more.

Color blindness can create challenges in recognizing visual cues, potentially affecting players’ performance, emotional involvement, and overall gaming experience. This study examines the impact of color blindness on player engagement and emotional experiences in digital games. The research aims to analyze how color-blind individuals engage with and emotionally respond to games, offering insights into more inclusive and accessible game design. An experiment-based study was conducted using a between-group design with a total of 13 participants, including 5 color-blind and 8 non-color-blind participants (aged 18–30). The sample was carefully selected to ensure participants had similar levels of digital gaming experience and familiarity with digital games, reducing potential biases related to skill or prior exposure. A custom-designed game, “Color Quest,” was developed to assess engagement and emotional responses. Emotional responses were measured through Emotion AI analysis, video recordings, and self-reported feedback forms. Participants were also asked to rate their engagement and emotional experience on a 1 to 5 scale, with additional qualitative feedback collected for deeper insights. The results indicate that color-blind players generally reported lower engagement levels compared to non-color-blind players. Although quantitative data did not reveal a direct correlation between color blindness and visual experience, self-reported feedback suggests that color-related design choices negatively impact emotional involvement and player immersion. Furthermore, in the survey responses from participants, color-blind individuals rated their experiences lower compared to individuals with normal vision. Participants emphasized that certain visual elements created difficulties in gameplay, and alternative sensory cues, such as audio feedback, helped mitigate these challenges. This study presents an experimental evaluation of color blindness in gaming, emphasizing how sensory adaptation strategies can support player engagement and emotional experience. This study contributes to game accessibility research by highlighting the importance of perceptual diversity and inclusive sensory design in enhancing player engagement for color-blind individuals. Full article

(This article belongs to the Special Issue Multimodal User Interfaces and Experiences: Challenges, Applications, and Perspectives—2nd Edition)

► Show Figures

Figure 1

18 pages, 5112 KiB

Open AccessArticle

Gaze–Hand Steering for Travel and Multitasking in Virtual Environments

by Mona Zavichi, André Santos, Catarina Moreira, Anderson Maciel and Joaquim Jorge

Multimodal Technol. Interact. 2025, 9(6), 61; https://doi.org/10.3390/mti9060061 - 13 Jun 2025

Viewed by 349

Abstract

As head-mounted displays (HMDs) with eye tracking become increasingly accessible, the need for effective gaze-based interfaces in virtual reality (VR) grows. Traditional gaze- or hand-based navigation often limits user precision or impairs free viewing, making multitasking difficult. We present a gaze–hand steering technique [...] Read more.

As head-mounted displays (HMDs) with eye tracking become increasingly accessible, the need for effective gaze-based interfaces in virtual reality (VR) grows. Traditional gaze- or hand-based navigation often limits user precision or impairs free viewing, making multitasking difficult. We present a gaze–hand steering technique that combines eye tracking with hand pointing: users steer only when gaze aligns with a hand-defined target, reducing unintended actions and enabling free look. Speed is controlled via either a joystick or a waist-level speed circle. We evaluated our method in a user study (n = 20) across multitasking and single-task scenarios, comparing it to a similar technique. Results show that gaze–hand steering maintains performance and enhances user comfort and spatial awareness during multitasking. Our findings support using gaze–hand steering in gaze-dominant VR applications requiring precision and simultaneous interaction. Our method significantly improves VR navigation in gaze–dominant, multitasking-intensive applications, supporting immersion and efficient control. Full article

► Show Figures

Figure 1

33 pages, 3913 KiB

Open AccessArticle

Rethinking the Bebras Challenge in Virtual Reality: Implementation and Usability Study of a Computational Thinking Game

by Jelena Nakić, Ivana Rogulj and Lada Maleš

Multimodal Technol. Interact. 2025, 9(6), 60; https://doi.org/10.3390/mti9060060 - 11 Jun 2025

Viewed by 392

Abstract

Virtual reality (VR) technology is becoming increasingly relevant as a modern educational tool. However, its application in teaching and learning computational thinking remains relatively underexplored. This paper presents the implementation of selected tasks from the international Bebras Challenge in a VR environment called [...] Read more.

Virtual reality (VR) technology is becoming increasingly relevant as a modern educational tool. However, its application in teaching and learning computational thinking remains relatively underexplored. This paper presents the implementation of selected tasks from the international Bebras Challenge in a VR environment called ThinkLand. A comparative study was conducted to evaluate the usability of the developed game across two interface types: mobile devices and desktop computers. A total of 100 participants, including high school and university students, took part in the study. The overall usability rating was classified as “good”, suggesting that ThinkLand holds promise as a platform for supporting computational thinking education. To assess specific aspects of interface usability, a custom Virtual Environment Usability Questionnaire (VEUQ) was developed. Regression analysis was performed to examine the relationship between participants’ age, gender, and interface type with both learning performance and perceived usability, as measured by the VEUQ. The analysis revealed statistically significant differences in interaction patterns between device types, providing practical insights for improving interface design. Validated in this study, the VEUQ proved to be an effective instrument for informing interaction design and guiding the development of educational VR applications for both mobile and desktop platforms. Full article

(This article belongs to the Special Issue Innovative Theories and Practices for Designing and Evaluating Inclusive Educational Technology and Online Learning)

► Show Figures

Figure 1

25 pages, 2939 KiB

Open AccessReview

by Jakub Swacha

Multimodal Technol. Interact. 2025, 9(6), 59; https://doi.org/10.3390/mti9060059 - 11 Jun 2025

Viewed by 501

Abstract

Video games come in many genres. Although the popularity of games that belong to different genres is the subject of various research and industry reports, so far, there have been no studies investigating their popularity in research papers. This paper addresses this gap [...] Read more.

Video games come in many genres. Although the popularity of games that belong to different genres is the subject of various research and industry reports, so far, there have been no studies investigating their popularity in research papers. This paper addresses this gap with an analysis of bibliographic data sourced from Scopus, spanning 45 years since the emergence of the topic till today and covering nine widely recognized genres: Action, Puzzle, Rhythm, Role-Playing, Simulation, Sports, Shooter, Strategy, and Traditional. The obtained results not only reveal the current popularity of these video game genres but also illustrate its change over time and geographic distribution as well as highlight the most impactful papers referring to the respective genres and their topics, providing a number of footholds for future studies, including regarding the identified disparities in the research interest in some genres and the number of available games belonging to them, the fluctuations in the relative popularity of the respective genres, and the disparities in the share of research output dedicated to video game genres in the total research output of different countries. Full article

► Show Figures

Figure 1

27 pages, 1540 KiB

Open AccessArticle

Designing Inclusive and Adaptive Content in Moodle: A Framework and a Case Study from Jordanian Higher Education

by Lamis F. Al-Qora’n, Julius T. Nganji and Fadi M. Alsuhimat

Multimodal Technol. Interact. 2025, 9(6), 58; https://doi.org/10.3390/mti9060058 - 10 Jun 2025

Viewed by 393

Abstract

Blended learning has introduced a more accessible and flexible teaching environment in higher education. However, ensuring that content is inclusive, particularly for students with learning difficulties, remains a challenge. This paper explores how Moodle, a widely adopted learning management system (LMS), can support [...] Read more.

Blended learning has introduced a more accessible and flexible teaching environment in higher education. However, ensuring that content is inclusive, particularly for students with learning difficulties, remains a challenge. This paper explores how Moodle, a widely adopted learning management system (LMS), can support inclusive and adaptive learning based on Universal Design for Learning (UDL) principles. A 16-week descriptive exploratory study was conducted with 70 undergraduate students during a software engineering fundamentals course at Philadelphia University in Jordan. The research combined weekly iterative focus groups, teaching reflections, and interviews with 16 educators to identify and address inclusion barriers. The findings highlight that the students responded positively to features such as conditional activities, flexible quizzes, and multimodal content. A UDL-based framework was developed to guide the design of inclusive Moodle content, and it was validated by experienced educators. To our knowledge, this is the first UDL-based framework designed for Moodle in Middle Eastern computing and engineering education. The findings indicate that Moodle features, such as conditional activities and flexible deadlines, can facilitate inclusive practices, but adoption remains hindered by institutional and workload constraints. This study contributes a replicable design model for inclusive blended learning and emphasizes the need for structured training, intentional course planning, and technological support for implementing inclusivity in blended learning environments. Moreover, this study provides a novel weekly iterative focus group methodology, which enables continuous course refinement based on evolving students’ feedback. Future work will look into generalizing the research findings and transferring the findings to other contexts. It will also explore AI-driven adaptive learning pathways within LMS platforms. This is an empirical study grounded in weekly student focus groups, educator interviews, and reflective teaching practice, offering evidence-based insights on the application of UDL in a real-world higher education setting. Full article

(This article belongs to the Special Issue Innovative Theories and Practices for Designing and Evaluating Inclusive Educational Technology and Online Learning)

► Show Figures

Figure 1

15 pages, 2437 KiB

Open AccessArticle

The Impacts of Incorporating Virtual Reality and Data Gloves in Exergames on Intrinsic Motivation in Upper-Extremity Assessments: A Study in a Young and Healthy Group

by He Kunze, Noppon Choosri and Supara Grudpan

Multimodal Technol. Interact. 2025, 9(6), 57; https://doi.org/10.3390/mti9060057 - 9 Jun 2025

Viewed by 784

Abstract

Virtual reality (VR) technology has shown potential as a viable tool for rehabilitation. VR is a well-recognized technology that creates immersive experiences to enhance engagement and encourage more effective participation in activities. In the current study, it has been shown that using a [...] Read more.

Virtual reality (VR) technology has shown potential as a viable tool for rehabilitation. VR is a well-recognized technology that creates immersive experiences to enhance engagement and encourage more effective participation in activities. In the current study, it has been shown that using a standard VR system setup can effectively increase participant motivation for various rehabilitation applications. However, there is a research gap in terms of participant motivation, relating to the intervention of integrating data gloves into VR to improve visibility in hand tracking for rehabilitation. This study presents and assesses an integrated approach utilizing VR and data glove technology to evaluate upper extremity function in a young, healthy population, comparing this to traditional methods. Participants’ intrinsic motivation was measured using the Intrinsic Motivation Inventory (IMI). The findings indicate that the combined immersive environment outperforms conventional practice in most aspects. Therefore, this research also sheds light on the fact that a data glove is promising technology in rehabilitation applications that can augment positive experiences while having no adverse effects on the VR system. Full article

► Show Figures

Graphical abstract

24 pages, 698 KiB

Open AccessSystematic Review

Multiple Ways in Which Video Games Make Education Inclusive: A Systematic Review of Cognitive Enhancement for Neurodivergent Learners

by Martina Basciano, Elisa Bisagno and Alessia Cadamuro

Multimodal Technol. Interact. 2025, 9(6), 56; https://doi.org/10.3390/mti9060056 - 7 Jun 2025

Viewed by 1079

Abstract

Children with neurodevelopmental disorders, such as ADHD, ASD and SLDs, often face challenges in executive functioning that can impact their inclusion and participation in educational and social experiences. Digital technologies, in particular video games, are becoming increasingly more integrated into children’s lives and [...] Read more.

Children with neurodevelopmental disorders, such as ADHD, ASD and SLDs, often face challenges in executive functioning that can impact their inclusion and participation in educational and social experiences. Digital technologies, in particular video games, are becoming increasingly more integrated into children’s lives and are receiving attention as tools to support cognitive development in these populations. This systematic review analyses the effects of video games on Working Memory (WM) and Executive Functions (EFs) in neurodivergent individuals, to evaluate their effectiveness as training tools and their impact in terms of inclusion. Following the PRISMA guidelines, 25 peer-reviewed studies published between 2013 and 2025 were analysed. The interventions included action-video games, exergames, serious games and ad hoc video game training with a pre- and post-assessment in neurodiverse participants. The results indicate that action-video games and exergames show promise in enhancing EFs, while serious games and ad hoc video game training seem to support WM. Despite a few contrasting results, overall, video games are emerging as promising tools of inclusive education thanks to their interactive, customisable and socially empowering nature, especially significant for neurodiverse children. The discussion will depict multiple ways in which video games can make education more inclusive for these populations. Full article

(This article belongs to the Special Issue Innovative Theories and Practices for Designing and Evaluating Inclusive Educational Technology and Online Learning)

► Show Figures

Figure 1

15 pages, 693 KiB

Open AccessArticle

Evaluating a UDL-Based Digital Escape Room for Bullying Awareness in Secondary Education

by Yosra Achab-Moukram, Paula Rodríguez-Rivera, José Manuel Aguilar-Parra, José M. Rodríguez-Ferrer, Javier Rodríguez-Moreno and Ana Manzano-León

Multimodal Technol. Interact. 2025, 9(6), 55; https://doi.org/10.3390/mti9060055 - 6 Jun 2025

Viewed by 944

Abstract

This study evaluates the effectiveness of a digital escape room designed according to the Universal Design for Learning (UDL) principles as an educational tool to raise awareness about school bullying and cyberbullying in lower secondary education. Using a quasi-experimental design with 137 students [...] Read more.

This study evaluates the effectiveness of a digital escape room designed according to the Universal Design for Learning (UDL) principles as an educational tool to raise awareness about school bullying and cyberbullying in lower secondary education. Using a quasi-experimental design with 137 students (66 in the control group and 71 in the experimental group), the gamified activity was compared to a traditional expository intervention. Empathy was measured using the Brief Basic Empathy Scale (BES-B), and bullying-related knowledge and experiences were assessed with the SPB-A scale. The escape room integrated key UDL principles, including multiple means of representation (e.g., video, images, text), action and expression (interactive tasks and puzzles), and engagement (narrative-based missions and collaborative challenges). The results indicate significant improvements in knowledge about behaviors related to school bullying among participants who completed the escape room. Although changes in affective and cognitive empathy were not statistically significant, positive trends were observed in the experimental group. The immersive, collaborative, and accessible nature of the activity positions the escape room as a promising tool within broader strategies for bullying prevention. Full article

(This article belongs to the Special Issue Video Games: Learning, Emotions, and Motivation)

► Show Figures

Figure 1

27 pages, 6914 KiB

Open AccessArticle

A New Serious Game (e-SoundWay) for Learning English Phonetics

by Alfonso Lago-Ferreiro, María Ángeles Gómez-González and José Carlos López-Ardao

Multimodal Technol. Interact. 2025, 9(6), 54; https://doi.org/10.3390/mti9060054 - 4 Jun 2025

Viewed by 1221

Abstract

This paper presents the design and evaluation of e-SoundWay, a cross-platform serious game developed to improve English phonetic competence through a multimodal and narrative-driven approach. While the platform is specifically tailored to meet the needs of Spanish-speaking learners, it is adaptable for [...] Read more.

This paper presents the design and evaluation of e-SoundWay, a cross-platform serious game developed to improve English phonetic competence through a multimodal and narrative-driven approach. While the platform is specifically tailored to meet the needs of Spanish-speaking learners, it is adaptable for a wider range of English as a Foreign Language (EFL) users. e-SoundWay offers over 600 interactive multimedia minigames that target three core competencies: perception, production, and transcription. Learners progress along a gamified version of the Camino de Santiago, interacting with characters representing diverse English accents. A mixed-methods evaluation combining pre- and post-tests with a user experience questionnaire revealed statistically significant improvements across all domains, particularly in perception. Reduced post-test variability indicated more equitable learning outcomes. User satisfaction was high, with 64% of participants reporting satisfaction with their phonetic progress and 91% stating they would recommend the platform. These findings highlight the educational effectiveness, accessibility, and motivational value of e-SoundWay, reinforcing the role of serious games and multimodal technologies in delivering inclusive and engaging pronunciation instruction. Full article

(This article belongs to the Special Issue Video Games: Learning, Emotions, and Motivation)

► Show Figures

Figure 1

25 pages, 5444 KiB

Open AccessArticle

Evaluating the Usability and Engagement of a Gamified, Desktop, Virtual Art Appreciation Module

by Sitong Li, Nurfaradilla Mohamad Nasri and Helmi Norman

Multimodal Technol. Interact. 2025, 9(6), 53; https://doi.org/10.3390/mti9060053 - 3 Jun 2025

Viewed by 976

Abstract

Traditional art appreciation instruction relies heavily on textbooks, slides, or videos, limiting student engagement and immersion. To address this issue, this study proposes a desktop VR (GDVR) art appreciation module based on a gamification approach. Unlike traditional VR art learning environments, the GDVR [...] Read more.

Traditional art appreciation instruction relies heavily on textbooks, slides, or videos, limiting student engagement and immersion. To address this issue, this study proposes a desktop VR (GDVR) art appreciation module based on a gamification approach. Unlike traditional VR art learning environments, the GDVR module combines real-time feedback and gamification elements to increase students’ motivation and understanding of information. This study used focus group interviews to evaluate the usability of the GDVR module, as well as student engagement. In addition, on-screen observations have been adopted to capture student interaction behavior and navigation patterns, providing greater insight into usability. Forty Chinese middle school students participated, and the data were analyzed thematically. The results show that the GDVR module demonstrates stable technical performance, intuitive navigation, and a high level of immersion. Moreover, most students find it more engaging than traditional methods, noting that the real-time feedback system significantly enhanced their engagement with and understanding of the material. Furthermore, this study highlights the practical application potential of utilizing low-cost, desktop-based virtual reality systems in the context of middle school art appreciation. Finally, the study acknowledges its limitations and provides recommendations for future research to further refine and expand the application of GDVR in the field of art education. Full article

► Show Figures

Figure 1

18 pages, 6479 KiB

Open AccessArticle

BabloXR: An Authoring Tool for Developing WebXR Educational Applications

by María Lucía Barrón-Estrada, Ramón Zatarain-Cabada, Manuel Alberto Sotelo-Rivas, María Blanca Ibáñez and Víctor Manuel Bátiz-Beltrán

Multimodal Technol. Interact. 2025, 9(6), 52; https://doi.org/10.3390/mti9060052 - 29 May 2025

Viewed by 756

Abstract

Different areas of education have undergone a significant transformation due to the rapid progress of extended reality in recent years. Nonetheless, existing eXtended Reality (XR) authoring tools often require advanced programming skills, limiting their accessibility to educators and students. To address this challenge, [...] Read more.

Different areas of education have undergone a significant transformation due to the rapid progress of extended reality in recent years. Nonetheless, existing eXtended Reality (XR) authoring tools often require advanced programming skills, limiting their accessibility to educators and students. To address this challenge, we introduce BabloXR, an intuitive authoring tool that leverages Blockly to enable visual, code-free programming for creating educational XR environments. BabloXR was evaluated through a user study involving teachers and students who successfully designed their own XR learning environments. The results indicate that users of BabloXR were able to create complete and detailed scenarios, with usability testing revealing high ease of use, positive user experience, and strong interest in extended reality. Full article

(This article belongs to the Special Issue Innovative Theories and Practices for Designing and Evaluating Inclusive Educational Technology and Online Learning)

► Show Figures

Figure 1

32 pages, 646 KiB

Open AccessArticle

Methods and Findings in the Analysis of Alignment of Bodily Motion in Cooperative Dyadic Dialogue

by Zohreh Khosrobeigi, Maria Koutsombogera and Carl Vogel

Multimodal Technol. Interact. 2025, 9(6), 51; https://doi.org/10.3390/mti9060051 - 27 May 2025

Viewed by 313

Abstract

This research analyses the temporal flow of motion energy (ME) in dyadic dialogues using alternating lagged correlation tests on consecutive windows and also Granger causality (GC) tests. This research considers both alternatives of lagged values, those of the more dominant party preceding those [...] Read more.

This research analyses the temporal flow of motion energy (ME) in dyadic dialogues using alternating lagged correlation tests on consecutive windows and also Granger causality (GC) tests. This research considers both alternatives of lagged values, those of the more dominant party preceding those of the less and vice versa (with relative dominance independently determined) and labels the resulting lagged windows according to the category of correlation (CC) that holds (positive, negative or none, if the correlation is not significant). Similarly, windows are labeled in relation to the significance of GC (one party causing the other, mutual causation, or no causation). Additionally, occurrences of gestures or speech within windows by an interlocutor whose ME precedes are identified. Then, the ME GC labels are compared with labels derived from simple lagged correlation of ME values to identify whether GC or CC is more efficacious in highlighting which participant independent observers classify as the more dominant party, potentially the “leader” for the conversation. In addition, the association between speech, gestures, dominance, and leadership is explored. This work aims to understand how the distributions of these labels interact with independent perceptions of dominance, to what extent dominant interlocutors lead, and the extent to which these labels “explain” variation in ME within any dialogue. Here, the focus is on between speakers dynamics. It shows dominant speakers have measurable influence on their conversation partners through bodily ME, as they are more likely to lead motion dynamics, though moments of mutual influence also occur. While GC and lagged correlation both capture aspects of leadership, GC reveals directional influence, whereas correlation highlights behavioural alignment. Furthermore, ME contrast during speaking and interaction of ME and gestures indicate that bodily movement synchronisation is shaped not only by dominance but also by gesture types and speaking states: speech affects leadership more than gestures. The interactions highlight the multimodal nature of conversational leadership, where verbal and nonverbal modalities interact to shape dialogue dynamics. Full article

► Show Figures

Figure 1

21 pages, 1796 KiB

Open AccessArticle

A Study of NLP-Based Speech Interfaces in Medical Virtual Reality

by Mohit Nayak, Jari Kangas and Roope Raisamo

Multimodal Technol. Interact. 2025, 9(6), 50; https://doi.org/10.3390/mti9060050 - 26 May 2025

Viewed by 815

Abstract

Applications of virtual reality (VR) have grown in significance in medicine, as they are able to recreate real-life scenarios in 3D while posing reduced risks to patients. However, there are several interaction challenges to overcome when moving from 2D screens to 3D VR [...] Read more.

Applications of virtual reality (VR) have grown in significance in medicine, as they are able to recreate real-life scenarios in 3D while posing reduced risks to patients. However, there are several interaction challenges to overcome when moving from 2D screens to 3D VR environments, such as complex controls and slow user adaptation. More intuitive techniques are needed for enhanced user experience. Our research explored the potential of intelligent speech interfaces to enhance user interaction while conducting complex medical tasks. We developed a speech-based assistant within a VR application for maxillofacial implant planning, leveraging natural language processing (NLP) to interpret user intentions and to execute tasks such as obtaining surgical equipment or answering questions related to the VR environment. The objective of the study was to evaluate the usability and cognitive load of the speech-based assistant. We conducted a mixed-methods within-subjects user study with 20 participants and compared the voice-assisted approach to traditional interaction methods, such as button panels on the VR view, across various tasks. Our findings indicate that NLP-driven speech-based assistants can enhance interaction and accessibility in medical VR, especially in areas such as locating controls, easiness of control, user comfort, and intuitive interaction. These findings highlight the potential benefits of augmenting traditional controls with speech interfaces, particularly in complex VR scenarios where conventional methods may limit usability. We identified key areas for future research, including improving the intelligence, accuracy, and user experience of speech-based systems. Addressing these areas could facilitate the development of more robust, user-centric, voice-assisted applications in virtual reality environments. Full article

► Show Figures

Figure 1

Journal Menu

Journal Browser

Multimodal Technol. Interact., Volume 9, Issue 6 (June 2025) – 15 articles

Further Information

Guidelines

MDPI Initiatives

Follow MDPI