Multimodal Technologies and Interaction

32 pages, 8144 KB

Open AccessArticle

Evaluating In-Vehicle Multimodal Interaction via Multimodal Behavioral Signals: A Theory-Driven Tool Chain and Sim-to-Real Pilot Study

by Xinyi Li, Gang Guo, Qihang Sun, Yingzhang Wu and Wenbo Li

Multimodal Technol. Interact. 2026, 10(7), 73; https://doi.org/10.3390/mti10070073 - 29 Jun 2026

Abstract

Multitasking is pervasive in multimodal interaction, particularly within safety-critical domains like driving. Evaluating the impact of In-Vehicle Multimodal Interaction (IVMI) on drivers is critical, yet existing methods predominantly rely on post hoc subjective surveys or coarse unimodal monitoring. Grounded in Multiple Resource Theory [...] Read more.

Multitasking is pervasive in multimodal interaction, particularly within safety-critical domains like driving. Evaluating the impact of In-Vehicle Multimodal Interaction (IVMI) on drivers is critical, yet existing methods predominantly rely on post hoc subjective surveys or coarse unimodal monitoring. Grounded in Multiple Resource Theory and following a Research through Design methodology, we operationalized this theory into a non-intrusive tool chain that evaluates IVMI impact from multimodal behavioral signals (visual, touch, and driving) and supports real-time, objective evaluation in both simulated and real-world domains. To mitigate the Sim-to-Real gap, the method combines real-world multimodal data acquisition with a modality-decoupled cross-domain calibration. Its feasibility was evaluated through a simulator study (

n = 27

) and a small-nscale real-world on-road pilot study (

n = 3

). The results suggest that the tool chain effectively acquires high-fidelity data to support the previously developed evaluation model (Quadratic Weighted Kappa = 0.916) and achieves a preliminary calibration of cross-domain latent feature spaces. As its reference labels are behaviorally derived and share a common basis with the model inputs, this agreement indicates internal consistency rather than independent construct validation. Crucially, while multimodal interaction behaviors (visual and touch) exhibited relatively high cross-domain consistency, real-world driving behaviors showed systematic magnitude suppression. This finding is tentatively interpreted, as a hypothesis to be tested in future work, through the lens of Risk Homeostasis Theory, and highlights the necessity of monitoring multimodal interaction behaviors rather than relying solely on vehicle telemetry. Overall, this research develops and provides preliminary feasibility evidence for a theory-driven cross-domain tool chain, indicating its potential to objectively quantify multimodal interaction impacts in real-world multitasking contexts. Given the small, homogeneous on-road sample, these pilot-stage results should be read as feasibility evidence and a methodological basis for future large-scale, demographically diverse validation. Full article

► Show Figures

Figure 1

15 pages, 6292 KB

Open AccessArticle

Evacuation Safety on Ships Through 360 Virtual Tour Familiarization

by Sebastià Nicolau Morey, Julio Rodríguez, José Antonio Orosa, Juan José Cartelle Barros, Laura Castro-Santos and María Isabel Lamas

Multimodal Technol. Interact. 2026, 10(7), 72; https://doi.org/10.3390/mti10070072 - 29 Jun 2026

Abstract

In the maritime industry, it is common for crew members to be unfamiliar with their workplace until they are on board. This limits evacuation time, which is critical in emergency situations such as fire or shipwreck. The aim of this work is therefore [...] Read more.

In the maritime industry, it is common for crew members to be unfamiliar with their workplace until they are on board. This limits evacuation time, which is critical in emergency situations such as fire or shipwreck. The aim of this work is therefore to enhance the preparedness and safety of new crew members by enabling them to become familiar with the location of escape routes before boarding the ship. For this purpose, the use of 360 technology is proposed. A 360 tour is a virtual, interactive experience that allows users to explore a location as if they were physically there. On a ship, this allows the crew to know the escape routes and make quicker and safer decisions on the most appropriate route. This paper shows how a 360 tour was developed and tested on a merchant vessel. The evacuation time from the engine room has been measured for a group of people. The results showed that the 360 tour had a very positive impact on the evacuation time for these people who had not previously been physically present in the area. Full article

(This article belongs to the Special Issue Educational Virtual/Augmented Reality)

► Show Figures

Figure 1

39 pages, 2176 KB

Open AccessArticle

Contrastive vs. Example-Based Explanations: Designing for Better User Comprehension in Smartphone Privacy Interfaces

by Ananya Bhadauria and Andreas Riener

Multimodal Technol. Interact. 2026, 10(7), 71; https://doi.org/10.3390/mti10070071 - 26 Jun 2026

Abstract

Smartphone applications routinely collect and share personal data, yet users often struggle to understand these practices, particularly when third-party data sharing is involved. Existing mechanisms such as privacy policies and notices provide limited support for user understanding. To address this gap, we investigated [...] Read more.

Smartphone applications routinely collect and share personal data, yet users often struggle to understand these practices, particularly when third-party data sharing is involved. Existing mechanisms such as privacy policies and notices provide limited support for user understanding. To address this gap, we investigated how explainability concepts can enhance contextual privacy policies for mobile apps. We designed two interface prototypes integrating explanation strategies: contrastive explanations, which clarify data-sharing boundaries, and example-based explanations, which illustrate counterfactual scenarios. In an exploratory between-subjects user study (N = 30), we evaluated their impact on comprehensibility, simplicity, cognitive load, and experience. Statistical analysis revealed no significant differences between the two types. Hence, the findings should be read as descriptive trends rather than confirmatory effects. These trends suggested example-based explanations better supported users mental models, while contrastive explanations better supported decision-making. Our findings contribute design recommendations on applying explanation strategies in privacy interfaces, offering guidance for developers and researchers seeking to improve user understanding and trust in digital systems. Full article

► Show Figures

Figure 1

18 pages, 3385 KB

Open AccessArticle

A Preliminary Investigation of Thai Clinical Attitudes Towards VR Adoption in Upper-Extremity Rehabilitation: Patient Usability and Clinician Perceived Usefulness

by Sanya Utthayotha and Noppon Choosri

Multimodal Technol. Interact. 2026, 10(7), 70; https://doi.org/10.3390/mti10070070 - 26 Jun 2026

Abstract

Virtual reality (VR) has shown promising potential for upper-extremity rehabilitation; however, its successful integration into clinical practice depends not only on therapeutic effectiveness but also on the acceptance of the technology by patients and healthcare professionals alike. Despite growing international research in this [...] Read more.

Virtual reality (VR) has shown promising potential for upper-extremity rehabilitation; however, its successful integration into clinical practice depends not only on therapeutic effectiveness but also on the acceptance of the technology by patients and healthcare professionals alike. Despite growing international research in this area, there is limited evidence on clinical attitudes toward VR rehabilitation in Thailand and other middle-income settings. This study investigates Thai patients’ and clinicians’ perceptions of VR for upper-extremity rehabilitation through two complementary studies focusing on perceived usability and usefulness. The first study evaluated the perceived usability of a VR rehabilitation game using the System Usability Scale (SUS) among 40 first-time VR users divided into younger and senior groups. The younger group reported a higher average SUS score (64.6) than the senior group (55.4). While both groups generally perceived VR rehabilitation positively, senior participants expressed greater concern regarding system complexity, consistency, and the need for technical assistance. Nevertheless, the findings indicate that VR remained an acceptable rehabilitation approach even among elderly first-time users in a population with relatively lower technological readiness. The second study explored clinicians’ perceptions of utilizing VR-generated movement data to support rehabilitation decision-making. Five rehabilitation professionals evaluated the potential usefulness of VR data visualizations for diagnosis and treatment monitoring. Clinicians generally perceived VR data as valuable, particularly for tracking rehabilitation progress rather than diagnostic decision-making. Feedback from interviews also highlighted practical considerations for future implementation, including the importance of normative data, simplified visualization formats, and the feasibility of clinical workflows. By combining patient usability perspectives with clinicians’ evaluations of clinical usefulness, this research provides a broader understanding of the factors influencing VR adoption for upper-extremity rehabilitation in Thailand. The findings contribute contextual evidence from an underrepresented healthcare environment and offer insights relevant to the future deployment of VR-assisted rehabilitation systems in similar socio-economic settings. Full article

► Show Figures

Figure 1

19 pages, 855 KB

Open AccessSystematic Review

Effectiveness of PhET Simulations on Learning Outcomes in Science and Chemistry Education: A Systematic Review

by Sinta Ayu Ningrum, Ijang Rohman, Gun Gun Gumilar, Ahmad Mudzakir, Muhammad Nurul Hana and Miarti Khikmatun Nais

Multimodal Technol. Interact. 2026, 10(7), 69; https://doi.org/10.3390/mti10070069 - 24 Jun 2026

Abstract

The development of digital learning technologies has introduced innovative tools to enhance science and chemistry education, including PhET simulations. This study aims to evaluate the effectiveness of PhET simulations on students’ learning outcomes through a systematic literature review following the PRISMA 2020 guidelines. [...] Read more.

The development of digital learning technologies has introduced innovative tools to enhance science and chemistry education, including PhET simulations. This study aims to evaluate the effectiveness of PhET simulations on students’ learning outcomes through a systematic literature review following the PRISMA 2020 guidelines. A systematic search of Scopus and Crossref databases was conducted (last search: January 2026) using predefined keywords. Eligible studies were empirical research published between 2020 and 2026 that investigated PhET simulations in science-related education and reported learning outcomes, while non-empirical studies and non-Scopus-indexed articles were excluded. Risk of bias was assessed using an adapted Joanna Briggs Institute critical appraisal tool. Due to heterogeneity in study designs and outcome measures, the results were synthesized using a narrative approach. A total of 14 studies across elementary to higher education levels were included. The findings indicate that PhET simulations consistently improve learning outcomes, particularly academic achievement and conceptual understanding, with effects generally favoring simulation-based instruction over traditional methods. However, higher-order skills and affective outcomes such as motivation and attitude remain less frequently investigated. The evidence is limited by variability in study designs, incomplete reporting of non-cognitive outcomes, and the absence of quantitative synthesis. Overall, PhET simulations demonstrate strong potential as an effective interactive learning medium, although their impact depends on instructional design, teacher facilitation, and technological accessibility. Full article

(This article belongs to the Special Issue Online Learning to Multimodal Era: Interfaces, Analytics and User Experiences)

► Show Figures

Figure 1

23 pages, 3967 KB

Open AccessArticle

Automating Spatial Visualisation of Handwritten Vector Equations Using Large Vision Models in Pre-Tertiary Mathematics

by Kenneth Y. T. Lim, Nguyen Thanh Minh Le and Sopheap Chanoudam

Multimodal Technol. Interact. 2026, 10(6), 68; https://doi.org/10.3390/mti10060068 - 14 Jun 2026

Abstract

Understanding advanced pre-tertiary mathematics, particularly three-dimensional vectors, demands robust spatial reasoning skills that many students find challenging to develop through traditional pedagogical methods. This study proposes and evaluates an innovative educational tool that leverages large vision models to automate the conversion of handwritten [...] Read more.

Understanding advanced pre-tertiary mathematics, particularly three-dimensional vectors, demands robust spatial reasoning skills that many students find challenging to develop through traditional pedagogical methods. This study proposes and evaluates an innovative educational tool that leverages large vision models to automate the conversion of handwritten vector equations into accurate 3D graphical representations. By interpreting students’ handwritten input using advanced computer vision, the system provides immediate, interactive visual feedback to bridge the cognitive gap between abstract symbolic notation and tangible geometric concepts. We evaluated the system using a dataset of 1000 handwritten vector equations typical of the Singapore-Cambridge GCE ‘A’ Level H2 Mathematics syllabus. Our findings demonstrate that while GPT-4o serves as a capable baseline, achieving 84.6% accuracy with multi-shot prompting, newer variants such as GPT-4.1-mini offer superior performance, reaching 91.4% accuracy with significantly higher computational efficiency. The results confirm that AI-powered visualisation tools can effectively interpret complex spatial mathematical layouts when guided by optimal prompt engineering. Implementing such technology in educational settings presents a viable, scalable, and cost-effective method to democratise learning support, fostering independent study and enhancing students’ conceptual comprehension of spatial mathematics. Full article

► Show Figures

Figure 1

40 pages, 19981 KB

Open AccessArticle

Digital Tools for Innovation in Craft Design: Lessons from a Multi-Domain European Design Pilot

by Arnaud Dubois, Zoé L’Évêque, Inés Moreno, Loïc Petitgirard, Danae Kaplanidi, Juan Carlos Bañón, Juan José Ortega, Nikolaos Partarakis and Xenophon Zabulis

Multimodal Technol. Interact. 2026, 10(6), 67; https://doi.org/10.3390/mti10060067 - 4 Jun 2026

Abstract

Traditional European craft practices face dual pressures: the erosion of tacit knowledge held by aging practitioners, and the risk of cultural homogenization through uninformed digital adoption. This paper presents a comparative analysis of a structured design pilot conducted across five Representative Craft Instances [...] Read more.

Traditional European craft practices face dual pressures: the erosion of tacit knowledge held by aging practitioners, and the risk of cultural homogenization through uninformed digital adoption. This paper presents a comparative analysis of a structured design pilot conducted across five Representative Craft Instances (RCIs): glassblowing, tapestry, marble/silversmithing, porcelain, and woodcarving within the Horizon Europe CRAEFT project. Drawing on co-creative workshops, motion capture pipelines, physically based rendering (PBR), interactive simulation, and additive manufacturing, we analyze how context-specific digital tools performed as mediators rather than modernizers across heterogeneous craft domains. Cross-domain analysis reveals that digital tools achieve cultural legitimacy only when introduced through co-creative, practitioner-led cycles; that gesture and tacit knowledge are transferable via structured computational pipelines; and that methodological portability, not workflow replication, is the appropriate model for cross-context scaling. Implications are discussed for sustainable heritage policy, design education, and the development of craft-sensitive digital infrastructure in Europe. A cross-RCI comparative assessment matrix evaluates all five domains across seven analytical dimensions: practitioner adoption, perceived usefulness, cultural legitimacy, technical maturity, sustainability impact, transferability potential, and educational effectiveness. Finally, practitioner reflective accounts from participating designers and craftspeople are presented to ground the analytical findings empirically. Full article

► Show Figures

Figure 1

18 pages, 1447 KB

Open AccessSystematic Review

Parental Communication Strategies During Screen Time in Early Childhood: A Scoping Review of Joint Media Engagement

by Litna A Varghese, Gagan Bajaj, Megha Mohan, Jayashree S. Bhat, Jayashree Kanthila and Aiswarya Liz Varghese

Multimodal Technol. Interact. 2026, 10(6), 66; https://doi.org/10.3390/mti10060066 - 4 Jun 2026

Abstract

Background: This scoping review aimed to systematically identify communication strategies used during Joint Media Engagement (JME) and examine their associations with developmental outcomes and contextual factors. Methods: A systematic search of seven databases (up to April 2025) was conducted using Rayyan, [...] Read more.

Background: This scoping review aimed to systematically identify communication strategies used during Joint Media Engagement (JME) and examine their associations with developmental outcomes and contextual factors. Methods: A systematic search of seven databases (up to April 2025) was conducted using Rayyan, following PRISMA-ScR guidelines; 26 studies met inclusion criteria and were synthesized to categorize parent communication strategies and their theoretical underpinnings. Results: Fifteen distinct communication strategies were identified and organized into four theoretical frameworks; Social Learning, Sociopragmatic, Behaviourist, and Theory of Mind along with a fifth category for technical scaffolding. Strategies aligned with Social Learning were most frequently reported and consistently associated with improvements in children’s language, cognitive, and socio-emotional outcomes. Findings also showed that JME strategies vary based on contextual factors, including parent type, geography, device type, media content, and child characteristics. Although most studies did not explicitly focus on JME, those employing mixed methods provided deeper insights. Conclusions: JME is shaped by both interaction quality and context, with Social Learning-based strategies playing a central role in supporting child development. The findings highlight the need for more rigorous, JME-focused research across diverse digital formats to strengthen the evidence-based parent coaching approaches to optimize JME practices in early childhood. Full article

► Show Figures

Figure 1

31 pages, 22757 KB

Open AccessArticle

Personalizing Live Avatar Interaction for Children with ASD Through Restricted Interests: A Feasibility Study

by Luis Fernando Guerrero-Vásquez, Martín López-Nores, Henry J. Jara-Quito, Dalila M. González-González and Jack Fernando Bravo-Torres

Multimodal Technol. Interact. 2026, 10(6), 65; https://doi.org/10.3390/mti10060065 - 2 Jun 2026

Abstract

Virtual avatars have shown potential as supports in Autism Spectrum Disorder (ASD) interventions, but many existing systems provide largely standardized interactions that do not account for individual variability. This study presents an exploratory evaluation of a virtual puppet system that enables real-time interaction [...] Read more.

Virtual avatars have shown potential as supports in Autism Spectrum Disorder (ASD) interventions, but many existing systems provide largely standardized interactions that do not account for individual variability. This study presents an exploratory evaluation of a virtual puppet system that enables real-time interaction by synchronously transmitting a human model’s movements, facial gestures, and voice to a digital avatar. The system was personalized using each participant’s restricted interests (RIs), identified through a clinical triangulation process involving therapist input, caregiver reports, and observation. After an initial technical validation with 16 neurotypical children, the system was evaluated in a proof-of-concept sample of 11 children with ASD (7 in an experimental group exposed to RI-based personalization and 4 in a control group interacting with a standard interface). Data sources included eye tracking and therapist-completed observational questionnaires. Across sessions, descriptive patterns in gaze fixation and therapist reports suggested that RI-based personalization may help sustain attention to the screen and support engagement with the therapeutic environment relative to non-personalized interaction. Heatmap patterns further indicated that children under the personalized condition visually explored RI-related elements within the scene. This study provides evidence of technical and procedural feasibility and generates hypotheses for future research. Full article

► Show Figures

Figure 1

29 pages, 3261 KB

Open AccessArticle

Illusionary Selves: Critiquing Online Persona Construction Through AI-Mediated Interaction Design

by Xueyi Li, Yonghong Liu and Yangcheng Wang

Multimodal Technol. Interact. 2026, 10(6), 64; https://doi.org/10.3390/mti10060064 - 1 Jun 2026

Abstract

Social media platforms have become central sites of identity construction, where visibility and legitimacy are shaped through algorithmic systems, aesthetic conventions, and platform economies. This paper approaches online personas through the lens of illusionary selves, understood here as online personas experienced as authentic [...] Read more.

Social media platforms have become central sites of identity construction, where visibility and legitimacy are shaped through algorithmic systems, aesthetic conventions, and platform economies. This paper approaches online personas through the lens of illusionary selves, understood here as online personas experienced as authentic while being shaped by sociotechnical processes, examining how they are produced through sociotechnical processes entangling design practices, generative artificial intelligence(AI), and cultural expectations. We present an AI-mediated critical design inquiry into how generative systems translate and normalize visual patterns of online self-imaging. Using a pix2pix-based model trained on 630 internet celebrity selfies, facial images are abstracted into dot-based representations and aggregated across selfie angles, foregrounding repetition and normalization. An interactive design installation links bodily orientation and numerical parameters to generative output in real time, introducing perceptual friction in self-imaging. A total of 30 participants engaged with the system in situated contexts, and their experiences were documented through observation, video recording, and a 5-point Likert questionnaire across three dimensions: perceptual friction, awareness of algorithmic mediation, and reflective responses to self-presentation. Results indicate high levels of perceptual friction (mean [M] = 4.21), strong awareness of algorithmic mediation (M = 4.29), and consistent reflective unease (M = 4.07). Through situated use, the system renders algorithmic mediation tangible and positions AI as an implicated actor in identity construction. This work contributes a conceptual framing of AI-mediated critical design, showing how generative and interactive systems operate as epistemic devices interrogating online persona construction. Full article

(This article belongs to the Special Issue Human-AI Collaborative Interaction Design: Rethinking Human-Computer Symbiosis in the Age of Intelligent Systems)

► Show Figures

Figure 1

33 pages, 10391 KB

Open AccessArticle

Computational Method for Predicting Visual Attention in Older Adults with Age-Related Features

by Xiangdong Li, Xinchi Shi, Haoyu Gu, Tianai Shen, Shiwei Cheng and Jing Wang

Multimodal Technol. Interact. 2026, 10(6), 63; https://doi.org/10.3390/mti10060063 - 1 Jun 2026

Abstract

Age-related changes in visual perception alter attentional deployment, yet computational models of visual attention have been validated almost exclusively on younger populations. This limits both the theoretical investigation of age-specific mechanisms and practical applications in age-inclusive design, where researchers depend on specialised eye-tracking [...] Read more.

Age-related changes in visual perception alter attentional deployment, yet computational models of visual attention have been validated almost exclusively on younger populations. This limits both the theoretical investigation of age-specific mechanisms and practical applications in age-inclusive design, where researchers depend on specialised eye-tracking equipment to observe such differences. Therefore, we present the Elderly Visual Attention Estimation (EVAE) model, a computational framework that predicts early visual attentional orienting in older adults by combining stimulus-driven image features with age-specific top-down priors. The framework models six dimensions of elderly visual attention from cross-age eye-tracking data: colour brightness sensitivity, centre bias, foreground–background differentiation, depth detection, early attentional prior, and sustained-attention spatial prior. On public datasets, EVAE achieves an AUC-Judd of 0.92, which outperforms existing saliency models and deep learning approaches such as DeepGaze II. The framework is optimised for an input resolution of 128 × 96 pixels, producing fixation probability maps that are upsampled to match the original stimulus resolution for practical interface evaluation. Cross-age validation confirms the model’s specificity, as EVAE predicts attentional behaviour in older adults but does not generalise to younger adults. An ablation study shows that image features and top-down spatial priors each contribute independently to prediction accuracy, and that bottom-up saliency alone cannot account for age-related attentional patterns. Centre bias and early attentional prior are the strongest predictors, indicating that visual ageing involves greater reliance on spatial strategies and compensatory processing. As an alternative to hardware-based eye-tracking, EVAE widens the scope of empirical research into older adults’ visual attention and informs the design of accessible digital interfaces. Full article

► Show Figures

Figure 1

15 pages, 547 KB

Open AccessArticle

First-Grade Students’ Perspectives on Digital and Traditional Learning

by Josipa Jurić, Branka Šegvić and Zoa Šimundić

Multimodal Technol. Interact. 2026, 10(6), 62; https://doi.org/10.3390/mti10060062 - 1 Jun 2026

Abstract

The development of digital technologies in early education raises important questions regarding how students perceive and experience learning in digital environments. The aim of this study was to explore first-grade students’ perspectives on learning using mobile devices, tablets, and digital textbooks, with particular [...] Read more.

The development of digital technologies in early education raises important questions regarding how students perceive and experience learning in digital environments. The aim of this study was to explore first-grade students’ perspectives on learning using mobile devices, tablets, and digital textbooks, with particular emphasis on their perceived advantages, limitations, and preferences. The research was conducted using a qualitative approach through four focus groups with a total of 20 first-grade students. The results indicate that students recognise the motivational and stimulating potential of digital technologies, particularly in terms of visualisation and the engaging nature of content. However, they simultaneously express a clear preference for traditional learning, emphasising the importance of concentration, independent thinking, and adult support. Digital tools are perceived as useful but secondary, and are often associated with distractions and reduced cognitive effort. The findings suggest that students do not perceive all forms of learning equally, but rather associate meaningful learning with effort, autonomy, and active cognitive engagement. The results highlight the need for the careful integration of digital technologies into the teaching process, particularly in early education, where attention, structure, and social interaction are crucial for effective learning. Full article

(This article belongs to the Special Issue Online Learning to Multimodal Era: Interfaces, Analytics and User Experiences)

► Show Figures

Figure 1

35 pages, 5851 KB

Open AccessArticle

AUMOR: Augmented-Reality-Based Mobile Application for University Orientation

by Muhammad Nadeem, Melinda Oroszlanyova, Pauly Awad, Hasan Ozkan and Svetlana Beryozkina

Multimodal Technol. Interact. 2026, 10(6), 61; https://doi.org/10.3390/mti10060061 - 29 May 2026

Abstract

Fresh engineering students are often required to absorb a large amount of new information within a short period of time, which can be academically and emotionally challenging. To address this challenge, this study introduces AUMOR, a mobile application designed to enhance university orientation [...] Read more.

Fresh engineering students are often required to absorb a large amount of new information within a short period of time, which can be academically and emotionally challenging. To address this challenge, this study introduces AUMOR, a mobile application designed to enhance university orientation by delivering contextual information at the point of need. It integrates GPS-based localization with QR code triggers to provide real-time, location-specific guidance and interactive content through an augmented reality (AR) interface. It uses GPS functionality to provide real-time location-based services, including information about academic buildings, student services, and recreational facilities. The QR codes on devices and laboratory equipment provide relevant information when scanned. A post-deployment user perception survey was conducted using a paper-based questionnaire involving 128 participants, including both students and faculty members. The results indicate that users perceived the application as helpful in enhancing their spatial awareness, navigation confidence, and ability to locate campus facilities, demonstrating high levels of usability and acceptance. The findings suggest that students perceived AUMOR as helpful for university orientation and suggest potential as a scalable solution. Full article

(This article belongs to the Special Issue Educational Virtual/Augmented Reality)

► Show Figures

Graphical abstract

20 pages, 6107 KB

Open AccessArticle

Improving Chatbot Usability Through Structured Prompt-Based Interaction Design

by Gisel Katerine Bastidas-Guacho, Edison Patricio Azogue Martínez, Marco Antonio Gabilanes Martínez and Patricio Xavier Moreno-Vallejo

Multimodal Technol. Interact. 2026, 10(6), 60; https://doi.org/10.3390/mti10060060 - 28 May 2026

Abstract

This study presents a comparative evaluation of the usability of an intelligent chatbot implemented in a childcare center management system, focusing on the impact of a prompt-enhanced conversational configuration on user experience. The Chatbot Usability Questionnaire (CUQ) was used to assess perceived usability [...] Read more.

This study presents a comparative evaluation of the usability of an intelligent chatbot implemented in a childcare center management system, focusing on the impact of a prompt-enhanced conversational configuration on user experience. The Chatbot Usability Questionnaire (CUQ) was used to assess perceived usability under two conditions: a baseline configuration and an enhanced configuration incorporating role-based prompting and preprocessing mechanisms. The results indicate a substantial increase in CUQ scores, from 69 in the baseline condition to 91 in the enhanced condition, suggesting improved perceived usability. Rather than isolating prompt engineering as a standalone variable, this work evaluates a system-level design approach that integrates structured prompts, role-based contextualization, and interaction refinement strategies. This study contributes to the understanding of how prompt-enhanced conversational designs can improve response clarity, relevance, and interaction quality in multi-role environments, including parents, teachers, and administrators. The findings provide empirical evidence that such configurations are associated with more coherent and role-appropriate interactions in service-oriented chatbot systems. Full article

► Show Figures

Figure 1

20 pages, 1030 KB

Open AccessArticle

The Pedagogical Transfer Chain in the DigCompEdu Framework from a Teacher-Reported Perspective: A Predictive Analysis Using PLS-SEM and ANN

by Daira Marizol Carvajal Morales, Jessica Mariela Carvajal Morales, Milton Alfonso Criollo Turusina, Santiago José Chele Delgado, Erika Jadira Romero Cardenas and Juan Diego Valenzuela Cobos

Multimodal Technol. Interact. 2026, 10(6), 59; https://doi.org/10.3390/mti10060059 - 26 May 2026

Abstract

The steady advancement of online education has not automatically translated into improved educational quality. Teacher training often continues to focus on the technical use of digital tools, while the pedagogical processes through which teachers report supporting students’ digital competence remain insufficiently understood. The [...] Read more.

The steady advancement of online education has not automatically translated into improved educational quality. Teacher training often continues to focus on the technical use of digital tools, while the pedagogical processes through which teachers report supporting students’ digital competence remain insufficiently understood. The objective of this study was to examine the sequential and predictive structure of teachers’ digital competence using the DigCompEdu framework as a reference. A quantitative cross-sectional study was conducted with a sample of 136 university teachers involved in online education. Data were collected through a self-reported questionnaire based on DigCompEdu and analyzed in two phases: Partial Least Squares Structural Equation Modeling (PLS-SEM) and Artificial Neural Networks (ANNs). The PLS-SEM results suggested a sequential pattern of associations among teacher-reported constructs: Professional Commitment (PC) was positively associated with Digital Resource Management (DR), which in turn was positively associated with Digital Pedagogy (DP) and Assessment and Feedback (AF). These dimensions were associated with Student Empowerment (SE), which showed the strongest positive relationship with teachers’ reported practices for Facilitating Students’ Digital Competence (FS). The ANN sensitivity analysis showed adequate predictive performance in the testing phase (RMSE = 0.155) and identified Student Empowerment as the predictor with the highest normalized importance within the specified model. These findings suggest that faculty development in online higher education may benefit from moving beyond basic digital literacy and platform management toward pedagogical design, formative assessment, inclusive participation, and learner agency. However, the results should be interpreted as evidence of teacher-reported facilitation practices within the analyzed sample, rather than as direct evidence of students’ actual digital competence development. Full article

(This article belongs to the Special Issue Online Learning to Multimodal Era: Interfaces, Analytics and User Experiences)

► Show Figures

Figure 1

36 pages, 2361 KB

Open AccessReview

A Comprehensive Review of Deep Learning Approaches for Video-Based Sign Language Recognition: Datasets, Challenges and Insights

by Ulmeken Berzhanova, Aigerim Yerimbetova, Marek Milosz, Bakzhan Sakenov, Dina Oralbekova, Elmira Daiyrbayeva and Daniyar Turgan

Multimodal Technol. Interact. 2026, 10(6), 58; https://doi.org/10.3390/mti10060058 - 22 May 2026

Abstract

This study presents a comprehensive review of more than 100 research papers on sign language recognition (SLR) published between 2020 and 2026. The analysis focuses on deep learning approaches applied to video-based SLR, including spatiotemporal feature extraction, temporal modeling, attention mechanisms, motion-based representations, [...] Read more.

This study presents a comprehensive review of more than 100 research papers on sign language recognition (SLR) published between 2020 and 2026. The analysis focuses on deep learning approaches applied to video-based SLR, including spatiotemporal feature extraction, temporal modeling, attention mechanisms, motion-based representations, hybrid frameworks, transfer learning methods and other methods. Particular attention is given to how these methods model spatiotemporal dynamics and capture subtle gesture characteristics in sign language communication. The review highlights several recent developments, such as the introduction of specialized datasets, the emergence of real-time recognition systems, and the integration of multimodal fusion strategies. At the same time, persistent challenges remain, including data scarcity in low-resource sign languages, limited linguistic standardization of datasets, and insufficient model interpretability. The findings underline the importance of developing scalable and generalizable models capable of handling diverse datasets and user variability. The distinct contributions of this review are fourfold: (1) a comprehensive synthesis of over 100 studies published between 2020 and 2026, covering the full spectrum of deep learning architectures for video-based SLR; (2) a structured six-category taxonomy enabling systematic cross-architectural comparison; (3) a comprehensive focus on low-resource sign languages, which remain underrepresented in the existing literature; and (4) a critical analysis of the current benchmark landscape for low-resource sign languages, identifying key gaps and outlining strategic directions for future dataset development. These contributions are intended to guide further research toward more robust, inclusive, and universally applicable SLR systems. Full article

► Show Figures

Figure 1

44 pages, 2602 KB

Open AccessArticle

From Prompt to Play: Examining Computational Thinking Through Vibe Coding in Game Making for Pre-Service Teacher Education

by Nikolaos Pellas

Multimodal Technol. Interact. 2026, 10(5), 57; https://doi.org/10.3390/mti10050057 - 21 May 2026

Abstract

Computational thinking (CT) is increasingly recognized as essential in education, yet teacher preparation programs struggle to develop both computational proficiency and pedagogical readiness in pre-service teachers (PSTs). This study examines an AI-mediated, game-making course grounded in the emerging “vibe coding” paradigm, where 24 [...] Read more.

Computational thinking (CT) is increasingly recognized as essential in education, yet teacher preparation programs struggle to develop both computational proficiency and pedagogical readiness in pre-service teachers (PSTs). This study examines an AI-mediated, game-making course grounded in the emerging “vibe coding” paradigm, where 24 novice PSTs iteratively constructed programs through natural language prompting. Adopting a mixed-methods design, the study drew on pre- and post-course attitude questionnaires, reflective accounts of prompting strategies, and open-ended responses. Results indicate that participants substantively engaged with core CT practices, particularly debugging, iterative refinement, and problem decomposition. Nonetheless, this downward recalibration in self-reported coding and teaching confidence represents a productive adjustment rather than a failure. Conversely, attitudes toward game-making improved significantly, with a statistically significant medium effect size for perceived instructional value (d = 0.51), the largest practical effect observed across dimensions. Most participants intended to integrate CT into future teaching. These findings suggest that prompt-driven learning environments support meaningful engagement with computational processes when carefully scaffolded, but do not inherently ensure pedagogical readiness, particularly for higher-order CT practices such as abstraction and pattern recognition. Unlike prior research that has examined game-making processes or PST attitudes toward CT in isolation, this study empirically integrates all three within a single scaffolded instructional design using vibe coding. This integration enables a process-level account of how CT is enacted—and how it develops—when code generation is partially delegated to AI systems. Beyond documenting attitude shifts, the study introduces an analytical rubric for identifying CT engagement in AI-mediated prompting and derives evidence-based design principles that specify the pedagogical conditions under which vibe coding supports, rather than bypasses, computational reasoning. Full article

(This article belongs to the Special Issue Technology-Enhanced Game-Based Approaches in Education: Learning, Emotions, and Motivation)

► Show Figures

Figure 1

17 pages, 1515 KB

Open AccessArticle

Attention-Based Multimodal Fusion for Salience-Aware Blended Emotion Recognition

by José Salas-Cáceres, Modesto Castrillón-Santana, Oliverio J. Santana, Daniel Hernández-Sosa and Javier Lorenzo-Navarro

Multimodal Technol. Interact. 2026, 10(5), 56; https://doi.org/10.3390/mti10050056 - 20 May 2026

Abstract

Blended emotion recognition introduces the challenge of identifying not only which emotions are present in an expressive display but also their relative salience. The proposed methodology builds upon the pre-extracted features provided with the dataset and enhances performance through a combination of temporal [...] Read more.

Blended emotion recognition introduces the challenge of identifying not only which emotions are present in an expressive display but also their relative salience. The proposed methodology builds upon the pre-extracted features provided with the dataset and enhances performance through a combination of temporal modeling and multimodal fusion strategies. Unimodal experiments revealed that visual encoders consistently outperformed audio ones, with the multimodal HiCMAE encoder achieving the strongest single-encoder results with 34% presence accuracy and 18.23% salience accuracy. Multimodal fusion further improved performance, with the best validation results obtained using a combination of simple concatenation and attention-based fusion, reaching 47.86% in presence accuracy and 27.92% in salience accuracy. Overall, the proposed methodology surpasses the chosen baseline introduced in the original paper across a k-fold experiment, confirming the effectiveness of multimodal attention-based fusion for the accurate prediction of both emotion presence and salience in blended affective behaviour. The experimental results further indicate that multimodal expression recognition consistently outperforms unimodal approaches, highlighting the complementary nature of cross-modal information. Full article

► Show Figures

Figure 1

18 pages, 4228 KB

Open AccessArticle

MAVAGEN: Multimodal Avatar Generation Framework for Personalized Human–Computer Interaction

by Alexandr Axyonov, Elena Ryumina, Dmitry Ryumin and Alexey Karpov

Multimodal Technol. Interact. 2026, 10(5), 55; https://doi.org/10.3390/mti10050055 - 18 May 2026

Abstract

Digital-avatar systems still provide limited control over emotionally expressive behavior in human–computer interaction, especially in Large Language Model (LLM)-based chatbots and virtual assistants with personalized visual embodiments. To address this problem, we propose Multimodal Avatar Generation (MAVAGEN), a multimodal avatar generation framework for [...] Read more.

Digital-avatar systems still provide limited control over emotionally expressive behavior in human–computer interaction, especially in Large Language Model (LLM)-based chatbots and virtual assistants with personalized visual embodiments. To address this problem, we propose Multimodal Avatar Generation (MAVAGEN), a multimodal avatar generation framework for synthesizing upper-body digital avatars with personalized appearance and controllable emotional expression. The user specifies the desired gender and age, as well as provides a short text input from which the target emotional state is inferred. MAVAGEN then retrieves an identity image from the HaGRIDv2-1M corpus and generates an avatar clip with synchronized facial expressions, hand gestures, and expressive speech. The framework uses the following six feature streams: textual features, emotion-distribution features, landmark-based pose features, depth-geometry features, RGB-appearance features, and acoustic features. In a quantitative evaluation against recent human animation methods, MAVAGEN achieves the best overall avatar quality, with FID 48.20, FVD 592.00, SSIM 0.741, Sync-C 7.40, HKC 0.929, HKV 25.30, CSIM 0.563, and EmoAcc 0.88. Ablation results show that emotion and acoustic features contribute most to emotional agreement, while landmark-based pose and depth features improve geometric and motion stability. These results support the practical use of MAVAGEN in personalized LLM-based assistants and other emotion-sensitive interactive systems. Full article

► Show Figures

Figure 1

Journal Description

Multimodal Technologies and Interaction

Latest Articles

Journal Menu

Journal Browser

Highly Accessed Articles

Latest Books

E-Mail Alert

News

Topics

Conferences

Special Issues

Further Information

Guidelines

MDPI Initiatives

Follow MDPI