Multimodal Technologies and Interaction

Acid-base analysis is a high-load diagnostic skill that many medical students struggle to master when taught using traditional text-based flowcharts. This multi-institution mixed-methods study evaluated a novel visual mnemonic algorithm that integrated Medimon characters, symbolic imagery, and pop-culture references into the standard acid-base diagnostic framework. First-year medical students (n = 273) at six distributed WWAMI campuses attended an identical lecture on acid-base physiology. Students at five control campuses received the original text-based algorithm, while students at one experimental campus received the Medimon algorithm in addition. Achievement was measured with a unit exam (nine focal items, day 7) and a final exam (four focal items, day 11). A Differences-in-Differences approach compared performance on focal items versus baseline items across sites. Students at the experimental campus showed no significant advantage on the unit exam (DiD = +1.2%, g = 0.12) but demonstrated a larger, but still non-significant, medium-to-large effect on the final exam (DiD = +11.0%, g = 0.85). At the experimental site, 39 students completed the Situational Interest Survey for Multimedia (SIS-M), revealing significantly higher triggered, maintained-feeling, maintained-value, and overall situational interest scores for the Medimon algorithm (all p < 0.001). Thematic analysis of open-ended responses identified four themes: enhanced clarity, improved memorability, increased engagement, and barriers to interpretation. Collectively, the findings suggest that embedding visual mnemonics and serious-game characters into diagnostic algorithms can enhance learner interest and may improve long-term retention in preclinical medical education. Full article

Background: Precision nutrition increasingly integrates mobile health (mHealth) and artificial intelligence (AI) tools. However, personalized hydration remains underdeveloped, particularly in accounting for both food- and beverage-derived water intake. Objective: This scoping review maps the existing literature on mHealth applications that incorporate machine learning (ML) or AI for personalized hydration. The focus is on systems that combine dietary (food-based) and fluid (beverage-based) water sources to generate individualized hydration assessments and recommendations. Methods: Following the PRISMA-ScR guidelines, we conducted a structured literature search across three databases (PubMed, Scopus, Web of Science) through March 2025. Studies were included if they addressed AI or ML within mHealth platforms for personalized hydration or nutrition, with an emphasis on systems using both beverage and food intake data. Results: Of the 43 included studies, most examined dietary recommender systems or hydration-focused apps. Few studies used hydration assessments focusing on both food and beverages or employed AI for integrated guidance. Emerging trends include wearable sensors, AR tools, and behavioral modeling. Conclusions: While numerous digital health tools address hydration or nutrition separately, there is a lack of comprehensive systems leveraging AI to guide hydration from both food and beverage sources. Bridging this gap is essential for effective, equitable, and precise hydration interventions. In this direction, we propose a hydration diet recommender system that integrates demographic, anthropometric, psychological, and socioeconomic data to create a truly personalized diet and hydration plan with a holistic approach. Full article

This paper presents a serious game designed to support the teaching of PID controllers. The game couples a visually clear Unity scene with a physics-accurate digital model of a drone with a single degree of freedom (called a one-axis drone) and helps prepare students to meet the demands of Industry 4.0 and 5.0. An analytics back-end logs system error at 10 Hz and interaction metrics, enabling instructors to diagnose common tuning issues from a plot and to provide actionable hints to students. The design process that led to choosing the one-axis drone and turbulence application via “turbulence balls” is explained, after which the implementation is described. The proposed solution is evaluated in a within-subjects study performed with 21 students from mixed technical backgrounds across two short, unsupervised tinkering sessions of up to 10 min framed by four quizzes of both general and theoretical content. Three questions shaped the analysis: (i) whether error traces can be visualized by instructors to generate actionable hints for students; (ii) whether brief, unsupervised play sessions yield measurable gains in knowledge or stability; and (iii) whether efficiency of tuning improves without measurable changes in tune performance. Results show that analysis of plotted error values exposes recognizable issues with PID tunes that map to concrete hints provided by the instructor. When it comes to unsupervised play sessions, no systematic pre/post improvement in quiz scores or normalized area under absolute error was observed. However, it required significantly less effort from students in the second session to reach the same tune performance, indicating improved tuning efficiency. Overall, the proposed serious game with the digital twin-inspired one-axis drone and custom analytics back-end has emerged as a practical, safe, and low-cost auxiliary tool for teaching PID controllers, helping bridge the gap between theory and practice. Full article

As AI systems become more integrated into society, the relationship between humans and AI is shifting from simple automation to co-creative collaboration. This evolution is particularly important in education, where human intuition and imagination can combine with AI’s computational power to enable innovative forms of learning and teaching. This study is grounded in the #ppAI6 model, a framework that describes six levels of creative engagement with AI in educational contexts, ranging from passive consumption to active, participatory co-creation of knowledge. The model highlights progression from initial interactions with AI tools to transformative educational experiences that involve deep collaboration between humans and AI. In this study, we explore how educators and learners can engage in deeper, more transformative interactions with AI technologies. The #ppAI6 model categorizes these levels of engagement as follows: level 1 involves passive consumption of AI-generated content, while level 6 represents expansive, participatory co-creation of knowledge. This model provides a lens through which we investigate how educational tools and practices can move beyond basic interactions to foster higher-order creativity. We conducted a systematic literature review following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines for reporting the levels of creative engagement with AI tools in education. This review synthesizes existing literature on various levels of engagement, such as interactive consumption through Intelligent Tutoring Systems (ITS), and shifts focus to the exploration and design of higher-order forms of creative engagement. The findings highlight varied levels of engagement across both learners and educators. For learners, a total of four studies were found at level 2 (interactive consumption). Two studies were found that looked at level 3 (individual content creation). Four studies focused on collaborative content creation at level 4. No studies were observed at level 5, and only one study was found at level 6. These findings show a lack of development in AI tools for more creative involvement. For teachers, AI tools mainly support levels two and three, facilitating personalized content creation and performance analysis with limited examples of higher-level creative engagement and indicating areas for improvement in supportive collaborative teaching practices. The review found that two studies focused on level 2 (interactive consumption) for teachers. In addition, four studies were identified at level 3 (individual content creation). Only one study was found at level 5 (participatory co-creation), and no studies were found at level 6. In practical terms, the review suggests that educators need professional development focused on building AI literacy, enabling them to recognize and leverage the different levels of creative engagement that AI tools offer. Full article

Dementia and mild cognitive impairment (MCI) are growing health concerns in Canada’s aging population. Over 700,000 Canadians currently live with dementia, and this number is expected to rise. As the older adult population increases, coupled with an already strained healthcare system, there is a pressing need for innovative tools that support aging in place. This study explored the feasibility and acceptability of using a Digital Human (DH) conversational agent, combined with AI-driven speech analysis, to monitor cognitive function, anxiety, and depression in cognitively healthy community-dwelling older adults (CDOA) aged 65 and older. Sixty older adults participated in up to three in-person sessions over six months, interacting with the DH through journaling and picture description tasks. Afterward, 51 of the participants completed structured interviews about their experiences and perceptions of the DH and AI more generally. Findings showed that 84% enjoyed interacting with the DH, and 96% expressed interest in learning more about AI in healthcare. While participants were open and curious about AI, 67% voiced concerns about AI replacing human interaction in healthcare. Most found the DH friendly, though reactions to its appearance varied. Overall, participants viewed AI as a promising tool, provided it complements, rather than replaces, human interactions. Full article

The increasing ubiquity of wearable computing and multimodal interaction technologies has created unprecedented opportunities for natural and seamless human–computer interaction. However, most existing systems adapt only to external user actions such as speech, gesture, or gaze, without considering internal cognitive or affective states. This limits their ability to provide intelligent and empathetic adaptations. This paper addresses this critical gap by proposing the Neuro-Adaptive Multimodal Architecture (NAMI), a principled, modular, and reproducible framework designed to integrate behavioral and neurophysiological signals in real time. NAMI combines multimodal behavioral inputs with lightweight EEG and peripheral physiological measurements to infer cognitive load and engagement and adapt the interface dynamically to optimize user experience. The architecture is formally specified as a three-layer pipeline encompassing sensing and acquisition, cognitive–affective state estimation, and adaptive interaction control, with clear data flows, mathematical formalization, and real-time performance on wearable platforms. A prototype implementation of NAMI was deployed in an augmented reality Java programming tutor for postgraduate informatics students, where it dynamically adjusted task difficulty, feedback modality, and assistance frequency based on inferred user state. Empirical evaluation with 100 participants demonstrated significant improvements in task performance, reduced subjective workload, and increased engagement and satisfaction, confirming the effectiveness of the neuro-adaptive approach. Full article

Hybrid learning spaces may be described as physical environments enhanced by digital technologies, which enable learning scenarios involving both in-person and online participation. This article presents a hybrid learning space designed for higher education. The design of the space has been informed by Lefebvre’s design principles: (a) spatial practice enabling flexible usage scenarios, (b) representations of space conveying openness and adaptability, and (c) representational spaces supporting experiences of presence in both physical and digital form. The article describes design characteristics guiding the implementation of the hybrid learning space and explains corresponding design decisions, such as the use of a wall-sized projection. Further, the article introduces affordances and usage scenarios of the hybrid learning space developed. Moreover, an evaluation study of the hybrid learning space is conducted by means of a 360°-based virtual field trip (VFT). The VFT, led by an educator, serves as preparation for a field trip (FT) to a composting plant two weeks later. Participants of both VFT and FT (N = 11) completed a questionnaire addressing psychological constructs related to learning, including motivation, emotion, immersion, presence, and cognitive load. We report the results of the VFT alongside those of the FT as a baseline. Some notable differences, for example in social presence, suggest areas for further development of the hybrid learning space. Overall, the study characterises key features of hybrid learning spaces, identifies their contribution to high-quality teaching and provides inspirations for their further development. Full article

This study explores and models the use of MATLAB technology in multimodal learning interactions to address the challenges of teaching and learning statistics in a multinational postgraduate classroom. The term multimodal refers to the deliberate integration of multiple representational and interaction modes, i.e., visual, textual, symbolic, and interactive computational modelling, within a coherent instructional design. MATLAB is utilised as it is a comprehensive tool for enhancing students’ understanding of statistical skills, practical applications, and data analysis—areas where traditional methods often fall short. International postgraduate students were chosen for this study because their diverse educational backgrounds present unique learning challenges. A qualitative case study design was employed, and data collection methods included classroom observations, interviews, and student work analysis. The collected data were analysed and modelled by conceptualising key elements and themes using thematic analysis, with findings verified through data triangulation and expert review. Emerging themes were structured into models that illustrate multimodal teaching and learning interactions. The novelty of this research lies in its contribution to multimodal teaching and learning strategies for multinational students in statistics education. The findings highlight significant challenges international students face, including language and technical barriers, limited prior content knowledge, time constraints, technical difficulties, and a lack of independent thinking. To address these challenges, MATLAB promotes collaborative learning, increases student engagement and discussion, boosts motivation, and develops essential skills. This study suggests that educators integrate multimodal interactions in their teaching strategies to better support multinational students in statistical learning environments. Full article

In aviation safety, pilots must efficiently process dynamic visual information and maintain a high level of attention. Any missed judgment of critical information or delay in decision-making may lead to mission failure or catastrophic consequences. Therefore, accurately detecting pilots’ attention states is the primary prerequisite for improving flight safety and performance. To better detect the attention state of pilots, this paper takes flight trainees as the research object and the simulated flight environment as the experimental background. It proposes a method for detecting the attention state of flight trainees based on a multi-modal dynamic depth network (M3D-Net). The M3D-Net architecture is a lightweight neural network architecture that integrates temporal image features, visual information features, and flight operation data features. It aligns image and text features through an attention mechanism to enhance the semantic association between modalities; it utilizes the Depth-wise Separable Convolution and LSTM (DSC-LSTM) module to model temporal information, dynamically capturing the contextual dependencies within the sequence, and achieving six-level attention state classification. This paper conducted ablation experiments to comparatively analyze the classification effects of the model and also evaluates the effectiveness of our proposed method through model evaluation metrics. Experiments show that the classification effect of the model architecture proposed in this paper reaches 97.56%, with a model size of 18.6 M. Compared with traditional algorithms, the M3D-Net architecture has better performance prospects in terms of application. Full article

An Adaptive Neuro-Fuzzy Inference System (ANFIS) framework for paediatric wrist injury classification (fracture versus sprain) was developed utilising infrared thermography (IRT). ANFIS combines artificial neural network (ANN) learning with interpretable fuzzy rules, mitigating the “black-box” limitation of conventional ANNs through explicit membership functions and Takagi–Sugeno rule consequents. Forty children (19 fractures, 21 sprains, confirmed by X-ray radiograph) provided thermal image sequences from which three statistically discriminative temperature distribution features namely standard deviation, inter-quartile range (IQR) and kurtosis were selected. A five-layer Sugeno ANFIS with Gaussian membership functions were trained using a hybrid least-squares/gradient descent optimisation and evaluated under three premise-parameter initialisation strategies: random seeding, K-means clustering, and fuzzy C-means (FCM) data partitioning. Five-fold cross-validation guided the selection of membership functions standard deviation (σ) and rule count, yielding an optimal nine-rule model. Comparative experiments show K-means initialisation achieved the best balance between convergence speed and generalisation versus slower but highly precise random initialisation and rapidly convergent yet unstable FCM. The proposed K-means–driven ANFIS offered data-efficient decision support, highlighting the potential of thermal feature fusion with neuro-fuzzy modelling to reduce unnecessary radiographs in emergency bone fracture triage. Full article

Pilot visual behavior safety assessment is a cross-disciplinary technology that analyzes pilots’ gaze behavior and neurocognitive responses. This paper proposes a multimodal analysis method for pilot visual behavior safety, specifically for cognitive state decoding. This method aims to achieve a quantitative and efficient assessment of pilots’ observational behavior. Addressing the subjective limitations of traditional methods, this paper proposes an observational behavior detection model that integrates facial images to achieve dynamic and quantitative analysis of observational behavior. It addresses the “Midas contact” problem of observational behavior by constructing a cognitive analysis method using multimodal signals. We propose a bidirectional long short-term memory (LSTM) network that matches physiological signal rhythmic features to address the problem of isolated features in multidimensional signals. This method captures the dynamic correlations between multiple physiological behaviors, such as prefrontal theta and chest-abdominal coordination, to decode the cognitive state of pilots’ observational behavior. Finally, the paper uses a decision-level fusion method based on an improved Dempster–Shafer (DS) evidence theory to provide a quantifiable detection strategy for aviation safety standards. This dual-dimensional quantitative assessment system of “visual behavior–neurophysiological cognition” reveals the dynamic correlations between visual behavior and cognitive state among pilots of varying experience. This method can provide a new paradigm for pilot neuroergonomics training and early warning of vestibular-visual integration disorders. Full article

Rotary dryers are widely used in industry for their ease of operation in processing large volumes of material continuously despite persistent challenges in energy efficiency, cost-effectiveness, and safety. Addressing the need for effective operator training, the purpose of this study is to develop virtual reality (VR) environments for industrial rotary dryers. Visual and behavioral aspects were considered in the methodology for developing the environments for two application cases—ammonium nitrate and low-rank coal drying. Visual aspects considered include the industrial-scale geometry and detailed components of the rotary dryer, while behavioral aspects were governed by mathematical modeling of heat and mass transfer phenomena. The case studies of ammonium nitrate and low-rank coal were selected due to their industrial relevance and contrasting drying characteristics, ensuring the versatility and applicability of the developed VR environments. The main contribution of this work is the embedding of validated mathematical models—expressed as ordinary differential equations—into these environments. The numerical integration of these models provides key process variables, such as solid temperature and moisture content along the rotary dryer, thereby enhancing the behavioral realism of the developed VR environments. Full article

This paper explores the integration of computing and multimodal technologies into personal reading practices to enhance student engagement and knowledge assimilation in higher education. In response to a documented decline in voluntary academic reading, we investigated how technology-enhanced reading environments can re-engage students through interactive and personalized experiences. Central to this research is SmartRead, a proposed multimodal eReading platform that incorporates gamification, adaptive content delivery, and real-time feedback mechanisms. Drawing on empirical data collected from students at a higher education institution, we examined how features such as progress tracking, motivational rewards, and interactive comprehension aids influence reading behavior, engagement levels, and information retention. Results indicate that such multimodal interventions can significantly improve learner outcomes and user satisfaction. This paper contributes actionable insights into the design of innovative, accessible, and pedagogically sound digital reading tools and proposes a framework for future eReading technologies that align with multimodal interaction principles. Full article

With the advancement of haptic technology, the use of pseudo-haptics to provide tactile feedback without physical contact has garnered significant attention. This paper aimed to investigate whether sliding fingers over onomatopoetic text strings with pseudo-haptic effects induces change in perception toward their symbolic semantics. To address this, we conducted an experiment using finger-point reading as our subject matter. The experimental results confirmed that the “neba-neba,” “puru-puru,” and “fusa-fusa” effects create a pseudo-haptic feeling for the associated texts on the “hard–soft,” “slippery–sticky,” and “elastic–inelastic” adjective pairs. Specifically, for “hard–soft,” it was found that the proposed effects could consistently produce an impact. Full article

The increasing use of immersive 3-D technologies in education raises critical questions about their cognitive impact on learners. This systematic review evaluates how electroencephalography (EEG) and eye-tracking have been used to objectively measure cognitive load in 3-D learning environments. We conducted a comprehensive literature search (2009–2025) across PubMed, Scopus, Web of Science, PsycInfo, and ERIC, identifying 51 studies that used EEG or eye-tracking in experimental contexts involving stereoscopic or head-mounted 3-D technologies. Our findings suggest that 3-D environments may enhance learning and engagement, particularly in spatial tasks, while affecting cognitive load in complex, task-dependent ways. Studies reported mixed patterns across psychophysiological measures, including spectral features (e.g., frontal theta, parietal alpha), workload indices (e.g., theta/alpha ratio), and gaze-based metrics (e.g., fixation duration, pupil dilation): some studies observed increased load, while others reported reductions or no difference. These discrepancies reflect methodological heterogeneity and underscore the value of time-sensitive assessments. While a moderate cognitive load supports learning, an excessive load may impair performance, and overload thresholds can vary across individuals. EEG and eye-tracking offer scalable methods for monitoring cognitive effort dynamically. Overall, 3-D and XR technologies hold promise but must be aligned with task demands and learner profiles and guided by real-time indicators of cognitive load in immersive environments. Full article

This study aimed to investigate the use of social robots as an interactive learning approach for treating children diagnosed with autism spectrum disorder (ASD). A review was conducted using the meta-analysis technique to compile pertinent research. An analysis was performed on the results of the online search process, which gathered information on pertinent research published until 31 January 2025, from three publication databases: IEEE Xplore, SCOPUS, and Google Scholar. One hundred and seven papers out of the 591 publications that were retrieved satisfied the previously established inclusion and exclusion criteria. Despite the differences in methodology and heterogeneity, the data were synthesized narratively. This review focuses on the various types of social robots used to treat ASD, as well as their communication mechanisms, development areas, target behaviors, challenges, and future directions. Both practitioners and seasoned researchers looking for a fresh approach to their next project will find this review a useful resource that offers broad summaries of state-of-the-art research in this field. Full article

Augmented reality (AR) technology offers a promising approach to providing consumers with detailed and personalized information about food products. The aim of this pilot study was to explore how the use of AR tools comprising visual and auditory formats affects consumers’ perception and understanding of nutrition labels of two commercially available products (lasagne ready meal and strawberry yogurt). The nutritional information of both the lasagne and yogurt product were presented to consumers (n = 30) under three experimental conditions: original packaging, visual AR, and visual and audio AR. Consumers answered questions about their perceptions of the products’ overall healthiness, caloric content, and macronutrient composition, as well as how the information was presented. The results showed that while nutritional information presented under the original packaging condition was more effective in changing consumer perceptions, the AR tools were found to be more “novel” and “memorable”. More specifically, for both lasagne and yogurt, the visual AR tool resulted in a more memorable experience compared to original packaging. The use of visual AR and visual and audio AR tools were considered novel experiences for both products. However, the provision of nutritional information had a greater impact on product perception than the specific experimental condition used to present it. These results provide evidence from a pilot study supporting the development of an AR tool for displaying and potentially improving the understanding of nutrition labels. Full article

Virtual reality (VR) has become increasingly popular and has started entering homes, schools, and clinics, yet evidence on how children interact during free-form, unguided play remains limited. Understanding how interaction dynamics relate to player performance is essential for designing more accessible and engaging VR experiences, especially in educational contexts. For this reason, we developed VRBloons, a child-friendly VR game about popping balloons. The game logs real-time gameplay telemetry such as total hand movement, accuracy, throw rate, and other performance related gameplay data. By analyzing several feature-engineered metrics using unsupervised clustering and non-parametric statistical validation, we aim to identify distinct behavioral patterns. The analysis revealed several associations between input preferences, movement patterns, and performance outcomes, forming clearly distinct clusters. From the performed analysis, input preference emerged as an independent dimension of play style, supporting the inclusion of redundant input mappings to accommodate diverse motor capabilities. Additionally, the results highlight the opportunities for performance-sensitive assistance systems that adapt the difficulty of the game in real time. Overall, this study demonstrates how telemetry-based profiling can shape the design decisions in VR experiences, offering a methodological framework for assessing varied interaction styles and a diverse player population. Full article

This paper presents an educational experience of using Poly-Universe, a game created by Janos Saxon, with the aim of developing computational thinking (CT) skills through unplugged activities. It was implemented in the course “Algorithm Analysis,” with the participation of students in the sixth period of Computer Science at a University Center for Higher Education in Brazil. These students were facing various cognitive difficulties in using the four pillars of CT, namely abstraction, pattern recognition, algorithm, and decomposition. To address the students’ learning gaps, unplugged activities were implemented using Poly-Universe pieces—geometric shapes such as triangles, squares, and circles—exploring the connection through the pillars of CT. A mixed methodology integrating quantitative and qualitative approaches was applied to compare the progress of the students and their reactions when developing the activities. The results obtained evidenced that the level of learning involving the computational pillars on “Algorithm Analysis” had a significant evolution, from 30% to almost 80% in terms of achievement in academic tests. In addition, an increase in students’ engagement and collaboration was also registered. Therefore, the implementation of unplugged activities with Poly-Universe revealed a promotion of skills related to the pillars of CT, especially in the analysis of algorithms. Full article