Multimodal Technologies and Interaction

24 pages, 5068 KB

Open AccessArticle

Multimodal Learning Interactions Using MATLAB Technology in a Multinational Statistical Classroom

by Qiaoyan Cai, Mohd Razip Bajuri, Kwan Eu Leong and Liangliang Chen

Multimodal Technol. Interact. 2025, 9(10), 106; https://doi.org/10.3390/mti9100106 (registering DOI) - 13 Oct 2025

Abstract

This study explores and models the use of MATLAB technology in multimodal learning interactions to address the challenges of teaching and learning statistics in a multinational postgraduate classroom. The term multimodal refers to the deliberate integration of multiple representational and interaction modes, i.e., [...] Read more.

This study explores and models the use of MATLAB technology in multimodal learning interactions to address the challenges of teaching and learning statistics in a multinational postgraduate classroom. The term multimodal refers to the deliberate integration of multiple representational and interaction modes, i.e., visual, textual, symbolic, and interactive computational modelling, within a coherent instructional design. MATLAB is utilised as it is a comprehensive tool for enhancing students’ understanding of statistical skills, practical applications, and data analysis—areas where traditional methods often fall short. International postgraduate students were chosen for this study because their diverse educational backgrounds present unique learning challenges. A qualitative case study design was employed, and data collection methods included classroom observations, interviews, and student work analysis. The collected data were analysed and modelled by conceptualising key elements and themes using thematic analysis, with findings verified through data triangulation and expert review. Emerging themes were structured into models that illustrate multimodal teaching and learning interactions. The novelty of this research lies in its contribution to multimodal teaching and learning strategies for multinational students in statistics education. The findings highlight significant challenges international students face, including language and technical barriers, limited prior content knowledge, time constraints, technical difficulties, and a lack of independent thinking. To address these challenges, MATLAB promotes collaborative learning, increases student engagement and discussion, boosts motivation, and develops essential skills. This study suggests that educators integrate multimodal interactions in their teaching strategies to better support multinational students in statistical learning environments. Full article

► Show Figures

Figure 1

16 pages, 4268 KB

Open AccessArticle

Research on the Detection Method of Flight Trainees’ Attention State Based on Multi-Modal Dynamic Depth Network

by Gongpu Wu, Changyuan Wang, Zehui Chen and Guangyi Jiang

Multimodal Technol. Interact. 2025, 9(10), 105; https://doi.org/10.3390/mti9100105 - 10 Oct 2025

Viewed by 166

Abstract

In aviation safety, pilots must efficiently process dynamic visual information and maintain a high level of attention. Any missed judgment of critical information or delay in decision-making may lead to mission failure or catastrophic consequences. Therefore, accurately detecting pilots’ attention states is the [...] Read more.

In aviation safety, pilots must efficiently process dynamic visual information and maintain a high level of attention. Any missed judgment of critical information or delay in decision-making may lead to mission failure or catastrophic consequences. Therefore, accurately detecting pilots’ attention states is the primary prerequisite for improving flight safety and performance. To better detect the attention state of pilots, this paper takes flight trainees as the research object and the simulated flight environment as the experimental background. It proposes a method for detecting the attention state of flight trainees based on a multi-modal dynamic depth network (M3D-Net). The M3D-Net architecture is a lightweight neural network architecture that integrates temporal image features, visual information features, and flight operation data features. It aligns image and text features through an attention mechanism to enhance the semantic association between modalities; it utilizes the Depth-wise Separable Convolution and LSTM (DSC-LSTM) module to model temporal information, dynamically capturing the contextual dependencies within the sequence, and achieving six-level attention state classification. This paper conducted ablation experiments to comparatively analyze the classification effects of the model and also evaluates the effectiveness of our proposed method through model evaluation metrics. Experiments show that the classification effect of the model architecture proposed in this paper reaches 97.56%, with a model size of 18.6 M. Compared with traditional algorithms, the M3D-Net architecture has better performance prospects in terms of application. Full article

► Show Figures

Figure 1

23 pages, 3467 KB

Open AccessArticle

Adaptive Neuro-Fuzzy Inference System Framework for Paediatric Wrist Injury Classification

by Olamilekan Shobayo, Reza Saatchi and Shammi Ramlakhan

Multimodal Technol. Interact. 2025, 9(10), 104; https://doi.org/10.3390/mti9100104 - 8 Oct 2025

Viewed by 178

Abstract

An Adaptive Neuro-Fuzzy Inference System (ANFIS) framework for paediatric wrist injury classification (fracture versus sprain) was developed utilising infrared thermography (IRT). ANFIS combines artificial neural network (ANN) learning with interpretable fuzzy rules, mitigating the “black-box” limitation of conventional ANNs through explicit membership functions [...] Read more.

An Adaptive Neuro-Fuzzy Inference System (ANFIS) framework for paediatric wrist injury classification (fracture versus sprain) was developed utilising infrared thermography (IRT). ANFIS combines artificial neural network (ANN) learning with interpretable fuzzy rules, mitigating the “black-box” limitation of conventional ANNs through explicit membership functions and Takagi–Sugeno rule consequents. Forty children (19 fractures, 21 sprains, confirmed by X-ray radiograph) provided thermal image sequences from which three statistically discriminative temperature distribution features namely standard deviation, inter-quartile range (IQR) and kurtosis were selected. A five-layer Sugeno ANFIS with Gaussian membership functions were trained using a hybrid least-squares/gradient descent optimisation and evaluated under three premise-parameter initialisation strategies: random seeding, K-means clustering, and fuzzy C-means (FCM) data partitioning. Five-fold cross-validation guided the selection of membership functions standard deviation (σ) and rule count, yielding an optimal nine-rule model. Comparative experiments show K-means initialisation achieved the best balance between convergence speed and generalisation versus slower but highly precise random initialisation and rapidly convergent yet unstable FCM. The proposed K-means–driven ANFIS offered data-efficient decision support, highlighting the potential of thermal feature fusion with neuro-fuzzy modelling to reduce unnecessary radiographs in emergency bone fracture triage. Full article

► Show Figures

Figure 1

26 pages, 4710 KB

Open AccessArticle

Research on Safe Multimodal Detection Method of Pilot Visual Observation Behavior Based on Cognitive State Decoding

by Heming Zhang, Changyuan Wang and Pengbo Wang

Multimodal Technol. Interact. 2025, 9(10), 103; https://doi.org/10.3390/mti9100103 - 1 Oct 2025

Viewed by 278

Abstract

Pilot visual behavior safety assessment is a cross-disciplinary technology that analyzes pilots’ gaze behavior and neurocognitive responses. This paper proposes a multimodal analysis method for pilot visual behavior safety, specifically for cognitive state decoding. This method aims to achieve a quantitative and efficient [...] Read more.

Pilot visual behavior safety assessment is a cross-disciplinary technology that analyzes pilots’ gaze behavior and neurocognitive responses. This paper proposes a multimodal analysis method for pilot visual behavior safety, specifically for cognitive state decoding. This method aims to achieve a quantitative and efficient assessment of pilots’ observational behavior. Addressing the subjective limitations of traditional methods, this paper proposes an observational behavior detection model that integrates facial images to achieve dynamic and quantitative analysis of observational behavior. It addresses the “Midas contact” problem of observational behavior by constructing a cognitive analysis method using multimodal signals. We propose a bidirectional long short-term memory (LSTM) network that matches physiological signal rhythmic features to address the problem of isolated features in multidimensional signals. This method captures the dynamic correlations between multiple physiological behaviors, such as prefrontal theta and chest-abdominal coordination, to decode the cognitive state of pilots’ observational behavior. Finally, the paper uses a decision-level fusion method based on an improved Dempster–Shafer (DS) evidence theory to provide a quantifiable detection strategy for aviation safety standards. This dual-dimensional quantitative assessment system of “visual behavior–neurophysiological cognition” reveals the dynamic correlations between visual behavior and cognitive state among pilots of varying experience. This method can provide a new paradigm for pilot neuroergonomics training and early warning of vestibular-visual integration disorders. Full article

► Show Figures

Figure 1

18 pages, 7318 KB

Open AccessArticle

Design of Enhanced Virtual Reality Training Environments for Industrial Rotary Dryers Using Mathematical Modeling

by Ricardo A. Gutiérrez-Aguiñaga, Jonathan H. Rosales-Hernández, Rogelio Salinas-Santiago, Froylán M. E. Escalante and Efrén Aguilar-Garnica

Multimodal Technol. Interact. 2025, 9(10), 102; https://doi.org/10.3390/mti9100102 - 30 Sep 2025

Viewed by 212

Abstract

Rotary dryers are widely used in industry for their ease of operation in processing large volumes of material continuously despite persistent challenges in energy efficiency, cost-effectiveness, and safety. Addressing the need for effective operator training, the purpose of this study is to develop [...] Read more.

Rotary dryers are widely used in industry for their ease of operation in processing large volumes of material continuously despite persistent challenges in energy efficiency, cost-effectiveness, and safety. Addressing the need for effective operator training, the purpose of this study is to develop virtual reality (VR) environments for industrial rotary dryers. Visual and behavioral aspects were considered in the methodology for developing the environments for two application cases—ammonium nitrate and low-rank coal drying. Visual aspects considered include the industrial-scale geometry and detailed components of the rotary dryer, while behavioral aspects were governed by mathematical modeling of heat and mass transfer phenomena. The case studies of ammonium nitrate and low-rank coal were selected due to their industrial relevance and contrasting drying characteristics, ensuring the versatility and applicability of the developed VR environments. The main contribution of this work is the embedding of validated mathematical models—expressed as ordinary differential equations—into these environments. The numerical integration of these models provides key process variables, such as solid temperature and moisture content along the rotary dryer, thereby enhancing the behavioral realism of the developed VR environments. Full article

► Show Figures

Figure 1

22 pages, 2952 KB

Open AccessArticle

SmartRead: A Multimodal eReading Platform Integrating Computing and Gamification to Enhance Student Engagement and Knowledge Retention

by Ifeoluwa Pelumi and Neil Gordon

Multimodal Technol. Interact. 2025, 9(10), 101; https://doi.org/10.3390/mti9100101 - 23 Sep 2025

Viewed by 446

Abstract

This paper explores the integration of computing and multimodal technologies into personal reading practices to enhance student engagement and knowledge assimilation in higher education. In response to a documented decline in voluntary academic reading, we investigated how technology-enhanced reading environments can re-engage students [...] Read more.

This paper explores the integration of computing and multimodal technologies into personal reading practices to enhance student engagement and knowledge assimilation in higher education. In response to a documented decline in voluntary academic reading, we investigated how technology-enhanced reading environments can re-engage students through interactive and personalized experiences. Central to this research is SmartRead, a proposed multimodal eReading platform that incorporates gamification, adaptive content delivery, and real-time feedback mechanisms. Drawing on empirical data collected from students at a higher education institution, we examined how features such as progress tracking, motivational rewards, and interactive comprehension aids influence reading behavior, engagement levels, and information retention. Results indicate that such multimodal interventions can significantly improve learner outcomes and user satisfaction. This paper contributes actionable insights into the design of innovative, accessible, and pedagogically sound digital reading tools and proposes a framework for future eReading technologies that align with multimodal interaction principles. Full article

► Show Figures

Graphical abstract

10 pages, 7955 KB

Open AccessArticle

Investigating the Effect of Pseudo-Haptics on Perceptions Toward Onomatopoeia Text During Finger-Point Tracing

by Satoshi Saga and Kanta Shirakawa

Multimodal Technol. Interact. 2025, 9(10), 100; https://doi.org/10.3390/mti9100100 - 23 Sep 2025

Viewed by 318

Abstract

With the advancement of haptic technology, the use of pseudo-haptics to provide tactile feedback without physical contact has garnered significant attention. This paper aimed to investigate whether sliding fingers over onomatopoetic text strings with pseudo-haptic effects induces change in perception toward their symbolic [...] Read more.

With the advancement of haptic technology, the use of pseudo-haptics to provide tactile feedback without physical contact has garnered significant attention. This paper aimed to investigate whether sliding fingers over onomatopoetic text strings with pseudo-haptic effects induces change in perception toward their symbolic semantics. To address this, we conducted an experiment using finger-point reading as our subject matter. The experimental results confirmed that the “neba-neba,” “puru-puru,” and “fusa-fusa” effects create a pseudo-haptic feeling for the associated texts on the “hard–soft,” “slippery–sticky,” and “elastic–inelastic” adjective pairs. Specifically, for “hard–soft,” it was found that the proposed effects could consistently produce an impact. Full article

► Show Figures

Figure 1

Journal Menu

Journal Browser

Multimodal Technol. Interact., Volume 9, Issue 10 (October 2025) – 7 articles

Further Information

Guidelines

MDPI Initiatives

Follow MDPI