Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
4.9
2.4
Journals
MTI
Special Issues
Multimodal User Interfaces and Experiences: Challenges, Applications, and Perspectives—2nd Edition
share announcement

Multimodal User Interfaces and Experiences: Challenges, Applications, and Perspectives—2nd Edition

A special issue of Multimodal Technologies and Interaction (ISSN 2414-4088).

Deadline for manuscript submissions: 15 August 2025 | Viewed by 17777

Special Issue Editors

Dr. Wei Liu

E-Mail Website
Guest Editor
School of Design, Southern University of Science and Technology, Shenzhen 518055, China
Interests: human–computer interaction; user experience; tangible interaction; engineering psychology
Special Issues, Collections and Topics in MDPI journals
Dr. Jan Auernhammer

E-Mail Website
Guest Editor
TD School, University of Technology Sydney, Broadway, Ultimo, NSW 2007, Australia
Interests: design thinking; organizational culture; innovation management; prototyping
Special Issues, Collections and Topics in MDPI journals
Dr. Takumi Ohashi

E-Mail Website
Guest Editor
Department of Transdisciplinary Science and Engineering, Tokyo Institute of Technology, Tokyo 152-8550, Japan
Interests: human-centered design; urban/rural sociology; qualitative research; engineering education
Special Issues, Collections and Topics in MDPI journals

Special Issue Information

Dear Colleagues,

Our previous Special Issue, entitled “Multimodal User Interfaces and Experiences: Challenges, Applications, and Perspectives” (https://www.mdpi.com/journal/mti/special_issues/F24T6HSRT9), was a successful compilation of research and review articles. As this is a rapidly evolving topic, we would like to further explore multimodal user interfaces and experiences with a follow-up Special Issue.

This Special Issue aims to explore the challenges and opportunities of understanding, designing, and evaluating user experience (UX) across and beyond disciplines. By soliciting contributions from theoretical and practical perspectives that explicitly address the exploration and evaluation of user interfaces and experiences, we will envision the future directions of UX/HCI research on multimodal technologies and the application of user-friendly interfaces. The context includes but is not limited to education, healthcare, transportation, finance, and environmental protection. In particular, we are interested in contributions addressing the research through design approach and the intersection between applied psychology, human–computer interaction (HCI), cognitive neuroscience, anthropology, and design, such as transdisciplinary teaching to specific student groups, digitalized child/elderly/patient care services, car/train/aircraft human–machine interfaces (HMIs), mobile banking applications, carbon peaking and carbon neutrality strategies.

We encourage authors to submit original research articles, works in progress, surveys, reviews, and viewpoint articles, presenting transdisciplinary frameworks, methods, and practices that may significantly impact the field for years to come. Topics of interest include, but are not limited to, the following:

  • Transdisciplinary teaching and learning;
  • Design thinking, doing, and tinkering;
  • Human factors and applied psychology;
  • Kansei, emotional, and affective engineering;
  • Psychological and digital wellbeing;
  • Clinical and counseling psychology;
  • Psychological and behavioral big data;
  • Brand, advertising, and consumer psychology;
  • Measurement and human resources;
  • Interaction design qualities and guidelines;
  • Usability evaluation methods;
  • Emerging and multimodal technologies;
  • Augmented, mixed, and extended realities;
  • Inclusion, resilience, and new normal;
  • Creativity, innovation, and entrepreneurship;
  • Human-centered design (HCD);
  • User experience design (UXD);
  • Emotion-driven design (EDD);
  • Collaborative design (Co-Design);
  • Industrial design (ID);
  • NeuroDesign (ND).

Dr. Wei Liu
Dr. Jan Auernhammer
Dr. Takumi Ohashi
Guest Editors

Manuscript Submission Information

Manuscripts should be submitted online at www.mdpi.com by registering and logging in to this website. Once you are registered, click here to go to the submission form. Manuscripts can be submitted until the deadline. All submissions that pass pre-check are peer-reviewed. Accepted papers will be published continuously in the journal (as soon as accepted) and will be listed together on the special issue website. Research articles, review articles as well as short communications are invited. For planned papers, a title and short abstract (about 100 words) can be sent to the Editorial Office for announcement on this website.

Submitted manuscripts should not have been published previously, nor be under consideration for publication elsewhere (except conference proceedings papers). All manuscripts are thoroughly refereed through a single-blind peer-review process. A guide for authors and other relevant information for submission of manuscripts is available on the Instructions for Authors page. Multimodal Technologies and Interaction is an international peer-reviewed open access monthly journal published by MDPI.

Please visit the Instructions for Authors page before submitting a manuscript. The Article Processing Charge (APC) for publication in this open access journal is 1600 CHF (Swiss Francs). Submitted papers should be well formatted and use good English. Authors may use MDPI's English editing service prior to publication or during author revisions.

Keywords

  • transdisciplinary teaching and learning
  • design thinking, doing, and tinkering
  • human factors and applied psychology
  • kansei, emotional, and affective engineering
  • psychological and digital wellbeing
  • clinical and counseling psychology
  • psychological and behavioral big data
  • brand, advertising, and consumer psychology
  • measurement and human resources
  • interaction design qualities and guidelines
  • usability evaluation methods
  • emerging and multimodal technologies
  • augmented, mixed, and extended realities
  • inclusion, resilience, and new normal
  • creativity, innovation, and entrepreneurship
  • human-centered design (HCD)
  • user experience design (UXD)
  • emotion-driven design (EDD)
  • collaborative design (Co-Design)
  • industrial design (ID)
  • neurodesign (ND)

Benefits of Publishing in a Special Issue

  • Ease of navigation: Grouping papers by topic helps scholars navigate broad scope journals more efficiently.
  • Greater discoverability: Special Issues support the reach and impact of scientific research. Articles in Special Issues are more discoverable and cited more frequently.
  • Expansion of research network: Special Issues facilitate connections among authors, fostering scientific collaborations.
  • External promotion: Articles in Special Issues are often promoted through the journal's social media, increasing their visibility.
  • e-Book format: Special Issues with more than 10 articles can be published as dedicated e-books, ensuring wide and rapid dissemination.

Further information on MDPI's Special Issue policies can be found here.

Published Papers (11 papers)

Download All Papers
Order results
Result details
Show export options expand_more Show export options expand_less
Select all
Export citation of selected articles as:

Research

Jump to: Review, Other

25 pages, 2317 KiB  
Article
diaLogic: A Multi-Modal Framework for Automated Team Behavior Modeling Based on Speech Acquisition
by Ryan Duke and Alex Doboli
Multimodal Technol. Interact. 2025, 9(3), 26; https://doi.org/10.3390/mti9030026 - 10 Mar 2025
Viewed by 508
Abstract
This paper presents diaLogic, a humans-in-the-loop system for modeling the behavior of teams during collective problem solving. Team behavior is modeled using multi-modal data about cognition, social interactions, and emotions acquired from speech inputs. The system includes methods for speaker diarization, speaker interaction [...] Read more.
This paper presents diaLogic, a humans-in-the-loop system for modeling the behavior of teams during collective problem solving. Team behavior is modeled using multi-modal data about cognition, social interactions, and emotions acquired from speech inputs. The system includes methods for speaker diarization, speaker interaction characterization, speaker emotion recognition, and speech-to-text conversion. Hypotheses about the invariant and differentiated aspects of teams are extracted using the similarities and dissimilarities of their behavior over time. Hypothesis extraction, a novel contribution of this work, uses a method to identify the clauses and concepts in each spoken sentence. Experiments present system performance for a broad set of cases of team behavior during problem solving. The average errors of the various methods are between 6% and 21%. The system can be used in a broad range of applications, from education to team research and therapy. Full article
(This article belongs to the Special Issue Multimodal User Interfaces and Experiences: Challenges, Applications, and Perspectives—2nd Edition)
►▼ Show Figures

Figure 1

23 pages, 2067 KiB  
Article
Choice Vectors: Streamlining Personal AI Alignment Through Binary Selection
by Eleanor Watson, Minh Nguyen, Sarah Pan and Shujun Zhang
Multimodal Technol. Interact. 2025, 9(3), 22; https://doi.org/10.3390/mti9030022 - 3 Mar 2025
Viewed by 691
Abstract
Value alignment for AI is not “one-size-fits-all”: even polite and friendly models can still fail to represent individual user contexts and preferences, and local cultural norms. This paper presents a modular workflow for personal fine-tuning, synthesizing four core components from our previous research: [...] Read more.
Value alignment for AI is not “one-size-fits-all”: even polite and friendly models can still fail to represent individual user contexts and preferences, and local cultural norms. This paper presents a modular workflow for personal fine-tuning, synthesizing four core components from our previous research: (1) robust vectorization of user values and preferences, (2) a binary choice user interface (UI) approach to capturing those preferences with minimal cognitive load, (3) contrastive activation methods for steering large language models (LLMs) via difference vectors, and (4) knowledge graph integration for more auditable and structured alignment. Our approach—descended from past research on “Towards an End-to-End Personal Fine-Tuning Framework”—demonstrates how these elements can be combined to create personalized, context-rich alignment solutions. We report on user studies for the forced-choice UI, describe an experimental pipeline for deriving “control vectors”, and propose a “moral graph” method for bridging symbolic and vector-based alignment. Our findings suggest that multi-pronged personalization can significantly reduce user annotation fatigue, improve alignment fidelity, and allow for more flexible, interpretable AI behaviors. Full article
(This article belongs to the Special Issue Multimodal User Interfaces and Experiences: Challenges, Applications, and Perspectives—2nd Edition)
►▼ Show Figures

Figure 1

14 pages, 981 KiB  
Article
Sensory Perception During Partial Pseudo-Haptics Applied to Adjacent Fingers
by Satoshi Saga and Kotaro Sakae
Multimodal Technol. Interact. 2025, 9(3), 19; https://doi.org/10.3390/mti9030019 - 26 Feb 2025
Viewed by 456
Abstract
Pseudo-haptics, the phenomenon of creating a simulated tactile sensation by introducing a discrepancy between a voluntary movement and its visual feedback, is well known. Typically, when inducing pseudo-haptics, the same control-display ratio (C/D ratio) is applied to all effectors. However, with the aim [...] Read more.
Pseudo-haptics, the phenomenon of creating a simulated tactile sensation by introducing a discrepancy between a voluntary movement and its visual feedback, is well known. Typically, when inducing pseudo-haptics, the same control-display ratio (C/D ratio) is applied to all effectors. However, with the aim of expanding the potential illusions that can be presented with pseudo-haptics, we investigated how perceived sensations change when partial pseudo-haptics are applied to adjacent body parts. In this research, we examined how perceived sensations change when pseudo-haptic stimuli are applied to adjacent body parts. Specifically, we investigated the correlation between finger states and the magnitude of illusory perception during both quasi-static and dynamic movements and identified the finger that experienced discomfort during dynamic movements with pseudo-haptics. Our findings revealed that: First, the magnitude of the illusion varied based on the contact state of adjacent fingers. Second, the illusion was more pronounced during dynamic movements compared to quasi-static movements. Third, regardless of the finger receiving the pseudo-haptic stimulus, the discomfort was primarily experienced in the finger exhibiting an overall inhibitory movement. The findings contribute to the practical application of pseudo-haptics as a virtual haptic display technology. Full article
(This article belongs to the Special Issue Multimodal User Interfaces and Experiences: Challenges, Applications, and Perspectives—2nd Edition)
►▼ Show Figures

Figure 1

10 pages, 8942 KiB  
Article
An Implementation of a Crime-Safety-Map Application Based on a Safety Index
by Seong-Cho Hong, Svetlana Kim and Sun-Young Ihm
Multimodal Technol. Interact. 2025, 9(2), 16; https://doi.org/10.3390/mti9020016 - 13 Feb 2025
Viewed by 644
Abstract
This paper presents the development of a crime-safety-map application and a safety index using the heatmap and geofence methods. The need for a tool that can satisfy safety needs has become more important than ever due to society’s growing fear of crime. One [...] Read more.
This paper presents the development of a crime-safety-map application and a safety index using the heatmap and geofence methods. The need for a tool that can satisfy safety needs has become more important than ever due to society’s growing fear of crime. One way to satisfy the general public’s safety needs is by informing them of crime data and the safety level of the surrounding environment, but it is not disclosed by law enforcement agencies. Therefore, this study focused on crime prevention through environmental design for developing a user-friendly, open to the public crime-safety-map application. Data from the Republic of Korean Open Government Data Portal’s nationwide safety and crime related data were used and the application was designed using Android Studio. The developed application visualizes the characteristic of the surrounding environment and can also inform crime safety level through a heatmap and the geofence technique. This application can reduce the general public’s fear of crime and crime incidents by informing and warning them about the crime prone areas. Full article
(This article belongs to the Special Issue Multimodal User Interfaces and Experiences: Challenges, Applications, and Perspectives—2nd Edition)
►▼ Show Figures

Figure 1

17 pages, 408 KiB  
Article
Craft-Based Methodologies in Human–Computer Interaction: Exploring Interdisciplinary Design Approaches
by Arminda Guerra
Multimodal Technol. Interact. 2025, 9(2), 13; https://doi.org/10.3390/mti9020013 - 10 Feb 2025
Viewed by 959
Abstract
Craft-based methodologies have emerged as a vital human-computer interaction (HCI) approach, bridging digital and physical materials in interactive system design. This study, born from a collaboration between two research networks focused on affective design and interaction design, investigates how diverse professionals use craft-based [...] Read more.
Craft-based methodologies have emerged as a vital human-computer interaction (HCI) approach, bridging digital and physical materials in interactive system design. This study, born from a collaboration between two research networks focused on affective design and interaction design, investigates how diverse professionals use craft-based approaches to transform design processes. Through carefully curated workshops, participants from varied backgrounds worked to identify specific problems, select technologies, and consider contextual factors within a creative framework. The workshops served as a platform for observing participant behaviors and goals in real-world settings, with researchers systematically collecting data through material engagement and visual problem-solving exercises. Drawing inspiration from concepts like Chindogu (Japanese “unuseless” inventions), the research demonstrates how reframing interaction design through craft-based methodologies can lead to more intuitive and contextually aware solutions. The findings highlight how interdisciplinary collaboration and sustainable and socially responsible design principles generate innovative solutions that effectively address user requirements. This integration of creative frameworks with physical and digital materials advances our understanding of meaningful technological interactions while establishing more holistic approaches to interactive system design that can inform future research directions in the field. Full article
(This article belongs to the Special Issue Multimodal User Interfaces and Experiences: Challenges, Applications, and Perspectives—2nd Edition)
►▼ Show Figures

Figure 1

12 pages, 2155 KiB  
Article
Human–Robot Interactions: A Pilot Study of Psychoaffective and Cognitive Factors to Boost the Acceptance and Usability of Assistive Wearable Devices
by Margherita Bertuccelli, Stefano Tortora, Edoardo Trombin, Liliana Negri, Patrizia Bisiacchi, Emanuele Menegatti and Alessandra Del Felice
Multimodal Technol. Interact. 2025, 9(1), 5; https://doi.org/10.3390/mti9010005 - 9 Jan 2025
Viewed by 792
Abstract
Robotic technology to assist rehabilitation provides practical advantages compared with traditional rehabilitation treatments, but its efficacy is still disputed. This controversial effectiveness is due to different factors, including a lack of guidelines to adapt devices to users’ individual needs. These needs include the [...] Read more.
Robotic technology to assist rehabilitation provides practical advantages compared with traditional rehabilitation treatments, but its efficacy is still disputed. This controversial effectiveness is due to different factors, including a lack of guidelines to adapt devices to users’ individual needs. These needs include the specific clinical conditions of people with disabilities, as well as their psychological and cognitive profiles. This pilot study aims to investigate the relationships between psychological, cognitive, and robot-related factors playing a role in human–robot interaction to promote a human-centric approach in robotic rehabilitation. Ten able-bodied volunteers were assessed for their anxiety, experienced workload, cognitive reserve, and perceived exoskeleton usability before and after a task with a lower-limb exoskeleton (i.e., 10 m path walking for 10 trials). Pre-trial anxiety levels were higher than post-trial ones (p < 0.01). While trait anxiety levels were predictive of the experienced effort (Adjusted-r2 = 0.43, p = 0.02), the state anxiety score was predictive of the perceived overall workload (Adjusted-r2 = 0.45, p = 0.02). High–average cognitive reserve scores were predictive of the perception of exoskeleton usability (Adjusted-r2 = 0.45, p = 0.02). A negative correlation emerged between the workload and the perception of personal identification with the exoskeleton (r = −0.67, p-value = 0.03). This study provides preliminary evidence of the impact of cognitive and psychoaffective factors on the perception of workload and overall device appreciation in exoskeleton training. It also suggests pragmatic measures such as familiarization time to reduce anxiety and end-user selection based on cognitive profiles. These assessments may provide guidance on the personalization of training. Full article
(This article belongs to the Special Issue Multimodal User Interfaces and Experiences: Challenges, Applications, and Perspectives—2nd Edition)
►▼ Show Figures

Figure 1

17 pages, 4004 KiB  
Article
Designing a Tactile Document UI for 2D Refreshable Tactile Displays: Towards Accessible Document Layouts for Blind People
by Sara Alzalabny, Omar Moured, Karin Müller, Thorsten Schwarz, Bastian Rapp and Rainer Stiefelhagen
Multimodal Technol. Interact. 2024, 8(11), 102; https://doi.org/10.3390/mti8110102 - 8 Nov 2024
Cited by 1 | Viewed by 1285
Abstract
Understanding document layouts is vital for enhancing document exploration and information retrieval for sighted individuals. However, for blind and visually impaired people, it becomes challenging to have access to layout information using typical assistive technologies such as screen readers. In this paper, we [...] Read more.
Understanding document layouts is vital for enhancing document exploration and information retrieval for sighted individuals. However, for blind and visually impaired people, it becomes challenging to have access to layout information using typical assistive technologies such as screen readers. In this paper, we examine the potential benefits of presenting documents on two-dimensional (2D) refreshable tactile displays. These displays enable the tactile perception of 2D data, offering the advantage of dynamic and interactive functionality. Despite their potential, the development of user interfaces (UIs) for such displays has not advanced significantly. Thus, we propose a design of an intelligent tactile user interface (TUI), incorporating touch and audio feedback to represent documents in a tactile format. Our exploratory study for evaluating this approach revealed satisfaction from participants with the experience of directly viewing documents in their true form, rather than relying on screen-reading interpretations. Additionally, participants offered recommendations for incorporating additional features and refining the approach in future iterations. To facilitate further research and development, we have made our dataset and models publicly available. Full article
(This article belongs to the Special Issue Multimodal User Interfaces and Experiences: Challenges, Applications, and Perspectives—2nd Edition)
►▼ Show Figures

Graphical abstract

Review

Jump to: Research, Other

32 pages, 475 KiB  
Review
Multimodal Interaction, Interfaces, and Communication: A Survey
by Elias Dritsas, Maria Trigka, Christos Troussas and Phivos Mylonas
Multimodal Technol. Interact. 2025, 9(1), 6; https://doi.org/10.3390/mti9010006 - 14 Jan 2025
Viewed by 4087
Abstract
Multimodal interaction is a transformative human-computer interaction (HCI) approach that allows users to interact with systems through various communication channels such as speech, gesture, touch, and gaze. With advancements in sensor technology and machine learning (ML), multimodal systems are becoming increasingly important in [...] Read more.
Multimodal interaction is a transformative human-computer interaction (HCI) approach that allows users to interact with systems through various communication channels such as speech, gesture, touch, and gaze. With advancements in sensor technology and machine learning (ML), multimodal systems are becoming increasingly important in various applications, including virtual assistants, intelligent environments, healthcare, and accessibility technologies. This survey concisely overviews recent advancements in multimodal interaction, interfaces, and communication. It delves into integrating different input and output modalities, focusing on critical technologies and essential considerations in multimodal fusion, including temporal synchronization and decision-level integration. Furthermore, the survey explores the challenges of developing context-aware, adaptive systems that provide seamless and intuitive user experiences. Lastly, by examining current methodologies and trends, this study underscores the potential of multimodal systems and sheds light on future research directions. Full article
(This article belongs to the Special Issue Multimodal User Interfaces and Experiences: Challenges, Applications, and Perspectives—2nd Edition)
►▼ Show Figures

Figure 1

24 pages, 9111 KiB  
Review
Bi-Directional Gaze-Based Communication: A Review
by Björn Rene Severitt, Nora Castner and Siegfried Wahl
Multimodal Technol. Interact. 2024, 8(12), 108; https://doi.org/10.3390/mti8120108 - 4 Dec 2024
Viewed by 1438
Abstract
Bi-directional gaze-based communication offers an intuitive and natural way for users to interact with systems. This approach utilizes the user’s gaze not only to communicate intent but also to obtain feedback, which promotes mutual understanding and trust between the user and the system. [...] Read more.
Bi-directional gaze-based communication offers an intuitive and natural way for users to interact with systems. This approach utilizes the user’s gaze not only to communicate intent but also to obtain feedback, which promotes mutual understanding and trust between the user and the system. In this review, we explore the state of the art in gaze-based communication, focusing on both directions: From user to system and from system to user. First, we examine how eye-tracking data is processed and utilized for communication from the user to the system. This includes a range of techniques for gaze-based interaction and the critical role of intent prediction, which enhances the system’s ability to anticipate the user’s needs. Next, we analyze the reverse pathway—how systems provide feedback to users via various channels, highlighting their advantages and limitations. Finally, we discuss the potential integration of these two communication streams, paving the way for more intuitive and efficient gaze-based interaction models, especially in the context of Artificial Intelligence. Our overview emphasizes the future prospects for combining these approaches to create seamless, trust-building communication between users and systems. Ensuring that these systems are designed with a focus on usability and accessibility will be critical to making them effective communication tools for a wide range of users. Full article
(This article belongs to the Special Issue Multimodal User Interfaces and Experiences: Challenges, Applications, and Perspectives—2nd Edition)
►▼ Show Figures

Graphical abstract

46 pages, 782 KiB  
Review
A Comprehensive Review of Multimodal XR Applications, Risks, and Ethical Challenges in the Metaverse
by Panagiotis Kourtesis
Multimodal Technol. Interact. 2024, 8(11), 98; https://doi.org/10.3390/mti8110098 - 6 Nov 2024
Cited by 9 | Viewed by 5902
Abstract
This scoping review examines the broad applications, risks, and ethical challenges associated with Extended Reality (XR) technologies, including Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR), within the context of Metaverse. XR is revolutionizing fields such as immersive learning in education, [...] Read more.
This scoping review examines the broad applications, risks, and ethical challenges associated with Extended Reality (XR) technologies, including Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR), within the context of Metaverse. XR is revolutionizing fields such as immersive learning in education, medical and professional training, neuropsychological assessment, therapeutic interventions, arts, entertainment, retail, e-commerce, remote work, sports, architecture, urban planning, and cultural heritage preservation. The integration of multimodal technologies—haptics, eye-, face-, and body tracking, and brain–computer interfaces—enhances user engagement and interactivity, playing a key role in shaping the immersive experiences in the Metaverse. However, XR’s expansion raises serious concerns, including data privacy risks, cybersecurity vulnerabilities, cybersickness, addiction, dissociation, harassment, bullying, and misinformation. These psychological, social, and security challenges are further complicated by intense advertising, manipulation of public opinion, and social inequality, which could disproportionately affect vulnerable individuals and social groups. This review emphasizes the urgent need for robust ethical frameworks and regulatory guidelines to address these risks while promoting equitable access, privacy, autonomy, and mental well-being. As XR technologies increasingly integrate with artificial intelligence, responsible governance is essential to ensure the safe and beneficial development of the Metaverse and the broader application of XR in enhancing human development. Full article
(This article belongs to the Special Issue Multimodal User Interfaces and Experiences: Challenges, Applications, and Perspectives—2nd Edition)
►▼ Show Figures

Figure 1

Other

Jump to: Research, Review

28 pages, 7066 KiB  
Systematic Review
A Systematic Review on Artificial Intelligence-Based Multimodal Dialogue Systems Capable of Emotion Recognition
by Luis Bravo, Ciro Rodriguez, Pedro Hidalgo and Cesar Angulo
Multimodal Technol. Interact. 2025, 9(3), 28; https://doi.org/10.3390/mti9030028 - 14 Mar 2025
Viewed by 1043
Abstract
In the current context, the use of technologies in applications for multimodal dialogue systems with computers and emotion recognition through artificial intelligence continues to grow rapidly. Consequently, it is challenging for researchers to identify gaps, propose new models, and increase user satisfaction. The [...] Read more.
In the current context, the use of technologies in applications for multimodal dialogue systems with computers and emotion recognition through artificial intelligence continues to grow rapidly. Consequently, it is challenging for researchers to identify gaps, propose new models, and increase user satisfaction. The objective of this study is to explore and analyze potential applications based on artificial intelligence for multimodal dialogue systems incorporating emotion recognition. The methodology used in selecting papers is in accordance with PRISMA and identifies 13 scientific articles whose research proposals are generally focused on convolutional neural networks (CNNs), Long Short-Term Memory (LSTM), GRU, and BERT. The research results identify the proposed models as Mindlink-Eumpy, RHPRnet, Emo Fu-Sense, 3FACRNNN, H-MMER, TMID, DKMD, and MatCR. The datasets used are DEAP, MAHNOB-HCI, SEED-IV, SEDD-V, AMIGOS, and DREAMER. In addition, the metrics achieved by the models are presented. It is concluded that emotion recognition models such as Emo Fu-Sense, 3FACRNNN, and H-MMER obtain outstanding results, with their accuracy ranging from 92.62% to 98.19%, and multimodal dialogue models such as TMID and the scene-aware model with BLEU4 metrics obtain values of 51.59% and 29%, respectively. Full article
(This article belongs to the Special Issue Multimodal User Interfaces and Experiences: Challenges, Applications, and Perspectives—2nd Edition)
►▼ Show Figures

Figure 1

Show export options expand_more Show export options expand_less
Select all
Export citation of selected articles as:
 
Displaying articles 1-11
Back to TopTop