Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
6.5
2.4
Journals
Information
Special Issues
Multimodal Human-Computer Interaction
share announcement

Multimodal Human-Computer Interaction

A special issue of Information (ISSN 2078-2489). This special issue belongs to the section "Artificial Intelligence".

Deadline for manuscript submissions: 31 July 2025 | Viewed by 7821

Special Issue Editors

Dr. Nuno Almeida

E-Mail Website
Guest Editor
Department of Electronics, Telecommunications and Informatics, Universidade de Aveiro, 3810-193 Aveiro, Portugal
Interests: multimodal interaction; speech; virtual assistants; smart environments; user-centered design
Prof. Dr. Samuel Silva

E-Mail Website
Guest Editor
Department of Electronics, Telecommunications and Informatics, University of Aveiro, 3810-193 Aveiro, Portugal
Interests: human centered computing; human-computer interaction; multimodal interaction; assistive technologies; computer graphics; visualization; virtual reality
Prof. Dr. António Joaquim da Silva Teixeira

E-Mail Website
Guest Editor
Department of Electronics, Telecommunications and Informatics, Universidade de Aveiro, 3810-193 Aveiro, Portugal
Interests: multimodal interaction; natural user interaction; natural language processing; speech and language processing
Special Issues, Collections and Topics in MDPI journals

Special Issue Information

Dear Colleagues,

Humans interact with one another using different senses. These capacities enable them to absorb information so that it can be interpreted and transmitted information to others. For instance, in human-to-human conversation, one interlocutor speaks and makes gestures to transmit a message, while the other uses the senses of hearing and sight to capture and interpret that message. Just as humans communicate and interact with each other, these senses and mechanisms might be used to interact with computers. Human-to-human communication provides a natural feeling of interaction, unlike interaction with a computer somehow does not, with primary interactions with computers nowadays being mostly oriented towards mouse use, keyboard and touch in mobile devices.

Each day that passes brings us a step closer to accomplishing more natural and intuitive ways of interaction with machines, including gestures, speech, touch, and facial expressions. Evolving technologies (e.g., deep learning and IoT) are allowing, as of today, humans to accomplish what some years ago were scenes from sci-fi cinema, such as having an intelligent personal assistant capable of interacting using voice and gestures. This evolution will continue to bring us new and increasingly evolved technologies that are very intuitive and fun.

Nowadays, several technologies exist that enable the creation of different modalities with the capability of interacting with machines in different ways. Furthermore, input modalities can be combined, and different inputs (fusion) from different modalities can create new meaning when used together. Also, two different output modalities can present information in complementary fashion (via fission).

This Special Issue aims to bring together prominent researchers, experts, and practitioners in the field to share their insights and findings, fostering a comprehensive understanding of the current state of the art. Potential topics include but are not limited to (alphabetical order):

  • Multimodal interaction
  • Technologies for new modalities
  • Speech-based interaction
  • Gestural interfaces
  • Virtual reality
  • Assistants
  • Interaction for IoT ecossystems
  • Smart environments (homes, buildings, cities)
  • Modality fusion
  • Fission
  • Applications
  • Evaluation
  • Context and user awareness and adaptation
  • Tools for developers
  • Human robot interaction
  • Interaction with multi-device systems
  • Migratory interfaces
  • User-centered design
  • Artificial intelligence tools and methods (for multimodal interaction)

Dr. Nuno Almeida
Prof. Dr. Samuel Silva
Prof. Dr. António Joaquim da Silva Teixeira
Guest Editors

Manuscript Submission Information

Manuscripts should be submitted online at www.mdpi.com by registering and logging in to this website. Once you are registered, click here to go to the submission form. Manuscripts can be submitted until the deadline. All submissions that pass pre-check are peer-reviewed. Accepted papers will be published continuously in the journal (as soon as accepted) and will be listed together on the special issue website. Research articles, review articles as well as short communications are invited. For planned papers, a title and short abstract (about 100 words) can be sent to the Editorial Office for announcement on this website.

Submitted manuscripts should not have been published previously, nor be under consideration for publication elsewhere (except conference proceedings papers). All manuscripts are thoroughly refereed through a single-blind peer-review process. A guide for authors and other relevant information for submission of manuscripts is available on the Instructions for Authors page. Information is an international peer-reviewed open access monthly journal published by MDPI.

Please visit the Instructions for Authors page before submitting a manuscript. The Article Processing Charge (APC) for publication in this open access journal is 1600 CHF (Swiss Francs). Submitted papers should be well formatted and use good English. Authors may use MDPI's English editing service prior to publication or during author revisions.

Keywords

  • human–computer interaction
  • multimodal interaction
  • fusion of modalities
  • fission
  • multi-device application
  • assistants

Benefits of Publishing in a Special Issue

  • Ease of navigation: Grouping papers by topic helps scholars navigate broad scope journals more efficiently.
  • Greater discoverability: Special Issues support the reach and impact of scientific research. Articles in Special Issues are more discoverable and cited more frequently.
  • Expansion of research network: Special Issues facilitate connections among authors, fostering scientific collaborations.
  • External promotion: Articles in Special Issues are often promoted through the journal's social media, increasing their visibility.
  • Reprint: MDPI Books provides the opportunity to republish successful Special Issues in book format, both online and in print.

Further information on MDPI's Special Issue policies can be found here.

Published Papers (5 papers)

Download All Papers
Order results
Result details
Show export options expand_more Show export options expand_less
Select all
Export citation of selected articles as:

Research

11 pages, 1513 KiB  
Article
How Human–Robot Interaction Can Influence Task Performance and Perceived Cognitive Load at Different Support Conditions
by Simone Varrasi, Roberto Vagnetti, Nicola Camp, John Hough, Alessandro Di Nuovo, Sabrina Castellano and Daniele Magistro
Information 2025, 16(5), 374; https://doi.org/10.3390/info16050374 - 30 Apr 2025
Viewed by 450
Abstract
Cognitive load refers to the mental resources used for executing simultaneous tasks. Since these resources are limited, individuals can only process a specific amount of information at a time. Daily activities often involve mentally demanding tasks, which is why social robots have been [...] Read more.
Cognitive load refers to the mental resources used for executing simultaneous tasks. Since these resources are limited, individuals can only process a specific amount of information at a time. Daily activities often involve mentally demanding tasks, which is why social robots have been proposed to simplify them and support users. This study aimed to verify whether and how a social robot can enhance the performance and support the management of cognitive load. Participants completed a baseline where a cognitive activity was carried out without support, and three other conditions where similar activities of increasing difficulty were collaboratively made with the NAO robot. In each condition, errors, time, and perceived cognitive load were measured. Results revealed that the robot improved performance and perceived cognitive load when compared to the baseline, but this support was then thwarted by excessive levels of cognitive load. Future research should focus on developing and designing collaborative human–robot interactions that consider the user’s mental demand, to promote effective and personalized robotic help for independent living. Full article
(This article belongs to the Special Issue Multimodal Human-Computer Interaction)
►▼ Show Figures

Figure 1

29 pages, 4988 KiB  
Article
Interaction Glove for 3-D Virtual Environments Based on an RGB-D Camera and Magnetic, Angular Rate, and Gravity Micro-Electromechanical System Sensors
by Pontakorn Sonchan, Neeranut Ratchatanantakit, Nonnarit O-Larnnithipong, Malek Adjouadi and Armando Barreto
Information 2025, 16(2), 127; https://doi.org/10.3390/info16020127 - 9 Feb 2025
Viewed by 3312
Abstract
This paper presents the theoretical foundation, practical implementation, and empirical evaluation of a glove for interaction with 3-D virtual environments. At the dawn of the “Spatial Computing Era”, where users continuously interact with 3-D Virtual and Augmented Reality environments, the need for a [...] Read more.
This paper presents the theoretical foundation, practical implementation, and empirical evaluation of a glove for interaction with 3-D virtual environments. At the dawn of the “Spatial Computing Era”, where users continuously interact with 3-D Virtual and Augmented Reality environments, the need for a practical and intuitive interaction system that can efficiently engage 3-D elements is becoming pressing. Over the last few decades, there have been attempts to provide such an interaction mechanism using a glove. However, glove systems are currently not in widespread use due to their high cost and, we propose, due to their inability to sustain high levels of performance under certain situations. Performance deterioration has been observed due to the distortion of the local magnetic field caused by ordinary ferromagnetic objects present near the glove’s operating space. There are several areas where reliable hand-tracking gloves could provide a next generation of improved solutions, such as American Sign Language training and automatic translation to text and training and evaluation for activities that require high motor skills in the hands (e.g., playing some musical instruments, training of surgeons, etc.). While the use of a hand-tracking glove toward these goals seems intuitive, some of the currently available glove systems may not meet the accuracy and reliability levels required for those use cases. This paper describes our concept of an interaction glove instrumented with miniature magnetic, angular rate, and gravity (MARG) sensors and aided by a single camera. The camera used is an off-the-shelf red, green, and blue–depth (RGB-D) camera. We describe a proof-of-concept implementation of the system using our custom “GMVDK” orientation estimation algorithm. This paper also describes the glove’s empirical evaluation with human-subject performance tests. The results show that the prototype glove, using the GMVDK algorithm, is able to operate without performance losses, even in magnetically distorted environments. Full article
(This article belongs to the Special Issue Multimodal Human-Computer Interaction)
►▼ Show Figures

Figure 1

20 pages, 21227 KiB  
Article
ShapeBand: Design of a Shape-Changing Wristband with Soft Materials and Physiological Sensors for Anxiety Regulation
by Yanting Liu, Zihan Xu, Ben Oldfrey and Youngjun Cho
Information 2025, 16(2), 101; https://doi.org/10.3390/info16020101 - 4 Feb 2025
Viewed by 946
Abstract
We introduce ShapeBand, a new shape-changing wristband designed for exploring multisensory and interactive anxiety regulation with soft materials and physiological sensing. Our approach takes a core principle of self-help psychotherapeutic intervention, aiming to help users to recognize anxiety triggers and engage in regulation [...] Read more.
We introduce ShapeBand, a new shape-changing wristband designed for exploring multisensory and interactive anxiety regulation with soft materials and physiological sensing. Our approach takes a core principle of self-help psychotherapeutic intervention, aiming to help users to recognize anxiety triggers and engage in regulation with attentional distraction. We conducted user-centered design activities to iteratively refine our design requirements and delve into users’ rich experiences, preferences, and feelings. With ShapeBand, we explored bidirectional and dynamic interaction flow in anxiety regulation and subjective factors influencing its use. Our findings suggest that integrating both active and passive modulations can significantly enhance user engagement for effective anxiety intervention. Further, different interactions, characterized by dynamic alterations in bubbles and water flow in the ShapeBand, can provide users with a gamified experience and convey more potent effects. This study provides valuable insights into the future design of tangible anxiety regulation interfaces that can be tailored to subjective feelings and individual needs. Full article
(This article belongs to the Special Issue Multimodal Human-Computer Interaction)
►▼ Show Figures

Graphical abstract

22 pages, 872 KiB  
Article
The Walk of Guilt: Multimodal Deception Detection from Nonverbal Motion Behaviour
by Sharifa Alghowinem, Sabrina Caldwell, Ibrahim Radwan, Michael Wagner and Tom Gedeon
Information 2025, 16(1), 6; https://doi.org/10.3390/info16010006 - 26 Dec 2024
Viewed by 984
Abstract
Detecting deceptive behaviour for surveillance and border protection is critical for a country’s security. With the advancement of technology in relation to sensors and artificial intelligence, recognising deceptive behaviour could be performed automatically. Following the success of affective computing in emotion recognition from [...] Read more.
Detecting deceptive behaviour for surveillance and border protection is critical for a country’s security. With the advancement of technology in relation to sensors and artificial intelligence, recognising deceptive behaviour could be performed automatically. Following the success of affective computing in emotion recognition from verbal and nonverbal cues, we aim to apply a similar concept for deception detection. Recognising deceptive behaviour has been attempted; however, only a few studies have analysed this behaviour from gait and body movement. This research involves a multimodal approach for deception detection from gait, where we fuse features extracted from body movement behaviours from a video signal, acoustic features from walking steps from an audio signal, and the dynamics of walking movement using an accelerometer sensor. Using the video recording of walking from the Whodunnit deception dataset, which contains 49 subjects performing scenarios that elicit deceptive behaviour, we conduct multimodal two-category (guilty/not guilty) subject-independent classification. The classification results obtained reached an accuracy of up to 88% through feature fusion, with an average of 60% from both single and multimodal signals. Analysing body movement using single modality showed that the visual signal had the highest performance followed by the accelerometer and acoustic signals. Several fusion techniques were explored, including early, late, and hybrid fusion, where hybrid fusion not only achieved the highest classification results, but also increased the confidence of the results. Moreover, using a systematic framework for selecting the most distinguishing features of guilty gait behaviour, we were able to interpret the performance of our models. From these baseline results, we can conclude that pattern recognition techniques could help in characterising deceptive behaviour, where future work will focus on exploring the tuning and enhancement of the results and techniques. Full article
(This article belongs to the Special Issue Multimodal Human-Computer Interaction)
►▼ Show Figures

Figure 1

18 pages, 2763 KiB  
Article
Impact of Robot Size and Number on Human–Robot Persuasion
by Abeer Alam, Michael Lwin, Aila Khan and Omar Mubin
Information 2024, 15(12), 782; https://doi.org/10.3390/info15120782 - 5 Dec 2024
Viewed by 1081
Abstract
Technological progress has seamlessly integrated digital assistants into our everyday lives, sparking an interest in social robots that communicate through both verbal and non-verbal means. The potential of these robots to influence human behaviour and attitudes holds significant implications for fields such as [...] Read more.
Technological progress has seamlessly integrated digital assistants into our everyday lives, sparking an interest in social robots that communicate through both verbal and non-verbal means. The potential of these robots to influence human behaviour and attitudes holds significant implications for fields such as healthcare, marketing, and promoting sustainability. This study investigates how the design and behavioural aspects of social robots affect their ability to persuade, drawing on principles from human interaction to enhance the quality of human–robot interactions. Conducted in three stages, the experiments involved 73 participants, offering a comprehensive view of human responses to robotic persuasion. Surprisingly, the findings reveal that individuals tend to be more receptive to a single robot than to groups of robots. Nao was identified as more effective and capable of persuasion than Pepper. This study shows that successful persuasion by robots depends on social influence, the robot’s appearance, and people’s past experiences with technology. Full article
(This article belongs to the Special Issue Multimodal Human-Computer Interaction)
►▼ Show Figures

Graphical abstract

Show export options expand_more Show export options expand_less
Select all
Export citation of selected articles as:
 
Displaying articles 1-5
Back to TopTop