Multimodal Interaction with Virtual Agents and Communication Robots

A special issue of Multimodal Technologies and Interaction (ISSN 2414-4088).

Deadline for manuscript submissions: closed (25 January 2024) | Viewed by 2173

Special Issue Editors

Faculty of Informatics, The University of Fukuchiyama, Fukuchiyama, Japan
Interests: multimodal interaction; intelligent virtual agents; social robotics; machine learning; communication science
Division of Advanced Science and Technology, Japan Advanced Institute of Science and Technology, Nomi, Japan
Interests: data mining; machine learning; human dynamics; multimodal interaction; social signal processing
Human Informatics Laboratories, NTT Corporation, Yokosuka, Japan
Interests: multimodal interaction; multimodal machine learning; conversation analysis; social signal processing
Graduate School of Informatics, Kyoto University, Kyoto, Japan
Interests: human–robot interaction; humanoid robots; spoken dialogue system; user experience; conversation analysis
HCI Lab, University of New Brunswick, Fredericton, NB, Canada
Interests: UX design; intelligent user interfaces; video game design; robotics; VR; human factors
Meidai Institute of Innovation for Future Society, Nagoya University, Nagoya, Japan
Interests: human–robot interaction; user experience; interface design; robotics; human factors

Special Issue Information

Dear Colleagues,

Multimodal information, including language, voice, gaze, posture, gestures, and biological signals, facilitates social interaction between humans. It reveals mechanisms of emotion, attitude, personality, skill, role, and other forms of human communication activities. On the other hand, virtual agents and communication robots are developed to save humans from simple and repetitive tasks for more valuable and more sophisticated ones. They imitate human-like outlooks and behaviors so that their users can interact with them using normal conversation without needing specific training. To reproduce such human behaviors and close the interaction loop, these artifacts have to perceive, understand, and generate them in a multimodal way.

In recent years, the advancement of machine learning techniques has enabled higher accuracy and brought out the potential of multimodal social signal processing. Much attention has been focused on such technologies and their applications for understanding and modeling the social aspects of human beings through their communication activities. Through these processes, there is the potential to develop new technologies for human–agent and human–robot interactions.

To further improve these studies, in addition computer science disciplines such as AI, NLP, signal processing, ML, and HCI, other disciplines such as linguistics, psychology, and sociology play important roles in providing the theoretical backgrounds of human communication. In other words, this is an exciting research area that has the potential to encourage collaborations between researchers in a wide variety of disciplines and to inspire new interdisciplinary ideas.

We welcome submissions from all research fields related to multimodal interactions with virtual agents and communication robots. For example, articles related to topics such as theoretical foundations, empirical verifications, analysis as well as component technologies, integrations, interface designs, and system developments are welcomed, as are submissions from behavioral science and other social sciences.

Dr. Hung-Hsuan Huang
Dr. Shogo Okada
Dr. Ryo Ishii
Dr. Divesh Lala
Dr. Daniel Rea
Dr. Nihan Karatas
Guest Editors

Manuscript Submission Information

Manuscripts should be submitted online at by registering and logging in to this website. Once you are registered, click here to go to the submission form. Manuscripts can be submitted until the deadline. All submissions that pass pre-check are peer-reviewed. Accepted papers will be published continuously in the journal (as soon as accepted) and will be listed together on the special issue website. Research articles, review articles as well as short communications are invited. For planned papers, a title and short abstract (about 100 words) can be sent to the Editorial Office for announcement on this website.

Submitted manuscripts should not have been published previously, nor be under consideration for publication elsewhere (except conference proceedings papers). All manuscripts are thoroughly refereed through a single-blind peer-review process. A guide for authors and other relevant information for submission of manuscripts is available on the Instructions for Authors page. Multimodal Technologies and Interaction is an international peer-reviewed open access monthly journal published by MDPI.

Please visit the Instructions for Authors page before submitting a manuscript. The Article Processing Charge (APC) for publication in this open access journal is 1600 CHF (Swiss Francs). Submitted papers should be well formatted and use good English. Authors may use MDPI's English editing service prior to publication or during author revisions.


  • multimodal interaction
  • virtual agents
  • communication robots
  • social robotics
  • machine learning
  • verbal and non-verbal

Published Papers (1 paper)

Order results
Result details
Select all
Export citation of selected articles as:


23 pages, 1523 KiB  
Furthering the Development of Virtual Agents and Communication Robot Devices through the Consideration of the Temporal Home
Multimodal Technol. Interact. 2023, 7(11), 104; - 13 Nov 2023
Viewed by 1346
This paper extends current research on Human–Robot Interaction (HRI) within Human–Computer Interaction (HCI), focusing on how future virtual agents and communication robots can support the temporal structures and routines within the home. We recruited representatives from 15 households with varied compositions, ranging from [...] Read more.
This paper extends current research on Human–Robot Interaction (HRI) within Human–Computer Interaction (HCI), focusing on how future virtual agents and communication robots can support the temporal structures and routines within the home. We recruited representatives from 15 households with varied compositions, ranging from single inhabitants to full nest families. Drawing upon P.G. Wodehouse’s The Inimitable Jeeves as an inspiration, the methodology sought to imitate the relationship between a recruitment agency worker (researcher) and an employer (participant) seeking to hire a personal assistant (e.g., a virtual agent or communication robot device) for their home. A ‘household audit’ comprising a guided household tour and an architectural survey was conducted to ascertain the nuanced spatiotemporal routines within the home. The study analysed the responses of participants using the Labovian narrative schema, a traditional method in linguistics research. The findings were then examined through the lens of Reddy et al.’s temporal features of work to understand how domestic work unfolds within the home from a temporal perspective. We argue that the temporal concepts discussed by Reddy et al. provided valuable insights into the temporal dynamics of everyday activities and could inform the design of virtual agents and communication robotic devices to fulfil their roles as domesticated ‘personal assistants’. Full article
(This article belongs to the Special Issue Multimodal Interaction with Virtual Agents and Communication Robots)
Show Figures

Figure 1

Back to TopTop