Next Issue
Volume 6, August
Previous Issue
Volume 6, June
 
 

Multimodal Technol. Interact., Volume 6, Issue 7 (July 2022) – 13 articles

  • Issues are regarded as officially published after their release is announced to the table of contents alert mailing list.
  • You may sign up for e-mail alerts to receive table of contents of newly released issues.
  • PDF is the official format for papers published in both, html and pdf forms. To view the papers in pdf format, click on the "PDF Full-text" link, and use the free Adobe Reader to open them.
Order results
Result details
Select all
Export citation of selected articles as:
26 pages, 2793 KiB  
Review
Design Considerations for Immersive Virtual Reality Applications for Older Adults: A Scoping Review
Multimodal Technol. Interact. 2022, 6(7), 60; https://doi.org/10.3390/mti6070060 - 20 Jul 2022
Cited by 8 | Viewed by 4087
Abstract
Immersive virtual reality (iVR) has gained considerable attention recently with increasing affordability and accessibility of the hardware. iVR applications for older adults present tremendous potential for diverse interventions and innovations. The iVR literature, however, provides a limited understanding of guiding design considerations and [...] Read more.
Immersive virtual reality (iVR) has gained considerable attention recently with increasing affordability and accessibility of the hardware. iVR applications for older adults present tremendous potential for diverse interventions and innovations. The iVR literature, however, provides a limited understanding of guiding design considerations and evaluations pertaining to user experience (UX). To address this gap, we present a state-of-the-art scoping review of literature on iVR applications developed for older adults over 65 years. We performed a search in ACM Digital Library, IEEE Xplore, Scopus, and PubMed (1 January 2010–15 December 2019) and found 36 out of 3874 papers met the inclusion criteria. We identified 10 distinct sets of design considerations that guided target users and physical configuration, hardware use, and software design. Most studies carried episodic UX where only 2 captured anticipated UX and 7 measured longitudinal experiences. We discuss the interplay between our findings and future directions to design effective, safe, and engaging iVR applications for older adults. Full article
(This article belongs to the Special Issue Virtual Reality and Augmented Reality)
Show Figures

Figure 1

16 pages, 12323 KiB  
Article
An Interdisciplinary Design of an Interactive Cultural Heritage Visit for In-Situ, Mixed Reality and Affective Experiences
Multimodal Technol. Interact. 2022, 6(7), 59; https://doi.org/10.3390/mti6070059 - 18 Jul 2022
Cited by 1 | Viewed by 1969
Abstract
Interactive technologies, such as mixed-reality and natural interactions with avatars, can enhance cultural heritage and the experience of visiting a museum. In this paper, we present the design rationale of an interactive experience for a cultural heritage place in the church of Roncesvalles [...] Read more.
Interactive technologies, such as mixed-reality and natural interactions with avatars, can enhance cultural heritage and the experience of visiting a museum. In this paper, we present the design rationale of an interactive experience for a cultural heritage place in the church of Roncesvalles at the beginning of Camino de Santiago. We followed a participatory design with a multidisciplinary team which resulted in the design of a spatial augmented reality system that employs 3D projection mapping and a conversational agent acting as the storyteller. Multiple features were identified as desirable for an interactive experience: interdisciplinary design team; in-situ; mixed reality; interactive digital storytelling; avatar; tangible objects; gestures; emotions and groups. The findings from a workshop are presented for guiding other interactive cultural heritage experiences. Full article
(This article belongs to the Special Issue Digital Cultural Heritage (Volume II))
Show Figures

Figure 1

16 pages, 791 KiB  
Article
The Appraisal Principle in Multimedia Learning: Impact of Appraisal Processes, Modality, and Codality
Multimodal Technol. Interact. 2022, 6(7), 58; https://doi.org/10.3390/mti6070058 - 18 Jul 2022
Cited by 1 | Viewed by 1359
Abstract
This paper presents two experiments examining the influences of media-specific appraisal and attribution on multimedia learning. The first experiment compares four different versions of learning material (text, text with images, animation with text, and animation with audio). Results reveal that the attributed type [...] Read more.
This paper presents two experiments examining the influences of media-specific appraisal and attribution on multimedia learning. The first experiment compares four different versions of learning material (text, text with images, animation with text, and animation with audio). Results reveal that the attributed type of appraisal, (i.e., the subjective impression of whether a medium is easy or difficult to learn with) impacts invested mental effort and learning outcomes. Though there was no evidence for the modality effect in the first experiment, we were able to identify it in a second study. We were also able to replicate appraisal and attribution findings from study 1 in study 2: if media appraisal leads to the result that learning with a specific medium is difficult, more mental effort will be invested in information processing. Consequently, learning outcomes are better, and learners are more likely to attribute knowledge acquisition to their own abilities. Outcomes also indicate that the modality effect can be explained by avoidance of split-attention rather than modality-specific information processing in working memory. Full article
Show Figures

Figure 1

22 pages, 813 KiB  
Article
Harvesting Context and Mining Emotions Related to Olfactory Cultural Heritage
Multimodal Technol. Interact. 2022, 6(7), 57; https://doi.org/10.3390/mti6070057 - 18 Jul 2022
Cited by 3 | Viewed by 2482
Abstract
This paper presents an Artificial Intelligence approach to mining context and emotions related to olfactory cultural heritage narratives, particularly to fairy tales. We provide an overview of the role of smell and emotions in literature, as well as highlight the importance of olfactory [...] Read more.
This paper presents an Artificial Intelligence approach to mining context and emotions related to olfactory cultural heritage narratives, particularly to fairy tales. We provide an overview of the role of smell and emotions in literature, as well as highlight the importance of olfactory experience and emotions from psychology and linguistic perspectives. We introduce a methodology for extracting smells and emotions from text, as well as demonstrate the context-based visualizations related to smells and emotions implemented in a novel smell tracker tool. The evaluation is performed using a collection of fairy tales from Grimm and Andersen. We find out that fairy tales often connect smell with the emotional charge of situations. The experimental results show that we can detect smells and emotions in fairy tales with an F1 score of 91.62 and 79.2, respectively. Full article
(This article belongs to the Special Issue Digital Cultural Heritage (Volume II))
Show Figures

Figure 1

16 pages, 2616 KiB  
Article
Detecting Emotions from Illustrator Gestures—The Italian Case
Multimodal Technol. Interact. 2022, 6(7), 56; https://doi.org/10.3390/mti6070056 - 17 Jul 2022
Viewed by 1855
Abstract
The evolution of computers in recent years has given a strong boost to research techniques aimed at improving human–machine interaction. These techniques tend to simulate the dynamics of the human–human interaction process, which is based on our innate ability to understand the emotions [...] Read more.
The evolution of computers in recent years has given a strong boost to research techniques aimed at improving human–machine interaction. These techniques tend to simulate the dynamics of the human–human interaction process, which is based on our innate ability to understand the emotions of other humans. In this work, we present the design of a classifier to recognize the emotions expressed by human beings, and we discuss the results of its testing in a culture-specific case study. The classifier relies exclusively on the gestures people perform, without the need to access additional information, such as facial expressions, the tone of a voice, or the words spoken. The specific purpose is to test whether a computer can correctly recognize emotions starting only from gestures. More generally, it is intended to allow interactive systems to be able to automatically change their behaviour based on the recognized mood, such as adapting the information contents proposed or the flow of interaction, in analogy to what normally happens in the interaction between humans. The document first introduces the operating context, giving an overview of the recognition of emotions and the approach used. Subsequently, the relevant bibliography is described and analysed, highlighting the strengths of the proposed solution. The document continues with a description of the design and implementation of the classifier and of the study we carried out to validate it. The paper ends with a discussion of the results and a short overview of possible implications. Full article
(This article belongs to the Special Issue Digital Cultural Heritage (Volume II))
Show Figures

Figure 1

37 pages, 9873 KiB  
Article
Customizing and Evaluating Accessible Multisensory Music Experiences with Pre-Verbal Children—A Case Study on the Perception of Musical Haptics Using Participatory Design with Proxies
Multimodal Technol. Interact. 2022, 6(7), 55; https://doi.org/10.3390/mti6070055 - 17 Jul 2022
Cited by 5 | Viewed by 2919
Abstract
Research on Accessible Digital Musical Instruments (ADMIs) has highlighted the need for participatory design methods, i.e., to actively include users as co-designers and informants in the design process. However, very little work has explored how pre-verbal children with Profound and Multiple Disabilities (PMLD) [...] Read more.
Research on Accessible Digital Musical Instruments (ADMIs) has highlighted the need for participatory design methods, i.e., to actively include users as co-designers and informants in the design process. However, very little work has explored how pre-verbal children with Profound and Multiple Disabilities (PMLD) can be involved in such processes. In this paper, we apply in-depth qualitative and mixed methodologies in a case study with four students with PMLD. Using Participatory Design with Proxies (PDwP), we assess how these students can be involved in the customization and evaluation of the design of a multisensory music experience intended for a large-scale ADMI. Results from an experiment focused on communication of musical haptics highlighted the diversity in employed interaction strategies used by the children, accessibility limitations of the current multisensory experience design, and the importance of using a multifaceted variety of qualitative and quantitative methods to arrive at more informed conclusions when applying a design with proxies methodology. Full article
(This article belongs to the Special Issue Musical Interactions (Volume II))
Show Figures

Figure 1

21 pages, 17802 KiB  
Article
A Study on Attention Attracting Elements of 360-Degree Videos Based on VR Eye-Tracking System
Multimodal Technol. Interact. 2022, 6(7), 54; https://doi.org/10.3390/mti6070054 - 14 Jul 2022
Viewed by 1838
Abstract
In 360-degree virtual reality (VR) videos, users possess increased freedom in terms of gaze movement. As a result, the users’ attention may not move according to the narrative intended by the director and miss out on important parts of the narrative of the [...] Read more.
In 360-degree virtual reality (VR) videos, users possess increased freedom in terms of gaze movement. As a result, the users’ attention may not move according to the narrative intended by the director and miss out on important parts of the narrative of the 360-degree video. Therefore, it is necessary to study a directing technique that can attract user attention in 360-degree VR videos. In this study, we analyzed the directing elements that can attract users’ attention in a 360-degree VR video and developed a 360 VR eye-tracking system to investigate the effect of the attention-attracting elements on the user. Elements that can attract user attention were classified into five categories: object movement, hand gesture, GUI insertion, camera movement, and gaze angle variation. We developed a 360 VR eye-tracking system to analyze whether five attention-attracting elements influence the user’s attention. Based on the eye tracking system, an experiment was conducted to analyze whether the user’s attention moves according to the five attention-attracting elements. Based on the experimental results, it can be seen that ‘hand gesture’ attracted the second most attention shift of the subjects, and ‘GUI insertion’ induced the smallest shift of attention of the subjects. Full article
Show Figures

Figure 1

20 pages, 1954 KiB  
Article
A Comparative Study of Methods for the Visualization of Probability Distributions of Geographical Data
Multimodal Technol. Interact. 2022, 6(7), 53; https://doi.org/10.3390/mti6070053 - 13 Jul 2022
Viewed by 1688
Abstract
Probability distributions are omnipresent in data analysis. They are often used to model the natural uncertainty present in real phenomena or to describe the properties of a data set. Designing efficient visual metaphors to convey probability distributions is, however, a difficult problem. This [...] Read more.
Probability distributions are omnipresent in data analysis. They are often used to model the natural uncertainty present in real phenomena or to describe the properties of a data set. Designing efficient visual metaphors to convey probability distributions is, however, a difficult problem. This fact is especially true for geographical data, where conveying the spatial context constrains the design space. While many different alternatives have been proposed to solve this problem, they focus on representing data variability. However, they are not designed to support spatial analytical tasks involving probability quantification. The present work aims to adapt recent non-spatial approaches to the geographical context, in order to support probability quantification tasks. We also present a user study that compares the efficiency of these approaches in terms of both accuracy and usability. Full article
Show Figures

Figure 1

20 pages, 2196 KiB  
Article
A Framework for Stakeholders’ Involvement in Digital Productions for Cultural Heritage Tourism
Multimodal Technol. Interact. 2022, 6(7), 52; https://doi.org/10.3390/mti6070052 - 27 Jun 2022
Cited by 1 | Viewed by 2194
Abstract
This paper proposes a new framework for the production and development of immersive and playful technologies in cultural heritage in which different stakeholders such as users and local communities are involved early on in the product development chain. We believe that an early [...] Read more.
This paper proposes a new framework for the production and development of immersive and playful technologies in cultural heritage in which different stakeholders such as users and local communities are involved early on in the product development chain. We believe that an early stage of co-creation in the design process produces a clear understanding of what users struggle with, facilitates the creation of community ownership and helps in better defining the design challenge at hand. We show that adopting such a framework has several direct and indirect benefits, including a deeper sense of site and product ownership as direct benefits to the individual, and the creation and growth of tangential economies to the community. Full article
(This article belongs to the Special Issue Co-Design Within and Between Communities in Cultural Heritage)
Show Figures

Figure 1

17 pages, 2606 KiB  
Article
Is Natural Necessary? Human Voice versus Synthetic Voice for Intelligent Virtual Agents
Multimodal Technol. Interact. 2022, 6(7), 51; https://doi.org/10.3390/mti6070051 - 27 Jun 2022
Cited by 6 | Viewed by 2941
Abstract
The use of intelligent virtual agents (IVA) to support humans in social contexts will depend on their social acceptability. Acceptance will be related to the human’s perception of the IVAs as well as the IVAs’ ability to respond and adapt their conversation appropriately [...] Read more.
The use of intelligent virtual agents (IVA) to support humans in social contexts will depend on their social acceptability. Acceptance will be related to the human’s perception of the IVAs as well as the IVAs’ ability to respond and adapt their conversation appropriately to the human. Adaptation implies computer-generated speech (synthetic speech), such as text-to-speech (TTS). In this paper, we present the results of a study to investigate the effect of voice type (human voice vs. synthetic voice) on two aspects: (1) the IVA’s likeability and voice impression in the light of co-presence, and (2) the interaction outcome, including human–agent trust and behavior change intention. The experiment included 118 participants who interacted with either the virtual advisor with TTS or the virtual advisor with human voice to gain tips for reducing their study stress. Participants in this study found the voice of the virtual advisor with TTS to be more eerie, but they rated both agents, with recorded voice and with TTS, similarly in terms of likeability. They further showed a similar attitude towards both agents in terms of co-presence and building trust. These results challenge previous studies that favor human voice over TTS, and suggest that even if human voice is preferred, TTS can deliver equivalent benefits. Full article
Show Figures

Figure 1

21 pages, 1612 KiB  
Article
Inter- and Transcultural Learning in Social Virtual Reality: A Proposal for an Inter- and Transcultural Virtual Object Database to be Used in the Implementation, Reflection, and Evaluation of Virtual Encounters
Multimodal Technol. Interact. 2022, 6(7), 50; https://doi.org/10.3390/mti6070050 - 25 Jun 2022
Cited by 3 | Viewed by 2281
Abstract
Visual stimuli are frequently used to improve memory, language learning or perception, and understanding of metacognitive processes. However, in virtual reality (VR), there are few systematically and empirically derived databases. This paper proposes the first collection of virtual objects based on empirical evaluation [...] Read more.
Visual stimuli are frequently used to improve memory, language learning or perception, and understanding of metacognitive processes. However, in virtual reality (VR), there are few systematically and empirically derived databases. This paper proposes the first collection of virtual objects based on empirical evaluation for inter-and transcultural encounters between English- and German-speaking learners. We used explicit and implicit measurement methods to identify cultural associations and the degree of stereotypical perception for each virtual stimuli (n = 293) through two online studies, including native German and English-speaking participants. The analysis resulted in a final well-describable database of 128 objects (called InteractionSuitcase). In future applications, the objects can be used as a great interaction or conversation asset and behavioral measurement tool in social VR applications, especially in the field of foreign language education. For example, encounters can use the objects to describe their culture, or teachers can intuitively assess stereotyped attitudes of the encounters. Full article
(This article belongs to the Special Issue Virtual Reality and Augmented Reality)
Show Figures

Figure 1

16 pages, 2923 KiB  
Article
Vocational Training in Virtual Reality: A Case Study Using the 4C/ID Model
Multimodal Technol. Interact. 2022, 6(7), 49; https://doi.org/10.3390/mti6070049 - 24 Jun 2022
Cited by 5 | Viewed by 2819
Abstract
Virtual reality (VR) is an emerging technology with a variety of potential benefits for vocational training. Therefore, this paper presents a VR training based on the highly validated 4C/ID model to train vocational competencies in the field of vehicle painting. The following 4C/ID [...] Read more.
Virtual reality (VR) is an emerging technology with a variety of potential benefits for vocational training. Therefore, this paper presents a VR training based on the highly validated 4C/ID model to train vocational competencies in the field of vehicle painting. The following 4C/ID components were designed using the associated 10 step approach: learning tasks, supportive information, procedural information, and part-task practice. The paper describes the instructional design process including an elaborated blueprint for a VR training application for aspiring vehicle painters. We explain the model’s principles and features and their suitability for designing a VR vocational training that fosters integrated competence acquisition. Following the methodology of design-based research, several research methods (e.g., a target group analysis) and the ongoing development of prototypes enabled agile process structures. Results indicate that the 4C/ID model and the 10 step approach promote the instructional design process using VR in vocational training. Implementation and methodological issues that arose during the design process (e.g., limited time within VR) are adequately discussed in the article. Full article
(This article belongs to the Special Issue Virtual Reality and Augmented Reality)
Show Figures

Figure 1

25 pages, 1963 KiB  
Article
AI Technologies for Machine Supervision and Help in a Rehabilitation Scenario
Multimodal Technol. Interact. 2022, 6(7), 48; https://doi.org/10.3390/mti6070048 - 22 Jun 2022
Cited by 6 | Viewed by 3125
Abstract
We consider, evaluate, and develop methods for home rehabilitation scenarios. We show the required modules for this scenario. Due to the large number of modules, the framework falls into the category of Composite AI. Our work is based on collected videos with high-quality [...] Read more.
We consider, evaluate, and develop methods for home rehabilitation scenarios. We show the required modules for this scenario. Due to the large number of modules, the framework falls into the category of Composite AI. Our work is based on collected videos with high-quality execution and samples of typical errors. They are augmented by sample dialogues about the exercise to be executed and the assumed errors. We study and discuss body pose estimation technology, dialogue systems of different kinds and the emerging constraints of verbal communication. We demonstrate that the optimization of the camera and the body pose allows high-precision recording and requires the following components: (1) optimization needs a 3D representation of the environment, (2) a navigation dialogue to guide the patient to the optimal pose, (3) semantic and instance maps are necessary for verbal instructions about the navigation. We put forth different communication methods, from video-based presentation to chit-chat-like dialogues through rule-based methods. We discuss the methods for different aspects of the challenges that can improve the performance of the individual components. Due to the emerging solutions, we claim that the range of applications will drastically grow in the very near future. Full article
Show Figures

Figure 1

Previous Issue
Next Issue
Back to TopTop