Multimodal Technol. Interact., Volume 6, Issue 9 (September 2022) – 14 articles

With the development of autonomous technology, the research into multimodal human-machine interaction (HMI) for autonomous vehicles (AVs) has attracted extensive attention, especially in automotive wellness. To support the design of HMIs for automotive wellness in AVs, this paper proposes a multimodal design framework. First, three elements of the framework were envisioned based on the typical composition of an interactive system. Second, a five-step process for utilizing the proposed framework was suggested. Third, the framework was applied in a design education course for exemplification. Finally, the AttrakDiff questionnaire was used to evaluate these interactive prototypes with 20 participants who had an affinity for HMI design. The questionnaire responses showed that the overall impression was positive and this framework can help design students to effectively identify research gaps and expand design concepts in a systematic way. The proposed framework offers a design approach for the development of multimodal HMIs for autonomous wellness in AVs. Full article

User experience (UX) has drawn the attention of the banking industry in the past few decades. Although banking systems have a complete service process to ensure financial safety for customers, the mobile banking UX has much potential to be improved. Most research in this field of study relies on existing criteria to describe a user’s experience. However, these criteria are focused more on usability measurement, which neglects to identify the requirements of end-users. Users are asked to give feedback on the provided application, limiting the scope of the user study. Therefore, this study uses mixed methods research and in-depth semi-structured interviews to collect end-user UX requirements to build a UX measurement framework of five main services transfers, including financial management, loans, account openings, and credit cards. This study uses an online survey to validate and revise the framework by applying analytic hierarchy process (AHP) analysis to quantify criteria. We interviewed 17 customers and collected 857 online validation surveys, and 400 customers attended the AHP analysis. As a result, this study proposes a three-level measurement framework for mobile banking applications in a Chinese context. The first-level criteria are scenario requirements (24.03%), data requirements (20.98%), and function requirements (54.99%). We hope that the framework will guide designers and researchers to design better user-friendly user interfaces and improve customer satisfaction rates in the future. Full article

In the context of automated vehicles, transparency of in-vehicle intelligent agents (IVIAs) is an important contributor to driver perception, situation awareness (SA), and driving performance. However, the effects of agent transparency on driver performance when the agent is unreliable have not been fully examined yet. This paper examined how transparency and reliability of the IVIAs affect drivers’ perception of the agent, takeover performance, workload and SA. A 2 × 2 mixed factorial design was used in this study, with transparency (Push: proactive vs. Pull: on-demand) as a within-subjects variable and reliability (high vs. low) as a between-subjects variable. In a driving simulator, 27 young drivers drove with two types of in-vehicle agents during the conditionally automated driving. Results suggest that transparency influenced participants’ perception on the agent and perceived workload. High reliability agent was associated with higher situation awareness and less effort, compared to low reliability agent. There was an interaction effect between transparency and reliability on takeover performance. These findings could have important implications for the continued design and development of IVIAs of the automated vehicle system. Full article

Computational fluid dynamics (CFD) are being used more and more in the industry to understand and optimize processes such as fluid flows. At the same time, tools such as augmented reality (AR) are becoming increasingly important with the realization of Industry 5.0 to make data and processes more tangible. Placing the two together paves the way for a new method of active learning and also for an interesting and engaging way of presenting industry processes. It also enables students to reinforce their understanding of the fundamental concepts of fluid dynamics in an interactive way. However, this is not really being utilized yet. For this reason, in this paper, we aim to combine these two powerful tools. Furthermore, we present the framework of a modular open-source library for scientific visualization of fluid flow “OpenVisFlow” which simplifies the creation of such applications and enables seamless visualization without other software by allowing users to integrate the visualization step into the simulation code. Using this framework and the open-source extension AR-Core, we show how a new markerless visualization tool can be implemented. Full article

Knowledge about human behaviour and motivation is essential for designing a positive user experience (UX). Theories of psychological needs have been profoundly researched and well-established in UX research. Experience categories are a rather new practical human-centred design method; they are not based on a psychological model, but instead on an empirical approach. Experience categories describe common positive experiences in a particular context. According to Hassenzahl, positive experiences result from the fulfilment of psychological needs. However, there has been no research on how experience categories and needs are associated. To fill this gap and enrich the knowledge about experience categories, we investigated which needs co-occur with them. In Study 1, we used a more general approach: experience categories of work context were operationalised with scenarios that were rated with a needs questionnaire. In Study 2, we aimed to replicate the findings of Study 1 in a more specific work context by investigating the relationship between experience categories and needs for an existing sample of experiences. Results show a consistent relationship between some experience categories and needs in both studies. Moreover, the need for competence was particularly relevant in the work context. Future studies should expand on this research to further work contexts. Full article

With this article, I would like to present a critical examination of my position on the role of design in museums by focusing on the influence of human-centred design practices (also known as design thinking) in the context of digital transformation in museums, a transformation that has been accelerated by two years of the COVID-19 pandemic and its restrictions. The article aims to offer a set of propositions about the contribution of design thinking (DT) to the main challenges museums are facing in a moment of digital transformation. A rigorous evaluation of the value of DT on museums is beyond the scope of this article, however, there is a sufficient body of academic and professional literature to hypothesise the contribution of DT in addressing digital transformation challenges. My argumentation is supported by evidence from the literature review in the museum sector and academia. Firstly, through a critical examination of facts (museum initiatives during the pandemic) and a critical reflection of existing literature, I will identify a set of key museum challenges. Secondly, by critically looking at design literature, I will suggest a set of propositions by discussing the contribution of DT practices in addressing those challenges. Full article

Here we describe a proof-of-concept case study focusing on the design and development of a novel computer interface that uses facial muscles to control interactivity within a virtual environment. We have developed a system comprised of skin-mounted electrodes that detect underlying muscle activity through electromyography. The signals from the electrodes are filtered and smoothed, then used as input data to an application that displays a virtual environment with a 3D animated avatar. The user’s expressions control the facial movements of the avatar, thus conveying user emotions through real-time animation of a representative face in a virtual scenario. To achieve this, we collaborated with our Public and Patient Involvement focus group to discuss concepts and design appropriate interactions, while simultaneously developing a prototype system. Programmers and 3D artists worked together to create a system whereby individual user facial muscles are connected to 3D animated models of the same muscle features represented in an avatar, providing the user with an option to receive visual and numerical feedback on the extent of their muscle control. Using the prototype system, people can communicate facial expressions virtually with each other, without the need for a camera. This research is part of an on-going project to develop a facial muscle rehabilitation system that can be optimized to help patients with conditions such as hypomimia. Full article

In second-language communication, emotional feedbacks play a preponderant role in instilling positive emotions and thereby facilitating the production of the target language by second-language learners. In contrast, facial expressions help convey emotion, intent, and sometimes even desired actions more effectively. Additionally, according to the facial feedback hypothesis, a major component of several contemporary theories of emotion, facial expressions can regulate emotional behavior and experience. The aim of this study was to determine whether and to what extent emotional expressions reproduced by virtual agents could provide empathetic support to second-language learners during communication tasks. To do so, using the Facial Coding Action System, we implemented a prototype virtual agent that can display a collection of nonverbal feedbacks, including Ekman’ six basic universal emotions and gazing and nodding behaviors. Then, we designed a Wizard of Oz experiment in which second-language learners were assigned independent speaking tasks with a virtual agent. In this paper, we outline our proposed method and report on an initial experimental evaluation which validated the meaningfulness of our approach. Moreover, we present our next steps for improving the system and validating its usefulness through large-scale experiments. Full article

We conducted a virtual reality study to explore virtual hand illusion through three levels of appearance (Appearance dimension: realistic vs. pixelated vs. toon hand appearances) and two levels of tactile feedback (Tactile dimension: no tactile vs. tactile feedback). We instructed our participants to complete a virtual assembly task in this study. Immediately afterward, we asked them to provide self-reported ratings on a survey that captured presence and five embodiment dimensions (hand ownership, touch sensation, agency and motor control, external appearance, and response to external stimuli). The results of our study indicate that (1) tactile feedback generated a stronger sense of presence, touch sensation, and response to external stimuli; (2) the pixelated hand appearance provided the least hand ownership and external appearance; and (3) in the presence of the pixelated hand, prior virtual reality experience of participants impacted their agency and motor control and their response to external stimuli ratings. This paper discusses our findings and provides design considerations for virtual reality applications with respect to the realistic appearance of virtual hands and tactile feedback. Full article

The progression and adoption of innovative learning methodologies signify that a respective part of society is open to new technologies and ideas and thus is advancing. The latest innovation in teaching is the use of Augmented Reality (AR). Applications using this technology have been deployed successfully in STEM (Science, Technology, Engineering, and Mathematics) education for delivering the practical and creative parts of teaching. Since AR technology already has a large volume of published studies about education that reports advantages, limitations, effectiveness, and challenges, classifying these projects will allow for a review of the success in the different educational settings and discover current challenges and future research areas. Due to COVID-19, the landscape of technology-enhanced learning has shifted more toward blended learning, personalized learning spaces and user-centered approach with safety measures. The main findings of this paper include a review of the current literature, investigating the challenges, identifying future research areas, and finally, reporting on the development of two case studies that can highlight the first steps needed to address these research areas. The result of this research ultimately details the research gap required to facilitate real-time touchless hand interaction, kinesthetic learning, and machine learning agents with a remote learning pedagogy. Full article

HCI research has explored a variety of technologies to support human-to-human touch by simulating or mediating the feeling of touch over distance. Restrictions in activities involving touch, close contact, and physical proximity due to the COVID-19 pandemic have made it more pertinent to understand and investigate the goals, opportunities, and limitations of technologically mediated touch. We conducted an exploratory interview study with 18 participants across the United States in April and June of 2021 to understand how the COVID-19 pandemic has changed people’s attitudes towards engaging in touch, being co-located, and sharing physical objects. We observed four themes: new efforts to evaluate safety, new preferences for physically co-locating without touch, technology solutions amplifying separation, and a newfound appreciation for human touch and touch-adjacent activities. These results suggest that future touch technologies may need to shift focus from simulating human touch convincingly to allowing for interactions that incorporate these emergent sensitivities to safety and preferences for physical co-location. Full article

Drivers of L3 automated vehicles (AVs) are not required to continuously monitor the AV system. However, they must be prepared to take over when requested. Therefore, it is necessary to design an in-vehicle environment that allows drivers to adapt their levels of preparedness to the likelihood of control transition. This study evaluates ambient in-vehicle lighting that continuously communicates the current level of AV reliability, specifically on how it could influence drivers’ take-over performance and mental workload (MW). We conducted an experiment in a driving simulator with 42 participants who experienced 10 take-over requests (TORs). The experimental group experienced a four-stage ambient light display that communicated the current level of AV reliability, which was not provided to the control group. The experimental group demonstrated better take-over performance, based on lower vehicle jerks. Notably, perceived MW did not differ between the groups, and the EEG indices of MW (frontal theta power, parietal alpha power, Task–Load Index) did not differ between the groups. These findings suggest that communicating the current level of reliability using ambient light might help drivers be better prepared for TORs and perform better without increasing their MW. Full article

Researchers have proposed a wide range of categorization schemes in order to characterize the space of VR locomotion techniques. In a previous work, a typology of VR locomotion techniques was proposed, introducing motion-based, roomscale-based, controller-based, and teleportation-based types of VR locomotion. The fact that (i) the proposed typology is used widely and makes a significant research impact in the field and (ii) the VR locomotion field is a considerably active research field, creates the need for this typology to be up-to-date and valid. Therefore, the present study builds on this previous work, and the typology’s consistency is investigated through a systematic literature review. Altogether, 42 articles were included in this literature review, eliciting 80 instances of 10 VR locomotion techniques. The results indicated that current typology cannot cover teleportation-based techniques enabled by motion (e.g., gestures and gazes). Therefore, the typology was updated, and a new type was added: “motion-based teleporting.” Full article

For many years, HCI research has been known to suffer from a replication crisis, due to the lack of openly available datasets and accompanying code. Recent research has identified several barriers that prevent the wider sharing of primary research materials in HCI, but such material does, in fact, exist. Interested in the field of mobile text entry research, and largely hindered by the lack of access to participants due to the COVID-19 pandemic, the exploration of a recently published open gaze and touch dataset became an appealing prospect. This paper demonstrates the numerous problems and the extent of required effort related to understanding, sanitising and utilising open data in order to produce meaningful outcomes from it, through a detailed account of working with this dataset. Despite these issues, the paper demonstrates the value of open data as a means to produce novel contributions, without the need for additional new data (in this case, an unsupervised learning pipeline for the robust detection of gaze clusters in vertically distinct areas of interest). Framing the experience of this case study under a dataset lifecycle model intended for ML open data, a set of useful guidelines for researchers wishing to exploit open data is derived. A set of recommendations is also proposed, about the handling of papers accompanied by data, by conferences and journals in the future, Finally, the paper proposes a set of actions for the mobile text entry community, in order to facilitate data sharing across its members. Full article