Multimodal Technologies and Interaction

Usability is a principal aspect of the system development process to improve and augment system facilities and meet users’ needs and necessities in all domains. It is no exception for cultural heritage. Usability problems of the interactive technology practice in cultural heritage museums should be recognized thoroughly from the viewpoints of experts and users. This paper reports on a two-phase empirical study to identify the usability problems in audio guides and websites of cultural heritage museums in Vietnam, as a developing country, and Australia, as a developed country. In phase one, five-user experience experts identified usability problems using the set of usability heuristics, and proposed suggestions to mitigate these issues. Ten usability heuristics identified a total of 176 problems for audio guides and websites. In phase two, we conducted field usability surveys to collect the real users’ opinions to detect the usability issues and examine the negative-ranked usability. The outstanding issues for audio guides and websites were pointed out. Identification of relevant usability issues and users’ and experts’ suggestions for these technologies should be given immediate attention to helping organizations and interactive service providers improve technologies’ adoptions. The paper’s findings are reliable inputs for our future study about the preeminent UX framework for interactive technology in the CH domain. Full article

In a case study, we transformed the existing learning program Language Shower, which is used in some Norwegian day-care centers in the Grorud district of Oslo municipality, into a digital solution using an app for smartphones or tablets with the option for further enhancement of the presentation by a NAO robot. The solution was tested in several iterations and multiple day-care centers over several weeks. Measurements of the children’s progress across learning sessions indicated a positive impact of the program using a robot as compared to the program without a robot. In situ observations and interviews with day-care center staff confirmed the solution’s many advantages, but also revealed some important areas for improvement. In particular, the speech recognition needs to be more flexible and robust, and special measures have to be in place to handle children speaking simultaneously. Full article

Eliciting knowledge from domain experts can play an important role throughout the machine learning process, from correctly specifying the task to evaluating model results. However, knowledge elicitation is also fraught with challenges. In this work, we consider why and how machine learning researchers elicit knowledge from experts in the model development process. We develop a taxonomy to characterize elicitation approaches according to the elicitation goal, elicitation target, elicitation process, and use of elicited knowledge. We analyze the elicitation trends observed in 28 papers with this taxonomy and identify opportunities for adding rigor to these elicitation approaches. We suggest future directions for research in elicitation for machine learning by highlighting avenues for further exploration and drawing on what we can learn from elicitation research in other fields. Full article

Despite the prevalence of game-based learning (GBL), most applications of GBL focus on teaching routine skills that are easily teachable, drill-able, and testable. Much less work has examined complex cognitive skills such as computational thinking, and even fewer are projects that have demonstrated commercial or critical success with complex learning in game contexts. Yet, recent successes in the games industry have provided examples of success in game-based complex learning. This article represents a series of case studies on those successes. We interviewed game designers Zach Gage and Jack Schlesinger, creators of Good Sudoku, and Zach Barth, creator of Zachtronics games, using reflexive thematic analysis to thematize findings. We additionally conducted a close play of Duolingo following Bizzocchi and Tanenbaum’s adaptation of close reading. Several insights result from these case studies, including the practice of game design as instructional design, the use of constructionist environments, the tensions between formal education and informal learning, and the importance of entrepreneurialism. Specific recommendations for GBL designers are provided. Full article

This study examines the effect of a COVID-19 Officer Robot (COR) on passersby compliance and the effects of its minor design manipulations on human–robot interaction. A robotic application was developed to ensure participants entering a public building comply with COVID restrictions of a green pass and wearing a face mask. The participants’ attitudes toward the robot and their perception of its authoritativeness were explored with video and questionnaires data. Thematic analysis was used to define unique behaviors related to human–COR interaction. Direct and extended interactions with minor design manipulation of the COR were evaluated in a public scenario setting. The results demonstrate that even minor design manipulations may influence users’ attitudes toward officer robots. The outcomes of this research can support manufacturers in rapidly adjusting their robots to new domains and tasks and guide future designs of authoritative socially assistive robots (SARs). Full article

Digital entertainment games frequently address current societal issues that are also dealt with in geography education, such as climate change or sustainable city development, and give various opportunities for learning. However, in order to be fully able to determine the games’ educational potential and to instruct meaningful reflection on them in class, the designers’ approaches to realism regarding these topics need to be understood. Therefore, we have developed a model of realism in games and conducted 9 interviews with 10 experts from the entertainment game industry about their understanding of and dealing with realism concerning the represented geographical topics. In many cases, the interviewees’ approach to incorporating real-world issues can be regarded as beneficial for their games’ educational potential, and some designers even pursued learning goals. However, we also identified approaches that can result in questionable presentations of real societal issues. We found the most problematic one to be the prioritization of player expectations for the sake of perceived realism. This approach may lead to the depiction of stereotypes and common misconceptions. The results presented in our study may help teachers to prepare reflection on such misrepresentations in class, or designers to become more aware of the educational implications of different forms of game realism. Full article

Cooperation between road users based on V2X communication has the potential to make road traffic safer and more efficient. The exchange of information enables the cooperative orchestration of critical traffic situations, such as truck overtaking maneuvers on freeways. With the benefit of such a system, questions arise concerning system failure or the abrupt and unexpected behavior of road users. A human-machine interface (HMI) organizes and negotiates the cooperation between drivers and maintains smooth interaction, trust, and system acceptance, even in the case of a possible system failure. A study was conducted with 30 truck drivers on a dynamic truck driving simulator to analyze the negotiation of cooperation requests and the reaction of truck drivers to potential system failures. The results show that an automated cooperation request does not translate into a significantly higher cooperation success rate. System failures in cooperative truck passing maneuvers are not considered critical by truck drivers in this simulated environment. The next step in the development process is to investigate how the success rate of truck overtaking maneuvers on freeways can be further increased as well as the implementation of the system in a real vehicle to investigate the reaction behavior of truck drivers in case of system failures in a real environment. Full article

One of the most tangible effects of music is its ability to alter our perception of time. Research on waiting times and time estimation of musical excerpts has attested its veritable effects. Nevertheless, there exist contrasting results regarding several musical features’ influence on time perception. When considering emotional valence and arousal, there is some evidence that positive affect music fosters time underestimation, whereas negative affect music leads to overestimation. Instead, contrasting results exist with regard to arousal. Furthermore, to the best of our knowledge, a systematic investigation has not yet been conducted within the audiovisual domain, wherein music might improve the interaction between the user and the audiovisual media by shaping the recipients’ time perception. Through the current between-subjects online experiment (n = 565), we sought to analyze the influence that four soundtracks (happy, relaxing, sad, scary), differing in valence and arousal, exerted on the time estimation of a short movie, as compared to a no-music condition. The results reveal that (1) the mere presence of music led to time overestimation as opposed to the absence of music, (2) the soundtracks that were perceived as more arousing (i.e., happy and scary) led to time overestimation. The findings are discussed in terms of psychological and phenomenological models of time perception. Full article

Smartphones are a constant companion in everyday life. Interacting with a smartphone calls for a multimodal input and often leads to a multisensory output. Combining research in human-computer interaction (HCI) and psychology, the present research explored the idea that a smartphone is more than a smart object but represents an object to which people feel emotionally attached to and which is even perceived as a part or an extension of a person’s self. To this end, we used an established rubber hand illusion paradigm to experimentally induce body ownership experiences in young adults (n = 76) in a 4-level mixed-design study. Our results revealed that in contrast to a neutral control object participants indeed felt attached to a smartphone, perceived it as a part of themselves and felt the need to interact with the device. This was specifically pronounced when hedonic characteristics were evaluated as high and when its usage for social communication was highlighted during the experiment. Psychological mechanisms of the incorporation of technologies are discussed and connected to positive and negative effects of smartphone usage on human behavior, its implications for technology design and marketing. Full article

Direct ophthalmoscopy (DO) is a medical procedure whereby a health professional, using a direct ophthalmoscope, examines the eye fundus. DO skills are in decline due to the use of interactive diagnostic equipment and insufficient practice with the direct ophthalmoscope. To address the loss of DO skills, physical and computer-based simulators have been developed to offer additional training. Among the computer-based simulations, virtual and augmented reality (VR and AR, respectively) allow simulated immersive and interactive scenarios with eye fundus conditions that are difficult to replicate in the classroom. VR and AR require employing 3D user interfaces (3DUIs) to perform the virtual eye examination. Using a combination of a between-subjects and within-subjects paradigm with two groups of five participants, this paper builds upon a previous preliminary usability study that compared the use of the HTC Vive controller, the Valve Index controller, and the Microsoft HoloLens 1 hand gesticulation interaction methods when performing a virtual direct ophthalmoscopy eye examination. The work described in this paper extends our prior work by considering the interactions with the Oculus Quest controller and Oculus Quest hand-tracking system to perform a virtual direct ophthalmoscopy eye examination while allowing us to compare these methods without our prior interaction techniques. Ultimately, this helps us develop a greater understanding of usability effects for virtual DO examinations and virtual reality in general. Although the number of participants was limited, n = 5 for Stage 1 (including the HTC Vive controller, the Valve Index controller, and the Microsoft HoloLens hand gesticulations), and n = 13 for Stage 2 (including the Oculus Quest controller and the Oculus Quest hand tracking), given the COVID-19 restrictions, our initial results comparing VR and AR 3D user interactions for direct ophthalmoscopy are consistent with our previous preliminary study where the physical controllers resulted in higher usability scores, while the Oculus Quest’s more accurate hand motion capture resulted in higher usability when compared to the Microsoft HoloLens hand gesticulation. Full article

Augmented reality (AR) is widely used to guide users when performing complex tasks, for example, in education or industry. Sometimes, these tasks are a succession of subtasks, possibly distant from each other. This can happen, for instance, in inspection operations, where AR devices can give instructions about subtasks to perform in several rooms. In this case, AR guidance is both needed to indicate where to head to perform the subtasks and to instruct the user about how to perform these subtasks. In this paper, we propose an approach based on user activity detection. An AR device displays the guidance for wayfinding when current user activity suggests it is needed. We designed the first prototype on a head-mounted display using a neural network for user activity detection and compared it with two other guidance temporality strategies, in terms of efficiency and user preferences. Our results show that the most efficient guidance temporality depends on user familiarity with the AR display. While our proposed guidance has not proven to be more efficient than the other two, our experiment hints toward several improvements of our prototype, which is a first step in the direction of efficient guidance for both wayfinding and complex task completion. Full article

This paper describes an overview of olfactory displays (human–computer interfaces that generate and diffuse an odor to a user to stimulate their sense of smell) that have been proposed and researched for supporting education and training. Past research has shown that olfaction (the sense of smell) can support memorization of information, stimulate information recall, and help immerse learners and trainees into educational virtual environments, as well as complement and/or supplement other human sensory channels for learning. This paper begins with an introduction to olfaction and olfactory displays, and a review of techniques for storing, generating and diffusing odors at the computer interface. The paper proceeds with a discussion on educational theories that support olfactory displays for education and training, and a literature review on olfactory displays that support learning and training. Finally, the paper summarizes the advantages and challenges regarding the development and application of olfactory displays for education and training. Full article

The increasing phenomenon of “cyberhate” is concerning because of the potential social implications of this form of verbal violence, which is aimed at already-stigmatized social groups. According to information collected by the Ministry of the Interior of Spain, the category of sexual orientation and gender identity is subject to the third-highest number of registered hate crimes, ranking behind racism/xenophobia and ideology. However, most of the existing computational approaches to online hate detection simultaneously attempt to address all types of discrimination, leading to weaker prototype performances. These approaches focus on other reasons for hate—primarily racism and xenophobia—and usually focus on English messages. Furthermore, few detection models have used manually generated databases as a training corpus. Using supervised machine learning techniques, the present research sought to overcome these limitations by developing and evaluating an automatic detector of hate speech motivated by gender and sexual orientation. The focus was Spanish-language posts on Twitter. For this purpose, eight predictive models were developed from an ad hoc generated training corpus, using shallow modeling and deep learning. The evaluation metrics showed that the deep learning algorithm performed significantly better than the shallow modeling algorithms, and logistic regression yielded the best performance of the shallow algorithms. Full article

(1) COVID-19 pandemic restrictions caused a dramatic shift in research activities, forcing the adoption of remote practices and methods. Despite the known benefits of remote testing, there is limited knowledge on how to prepare and conduct such studies in the industrial context where the target users are experts and company employees. (2) In this article, we detail how we organized VR user tests in five industrial cases during the pandemic, focusing on practicalities and procedures. We cover both on-site testing, including disinfecting and other safety protocols, as well as remote and hybrid setups where both remote and on-site participants were involved. Subject matter experts from eight countries were involved in a total of 22 tests. (3) We share insights for VR user test arrangements relevant to the pandemic, remote and hybrid setups, and an industrial context, among others. (4) Our work confirms that with careful planning it is possible to organize user tests remotely. There are also some limitations in remote user testing, such as reduced visibility and interaction with participants. Most importantly, we list practical recommendations for organizing hybrid user tests with safety and disinfecting procedures for on-site VR use. Full article

As a precursor to future public transportation, automated shuttle buses can already be experienced in some test regions, but the general public still has reservations and may not yet be ready for this change. For example, the fact that such vehicles might operate independently (without a human driver) creates a barrier of uncertainty and mistrust among people. In this work, we aim to identify and classify the prevailing reservations and propose solutions. We followed the User Centered Design (UCD) process to design concepts that are specifically tailored to the needs of future public transport users. After related work analysis, on-site research, and pre-studies, two main studies were conducted specifically to address communication in the exterior (n = 24) and interior/service design (n = 21). For both studies, we applied a mixed-methods approach combining quantitative and qualitative measures. Our results indicate that, in general, existing ways of communication in the exterior are insufficient to meet future needs. The two visualization concepts for external communication developed in this work were rated (significantly) better in most dimensions of the User Experience Questionnaire (UEQ), when compared to the baseline condition with no additional visualization. Furthermore, preferences among the study participants towards simple, highly visible, and well-known lighting concepts could be observed. As for the interior, the results show that participants rated attractiveness highly for the two design concepts (closer, further in the future) as compared to current, state-of-the-art solutions (automated buses currently in operation). For the “near future” concept, the pragmatic quality dominated, while in the other (the “far future”) concept the hedonic quality was in the foreground. From the results, design recommendations in different categories were derived, which reflect the general openness of the public towards new technologies and interior approaches, but also point out the importance for privacy and designated personal spaces inside an (automated) shuttle bus. Some of the results do not strictly apply to automated shuttle buses, and can serve as valuable suggestions for improving conventional shuttle buses. Full article

Augmenting online interpersonal communication with biosignals, often in the form of heart rate sharing, has shown promise in increasing affiliation, feelings of closeness, and intimacy. Increasing empathetic awareness in the professional domain and in the customer interface could benefit both customer and employee satisfaction, but heart rate sharing in this context needs to consider issues around physiological monitoring of employees, appropriate level of intimacy, as well as the productivity outlook. In this study, we explore heart rate sharing at the workplace and study its effects on task performance. Altogether, 124 participants completed a collaborative visual guidance task using a chat box with heart rate visualization. Participants’ feedback about heart rate sharing reveal themes such as a stronger sense of human contact and increased self-reflection, but also raise concerns around unnecessity, intimacy, privacy and negative interpretations. Live heart rate was always measured, but to investigate the effect of heart rate sharing on task performance, half of the customers were told that they were seeing a recording, and half were told that they were seeing the advisor’s live heart beat. We found a negative link between awareness and task performance. We also found that higher ratings of usefulness of the heart rate visualization were associated with increased feelings of closeness. These results reveal that intimacy and privacy issues are particularly important for heart rate sharing in professional contexts, that preference modulates the effects of heart rate sharing on social closeness, and that heart rate sharing may have a negative effect on performance. Full article

Understanding the behavioral dynamics that underline human-robot interactions in groups remains one of the core challenges in social robotics research. However, despite a growing interest in this topic, there is still a lack of established and validated measures that allow researchers to analyze human-robot interactions in group scenarios; and very few that have been developed and tested specifically for research conducted in-the-wild. This is a problem because it hinders the development of general models of human-robot interaction, and makes the comprehension of the inner workings of the relational dynamics between humans and robots, in group contexts, significantly more difficult. In this paper, we aim to provide a reflection on the current state of research on human-robot interaction in small groups, as well as to outline directions for future research with an emphasis on methodological and transversal issues. Full article

In emergency contexts such as the COVID-19 pandemic, mental health risk factors affect children and may affect behavioral and emotional problems including anxiety, self-blame, emotional disturbance, and stress. Preventive measures are crucial to address these health risks. Research highlights strength-based factors, such as socio-emotional skills and prosocial behaviors, as important for childhood development and socio-emotional wellbeing. Yet, the initial evidence base shows mixed effectiveness and insufficient behavior change theory application into socio-emotional and prosocial digital game interventions. Additionally, few interventions are designed with a clear process to convert theory into practical game solutions and very limited guidance on the digital game development process exists. This study describes the Intervention Mapping Protocol (IMP) and theory application in the design of a digital game intervention which aims to strengthen social-emotional skills development and promote prosocial behavior in 8–11-year-old children. The method systematically describes the steps of the IMP process in detail to guide future theory-based game design. The results explain the final digital game prototype that was co-designed with continuous input and insights provided by stakeholders and feedback from children. This paper contributes to our understanding of theory application in the design of digital game interventions focused on health and behavior change and provides much needed guidance on how theory and stakeholder input can be incorporated into a final game design. Full article

Facial expressions are of utmost importance in social interactions, allowing communicative prompts for a speaking turn and feedback. Nevertheless, not all have the ability to express themselves socially and emotionally in verbal and non-verbal communication. In particular, individuals with Autism Spectrum Disorder (ASD) are characterized by impairments in social communication, repetitive patterns of behaviour, and restricted activities or interests. In the literature, the use of robotic tools is reported to promote social interaction with children with ASD. The main goal of this work is to develop a system capable of automatic detecting emotions through facial expressions and interfacing them with a robotic platform (Zeno R50 Robokind^® robotic platform, named ZECA) in order to allow social interaction with children with ASD. ZECA was used as a mediator in social communication activities. The experimental setup and methodology for a real-time facial expression (happiness, sadness, anger, surprise, fear, and neutral) recognition system was based on the Intel^® RealSense™ 3D sensor and on facial features extraction and multiclass Support Vector Machine classifier. The results obtained allowed to infer that the proposed system is adequate in support sessions with children with ASD, giving a strong indication that it may be used in fostering emotion recognition and imitation skills. Full article