Multimodal Technol. Interact., Volume 6, Issue 10 (October 2022) – 11 articles

Smart technology is already present in many areas of everyday life. People rely on algorithms in crucial life domains such as finance and healthcare, and the smart car promises a more relaxed driving experience—all the while, the technology recedes further into the background. The smarter the technology, the more intransparent it tends to become. Users no longer understand how the technology works, what its limits are, and what consequences regarding autonomy and privacy emerge. Both extremes, total invisibility and total transparency, come with specific challenges and do not form reasonable design goals. This research explores the potential tension between smart and invisible versus transparent and understandable technology. We discuss related theories from the fields of explainable AI (XAI) as well as trust psychology, and then introduce transparency in smart spaces as a special field of application. A case study explores specific challenges and design approaches through the example of a so-called room intelligence (RI), i.e., a special kind of smart living room. We conclude with research perspectives for more general design approaches and implications for future research. Full article

Over decades, many researchers developed complex in-lab systems with the overall goal to track multiple body parts of the user for a richer and more powerful 2D/3D interaction with a distant display. In this work, we introduce a novel smartphone-based tracking approach that eliminates the need for complex tracking systems. Relying on simultaneous usage of the front and rear smartphone cameras, our solution enables rich spatial interactions with distant displays by combining touch input with hand-gesture input, body and head motion, as well as eye-gaze input. In this paper, we firstly present a taxonomy for classifying distant display interactions, providing an overview of enabling technologies, input modalities, and interaction techniques, spanning from 2D to 3D interactions. Further, we provide more details about our implementation—using off-the-shelf smartphones. Finally, we validate our system in a user study by a variety of 2D and 3D multimodal interaction techniques, including input refinement. Full article

Digital games can be defined as games supported by audiovisual apparatus and based on storytelling. This work aims to frame video games in the recent perspective offered by positive psychology and focuses on risk perception and on the promotion of protective behaviors in adolescent students, by means of a new Italian ad hoc created digital game named ‘Scare-away risks’ (Scacciarischi). In its storytelling, the different characters must defeat monsters that symbolically represent potential risks in home, school and work contexts. On this basis, the present study assumes that playing video games, as an engaging and motivating activity, can improve the adolescents’ psychological perception of home, school and work risks. To verify these hypotheses, a quasi-experiment comparing students who played Scacciarischi (experimental group) with a control group was conducted. At the end of the game, a questionnaire was administered to 149 participants, balanced for gender, investigating (a) the level of engagement during the game, (b) the perceived risks, (c) the locus of control, (d) the protective behavior intentions. The results showed that playing Scacciarischi is related to higher levels of engagement, internal locus of control, risk perception and protective behavioral intentions. In addition, both engagement and internal locus of control act as precursors of the other two variables. Finally, reflections about the importance of the engaging experience of Scacciarischi in the domain of safety, prevention and health issues are proposed. Full article

Augmented Reality (AR) solutions are emerging in multiple scenarios of application as Industry 4.0 takes shape. In particular, for remote collaboration, flexible mechanisms such as authoring tools can be used to generate instructions and assist human operators as they experience increased complexity in their daily tasks. In addition to the traditional handicap of ensuring instructions can be intuitively created without having to understand complicated AR concepts, another relevant topic is the fact that the quality of said instructions is not properly analyzed prior to the tools being evaluated. This means that the characteristics of the visual content are not adequately assessed beforehand. Hence, it is essential to be aware of the cognitive workload associated with AR instructions to assert if they can be easily understood and accepted before being deployed in real-world scenarios. To address this, we focused on AR during sessions of remote guidance. Based on a participatory process with domain experts from the industry sector, a prototype for creating AR-based instructions was developed, and a user study with two parts was conducted: (1) first, a set of step-by-step instructions was produced, and their visual characteristics were evaluated by 129 participants based on a set of relevant dimensions; (2) afterward, these instructions were used by nine participants to understand if they could be used to assist on-site collaborators during real-life remote maintenance tasks. The results suggest that the AR instructions offer low visual complexity and considerable visual impact, clarity, and directed focus, thus improving situational understanding and promoting task resolution. Full article

When designing rule-based dialogue systems, the need for the creation of an elaborate design by the designer is a challenge. One way to reduce the cost of creating content is to generate utterances from data collected in an objective and reproducible manner. This study focuses on rule-based dialogue systems using survey data and, more specifically, on opinion dialogue in which the system models the user. In the field of opinion dialogue, there has been little study on the topic of transition methods for modeling users while maintaining their motivation to engage in dialogue. To model them, we adopted information content. Our contribution includes the design of a rule-based dialogue system that does not require an elaborate design. We also reported an appropriate topic transition method based on information content. This is confirmed by the influence of the user’s personality characteristics. The content of the questions gives the user a sense of the system’s intention to understand them. We also reported the possibility that the system’s rational intention contributes to the user’s motivation to engage in dialogue with the system. Full article

With working from home becoming more normalized, creative workshops are increasingly taking place in digital and hybrid form. However, participants are usually less engaged and motivated in these contexts. This is due to less physical presence and activity, complex technical systems and a lack of social interaction and communication. This leaves the facilitators with the challenge that these creative workshops are sometimes not experienced as positively, and therefore participants are not able to work as creatively. An important approach that can strengthen these factors in workshops is the use of warm-ups as a type of playful intervention. Although some research on and compilations of warm-ups exist and may help the situation, they do not yet provide direction on how to specifically promote a positive experience in creative workshops with warm-ups. An important link here is user experience research, which assumes that positive experiences are due to the fulfilment of psychological needs. Based on research about warm-ups and playful interventions in general, we derive categories for classifying warm-ups that can potentially address several specific psychological needs. Then, 28 warm-ups are selected according to their applicability in analogue, digital and hybrid application spaces. Moreover, those 28 warm-ups are assigned to the two most relevant classifying categories. The results are mapped in the form of a ready-to-use Warm UP Set, which is then evaluated for applicability from the facilitators’ perspective and regarding the influence on the emotional experience of the participants. The evaluation shows that the developed Warm UP Set with its categories seems to be suitable to support facilitators in systematically inducing positive experiences in participants in creative workshops. Full article

Background: Chemistry and biology students often have difficulty understanding molecular structures. Sonification (the rendition of data into non-speech sounds that convey information) can be used to support molecular understanding by complementing scientific visualization. A proper sonification design is important for its effective educational use. This paper describes a participatory design (PD) approach to designing and developing the sonification of a molecular structure model to be used in an educational setting. Methods: Biology, music, and computer science students and specialists designed a sonification of a model of an insulin molecule, following Spinuzzi’s PD methodology and involving evolutionary prototyping. The sonification was developed using open-source software tools used in digital music composition. Results and Conclusions: We tested our sonification played on a virtual reality headset with 15 computer science students. Questionnaire and observational results showed that multidisciplinary PD was useful and effective for developing an educational scientific sonification. PD allowed for speeding up and improving our sonification design and development. Making a usable (effective, efficient, and pleasant to use) sonification of molecular information requires the multidisciplinary participation of people with music, computer science, and molecular biology backgrounds. Full article

Contact tracing apps (CTAs) were deployed worldwide in 2020 to combat COVID-19. Due to their low uptake, a growing amount of empirical research is being conducted to understand the factors that drive their adoption. For CTAs to be adopted, users must, first and foremost, find them persuasive. However, there is little research to understand the role user experience (UX) plays in their perceived persuasiveness. Consequently, we conducted an online study on Amazon Mechanical Turk among Canadian and American residents (n = 446) to investigate the most important UX design attributes associated with the perceived persuasiveness of CTAs. The study was based on two app designs (control and persuasive), each of which comprises three use cases: no exposure, exposure, and diagnosis report interfaces. One interface (screenshot) was randomly presented to a participant to view and provide their responses on the perceived UX design attributes and perceived persuasiveness of the interface. In the overall model, we found that perceived usefulness is the most important and consistent UX design attribute that influences perceived persuasiveness (β = 0.29, p < 0.001), followed by perceived trustworthiness (β = 0.24, p < 0.001) and perceived privacy protection (β = 0.16, p < 0.05). Respectively, the three predictors were consistently significant in two-thirds, half, and one-third of the 12 submodels based on app design, adoption status, and country of residence. The relationships regarding the persuasive designs are more likely to be significant, with the variance of the target construct explained by the predictors ranging from 71% to 89% compared with 54% to 69% for the control designs. The three significant attributes will help designers know which UX design attributes to focus on when designing CTAs for future epidemics. More importantly, in predictive modeling, if their ratings are known, they hold potential in predicting new users’ responsiveness to multiple persuasive strategies/messages featured in behavior-change support systems. Full article

Wellbeing at work can be achieved through different strategies; designing for a positive user experience (UX) is one way. However, the relationship between wellbeing and professionally used technology is rather unexplored, especially in work areas that are far from desktop work such as worker guidance systems (WGSs) used in assembly processes. In this paper, we first described a qualitative evaluation (using the valence method) of a prototype WGS designed for a positive UX. The evaluation showed that it elicited far more positive than negative feelings. Based on the results, we improved and redesigned the prototype. We then implemented it in a realistic setting and quantitatively compared it with an established WGS. It was shown that the prototype elicited more positive feelings than the established system, whereas there were no differences in the number of negative markers. Thus, one can assume that the improvement of UX in the redesigned system was due to the positive UX design concepts. However, there were no significant differences in the mood questionnaires. The paper showed that positive experiences at work can be achieved when the design of professional technology is focused on a positive UX. Long-term studies should further investigate whether these experiences lead to a generally elevated mood. Full article

With the recent deployment of the latest generation of Tesla’s Full Self-Driving (FSD) mode, consumers are using semi-autonomous vehicles in both highway and residential driving for the first time. As a result, drivers are facing complex and unanticipated situations with an unproven technology, which is a central challenge for cooperative cognition. One way to support cooperative cognition in such situations is to inform and educate the user about potential limitations. Because these limitations are not always easily discovered, users have turned to the internet and social media to document their experiences, seek answers to questions they have, provide advice on features to others, and assist other drivers with less FSD experience. In this paper, we explore a novel approach to supporting cooperative cognition: Using social media posts can help characterize the limitations of the automation in order to get information about the limitations of the system and explanations and workarounds for how to deal with these limitations. Ultimately, our goal is to determine the kinds of problems being reported via social media that might be useful in helping users anticipate and develop a better mental model of an AI system that they rely on. To do so, we examine a corpus of social media posts about FSD problems to identify (1) the typical problems reported, (2) the kinds of explanations or answers provided by users, and (3) the feasibility of using such user-generated information to provide training and assistance for new drivers. The results reveal a number of limitations of the FSD system (e.g., lane-keeping and phantom braking) that may be anticipated by drivers, enabling them to predict and avoid the problems, thus allowing better mental models of the system and supporting cooperative cognition of the human-AI system in more situations. Full article

In science education laboratory experimentation has a vital role for students’ learning enhancement. Keeping in view the importance of modern day technologies in teaching learning process, various interactive laboratories (ISLs) have been developed to assist students in hands-on experiments in science education. In this paper we describe the potential contributions of existing interactive science laboratories (ISLs) in the major subjects of science, i.e., chemistry, biology and physics. The existing ISLs include virtual labs and simulation software where users performed their experiments. Important problems and challenges in the existing ISLs are highlighted. The systematic literature review (SLR) methodology is used for article searching, selection, and quality assessments. For this study, 86 articles after final selection using SLR are selected and classified into different categories. Each article is selected after briefly studying its different information, including category of the article, key idea, evaluation criterion, and its strengths and weaknesses. A subjective study with field experts was also conducted to investigate one of our existing virtual lab about the practical implementation and to find out the key issues in its implementation and use. Then, considering the suggestions of the subjective study, some guidelines are proposed for the improvement of future ISLs. Full article