Multimodal Technologies and Interaction

Speech technology has matured so that voice-based reporting utilizing speech-to-text can be applied in various domains. Speech has two major benefits: it enables efficient reporting and speech input improves the quality of the reports since reporting can be done as a part of the workflow without delays between work and reporting. However, designing reporting voice user interfaces (VUIs) for professional use is challenging, as there are numerous aspects from technology to organization and language that need to be considered. Based on our experience in developing professional reporting VUIs with different stakeholders representing both commercial and public sector, we define a design space for voice-based reporting systems. The design space consists of 28 dimensions grouped into five categories: Language Processing, Structure of Reporting, Technical Limitations in the Work Domain, Interaction Related Aspects in the Work Domain, and Organization. We illustrate the design space by discussing four voice-based reporting systems, designed and implemented by us, and describing a design process that utilizes it. The design space enables designers to identify critical aspects of professional reporting VUIs and optimize those for their target domain. The design space can be used as a practical tool especially by designers with limited experience on speech technologies. Full article

Ontology datasets, which encode the expert-defined complex objects mapping the entities, relations, and structures of a domain ontology, are increasingly being integrated into the performance of challenging knowledge-based tasks. Yet, it is hard to use ontology datasets within our tasks without first understanding the ontology which it describes. Using visual representation and interaction design, interactive visualization tools can help us learn and develop our understanding of unfamiliar ontologies. After a review of existing tools which visualize ontology datasets, we find that current design practices struggle to support learning tasks when attempting to build understanding of the ontological spaces within ontology datasets. During encounters with unfamiliar spaces, our cognitive processes align with the theoretical framework of cognitive map formation. Furthermore, designing encounters to promote cognitive map formation can improve our performance during learning tasks. In this paper, we examine related work on cognitive load, cognitive map formation, and the use of interactive visualizations during learning tasks. From these findings, we formalize a set of high-level design criteria for visualizing ontology datasets to promote cognitive map formation during learning tasks. We then perform a review of existing tools which visualize ontology datasets and assess their interface design towards their alignment with the cognitive map framework. We then present PRONTOVISE (PRogressive ONTOlogy VISualization Explorer), an interactive visualization tool which applies the high-level criteria within its design. We perform a task-based usage scenario to illustrate the design of PRONTOVISE. We conclude with a discussion of the implications of PRONTOVISE and its use of the criteria towards the design of interactive visualization tools which help us develop understanding of the ontological space within ontology datasets. Full article

Designing for slow cities and the need to design for future urban environments that include the more than human is a major priority for our times. This position paper problematizes the nature–culture divide in research about place and place-making, where place is understood to be about the sense of meaning we layer on locations in the physical world. It emphasizes the importance of narrative identity and place-making in the context of designing for urban environmental futures and creation of slow cities. We present an overview of a methodology to re-emplace place-making with animals in the context of slow cities and designing for the more than human. The work discussed here explores the use of narrative inquiry with some early narrative data (in the form of stories) about dog walks and those moments where our companion animals demonstrate agentic place-based meaning-making. The problem of understanding “what animals want” and how they make might ”make sense” of an experience is approached via a focus on a rich exemplar case in order to distinguish between emplotment (narrative meaning-making as self) and emplacement (narrative meaning-making as an aspect of place). This is used to create a framework for future evaluation with a view to revealing how “more than human stories”—just like our own familiar human stories—are also about agency and meaning in place. This recognition has import for ways in which we might approach decentring the human when we frame urban design activities. Full article

Language, speech and conversational behaviours reflect cognitive changes that may precede physiological changes and offer a much more cost-effective option for detecting preclinical cognitive decline. Artificial intelligence and machine learning have been established as a means to facilitate automated speech-based cognitive screening through automated recording and analysis of linguistic, speech and conversational behaviours. In this work, a scoping literature review was performed to document and analyse current automated speech-based implementations for cognitive screening from the perspective of human–computer interaction. At this stage, the goal was to identify and analyse the characteristics that define the interaction between the automated speech-based screening systems and the users, potentially revealing interaction-related patterns and gaps. In total, 65 articles were identified as appropriate for inclusion, from which 15 articles satisfied the inclusion criteria. The literature review led to the documentation and further analysis of five interaction-related themes: (i) user interface, (ii) modalities, (iii) speech-based communication, (iv) screening content and (v) screener. Cognitive screening through speech-based interaction might benefit from two practices: (1) implementing more multimodal user interfaces that facilitate—amongst others—speech-based screening and (2) introducing the element of motivation in the speech-based screening process. Full article

The purpose of this perspective paper and technology overview is to encourage collaboration between designers and animal carers in zoological institutions, sanctuaries, research facilities, and in soft-release scenarios for the benefit of all stakeholders, including animals, carers, managers, researchers, and visitors. We discuss the evolution of animal-centered technology (ACT), including more recent animal-centered computing to increase animal wellbeing by providing increased opportunities for choice and control for animals to gain greater self-regulation and independence. We believe this will increase animal welfare and relative freedom, while potentially improving conservation outcomes. Concurrent with the benefits to the animals, this technology may benefit human carers by increasing workplace efficiency and improving research data collection using automated animal monitoring systems. These benefits are balanced against cultural resistance to change, the imposition of greater staff training, a potential reduction in valuable animal-carer interaction, and the financial costs for technology design, acquisition, obsolescence, and maintenance. Successful applications will be discussed to demonstrate how animal-centered technology has evolved and, in some cases, to suggest future opportunities. We suggest that creative uses of animal-centered technology, based upon solid animal welfare science, has the potential for greatly increasing managed animal welfare, eventually growing from individual animal enrichment features to facility-wide integrated animal movement systems and transitions to wildlife release and rewilding strategies. Full article

Hand tracking enables controller-free interaction with virtual environments, which can, compared to traditional handheld controllers, make virtual reality (VR) experiences more natural and immersive. As naturalness hinges on both technological and user-based features, fine-tuning the former while assessing the latter can be used to increase usability. For a grab-and-place use case in immersive VR, we compared a prototype of a camera-based hand tracking interface (Leap Motion) with customized design elements to the standard Leap Motion application programming interface (API) and a traditional controller solution (Oculus Touch). Usability was tested in 32 young healthy participants, whose performance was analyzed in terms of accuracy, speed and errors as well as subjective experience. We found higher performance and overall usability as well as overall preference for the handheld controller compared to both controller-free solutions. While most measures did not differ between the two controller-free solutions, the modifications made to the Leap API to form our prototype led to a significant decrease in accidental drops. Our results do not support the assumption of higher naturalness for hand tracking but suggest design elements to improve the robustness of controller-free object interaction in a grab-and-place scenario. Full article

This article reports from a project introducing a virtual reality simulator with patient-specific input for endovascular aneurysm repair (EVAR) into a surgical environment at a university hospital in Norway during 2016–2019. The project includes acquisition of the simulator, training of personnel, and a mapping of the effects. We followed the process, adopting ethnographic methods including participation in the operating room, simulated patient-specific rehearsals, preparations of the rehearsals, meetings with the simulator company, scientific meetings and scientific work related to the clinical trials (the second author led the clinical trial), in addition to open-ended interviews with vascular surgeons and interventional radiologists. We used the concepts of boundary work and sensework as conceptual lenses through which we studied the introduction of the simulator and how it influenced the nature of work and the professional relationship between the vascular surgeons and the interventional radiologists. We found that the simulator facilitated professional integration, at the same time as it served as a material resource for professional identity development. This study is the first to our knowledge that investigates the role of simulators for professional identity and relationship among surgeons and radiologists. Further studies of simulators in similar and different social contexts may contribute to deeper and more generic understanding of the way simulators influence our working life. Full article

As gaming spectatorship has become a worldwide phenomenon, keeping the spectator in mind while designing games is becoming more important. Here, we explore the factors that influence spectators’ engagement. Through the use of GRiD Crowd, a game akin to life-size Pong, different levels of spectator influence on the game were tested and their impact on engagement via arousal measures were analyzed. Spectator influence on the game was accomplished via smartphone, where 78 participants put in different audience compositions (alongside friends or strangers) were tested. We found that when the spectators had an impact on the game, higher levels of emotional arousal were recorded, which generated an increase in engagement. These results provide a suggestion of design that could be used by game designers who wish to engage their spectatorship, a segment of their target market that is becoming impossible to ignore. Full article

Gesture elicitation studies are a popular means of gaining valuable insights into how users interact with novel input devices. One of the problems elicitation faces is that of legacy bias, when elicited interactions are biased by prior technologies use. In response, methodologies have been introduced to reduce legacy bias. This is the first study that formally examines the production method of reducing legacy bias (i.e., repeated proposals for a single referent). This is done through a between-subject study that had 27 participants per group (control and production) with 17 referents placed in a virtual environment using a head-mounted display. This study found that over a range of referents, legacy bias was not significantly reduced over production trials. Instead, production reduced participant consensus on proposals. However, in the set of referents that elicited the most legacy biased proposals, production was an effective means of reducing legacy bias, with an overall reduction of 11.93% for the chance of eliciting a legacy bias proposal. Full article

The research on children’s learning of science, technology, engineering, and math (STEM) topics from electronic applications (apps) is limited, though it appears that children can reasonably transfer learning from tablet games to particular tasks. We were interested to determine whether these findings would translate to the emerging technology of haptic feedback tablets. The research on haptic feedback technology, specifically, has found that this type of feedback is effective in teaching physics concepts to older students. However, haptic feedback has not yet been sufficiently explored with younger groups (e.g., preschoolers). To determine the effect of playing a STEM game enhanced with haptic technology on learning outcomes, we designed an experiment where preschool participants were randomly exposed to one of three different conditions: (a) STEM game with no haptic feedback (tablet), (b) STEM game enabled with haptic feedback (haptics), or (c) a puzzle game (control). Results revealed no significant differences in comprehension or transfer by condition. Results from this study contribute to the literature on the effectiveness of haptic feedback for preschool STEM learning. Full article

This paper investigates the possible visualization using transparent displays, which could be placed on the excavator’s windshield. This way, the information could be presented closer to operators’ line of sight, without fully obstructing their view. Therefore, excavator operators could acquire the supportive information provided by the machine without diverting their attention from operational areas. To ensure that there is a match between the supportive information and operators’ contextual needs, we conducted four different activities as parts of our design process. Firstly, we looked at four relevant safety guidelines to determine which information is essential to perform safe operations. Secondly, we reviewed all commercially available technologies to discover their suitability in the excavator context. Thirdly, we conducted a design workshop to generate ideas on how the essential information should look like and behave based on the performed operation and the chosen available technology. Fourthly, we interviewed seven excavator operators to test their understanding and obtain their feedback on the proposed visualization concepts. The results indicated that four out of six visualization concepts that we proposed could be understood easily by the operators and we also revised them to better suit the operators’ way of thinking. All the operators also positively perceived this approach, since all of them included at least three visualization concepts to be presented on the windshield. Full article

Intelligent virtual agents (VAs) already support us in a variety of everyday tasks such as setting up appointments, monitoring our fitness, and organizing messages. Adding a humanoid body representation to these mostly voice-based VAs has enormous potential to enrich the human–agent communication process but, at the same time, raises expectations regarding the agent’s social, spatial, and intelligent behavior. Embodied VAs may be perceived as less human-like if they, for example, do not return eye contact, or do not show a plausible collision behavior with the physical surroundings. In this article, we introduce a new model that extends human-to-human interaction to interaction with intelligent agents and covers different multi-modal and multi-sensory channels that are required to create believable embodied VAs. Theoretical considerations of the different aspects of human–agent interaction are complemented by implementation guidelines to support the practical development of such agents. In this context, we particularly emphasize one aspect that is distinctive of embodied agents, i.e., interaction with the physical world. Since previous studies indicated negative effects of implausible physical behavior of VAs, we were interested in the initial responses of users when interacting with a VA with virtual–physical capabilities for the first time. We conducted a pilot study to collect subjective feedback regarding two forms of virtual–physical interactions. Both were designed and implemented in preparation of the user study, and represent two different approaches to virtual–physical manipulations: (i) displacement of a robotic object, and (ii) writing on a physical sheet of paper with thermochromic ink. The qualitative results of the study indicate positive effects of agents with virtual–physical capabilities in terms of their perceived realism as well as evoked emotional responses of the users. We conclude with an outlook on possible future developments of different aspects of human–agent interaction in general and the physical simulation in particular. Full article

People with physical disabilities in their upper extremities face serious issues in using classical input devices due to lacking movement possibilities and precision. This article suggests an alternative input concept and presents corresponding input devices. The proposed interface combines an inertial measurement unit and force sensing resistors, which can replace mouse and keyboard. Head motions are mapped to mouse pointer positions, while mouse button actions are triggered by contracting mastication muscles. The contact pressures of each fingertip are acquired to replace the conventional keyboard. To allow for complex text entry, the sensory concept is complemented by an ambiguous keyboard layout with ten keys. The related word prediction function provides disambiguation at word level. Haptic feedback is provided to users corresponding to their virtual keystrokes for enhanced closed-loop interactions. This alternative input system enables text input as well as the emulation of a two-button mouse. Full article

The present study introduces a new MATLAB toolbox, called MatMouse, suitable for the performance of experimental studies based on mouse movements tracking and analysis. MatMouse supports the implementation of task-based visual search experiments. The proposed toolbox provides specific functions which can be utilized for the experimental building and mouse tracking processes, the analysis of the recorded data in specific metrics, the production of related visualizations, as well as for the generation of statistical grayscale heatmaps which could serve as an objective ground truth product. MatMouse can be executed as a standalone package or integrated in existing MATLAB scripts and/or toolboxes. In order to highlight the functionalities of the introduced toolbox, a complete case study example is presented. MatMouse is freely distributed to the scientific community under the third version of GNU General Public License (GPL v3) on GitHub platform. Full article

From looking up at a skyscraper to the Grand Canyon’s vastness, you may have experienced awe in one way or another. Awe is experienced when one encounters something greater or more powerful than themselves and is associated with prosocial behavior through a diminishment of self-importance. In design research, most studies on awe have been conducted in lab conditions by using technologies such as virtual reality because of its efficiency to simulate typical awe-stimulating conditions (e.g., nature scenes). While useful in inducing awe and assessing its effects on users, they give little guidance about how design can deliberately evoke awe. Most attempts focus on the response of awe instead of its eliciting conditions. With an aim to support designers to facilitate awe, this paper explores design strategies to evoke awe. Based on appraisal theory, the cause of awe was formulated, and its relevance to designing for awe was investigated. The conditions that underlie awe in design were explored through a survey in which participants reported 150 awe experiences, resulting in six design strategies. The paper describes these strategies and discusses how they can be used in a design process, giving attention to addressing the experiential value of awe. Full article

The way we interact with computers has significantly changed over recent decades. However, interaction with computers still falls behind human to human interaction in terms of seamlessness, effortlessness, and satisfaction. We argue that simultaneously using verbal, nonverbal, explicit, implicit, intentional, and unintentional communication channels addresses these three aspects of the interaction process. To better understand what has been done in the field of Human Computer Interaction (HCI) in terms of incorporating the type channels mentioned above, we reviewed the literature on implicit nonverbal interaction with a specific emphasis on the interaction between humans on the one side, and robot and virtual humans on the other side. These Artificial Social Agents (ASA) are increasingly used as advanced tools for solving not only physical but also social tasks. In the literature review, we identify domains of interaction between humans and artificial social agents that have shown exponential growth over the years. The review highlights the value of incorporating implicit interaction capabilities in Human Agent Interaction (HAI) which we believe will lead to satisfying human and artificial social agent team performance. We conclude the article by presenting a case study of a system that harnesses subtle nonverbal, implicit interaction to increase the state of relaxation in users. This “Virtual Human Breathing Relaxation System” works on the principle of physiological synchronisation between a human and a virtual, computer-generated human. The active entrainment concept behind the relaxation system is generic and can be applied to other human agent interaction domains of implicit physiology-based interaction. Full article

It is becoming common for several devices to be utilised together to access and manipulate shared information spaces and migrate tasks between devices. Despite the increased worldwide use of cross-platform services, there is limited research into how cross-platform service usability can be assessed. This paper presents a novel cross-platform usability model. The model employs the think-aloud protocol, observations, and questionnaires to reveal cross-platform usability problems. Two Likert scales were developed for measuring overall user satisfaction of cross-platform usability and user satisfaction with the seamlessness of the transition between one device and another. The paper further employs a series of objective measures for the proposed model. The viability and performance of the model were examined in the context of evaluating three cross-platform services across three devices. The results demonstrate that the model is a valuable method for assessing and quantifying cross-platform usability. The findings were thoroughly analysed and discussed, and subsequently used to refine the model. The model was also evaluated by eight user experience experts and seven out of the eight agreed that it is useful. Full article

Although most readers associate the term virtual reality (VR) with visually appealing entertainment content, this technology also promises to be helpful to disadvantaged people like blind or visually impaired people. While overcoming physical objects’ and spaces’ limitations, virtual objects and environments that can be spatially explored have a particular benefit. To give readers a complete, clear and concise overview of current and past publications on touchable and walkable audio supplemented VR applications for blind and visually impaired users, this survey paper presents a high-level taxonomy to cluster the work done up to now from the perspective of technology, interaction and application. In this respect, we introduced a classification into small-, medium- and large-scale virtual environments to cluster and characterize related work. Our comprehensive table shows that especially grounded force feedback devices for haptic feedback (‘small scale’) were strongly researched in different applications scenarios and mainly from an exocentric perspective, but there are also increasingly physically (‘medium scale’) or avatar-walkable (‘large scale’) egocentric audio-haptic virtual environments. In this respect, novel and widespread interfaces such as smartphones or nowadays consumer grade VR components represent a promising potential for further improvements. Our survey paper provides a database on related work to foster the creation process of new ideas and approaches for both technical and methodological aspects. Full article