Multimodal Technologies and Interaction — Open Access Journal

Portuguese schools have high student failure and early school leaving rates (Pordata, 2017) giving rise to a number of initiatives aimed at their reduction. The “Alternative Curricular Course” (ACC) promotes the learning of basic skills, specifically in Portuguese language and Mathematics, to support logical reasoning and artistic, vocational, and professional development. Its main goal is the fulfilment of compulsory schooling and the reduction of academic failure. Research based on attachment theory (Bowlby, 1969) suggests that different internal working models of attachment are associated with different characteristics of social, academic, emotional, and behavioural competencies that may interfere in the quality of relationships that young people establish in school, especially with teachers, and also influence their academic performance. This study evaluates the relationship between internal working models of students, their perceptions of the quality of their relationships with teachers, and their academic performance using three measures: (i) the “Inventory of Attachment in Childhood and Adolescence” (IACA) measure, (ii) the “Inventory of Parent and Peer Attachment” (IPPA) measure—concerning the attachment to teacher”, and (iii) a socio-demographic questionnaire on a sample of 305 students from the 8th grade of regular education (RE) and the ACC. The results reveal that students on the ACC exhibit a less secure internal working model than students in RE, and that the perception of the quality of the student-teacher relationship, regarding the dimension of acceptance and understanding by the teachers, is associated with a better academic performance. These results align with those of other recent studies in support of the conclusion that the process of attachment has a significant influence on educational contexts, consistent with attachment and related theories. Full article

Due to rapid technological development, virtual reality (VR) is becoming an accessible and important tool for many applications in science, industry, and economy. Being immersed in a 3D environment offers numerous advantages especially for the presentation of geographical data that is usually depicted in 2D maps or pseudo 3D models on the monitor screen. This study investigated advantages, limitations, and possible applications for immersive and intuitive 3D terrain visualizations in VR. Additionally, in view of ever-increasing data volumes, this study developed a workflow to present large scale terrain datasets in VR for current mid-end computers. The developed immersive VR application depicts the Arctic fjord Clyde Inlet in its 160 km × 80 km dimensions at 5 m spatial resolution. Techniques, such as level of detail algorithms, tiling, and level streaming, were applied to run the more than one gigabyte large dataset at an acceptable frame rate. The immersive VR application offered the possibility to explore the terrain with or without water surface by various modes of locomotion. Terrain textures could also be altered and measurements conducted to receive necessary information for further terrain analysis. The potential of VR was assessed in a user survey of persons from six different professions. Full article

Social robots are emerging as an important intervention for a variety of vulnerable populations. However, engaging participants in the design of social robots in a way that is ethical, meaningful, and rigorous can be challenging. Many current methods in human–robotic interaction rely on laboratory practices, often experimental, and many times involving deception which could erode trust in vulnerable populations. Therefore, in this paper, we share our human-centered design methodology informed by a participatory approach, drawing on three years of data from a project aimed to design and develop a social robot to improve the mental health of teens. We present three method cases from the project that describe creative and age appropriate methods to gather contextually valid data from a teen population. Specific techniques include design research, scenario and script writing, prototyping, and teens as operators and collaborative actors. In each case, we describe the method and its implementation and discuss the potential strengths and limitations. We conclude by situating these methods by presenting a set of recommended participatory research principles that may be appropriate for designing new technologies with vulnerable populations. Full article

This paper discusses the enactment and evaluation of Embodied Conversational Agents (ECA) capable of affective communication in Collaborative Virtual Environments (CVE) for learning. The CVE discussed is a reconstruction of the European Parliament in Brussels developed using the REVERIE (Real and Virtual Engagement In Realistic Immersive Environment) framework. REVERIE is a framework designed to support the creation of CVEs populated by ECAs capable of natural human-like behaviour, physical interaction and engagement. The ECA provides a tour of the virtual parliament and participates in the learning activity as an intervention mechanism to engage students. The ECA is capable of immediacy behaviour (verbal and non-verbal) and interactions to support a dialogic learning scenario. The design of the ECA is grounded on a theoretical framework that addresses the required characteristics of the ECA to successfully support collaborative learning. In this paper, we discuss the Heuristic Evaluation of the REVERIE ECA which revealed a wealth of usability problems that led to the development of a list of design recommendations to improve their usability, including its immediacy behaviours and interactions. An ECA capable of effectively creating rapport should result in more positive experiences for participants and better learning results for students in dialogic learning scenarios. Future work aims to evaluate this hypothesis in real-world scenarios with teachers and students participating in a shared virtual educational experience. Full article

Multiple sclerosis (MS) is a disease that affects the central nervous system, which consists of the brain and spinal cord. Although this condition cannot be cured, proper treatment of persons with MS (PwMS) can help control and manage the relapses of several symptoms. In this survey article, we focus on the different technologies used for the assessment and rehabilitation of motor impairments for PwMS. We discuss sensor-based and robot-based solutions for monitoring, assessment and rehabilitation. Among MS symptoms, fatigue is one of the most disabling features, since PwMS may need to put significantly more intense effort toward achieving simple everyday tasks. While fatigue is a common symptom across several neurological chronic diseases, it remains poorly understood for various reasons, including subjectivity and variability among individuals. To this end, we also investigate recent methods for fatigue detection and monitoring. The result of this survey will provide both clinicians and researchers with valuable information on assessment and rehabilitation technologies for PwMS, as well as providing insights regarding fatigue and its effect on performance in daily activities for PwMS. Full article

Driver fatigue and inattention accounts for up to 20% of all traffic accidents, therefore any system that can warn the driver whenever fatigue occurs proves to be useful. Several systems have been devised to detect driver fatigue symptoms, such as measuring physiological parameters, which can be uncomfortable, or using a video or infrared camera pointed at the driver’s face, which in some cases, may cause privacy concerns for the driver. Usually these systems are expensive, therefore a brief discussion on low-cost fatigue detection systems is presented, followed by a proposal for a non-intrusive low-cost prototype, that aims to detect driver fatigue symptoms. The prototype consists of several sensors that monitor driver physical parameters and vehicle behaviour, with a total system price close to 30 euros. The prototype is discussed and compared with similar systems, pointing out its strengths and weaknesses. Full article

Quality care for patients requires effective communication amongst medical teams. Increasingly, communication is required not only between team members themselves, but between members and the medical devices monitoring and managing patient well-being. Most human–computer interfaces use either auditory or visual displays, and despite significant experimentation, they still elicit well-documented concerns. Curiously, few interfaces explore the benefits of multimodal communication, despite extensive documentation of the brain’s sensitivity to multimodal signals. New approaches built on insights from basic audiovisual integration research hold the potential to improve future human–computer interfaces. In particular, recent discoveries regarding the acoustic property of amplitude envelope illustrate that it can enhance audiovisual integration while also lowering annoyance. Here, we share key insights from recent research with the potential to inform applications related to human–computer interface design. Ultimately, this could lead to a cost-effective way to improve communication in medical contexts—with signification implications for both human health and the burgeoning medical device industry. Full article

The intensive care unit (ICU) of a hospital is an environment subjected to ceaseless noise. Patient alarms contribute to the saturated auditory environment and often overwhelm healthcare providers with constant and false alarms. This may lead to alarm fatigue and prevent optimum patient care. In response, a multisensory alarm system developed with consideration for human neuroscience and basic music theory is proposed as a potential solution. The integration of auditory, visual, and other sensory output within an alarm system can be used to convey more meaningful clinical information about patient vital signs in the ICU and operating room to ultimately improve patient outcomes. Full article

An increasing number of museum exhibits incorporate multi-modal technologies and interactions; yet these media divert visitors’ attention away from the cultural heritage artifacts on display. This paper proposes an overarching conceptual structure for designing tangible and embodied narrative interaction with cultural heritage artifacts within a museum exhibit so that visitors can interact with them to comprehend their cultural context. The Tangible and Embodied Narrative Framework (TENF) consists of three spectra (diegetic vs. non-diegetic, internal vs. external, and ontological vs. exploratory) and, considering how different interactions map along these three spectra, can guide designers in the way they integrate digital media, narrative, and embodiment. In this paper, we examine interactive narrative scholarship, existing frameworks for tangible and embodied interactions, and tangible and embodied narrative projects. We then describe the design of the TENF and its application to the pilot project, Mapping Place, and to the case study project, Multi-Sensory Prayer Nuts. The findings indicate that embodied engagement with artifacts through a narrative role can help visitors (1) contextualize the meaning of artifacts and (2) make personalized connections to the artifacts. Based on this work, we suggest design recommendations for tailoring the use of the TENF in the cultural heritage domain: simulate cultural practices, associate visitors with cultural perspectives, and provide simultaneous digital feedback. We conclude by describing future directions for the research, which include generating other possible projects using the TENF; collaborating with other designers and museum professionals; and exploring applications of the TENF in museum spaces. Full article

Social robots are expected gradually to be used by more and more people in a wider range of settings, domestic as well as professional. As a consequence, the features and quality requirements on human–robot interaction will increase, comprising possibilities to communicate emotions, establishing a positive user experience, e.g., using touch. In this paper, the focus is on depicting how humans, as the users of robots, experience tactile emotional communication with the Nao Robot, as well as identifying aspects affecting the experience and touch behavior. A qualitative investigation was conducted as part of a larger experiment. The major findings consist of 15 different aspects that vary along one or more dimensions and how those influence the four dimensions of user experience that are present in the study, as well as the different parts of touch behavior of conveying emotions. Full article

Semantic fusion is a central requirement of many multimodal interfaces. Procedural methods like finite-state transducers and augmented transition networks have proven to be beneficial to implement semantic fusion. They are compliant with rapid development cycles that are common for the development of user interfaces, in contrast to machine-learning approaches that require time-costly training and optimization. We identify seven fundamental requirements for the implementation of semantic fusion: Action derivation, continuous feedback, context-sensitivity, temporal relation support, access to the interaction context, as well as the support of chronologically unsorted and probabilistic input. A subsequent analysis reveals, however, that there is currently no solution for fulfilling the latter two requirements. As the main contribution of this article, we thus present the Concurrent Cursor concept to compensate these shortcomings. In addition, we showcase a reference implementation, the Concurrent Augmented Transition Network (cATN), that validates the concept’s feasibility in a series of proof of concept demonstrations as well as through a comparative benchmark. The cATN fulfills all identified requirements and fills the lack amongst previous solutions. It supports the rapid prototyping of multimodal interfaces by means of five concrete traits: Its declarative nature, the recursiveness of the underlying transition network, the network abstraction constructs of its description language, the utilized semantic queries, and an abstraction layer for lexical information. Our reference implementation was and is used in various student projects, theses, as well as master-level courses. It is openly available and showcases that non-experts can effectively implement multimodal interfaces, even for non-trivial applications in mixed and virtual reality. Full article

In some circumstances, immersion in virtual environments with the aid of virtual reality (VR) equipment can create feelings of anxiety in users and be experienced as something “frightening”, “oppressive”, “alienating”, “dehumanizing”, or “dystopian”. Sometimes (e.g., in exposure therapy or VR gaming), a virtual environment is intended to have such psychological impacts on users; however, such effects can also arise unintentionally due to the environment’s poor architectural design. Designers of virtual environments may employ user-centered design (UCD) to incrementally improve a design and generate a user experience more closely resembling the type desired; however, UCD can yield suboptimal results if an initial design relied on an inappropriate architectural approach. This study developed a framework that can facilitate the purposeful selection of the most appropriate architectural approach by drawing on Norberg-Schulz’s established phenomenological account of real-world architectural modes. By considering the unique possibilities for structuring and experiencing space within virtual environments and reinterpreting Norberg-Schulz’s schemas in the context of virtual environment design, a novel framework was formulated that explicates six fundamental “architectural paradigms” available to designers of virtual environments. It was shown that the application of this framework could easily be incorporated as an additional step within the UCD process. Full article

An understanding of the need for user-centred and participatory design continues to gain universal momentum both in academia and industry. It is essential this momentum is maintained as the population changes and technology develops. The contribution of this work draws on research from different disciplines to provide the design community with new knowledge and an awareness of the diversity of user needs, particularly the needs and skills of older people. A collection of usability and accessibility guidelines are referenced in terms of their applicability toward designing interfaces and interaction for an ageing population, in conjunction with results from studies that highlight the extent to which familiarity and successful interaction with contemporary products decreases according to age and prior experience, and identifies the problems users experience during interaction with technology. The hope is that more widespread awareness of this knowledge will encourage greater understanding and assist in the development of better design methods and better on- and offline products and tools for those of any age, but particularly those within an increasingly ageing demographic. Full article

The concept of affordance perception is one of the distinctive traits of human cognition; and its application to robots can dramatically improve the quality of human-robot interaction (HRI). In this paper we explore and discuss the idea of “emotional affordances” by proposing a viable model for implementation into HRI; which considers allocentric and multimodal perception. We consider “2-ways” affordances: perceived object triggering an emotion; and perceived human emotion expression triggering an action. In order to make the implementation generic; the proposed model includes a library that can be customised depending on the specific robot and application scenario. We present the AAA (Affordance-Appraisal-Arousal) model; which incorporates Plutchik’s Wheel of Emotions; and we outline some numerical examples of how it can be used in different scenarios. Full article

This article discusses the use of game design as a method for interdisciplinary project-based teaching in secondary school education to convey informatics and society topics, which encompass the larger social context of computing. There is a lot of knowledge about learning games but little background on using game design as a method for project-based teaching of social issues in informatics. We present the results of an analysis of student-created games and an evaluation of a student-authored database on learning contents found in commercial off-the-shelf games. We further contextualise these findings using a group discussion with teachers. The results underline the effectiveness of project-based teaching to raise awareness for informatics and society topics. We further outline informatics and society topics that are particularly interesting to students, genre preferences, and potentially engaging game mechanics stemming from our analyses. Full article

In recent years, tangible user interfaces (TUI) have gained in popularity in educational contexts, among others to implement problem-solving and discovery learning science activities. In the context of an interdisciplinary and cross-institutional collaboration, we conducted a multimodal EMCA-based video user study involving a TUI-mediated bicycle mechanics simulation. This article focusses on the discovering work of a group of three students with regard to a particular tangible object (a red button), designed to support participants engagement with the underlying physics aspects and its consequences with regard to their engagement with the targeted mechanics aspects. Full article

In this paper, we present the results of an online survey (N = 170) on emotional situations on the road. In particular, we asked potential early adopters to remember a situation where they felt either an intense positive or negative emotion while driving. Our research is motivated by imminent disruptions in the automotive sector due to automated driving and the accompanying switch to selling driving experiences over horsepower. This creates a need to focus on the driver’s emotion when designing in-car interfaces. As a result of our research, we present a set of propositions for affective car interfaces based on real-life experiences. With our work we aim to support the design of affective car interfaces and give designers a foundation to build upon. We find respondents often connect positive emotions with enjoying their independence, while negative experiences are associated mostly with traffic behavior. Participants who experienced negative situations wished for better information management and a higher degree of automation. Drivers with positive emotions generally wanted to experience the situation more genuinely, for example, by switching to a “back-to-basic” mode. We explore these statements and discuss recommendations for the design of affective interfaces in future cars. Full article

This study reports on the empirical findings of participatory design workshops for the development of a supportive automotive user experience design system. Identifying and addressing this area with traditional research methods is problematic due to the different user experience (UX) design perspectives that might conflict and the related limitations of the automotive domain. To help resolve this problem, we conducted research with 12 user experience (UX) designers through individual participatory prototyping activities to gain insights into their explicit, observable, tacit and latent needs. These activities allowed us to explore their motivation to use different technologies; the system’s architecture; detailed features of interactivity; and to describe user needs including efficiency, effectiveness, engagement, naturalness, ease of use, information retrieval, self-image awareness, politeness, and flexibility. Our analysis led us to design implications that translate participants’ needs into UX design goals, informing practitioners on how to develop relevant systems further. Full article