Multimodal Technologies and Interaction — Open Access Journal

Exploring geographic maps on touchscreens is a difficult task in the absence of vision as those devices miss tactile cues. Prior research has therefore introduced non-visual interaction techniques designed to allow visually impaired people to explore spatial configurations on tactile devices. In this paper, we present a study in which six blind and six blindfolded sighted participants evaluated three of those interaction techniques compared to a screen reader condition. We observed that techniques providing guidance result in a higher user satisfaction and more efficient exploration. Adding a grid-like structure improved the estimation of distances. None of the interaction techniques improved the reconstruction of the spatial configurations. The results of this study allow improving the design of non-visual interaction techniques that support a better exploration and memorization of maps in the absence of vision. Full article

The loss of peripheral vision is experienced by millions of people with glaucoma or retinitis pigmentosa, and has a major impact in everyday life, specifically to locate visual targets in the environment. In this study, we designed a wearable interface to render the location of specific targets with private and non-intrusive tactile cues. Three experimental studies were completed to design and evaluate the tactile code and the device. In the first study, four different tactile codes (single stimuli or trains of pulses rendered either in a Cartesian or a Polar coordinate system) were evaluated with a head pointing task. In the following studies, the most efficient code, trains of pulses with Cartesian coordinates, was used on a bracelet located on the wrist, and evaluated during a visual search task in a complex virtual environment. The second study included ten subjects with a simulated restrictive field of view (10°). The last study consisted of proof of a concept with one visually impaired subject with restricted peripheral vision due to glaucoma. The results show that the device significantly improved the visual search efficiency with a factor of three. Including object recognition algorithm to smart glass, the device could help to detect targets of interest either on demand or suggested by the device itself (e.g., potential obstacles), facilitating visual search, and more generally spatial awareness of the environment. Full article

The number of scientific publications combining robotic user interfaces and mixed reality highly increased during the 21st Century. Counting the number of yearly added publications containing the keywords “mixed reality” and “robot” listed on Google Scholar indicates exponential growth. The interdisciplinary nature of mixed reality robotic user interfaces (MRRUI) makes them very interesting and powerful, but also very challenging to design and analyze. Many single aspects have already been successfully provided with theoretical structure, but to the best of our knowledge, there is no contribution combining everything into an MRRUI taxonomy. In this article, we present the results of an extensive investigation of relevant aspects from prominent classifications and taxonomies in the scientific literature. During a card sorting experiment with professionals from the field of human–computer interaction, these aspects were clustered into named groups for providing a new structure. Further categorization of these groups into four different categories was obvious and revealed a memorable structure. Thus, this article provides a framework of objective, technical factors, which finds its application in a precise description of MRRUIs. An example shows the effective use of the proposed framework for precise system description, therefore contributing to a better understanding, design, and comparison of MRRUIs in this growing field of research. Full article

The hip joint is highly prone to traumatic and degenerative pathologies resulting in irregular locomotion. Monitoring and treatment depend on high-end technology facilities requiring physician and patient co-location, thus limiting access to specialist monitoring and treatment for populations living in rural and remote locations. Telemedicine offers an alternative means of monitoring, negating the need for patient physical presence. In addition, emerging technologies, such as virtual reality (VR) and immersive technologies, offer potential future solutions through virtual presence, where the patient and health professional can meet in a virtual environment (a virtual clinic). To this end, a prototype asynchronous telemedicine VR gait analysis system was designed, aiming to transfer a full clinical facility within the patients’ local proximity. The proposed system employs cost-effective alternative motion capture combined with the system’s immersive 3D virtual gait analysis clinic. The user interface and the tools in the application offer health professionals asynchronous, objective, and subjective analyses. This paper investigates the requirements for the design of such a system and discusses preliminary comparative data of its performance evaluation against a high-fidelity gait analysis clinical application. Full article

In this article, we compare a Google Tango tablet with the Microsoft HoloLens smartglasses in the context of the visualisation and interaction with Building Information Modeling data. A user test was conducted where 16 participants solved four tasks, two for each device, in small teams of two. Two aspects are analysed in the user test: the visualisation of interior designs and the visualisation of Building Information Modeling data. The results show that the Tango tablet is surprisingly preferred by most users when it comes to collaboration and discussion in our scenario. While the HoloLens offers hands-free operation and a stable tracking, users mentioned that the interaction with the Tango tablet felt more natural. In addition, users reported that it was easier to get an overall impression with the Tango tablet rather than with the HoloLens smartglasses. Full article

In recent years, pedestrian navigation assistance has been used by an increasing number of people to support wayfinding tasks. Especially in unfamiliar and complex indoor environments such as universities and hospitals, the importance of an effective navigation assistance becomes apparent. This paper investigates the feasibility of the indoor landmark navigation model (ILNM), a method for generating landmark-based routing instructions, by combining it with indoor route maps and conducting a wayfinding experiment with human participants. Within this context, three different cartographic visualization scenarios were designed and evaluated. Two of these scenarios were based on the implementation of the ILNM algorithm, with the concurrent effort to overcome the challenge of representing the semantic navigation instructions in two different ways. In the first scenario, the selected landmarks were visualized as pictograms, while in the second scenario, an axonometric-based design philosophy for the depiction of landmarks was followed. The third scenario was based on the benchmark approach (metric-based routing instructions) for conveying routing instructions to the users. The experiment showed that the implementation of the ILNM was feasible, and, more importantly, it was beneficial in terms of participants’ navigation performance during the wayfinding experiment, compared to the metric-based instructions scenario (benchmark for indoor navigation). Valuable results were also obtained, concerning the most suitable cartographic approach for visualizing the selected landmarks, while implementing this specific algorithm (ILNM). Finally, our findings confirm that the existence of landmarks, not only within the routing instructions, but also as cartographic representations on the route map itself, can significantly help users to position themselves correctly within an unfamiliar environment and to improve their navigation performance. Full article

Drivers in negative emotional states, such as anger or sadness, are prone to perform bad at driving, decreasing overall road safety for all road users. Recent advances in affective computing, however, allow for the detection of such states and give us tools to tackle the connected problems within automotive user interfaces. We see potential in building a system which reacts upon possibly dangerous driver states and influences the driver in order to drive more safely. We compare different interaction approaches for an affective automotive interface, namely Ambient Light, Visual Notification, a Voice Assistant, and an Empathic Assistant. Results of a simulator study with 60 participants (30 each with induced sadness/anger) indicate that an emotional voice assistant with the ability to empathize with the user is the most promising approach as it improves negative states best and is rated most positively. Qualitative data also shows that users prefer an empathic assistant but also resent potential paternalism. This leads us to suggest that digital assistants are a valuable platform to improve driver emotions in automotive environments and thereby enable safer driving. Full article

The present research explores how autonomous vehicle voice agent (AVVA) design influences autonomous vehicle passenger (AVP) intentions to adopt autonomous vehicles. An online experiment (N = 158) examined the role of gender stereotypes in response to an AVVA with respect to the technology acceptance model. The findings indicate that characteristics of the AVVA that are more consistent with the stereotypical expectation of the social role (informative male AVVA and social female AVVA) foster greater perceived ease of use (PEU) and perceived usefulness (PU) than inconsistent conditions (social male AVVA and informative female AVVA). The study offers theoretical implications regarding the technology acceptance model in the context of autonomous technologies as well as practical implications for the design of autonomous vehicle voice agents. Full article

In Cinematic Virtual Reality (CVR), the viewer of an omnidirectional movie can freely choose the viewing direction when watching a movie. Therefore, traditional techniques in filmmaking for guiding the viewers’ attention cannot be adapted directly to CVR. Practices such as panning or changing the frame are no longer defined by the filmmaker; rather it is the viewer who decides where to look. In some stories, it is necessary to show certain details to the viewer, which should not be missed. At the same time, the freedom of the viewer to look around in the scene should not be destroyed. Therefore, techniques are needed which guide the attention of the spectator to visual information in the scene. Attention guiding also has the potential to improve the general viewing experience, since viewers will be less afraid to miss something when watching an omnidirectional movie where attention-guiding techniques have been applied. In recent years, there has been a lot of research about attention guiding in images, movies, virtual reality, augmented reality and also in CVR. We classify these methods and offer a taxonomy for attention-guiding methods. Discussing the different characteristics, we elaborate the advantages and disadvantages, give recommendations for use cases and apply the taxonomy to several examples of guiding methods. Full article

Excessive numbers of clinical alarms in the intensive care unit (ICU) contribute to alarm fatigue. Efforts to eliminate unnecessary alarms, including during end of life (EOL) care, are pivotal. This study describes electrocardiographic (ECG) arrhythmia alarm usage following the decision for comfort care. We conducted a review of electronic health records (EHR) in patients who died and had comfort care orders that were in place during our study. The occurrences of ECG arrhythmia alarms among these patients were examined. We found 151 arrhythmia alarms that were generated in 11 patients after comfort care was initiated: 72% were audible, 21% were manually muted, and 7% had an unknown audio label. Level of alarm: 33% crisis, 58% warning, 1% message, and 8% were labeled unknown. Our report shows that ECG monitoring was commonly maintained during the EOL care. Since the goal of care during this phase is for both patient and family comfort, it is important for the clinicians to weigh the benefits versus harms of the continuous ECG monitoring. Full article

Preserving older pedestrians’ navigation skills in urban environments is a challenge for maintaining their quality of life. However, the maps that are usually used by older pedestrians might be unsuitable to their specificities and the existing digital aids do not consider older people’s perceptual and cognitive declines or user experience. This study presents a rich description of the navigation experience of older pedestrians either with a visual (augmented reality glasses), auditory (bone conduction headphones), or a visual and haptic (smartwatch) wearable device adapted to age-related declines. These wearable devices are compared to the navigation aid older people usually use when navigating the city (their own digital or paper map). The study, with 18 participants, measured the navigation performance and captured detailed descriptions of the users’ experience using interviews. We highlight three main phenomena which impact the quality of the user experience with the four aids: (1) the shifts in attention over time, (2) the understanding of the situation over time, and (3) the emergence of affective and aesthetic feelings over time. These findings add a new understanding of the specificities of navigation experience by older people and are discussed in terms of design recommendations for navigation devices. Full article

This paper presents an alternative approach for the diagnosis of learning difficulties in children. A game-based evaluation study, using Kinaesthetic Learning Difficulties Diagnosis (Kin-LDD), was performed during the actual diagnosis procedure for the identification of learning difficulties. Kin-LDD is a serious game that provides a gesture-based interface and incorporates spatial and time orientation activities. These activities assess children’s cognitive attributes while they are using their motor skills to interact with the game. The aim of this work was to introduce the fun parameter to the diagnostic process, provide a useful tool for the special educators and investigate potential correlations between in-game metrics and the diagnosis outcome. An experiment was conducted in which 30 children played the game during their official assessment for the diagnosis of learning difficulties at the Center for Differential Diagnosis, Diagnosis and Support. Performance metrics were collected automatically while the children were playing the game. These metrics, along with questionnaires appropriate for children and post-session interviews were later analyzed and the findings are presented in the paper. According to the results: (a) children evaluated the game as a fun experience, (b) special educators claimed it was helpful to the diagnostic procedure, and (c) there were statistically significant correlations between in-game metrics and the category of learning difficulty the child was characterized with. Full article

Technology plays a prominent role in intensive care units (ICU), with a variety of sensors monitoring both patients and devices. A serious problem exists, however, that can reduce the sensors’ effectiveness. When important values exceed or fall below a certain threshold or sensors lose their signal, up to 350 alarms per patient a day are issued. These frequent alarms are audible in several locations on the ICU, resulting in a massive cognitive load for ICU nurses, as they must evaluate and acknowledge each alarm. “Alarm fatigue” sets in, a desensitization and delayed response time for alarms that can have severe consequences for patients and nurses. To counteract the acoustic load on ICUs, we designed and evaluated personal multimodal alarms for a wearable alarm system (WAS). The result was a lower response time and higher ratings on suitability and feasibility, as well as a lower annoyance level, compared to acoustic alarms. We find that multimodal alarms are a promising new approach to alert ICU nurses, reduce cognitive load, and avoid alarm fatigue. Full article

Workflow in a neonatal intensive care unit (NICU) is relatively unpredictable, which makes it difficult to plan activities. Simple tasks, such as checking device statuses may be forgotten, resulting in disturbing alarms. In this paper, we will present CheckMates, ambient lighting displays, which visualize device statuses to provide nurses with more overview. We performed expert reviews to obtain insights into the different potentials of CheckMates. Additionally, we performed a simulation study to gather user experiences regarding the functioning of CheckMates and their capacity to improve planning in an NICU environment. The results showed a variety of potential benefits for increasing nurses’ overview of device statuses and their opportunities for workflow planning. Furthermore, CheckMates did not appear to be distracting. Full article

The interpretation and understanding of physical gestures play a significant role in various forms of art. Interactive technology and digital devices offer a plethora of opportunities for personal gesture-based experience and they assist in the creation of collaborative artwork. In this study, three prototypes for use with different digital devices (digital camera, PC camera, and Kinect) were designed. Subsequently, a series of workshops were conducted and in-depth interviews with participants from different cultural and occupational backgrounds. The latter were designed to explore how to specifically design personalised gesture-based expressions and how to engage the creativity of the participants in their gesture-based experiences. The findings indicated that, in terms of gesture-based interaction, the participants preferred to engage with the visual traces that were displayed at specific timings in multi-experience spaces. Their gesture-based interactions could effectively support non-verbal emotional expression. In addition, the participants were shown to be strongly inclined to combine their personal stories and emotions into their own gesture-based artworks. Based on the participants’ different cultural and occupational backgrounds, their artistic creation could be spontaneously formed. Full article

Navigation is done through obtaining spatial information from the environment and forming a spatial map about it. The visually impaired rely mainly on orientation and mobility training by a certified specialist to acquire spatial navigation skills. However, it is manpower intensive and costly. This research designed and developed a serious game, Hungry Cat. This game can convey spatial information of virtual rooms to children with visual impairment through game playing. An evaluation with 30 visually impaired participants was conducted by allowing them to explore each virtual room in Hungry Cat. After exploration, the food finding test, which is a game mode available in Hungry Cat, was conducted, followed by the physical wire net test to evaluate their ability in forming the spatial mental maps of the virtual rooms. The positive results of the evaluation obtained demonstrate the ability of Hungry Cat, in conveying spatial information about virtual rooms and aiding the development of spatial mental maps of these rooms through game playing. Full article

Physiologic monitors generate alarms to alert clinicians to signs of instability. However, these monitors also create alarm fatigue that places patients at risk. Redundant alarms have contributed to alarm fatigue without improving patient safety. In this study, our specific aim was to decrease the median percentage of redundant alarms by 50% within 6 months using the Model for Improvement. Our primary outcome was to lower the percentage of redundant alarms. We used the overall alarm rate per patient per day and code blue events as balancing metrics. We completed three Plan-Do-Study-Act cycles and generated run charts using standard industry criteria to determine the special cause. Ultimately, we decreased redundant alarms from a baseline of 6.4% of all alarms to 1.8%, surpassing our aim of a 50% reduction. Our overall alarm rate, one of our balancing metrics, decreased from 137 alarms/patient day to 118 alarms/patient day during the intervention period. No code blue events were determined to be related to incorrect setting of alarms. Decreasing redundant alarms is safe and feasible. Following a reduction in redundant alarms, more intensive alarm reduction methods are needed to continue to reduce alarm fatigue while keeping patients safe. Full article

Portuguese schools have high student failure and early school leaving rates (Pordata, 2017) giving rise to a number of initiatives aimed at their reduction. The “Alternative Curricular Course” (ACC) promotes the learning of basic skills, specifically in Portuguese language and Mathematics, to support logical reasoning and artistic, vocational, and professional development. Its main goal is the fulfilment of compulsory schooling and the reduction of academic failure. Research based on attachment theory (Bowlby, 1969) suggests that different internal working models of attachment are associated with different characteristics of social, academic, emotional, and behavioural competencies that may interfere in the quality of relationships that young people establish in school, especially with teachers, and also influence their academic performance. This study evaluates the relationship between internal working models of students, their perceptions of the quality of their relationships with teachers, and their academic performance using three measures: (i) the “Inventory of Attachment in Childhood and Adolescence” (IACA) measure, (ii) the “Inventory of Parent and Peer Attachment” (IPPA) measure—concerning the attachment to teacher”, and (iii) a socio-demographic questionnaire on a sample of 305 students from the 8th grade of regular education (RE) and the ACC. The results reveal that students on the ACC exhibit a less secure internal working model than students in RE, and that the perception of the quality of the student-teacher relationship, regarding the dimension of acceptance and understanding by the teachers, is associated with a better academic performance. These results align with those of other recent studies in support of the conclusion that the process of attachment has a significant influence on educational contexts, consistent with attachment and related theories. Full article

Due to rapid technological development, virtual reality (VR) is becoming an accessible and important tool for many applications in science, industry, and economy. Being immersed in a 3D environment offers numerous advantages especially for the presentation of geographical data that is usually depicted in 2D maps or pseudo 3D models on the monitor screen. This study investigated advantages, limitations, and possible applications for immersive and intuitive 3D terrain visualizations in VR. Additionally, in view of ever-increasing data volumes, this study developed a workflow to present large scale terrain datasets in VR for current mid-end computers. The developed immersive VR application depicts the Arctic fjord Clyde Inlet in its 160 km × 80 km dimensions at 5 m spatial resolution. Techniques, such as level of detail algorithms, tiling, and level streaming, were applied to run the more than one gigabyte large dataset at an acceptable frame rate. The immersive VR application offered the possibility to explore the terrain with or without water surface by various modes of locomotion. Terrain textures could also be altered and measurements conducted to receive necessary information for further terrain analysis. The potential of VR was assessed in a user survey of persons from six different professions. Full article