Multimodal Technol. Interact., Volume 8, Issue 6 (June 2024) – 12 articles

Soft robotics is used in real-world clinical situations, including surgery, rehabilitation, and diagnosis. However, several challenges remain to make soft robots more viable, especially for clinical interventions such as improving sleep quality, which impacts physiological and mental health. This paper presents an autoethnographic account of the experience of sleeping with a companion robot (Somnox), which mimics breathing to promote better sleep. The study is motivated by the key author’s experience with insomnia and a desire to better understand how Somnox is used in different social contexts. Data were collected through diary entries for 16 weeks (8 weeks without, 8 weeks with) and analysed thematically. The findings indicate improved sleep and observations about the relationship developed with the companion robot, including emotional connection and empathy for the technology. Furthermore, Somnox is a multidimensional family companion robot that can ease stomach discomfort and stress, reduce anxiety, and provide holistic care. Full article

The recent advancements in mobile technologies have enabled the widespread adoption of augmented reality (AR) to enrich cultural heritage (CH) digital experiences. Mobile AR leverages visual recognition capabilities and sensor data to superimpose digital elements into the user’s view of their surroundings. The pervasive nature of AR serves several purposes in CH: visitor guidance, 3D reconstruction, educational experiences, and mobile location-based games. While most literature reviews on AR in CH focus on technological aspects such as tracking algorithms and software frameworks, there has been little exploration of the expressive affordances of AR for the delivery of meaningful interactions. This paper (based on the PRISMA guidelines) considers 64 selected publications, published from 2016 to 2023, that present mobile AR applications in CH, with the aim of identifying and analyzing the (mobile) AR (interaction) design patterns that have so far been discussed sporadically in the literature. We identify sixteen (16) main UX design patterns, as well as eight (8) patterns with a single occurrence in the paper corpus, that have been employed—sometimes in combination—to address recurring design problems or contexts, e.g., user navigation, representing the past, uncovering hidden elements, etc. We analyze each AR design pattern by providing a title, a working definition, principal use cases, and abstract illustrations that indicate the main concept and its workings (where applicable) and explanation with respect to examples from the paper corpus. We discuss the AR design patterns in terms of a few broader design and development concerns, including the AR recognition approach, content production and development requirements, and affordances for storytelling, as well as possible contexts and experiences, including indoor/outdoor settings, location-based experiences, mobile guides, and mobile games. We envisage that this work will thoroughly inform AR designers and developers abot the current state of the art and the possibilities and affordances of mobile AR design patterns with respect to particular CH contexts. Full article

Subtitles play a crucial role in facilitating the understanding of visual content when watching films and television programs. In this study, we propose a method for presenting subtitles in a way that considers cognitive load when viewing video content in a non-native language. Subtitles are generally displayed at the bottom of the screen, which causes frequent eye focus switching between subtitles and video, increasing the cognitive load. In our proposed method, we focused on the position, display time, and amount of information contained in the subtitles to reduce the cognitive load and to avoid disturbing the viewer’s concentration. We conducted two experiments to investigate the effects of our proposed subtitle method on gaze distribution, comprehension, and cognitive load during English-language video viewing. Twelve non-native English-speaking subjects participated in the first experiment. The results show that participants’ gazes were more focused around the center of the screen when using our proposed subtitles compared to regular subtitles. Comprehension levels recorded using LightSub were similar, but slightly inferior to those recorded using regular subtitles. However, it was confirmed that most of the participants were viewing the video with a higher cognitive load using the proposed subtitle method. In the second experiment, we investigated subtitles considering connected speech form in English with 18 non-native English speakers. The results revealed that the proposed method, considering connected speech form, demonstrated an improvement in cognitive load during video viewing but it remained higher than that of regular subtitles. Full article

The main purpose of this study is to examine the implementation of active methodologies in the teaching–learning process in computer science. To achieve this objective, a systematic review using the PRISMA method was performed; the search for articles was conducted through the Scopus and Web of Science databases and the scientific search engine Google Scholar. By establishing inclusion and exclusion criteria, 15 research papers were selected addressing the use of various active methodologies which have had a positive impact on students’ learning processes. Among the principal active methodologies highlighted are problem-based learning, flipped classrooms, and gamification. The results of the review show how active methodologies promote significant learning, in addition to fostering more outstanding commitment, participation, and motivation on the students’ part. It was observed that active methodologies contribute to the development of fundamental cognitive and socio-emotional skills for their professional growth. Full article

In the present study, a new eye-tracking dataset (OnMapGaze) and a graph-based metric (GraphGazeD) for modeling visual perception differences are introduced. The dataset includes both experimental and analyzed gaze data collected during the observation of different cartographic backgrounds used in five online map services, including Google Maps, Wikimedia, Bing Maps, ESRI, and OSM, at three different zoom levels (12z, 14z, and 16z). The computation of the new metric is based on the utilization of aggregated gaze behavior data. Our dataset aims to serve as an objective ground truth for feeding artificial intelligence (AI) algorithms and developing computational models for predicting visual behavior during map reading. Both the OnMapGaze dataset and the source code for computing the GraphGazeD metric are freely distributed to the scientific community. Full article

Driven by technological advances in various fields (AI, 5G, VR, IoT, etc.) together with the emergence of digital twins technologies (HDT, HAL, BIM, etc.), the Metaverse has attracted growing attention from scientific and industrial communities. This interest is due to its potential impact on people lives in different sectors such as education or medicine. Specific solutions can also increase inclusiveness of people with disabilities that are an impediment to a fulfilled life. However, security and privacy concerns remain the main obstacles to its development. Particularly, the data involved in the Metaverse can be comprehensive with enough granularity to build a highly detailed digital copy of the real world, including a Human Digital Twin of a person. Existing security countermeasures are largely ineffective and lack adaptability to the specific needs of Metaverse applications. Furthermore, the virtual worlds in a large-scale Metaverse can be highly varied in terms of hardware implementation, communication interfaces, and software, which poses huge interoperability difficulties. This paper aims to analyse the risks and opportunities associated with adopting digital replicas of humans (HDTs) within the Metaverse and the challenges related to managing digital identities in this context. By examining the current technological landscape, we identify several open technological challenges that currently limit the adoption of HDTs and the Metaverse. Additionally, this paper explores a range of promising technologies and methodologies to assess their suitability within the Metaverse context. Finally, two example scenarios are presented in the Medical and Education fields. Full article

This paper researches the classification of human emotions in a virtual reality (VR) context by analysing psychophysiological signals and facial expressions. Key objectives include exploring emotion categorisation models, identifying critical human signals for assessing emotions, and evaluating the accuracy of these signals in VR environments. A systematic literature review was performed through peer-reviewed articles, forming the basis for our methodologies. The integration of various emotion classifiers employs a ‘late fusion’ technique due to varying accuracies among classifiers. Notably, facial expression analysis faces challenges from VR equipment occluding crucial facial regions like the eyes, which significantly impacts emotion recognition accuracy. A weighted averaging system prioritises the psychophysiological classifier over the facial recognition classifiers due to its higher accuracy. Findings suggest that while combined techniques are promising, they struggle with mixed emotional states as well as with fear and trust emotions. The research underscores the potential and limitations of current technologies, recommending enhanced algorithms for effective interpretation of complex emotional expressions in VR. The study provides a groundwork for future advancements, aiming to refine emotion recognition systems through systematic data collection and algorithm optimisation. Full article

Police communication is a field with unique challenges and specific requirements. Police officers depend on effective communication, particularly in high-stress operations, but current training methods are not focused on communication and provide only limited evaluation methods. This work explores the potential of virtual reality (VR) for enhancing police communication training. The rise of VR training, especially in specific application areas like policing, provides benefits. We conducted a field study during police training to assess VR approaches for training communication. The results show that VR is suitable for communication training if factors such as realism, reflection and repetition are given in the VR system. Trainer feedback shows that assistive systems for evaluation and visualization of communication are highly needed. We present ideas and approaches for evaluation in communication training and concepts for visualization and exploration of the data. This research contributes to improving VR police training and has implications for communication training in VR in challenging contexts. Full article

This paper investigates the versatility of force feedback (FF) technology in enhancing user interfaces across a spectrum of applications. We delve into the human finger pad’s sensitivity to FF stimuli, which is critical to the development of intuitive and responsive controls in sectors such as medicine, where precision is paramount, and entertainment, where immersive experiences are sought. The study presents a case study in the automotive domain, where FF technology was implemented to simulate mechanical button presses, reducing the JND FF levels that were between 0.04 N and 0.054 N to the JND levels of 0.254 and 0.298 when using a linear force feedback scale and those that were 0.028 N and 0.033 N to the JND levels of 0.074 and 0.164 when using a logarithmic force scale. The results demonstrate the technology’s efficacy and potential for widespread adoption in various industries, underscoring its significance in the evolution of haptic feedback systems. Full article

Humans interact with the environment through a variety of senses. Touch in particular contributes to a sense of presence, enhancing perceptual experiences, and establishing causal relations between events. Many human–machine interfaces only allow for one-way communication, which does not do justice to the complexity of the interaction. To address this, we developed a bidirectional human–machine interface featuring a bracelet equipped with linear resonant actuators, controlled via a Robot Operating System (ROS) program, to simulate haptic feedback. Further, the wireless interface includes a motion sensor and a sensor to quantify the tightness of the bracelet. Our functional experiments, which compared stimulation with three and five intensity levels, respectively, were performed by four healthy participants in their twenties and thirties. The participants achieved an average accuracy of 88% estimating three vibration intensity levels. While the estimation accuracy for five intensity levels was only 67%, the results indicated a good performance in perceiving relative vibration changes with an accuracy of 82%. The proposed haptic feedback bracelet will facilitate research investigating the benefits of bidirectional human–machine interfaces and the perception of vibrotactile feedback in general by closing the gap for a versatile device that can provide high-density user feedback in combination with sensors for intent detection. Full article

Augmented Reality (AR) enhances learning by integrating interactive and immersive elements that bring content to life, thus increasing motivation and improving retention. AR also supports personalized learning, allowing learners to interact with content at their own pace and according to their preferred learning styles. This adaptability not only promotes self-directed learning but also empowers learners to take charge of their educational journey. Effective interface design is crucial for these AR applications, requiring careful integration of user interactions and visual cues to blend AR elements seamlessly with reality. This paper explores the impact of AR on user experience within educational settings, examining engagement, motivation, and learning outcomes to determine how AR can enhance the educational experience. Additionally, it addresses design considerations and challenges in developing AR user interfaces, drawing on current research and best practices to propose effective and adaptable solutions for educational AR applications. As AR technology evolves, its potential to transform educational experiences continues to grow, promising significant advancements in how users interact with, personalize, and immerse themselves in learning content. Full article

The aim was to investigate how the congruence of odors and visual objects in virtual reality (VR) affects later memory recall of the objects. Participants (N = 30) interacted with 12 objects in VR. The interaction was varied by odor congruency (i.e., the odor matched the object’s visual appearance, the odor did not match the object’s visual appearance, or the object had no odor); odor quality (i.e., an authentic or a synthetic odor); and interaction type (i.e., participants could look and manipulate or could only look at objects). After interacting with the 12 objects, incidental memory performance was measured with a free recall task. In addition, the participants rated the pleasantness and arousal of the interaction with each object. The results showed that the participants remembered significantly more objects with congruent odors than objects with incongruent odors or odorless objects. Furthermore, interaction with congruent objects was rated significantly more pleasant and relaxed than interaction with incongruent objects. Odor quality and interaction type did not have significant effects on recall or emotional ratings. These results can be utilized in the development of multisensory VR applications. Full article