Multimodal Technologies and Interaction

Ideally, language and reading skills in bilingual children are assessed in both languages spoken in order to avoid misdiagnoses of communication or learning disorders. Due to limited capacity of clinical and educational staff, computerized screenings that allow for automatic evaluation of the children’s performance on reading tasks (accuracy and speed) might pose a useful alternative in clinical and school settings. In this study, a novel web-based screening platform for language and reading assessment is presented. This tool has been preliminarily validated with monolingual Italian, Mandarin–Italian and English–Italian speaking primary school children living and schooled in Italy. Their performances in the screening tasks in Italian and—if bilingual—in their native language were compared to the results of standardized/conventional reading assessment tests as well as parental and teacher questionnaires. Correlations revealed the tasks that best contributed to the identification of risk for the presence of reading disorders and showed the general feasibility and usefulness of the computerized screening. In a further step, both screening administrators (Examiners) and child participants (Examinees) were invited to participate in usability studies, which revealed general satisfaction and provided suggestions for further improvement of the screening platform. Based on these findings, the potential of the novel web-based screening platform is discussed. Full article

Virtual reality devices are used for several application domains, such as medicine, entertainment, marketing and training. A handheld controller is the common interaction method for direct object manipulation in virtual reality environments. Using hands would be a straightforward way to directly manipulate objects in the virtual environment if hand-tracking technology were reliable enough. In recent comparison studies, hand-based systems compared unfavorably against the handheld controllers in task completion times and accuracy. In our controlled study, we compare these two interaction techniques with a new hybrid interaction technique which combines the controller tracking with hand gestures for a rigid object manipulation task. The results demonstrate that the hybrid interaction technique is the most preferred because it is intuitive, easy to use, fast, reliable and it provides haptic feedback resembling the real-world object grab. This suggests that there is a trade-off between naturalness, task accuracy and task completion time when using these direct manipulation interaction techniques, and participants prefer to use interaction techniques that provide a balance between these three factors. Full article

The article presents a contextual survey of eight contributions in the special issue Musical Interactions (Volume I) in Multimodal Technologies and Interaction. The presentation includes (1) a critical examination of what it means to be musical, to devise the concept of music proper to MTI as well as multicultural proximity, and (2) a conceptual framework for instrumentation, design, and assessment of musical interaction research through five enabling dimensions: Affordance; Design Alignment; Adaptive Learning; Second-Order Feedback; Temporal Integration. Each dimension is discussed and applied in the survey. The results demonstrate how the framework provides an interdisciplinary scope required for musical interaction, and how this approach may offer a coherent way to describe and assess approaches to research and design as well as implementations of interactive musical systems. Musical interaction stipulates musical liveness for experiencing both music and technologies. While music may be considered ontologically incomplete without a listener, musical interaction is defined as ontological completion of a state of music and listening through a listener’s active engagement with musical resources in multimodal information flow. Full article

Numerous statistics show that cyclists are often involved in road traffic accidents, often with serious outcomes. One potential hazard of cycling, especially in cities, is “dooring”—passing parked vehicles that still have occupants inside. These occupants could open the vehicle door unexpectedly in the cyclist’s path—requiring a quick evasive response by the cyclist to avoid a collision. Dooring can be very poorly anticipated; as a possible solution, we propose in this work a system that notifies the cyclist of opening doors based on a networked intelligent transportation infrastructure. In a user study with a bicycle simulator (N = 24), we examined the effects of three user interface designs compared to a baseline (no notifications) on cycling behavior (speed and lateral position), perceived safety, and ease of use. Awareness messages (either visual message, visual message + auditory icon, or visual + voice message) were displayed on a smart bicycle helmet at different times before passing a parked, still-occupied vehicle. Our participants found the notifications of potential hazards very easy to understand and appealing and felt that the alerts could help them navigate traffic more safely. Those concepts that (additionally) used auditory icons or voice messages were preferred. In addition, the lateral distance increased significantly when a potentially opening door was indicated. In these situations, cyclists were able to safely pass the parked vehicle without braking. In summary, we are convinced that notification systems, such as the one presented here, are an important component for increasing road safety, especially for vulnerable road users. Full article

The ubiquity of mobile devices in peoples’ everyday life makes them a feasible tool for language learning. Learning anytime and anywhere creates great flexibility but comes with the inherent risk of infrequent learning and learning in interruption-prone environments. No matter the length of the learning break, it can negatively affect knowledge consolidation and recall. This work presents the design and implementation of memory cues to support task resumption in mobile language learning applications and two evaluations to assess their impact on user experience. An initial laboratory experiment ($N=15$) revealed that while the presentation of the cues had no significant effect on objective performance measures (task completion time and error rate), the users still perceived the cues as helpful and would appreciate them in a mobile learning app. A follow-up study ($N=16$) investigated revised cue designs in a real-world field setting and found that users particularly appreciated our interactive test cue design. We discuss strengths and limitations of our concept and implications for the application of task resumption cues beyond the scope of mobile language learning. Full article

The ubiquity of multimodal smart devices affords new opportunities for eyes-free applications for conveying graphical information to both sighted and visually impaired users. Using previously established haptic design guidelines for generic rendering of graphical content on touchscreen interfaces, the current study evaluates the learning and mental representation of digital maps, representing a key real-world translational eyes-free application. Two experiments involving 12 blind participants and 16 sighted participants compared cognitive map development and test performance on a range of spatio-behavioral tasks across three information-matched learning-mode conditions: (1) our prototype vibro-audio map (VAM), (2) traditional hardcopy-tactile maps, and (3) visual maps. Results demonstrated that when perceptual parameters of the stimuli were matched between modalities during haptic and visual map learning, test performance was highly similar (functionally equivalent) between the learning modes and participant groups. These results suggest equivalent cognitive map formation between both blind and sighted users and between maps learned from different sensory inputs, providing compelling evidence supporting the development of amodal spatial representations in the brain. The practical implications of these results include empirical evidence supporting a growing interest in the efficacy of multisensory interfaces as a primary interaction style for people both with and without vision. Findings challenge the long-held assumption that blind people exhibit deficits on global spatial tasks compared to their sighted peers, with results also providing empirical support for the methodological use of sighted participants in studies pertaining to technologies primarily aimed at supporting blind users. Full article

In recent years, researchers and manufacturers have started to investigate ways to enable autonomous vehicles (AVs) to interact with nearby pedestrians in compensation for the absence of human drivers. The majority of these efforts focuses on external human–machine interfaces (eHMIs), using different modalities, such as light patterns or on-road projections, to communicate the AV’s intent and awareness. In this paper, we investigate the potential role of affective interfaces to convey emotions via eHMIs. To date, little is known about the role that affective interfaces can play in supporting AV–pedestrian interaction. However, emotions have been employed in many smaller social robots, from domestic companions to outdoor aerial robots in the form of drones. To develop a foundation for affective AV–pedestrian interfaces, we reviewed the emotional expressions of non-humanoid robots in 25 articles published between 2011 and 2021. Based on findings from the review, we present a set of considerations for designing affective AV–pedestrian interfaces and highlight avenues for investigating these opportunities in future studies. Full article

Millions of people worldwide are taking up foreign languages with logographic writing systems, such as Japanese or Chinese. Learning thousands of characters necessary for literacy in those languages is a unique challenge to those coming from alphabetic backgrounds, and sustaining motivation in the face of such a momentous task is a struggle for many students. Many games exist for this purpose, but few offer production memory practice such as writing, and the vast majority are thinly veiled flashcards. To address this gap, we created Radical Tunes—a musical kanji-writing game—which combines production practice with musical mnemonic by assigning a melody to each element of a character. We chose to utilize music as it is a powerful tool that can be employed to enhance learning and memory. In this article, we explore whether incorporating melodies into a kanji learning game can positively affect the memorization of the stroke order/direction and overall shape of several Japanese characters, similar to the mnemonic effect of adding music to text. Specifically, we conducted two experimental studies, finding that (1) music improved immersion—an important factor related to learning; and (2) there was a positive correlation between melody presence and character production, particularly for more complex characters. Full article

Mobile augmented reality applications are gaining prominence in education, but there is a need to design appropriate and enjoyable games to be used in educational contexts such as classrooms. This paper presents an interactive information system designed to support the implementation of an augmented reality application in the context of game-based learning. PlanetarySystemGO includes a location-based mobile augmented reality game designed to promote learning about the celestial bodies and planetary systems of the Universe, and a web application that interacts with the mobile device application. Besides face-to-face classes, this resource can also be used in online classes, which is very useful in social isolation situations as the ones caused by the COVID-19 pandemic. Furthermore, it is the inclusion of the web application, with a back-office, in the information system that makes it possible to include curricula contents according to the grade level of students. Moreover, it is intended that teachers use the information system to include the contents they find appropriate to the grade level they teach. Therefore, it is crucial to provide their professional development to be able to use this resource. In this regard, a pilot study was conducted with teachers who participated in a STEM professional development programme in order to assess if the system is appropriate to be used by them. It is concluded that teachers found this resource relevant to motivate students to learn, and also acknowledged that the web application facilitated the introduction of appropriate curricula contents and also was useful to assess student performance during the game. Teachers need support, however, to implement these types of technologies which are not familiar to them. The necessary support can be provided through collaboration among the researchers and teachers in their schools. Besides engaging students to learn about celestial bodies, it is concluded that the information system can be used by teachers to introduce appropriate curricula contents and to be implemented in class. Full article

When designing extended reality (XR) applications, it is important to consider multimodal interaction techniques, which employ several human senses simultaneously. Multimodal interaction can transform how people communicate remotely, practice for tasks, entertain themselves, process information visualizations, and make decisions based on the provided information. This scoping review summarized recent advances in multimodal interaction technologies for head-mounted display-based (HMD) XR systems. Our purpose was to provide a succinct, yet clear, insightful, and structured overview of emerging, underused multimodal technologies beyond standard video and audio for XR interaction, and to find research gaps. The review aimed to help XR practitioners to apply multimodal interaction techniques and interaction researchers to direct future efforts towards relevant issues on multimodal XR. We conclude with our perspective on promising research avenues for multimodal interaction technologies. Full article

Vocabulary learning has been traditionally considered central to second language learning. It may take place either intentionally, by means of deliberate attempts to commit factual information to memory, or incidentally, as a consequence of other cognitive processes involving comprehension. Video games, which have been extensively employed in educational contexts to understand lexical development in foreign languages, foster both exposure to and the production of authentic and meaning-focused vocabulary. An empirical study was conducted to explore the effect of playing an online multiplayer social deduction game (i.e., a game in which players attempt to uncover each other’s hidden role) on incidental and intentional second language (L2) vocabulary learning. Secondary school pre-intermediate English as a Foreign Language (EFL) students (n = 54) took a vocabulary pre-test that identified eight unknown words likely to appear in the video game Among Us. Then, students were randomly assigned to different groups of players and to different learning conditions—within each group, half of the players were given a list of phrases containing such target words, which they were encouraged to meaningfully use in the game by means of written interaction. In doing so, students learnt some target words intentionally and provided contextualized incidental exposure to other players. They took a vocabulary test after two sessions of practice with the game to explore intentional and incidental L2 vocabulary learning gains. The pre- and post-tests suggested, among other results, that players using new L2 words in the game Among Us would retain more vocabulary than players only encountering them, that vocabulary intentionally input helped other users trigger incidental vocabulary learning, and that repetition had a positive effect on L2 vocabulary learning. Full article

Studies in the virtual heritage (VH) domain identify collaboration (social interaction), engagement, and a contextual relationship as key elements of interaction design that influence users’ experience and cultural learning in VH applications. The purpose of this study is to validate whether collaboration (social interaction), engaging experience, and a contextual relationship enhance cultural learning in a collaborative and multi-modal mixed reality (MR) heritage environment. To this end, we have designed and implemented a cloud-based collaborative and multi-modal MR application aiming at enhancing user experience and cultural learning in museums. A conceptual model was proposed based on collaboration, engagement, and relationship in the context of MR experience. The MR application was then evaluated at the Western Australian Shipwrecks Museum by experts, archaeologists, and curators from the gallery and the Western Australian Museum. Questionnaire, semi-structured interview, and observation were used to collect data. The results suggest that integrating collaborative and multi-modal interaction methods with MR technology facilitates enhanced cultural learning in VH. Full article

This study presents a co-design project that invites participants with little or no background in artificial intelligence (AI) and machine learning (ML) to design their ideal virtual assistants (VAs) for everyday (/daily) use. VAs are differently designed and function when integrated into people’s daily lives (e.g., voice-controlled VAs are designed to blend in based on their natural qualities). To further understand users’ ideas of their ideal VA designs, participants were invited to generate designs of personal VAs. However, end users may have unrealistic expectations of future technologies. Therefore, design fiction was adopted as a method of guiding the participants’ image of the future and carefully managing their realistic, as well as unrealistic, expectations of future technologies. The result suggests the need for a human–AI relationship based on controls with various dimensions (e.g., vocalness degree and autonomy level) instead of specific features. The design insights are discussed in detail. Additionally, the co-design process offers insights into how users can participate in AI/ML designs. Full article

In this paper, we examine to what degree children of 3–4 years old engage with a task and with a social robot during a second-language tutoring lesson. We specifically investigated whether children’s task engagement and robot engagement were influenced by three different feedback types by the robot: adult-like feedback, peer-like feedback and no feedback. Additionally, we investigated the relation between children’s eye gaze fixations and their task engagement and robot engagement. Fifty-eight Dutch children participated in an English counting task with a social robot and physical blocks. We found that, overall, children in the three conditions showed similar task engagement and robot engagement; however, within each condition, they showed large individual differences. Additionally, regression analyses revealed that there is a relation between children’s eye-gaze direction and engagement. Our findings showed that although eye gaze plays a significant role in measuring engagement and can be used to model children’s task engagement and robot engagement, it does not account for the full concept and engagement still comprises more than just eye gaze. Full article

Robot-assisted language learning (RALL) is a promising application when employing social robots to help both children and adults acquire a language and is an increasingly widely studied area of child–robot interaction. By introducing prosodic entrainment, i.e., converging the robot’s pitch with that of the learner, the present study aimed to provide new insights into RALL as a facilitative method for interactive tutoring. It is hypothesized that pitch-level entrainment by a Nao robot during a word learning task in a foreign language will result in increased learning in school-aged children. The results indicate that entrainment has no significant effect on participants’ learning, contra the hypothesis. Research on the implementation of entrainment in the context of RALL is new. This study highlights constraints in currently available technologies for voice generation and methodological limitations that should be taken into account in future research. Full article

Usability is a principal aspect of the system development process to improve and augment system facilities and meet users’ needs and necessities in all domains. It is no exception for cultural heritage. Usability problems of the interactive technology practice in cultural heritage museums should be recognized thoroughly from the viewpoints of experts and users. This paper reports on a two-phase empirical study to identify the usability problems in audio guides and websites of cultural heritage museums in Vietnam, as a developing country, and Australia, as a developed country. In phase one, five-user experience experts identified usability problems using the set of usability heuristics, and proposed suggestions to mitigate these issues. Ten usability heuristics identified a total of 176 problems for audio guides and websites. In phase two, we conducted field usability surveys to collect the real users’ opinions to detect the usability issues and examine the negative-ranked usability. The outstanding issues for audio guides and websites were pointed out. Identification of relevant usability issues and users’ and experts’ suggestions for these technologies should be given immediate attention to helping organizations and interactive service providers improve technologies’ adoptions. The paper’s findings are reliable inputs for our future study about the preeminent UX framework for interactive technology in the CH domain. Full article

In a case study, we transformed the existing learning program Language Shower, which is used in some Norwegian day-care centers in the Grorud district of Oslo municipality, into a digital solution using an app for smartphones or tablets with the option for further enhancement of the presentation by a NAO robot. The solution was tested in several iterations and multiple day-care centers over several weeks. Measurements of the children’s progress across learning sessions indicated a positive impact of the program using a robot as compared to the program without a robot. In situ observations and interviews with day-care center staff confirmed the solution’s many advantages, but also revealed some important areas for improvement. In particular, the speech recognition needs to be more flexible and robust, and special measures have to be in place to handle children speaking simultaneously. Full article

Eliciting knowledge from domain experts can play an important role throughout the machine learning process, from correctly specifying the task to evaluating model results. However, knowledge elicitation is also fraught with challenges. In this work, we consider why and how machine learning researchers elicit knowledge from experts in the model development process. We develop a taxonomy to characterize elicitation approaches according to the elicitation goal, elicitation target, elicitation process, and use of elicited knowledge. We analyze the elicitation trends observed in 28 papers with this taxonomy and identify opportunities for adding rigor to these elicitation approaches. We suggest future directions for research in elicitation for machine learning by highlighting avenues for further exploration and drawing on what we can learn from elicitation research in other fields. Full article