Multimodal Technologies and Interaction — Open Access Journal

Most children can naturally engage in play and by this, develop skills while interacting with their peers and toys. However, children with Autism Spectrum Disorder (ASD) often show impairments in play skills which result in limited opportunities for interaction with others and the learning of social skills. In this regard, robotic devices/toys that can provide simple and attractive indications are advantageous to engage children with ASD in play activities that require social and interaction skills. This project proposes a new interaction method using paired robotic devices called COLOLO to facilitate a fundamental exchange of intention in communication so-called turn-taking. These tangible devices are designed to sense the user’s manipulation, send a message to the paired device, and display visual cues for assisting children to achieve turn-taking through play. On the sessions with COLOLO there are two devices, one held by the therapist and one by the child, and they take turns to manipulate the toys and change their colors. In this article, two experimental conditions or interaction rules: the “two-sided lighting rule” and the “one-sided lighting rule" were introduced. The two interactions rules differ from each on the way the devices used the visual cues to indicate the turn-holder. The effect of each interaction rule on children’s turn-taking behaviors was investigated through an experimental study with four children with ASD. From the results, we found that with the one-sided lighting rule participants tended to shift their gaze more and to decrease the failed attempts of turn-taking. The discussion covers the possibilities of using paired devices to describe participants’ behaviors related to turn-taking quantitatively. Full article

In 1950, Alan Turing proposed his concept of universal machines, emphasizing their abilities to learn, think, and behave in a human-like manner. Today, the existence of intelligent agents imitating human characteristics is more relevant than ever. They have expanded to numerous aspects of daily life. Yet, while they are often seen as work simplifiers, their interactions usually lack social competence. In particular, they miss what one may call authenticity. In the study presented in this paper, we explore how characteristics of social intelligence may enhance future agent implementations. Interviews and an open question survey with experts from different fields have led to a shared understanding of what it would take to make intelligent virtual agents, in particular messaging agents (i.e., chat bots), more authentic. Results suggest that showcasing a transparent purpose, learning from experience, anthropomorphizing, human-like conversational behavior, and coherence, are guiding characteristics for agent authenticity and should consequently allow for and support a better coexistence of artificial intelligence technology with its respective users. Full article

Educational content of many kinds and from many disciplines are increasingly presented in the form of short videos made broadly accessible via platforms such as YouTube. We argue that understanding how such communicative forms function effectively (or not) demands a more thorough theoretical foundation in the principles of multimodal communication that is also capable of engaging with, and driving, empirical studies. We introduce the basic concepts adopted and discuss an empirical study showing how functional measures derived from the theory of multimodality we employ and results from a recipient-based study that we conducted align. We situate these results with respect to the state of the art in cognitive research in multimodal learning and argue that the more complex multimodal interactions and artifacts become, the more a fine-grained view of multimodal communication of the kind we propose will be essential for engaging with such media, both theoretically and empirically. Full article

For robots to attain more general-purpose utility, grasping is a necessary skill to master. Such general-purpose robots may use their perception abilities to visually identify grasps for a given object. A grasp describes how a robotic end-effector can be arranged to securely grab an object and successfully lift it without slippage. Traditionally, grasp detection requires expert human knowledge to analytically form the task-specific algorithm, but this is an arduous and time-consuming approach. During the last five years, deep learning methods have enabled significant advancements in robotic vision, natural language processing, and automated driving applications. The successful results of these methods have driven robotics researchers to explore the use of deep learning methods in task-generalised robotic applications. This paper reviews the current state-of-the-art in regards to the application of deep learning methods to generalised robotic grasping and discusses how each element of the deep learning approach has improved the overall performance of robotic grasp detection. Several of the most promising approaches are evaluated and the most suitable for real-time grasp detection is identified as the one-shot detection method. The availability of suitable volumes of appropriate training data is identified as a major obstacle for effective utilisation of the deep learning approaches, and the use of transfer learning techniques is proposed as a potential mechanism to address this. Finally, current trends in the field and future potential research directions are discussed. Full article

This paper describes the technical development of CrystalWalk, a crystal editor and visualization software designed for teaching materials science and engineering aiming to provide an accessible and interactive platform to students, professors and researchers. Justified by the lack of proper didactic tools, an evaluation of the existing crystallographic software has further shown opportunities for the development of a new software, more focused on the educational approach. CrystalWalk’s was guided by principles of free software, accessibility and democratization of knowledge, which was reflected in the application’s architecture strategy and the adoption of state-of-the-art technologies for the development of interactive web applications, such as HTML5/WebGL, service-oriented architecture (SOA) and responsive, resilient and elastic distributed systems. CrystalWalk’s architecture was successful in supporting the implementation of all specified software requirements proposed by state-of-the-art research and deemed to exert a positive impact in building accessible 3D interactive educational applications for the web. Full article

Tele-guidance systems for the remote monitoring and maintenance of equipment have been extensively investigated. Such systems enable a remote helper to provide guidance to a local worker while perceiving local conditions. In this study, we propose a tele-guidance system that supports the anticipation of an interlocutor’s actions during communication. Our proposed system enables a helper and worker to anticipate each other’s actions by allowing them to move around in the workspace freely and observe each other’s non-verbal cues (e.g., body motions and other gestures) through a head-mounted display. We conducted an experiment to compare the effectiveness of our proposed method with that of existing methods (a simple tele-pointer) that support anticipation during communication. Full article

Tangible technologies are considered promising tools for learning, by enabling multimodal interaction through physical action and manipulation of physical and digital elements, thus facilitating representational concrete–abstract links. A key concept in a tangible system is that its physical components are objects of interest, with associated meanings relevant to the context. Tangible technologies are said to provide ‘natural’ mappings that employ spatial analogies and adhere to cultural standards, capitalising on people’s familiarity with the physical world. Students with intellectual disabilities particularly benefit from interaction with tangibles, given their difficulties with perception and abstraction. However, symbolic information does not always have an obvious physical equivalent, and meanings do not reside in the representations used in the artefacts themselves, but in the ways they are manipulated and interpreted. In educational contexts, meaning attached to artefacts by designers is not necessarily transparent to students, nor interpreted by them as the designer predicted. Using artefacts and understanding their significance is of utmost importance for the construction of knowledge within the learning process; hence the need to study the use of the artefacts in contexts of practice and how they are transformed by the students. This article discusses how children with intellectual disabilities conceptually interpreted the elements of four tangible artefacts, and which characteristics of these tangibles were key for productive, multimodal interaction, thus potentially guiding designers and educators. Analysis shows the importance of designing physical-digital semantic mappings that capitalise on conceptual metaphors related to children’s familiar contexts, rather than using more abstract representations. Such metaphorical connections, preferably building on physical properties, contribute to children’s comprehension and facilitate their exploration of the systems. Full article

The purpose of this paper is to review the scholarly works regarding social embodiment aligned with the design of non-player characters in virtual reality (VR)-based social skill training for autistic children. VR-based social skill training for autistic children has been a naturalistic environment, which allows autistic children themselves to shape socially-appropriate behaviors in real world. To build up the training environment for autistic children, it is necessary to identify how to simulate social components in the training. In particular, designing non-player characters (NPCs) in the training is essential to determining the quality of the simulated social interactions during the training. Through this literature review, this study proposes multiple design themes that underline the nature of social embodiment in which interactions with NPCs in VR-based social skill training take place. Full article

Social robots are being designed to help support people’s well-being in domestic and public environments. To address increasing incidences of psychological and emotional difficulties such as loneliness, and a shortage of human healthcare workers, we believe that robots will also play a useful role in engaging with people in therapy, on an emotional and creative level, e.g., in music, drama, playing, and art therapy. Here, we focus on the latter case, on an autonomous robot capable of painting with a person. A challenge is that the theoretical foundations are highly complex; we are only just beginning ourselves to understand emotions and creativity in human science, which have been described as highly important challenges in artificial intelligence. To gain insight, we review some of the literature on robots used for therapy and art, potential strategies for interacting, and mechanisms for expressing emotions and creativity. In doing so, we also suggest the usefulness of the responsive art approach as a starting point for art therapy robots, describe a perceived gap between our understanding of emotions in human science and what is currently typically being addressed in engineering studies, and identify some potential ethical pitfalls and solutions for avoiding them. Based on our arguments, we propose a design for an art therapy robot, also discussing a simplified prototype implementation, toward informing future work in the area. Full article

The use of musical instruments and interfaces that involve animals in the interaction process is an emerging, yet not widespread practice. The projects that have been implemented in this unusual field are raising questions concerning ethical principles, animal-centered design processes, and the possible benefits and risks for the animals involved. Animal–Computer Interaction is a novel field of research that offers a framework (ACI manifesto) for implementing interactive technology for animals. Based on this framework, we have examined several projects focusing on the interplay between animals and music technology in order to arrive at a better understanding of animal-based musical projects. Building on this, we will discuss how the implementation of new musical instruments and interfaces could provide new opportunities for improving the quality of life for grey parrots living in captivity. Full article

In this paper, the potential relationship between creative citizenship and what may be termed ‘maker literacies’ is examined in the light of emergent findings from an international project on the use of makerspaces in early childhood, “MakEY” (see http://makeyproject.eu). The paper outlines the concept of creative citizenship and considers the notion of maker literacies before moving on to examine how maker literacies might be developed in early-years curricula in ways that foster civic engagement. Three vignettes are offered of makerspaces in early-years settings and a museum in Finland, Norway, and the UK. The activities outlined in the vignettes might be conceived of as ‘maker citizenship’, a concept which draws together understandings of making, digital literacies, and citizenship. The paper considers the implications of this analysis for future research and practice. Full article

This article is concerned with digital, multimodal feedback that supports learning and assessment within education. Drawing on the research literature alongside a case study from a postgraduate program in digital education, I argue that approaching feedback as an ongoing dialogue presented in richly multimodal and digital form can support opportunities for learning that are imaginative, critical, and in-tune with our increasingly digitally-mediated society. Using the examples of a reflective blogging exercise and an assignment built in the Second Life virtual world, I demonstrate how the tutor’s emphasis on providing feedback in multimodal form, alongside more conventional print-based approaches, inspired and emboldened students towards the creation of apt and sophisticated coursework. At the same time, the crafting of multimodal feedback carries resource implications and can sit uncomfortably with some deep-rooted assumptions around language-based representations of academic knowledge. This article should be seen in the context of a growing pedagogic and institutional interest in feedback around assessment, alongside the emergence of new ways of communicating and consuming academic content in richly multimodal ways. In this setting, multimodality, technology, and interaction refers to the digitally-mediated dialogue that takes place between the student and tutor around assessment. Full article

There is an increasing interest in the use of intelligent virtual agents (IVAs) to work in teams with humans. To achieve successful outcomes for these heterogeneous teams, many of the aspects found in successful human teams will need to be supported. These aspects include behavioural (i.e., multimodal communication), cognitive (i.e., a shared mental model (SMM)), and social (trust and commitment). Novelly, this paper aims to investigate the impact of IVA’s multimodal communication on the development of a SMM between humans and IVAs. Moreover, this paper aims to explore the impact of the developed SMM on a human’s trust in an IVA’s decisions and a human’s commitment to honour his/her promises to an IVA. The results from two studies involving a collaborative activity showed a significant positive correlation between team multimodal communication (i.e., behavioural aspect) and a SMM between teammates (i.e., cognitive aspect). Moreover, the result showed that there is a significant positive correlation between the developed SMM and a human’s trust in the IVA’s decision and the human’s commitment to honour his/her promises (the establishment of the social aspect of teamwork). Additionally, the results showed a cumulative effect of all of these aspects on human–agent team performance. These results can guide the design of human–agent teamwork multimodal communication models. Full article

In this review the application of deep learning for medical diagnosis is addressed. A thorough analysis of various scientific articles in the domain of deep neural networks application in the medical field has been conducted. More than 300 research articles were obtained, and after several selection steps, 46 articles were presented in more detail. The results indicate that convolutional neural networks (CNN) are the most widely represented when it comes to deep learning and medical image analysis. Furthermore, based on the findings of this article, it can be noted that the application of deep learning technology is widespread, but the majority of applications are focused on bioinformatics, medical diagnosis and other similar fields. Full article

That games can be used to teach specific content has been demonstrated numerous times. However, although specific game features have been conjectured to have an impact on learning outcomes, little empirical research exists on the impact of iterative design on learning outcomes. This article analyzes two games that have been developed to train an adult audience to recognize and avoid relying on six cognitive biases (three per game) in their decision making. The games were developed iteratively and were evaluated through a series of experiments. Although the experimental manipulations did not find a significant impact of the manipulated game features on the learning outcomes, each game iteration proved more successful than its predecessors at training players. Here, we outline a mixed-methods approach to postmortem game design analysis that helps us understand what might account for the improvement across games, and to identify new variables for future experimental training game studies. Full article

As mobile devices allow access to an increasing amount of private data, using them in public can potentially leak sensitive information through shoulder surfing. This includes personal private data (e.g., in chat conversations) and business-related content (e.g., in emails). Leaking the former might infringe on users’ privacy, while leaking the latter is considered a breach of the EU’s General Data Protection Regulation as of May 2018. This creates a need for systems that protect sensitive data in public. We introduce EyeSpot, a technique that displays content through a spot that follows the user’s gaze while hiding the rest of the screen from an observer’s view through overlaid masks. We explore different configurations for EyeSpot in a user study in terms of users’ reading speed, text comprehension, and perceived workload. While our system is a proof of concept, we identify crystallized masks as a promising design candidate for further evaluation with regard to the security of the system in a shoulder surfing scenario. Full article

Flavours of Glenroy (2013–4) was an action research project where artists imagined mobile edible gardens as a way to connect and engage with locals through project presentation and execution. As a socially engaged art project, it focused on developing ways to connect the mobile, diverse and transforming community of Glenroy, Victoria, Australia. The transnational, Australian dream suburb, reflecting the fluid and globalizing conditions of our cities, was emphasized through the strategy of growing and distributing plants using a mobile system that aligned with the mobility and diversity of the suburb. The project emphasized how social relations, encouraged through art, has the capacity to transform public spaces, providing a platform to introduce new voices and narratives of a community and encourage inclusive participation in sustainable citizenship. Full article

The impact of an aging population on healthcare and the sustainability of our healthcare system are pressing issues in contemporary society. Technology has the potential to address these challenges, alleviating pressures on the healthcare system and empowering individuals to have greater control over monitoring their own health. Importantly, mobile devices such as smartphones and tablets can allow older adults to have “on the go” access to health-related information. This paper explores mobile health apps that enable older adults and those who care for them to track health-related factors such as body readings and medication adherence, and it serves as a review of the literature on the usability and acceptance of mobile health apps in an older population. Full article