Multimodal Technologies and Interaction

Several studies have researched how people can use the anonymity of the Internet to explore different aspects of their identity. There are many different platforms where one can create a “virtual self” by actively choosing how one presents themselves to others, and each one is designed with different characteristics depending on their purpose: from socialization to professional networking or even entertainment. Different studies have usually focused on specific platforms, but there is no one comparing how people construe their virtual self across different online environments. In this review, we aimed to synthesize research studies carried out about the construal of one’s identity within online platforms and how it can relate to different aspects of participants’ offline identity, such as their self-esteem or self-concept clarity. Data were gathered from 34 publications that researched this topic across different kinds of online platforms. We conducted a quality assessment and a narrative synthesis, reporting and comparing the main findings, as well as identifying possible gaps in the literature. Many studies have explored the idea of people creating either an idealized or accurate version of themselves when construing their online identities. Others have also contemplated the possibility of exploring identities that diverge from both the actual and the ideal self or even an actively negative one. The latter was studied more in the context of video games and other avatar-mediated worlds. We found that people with low self-esteem create a more idealized virtual identity rather than a virtual self that is similar to their offline self. Other variables such as the purpose for using the online platform or self-concept clarity also had a role in virtual self construal, although the latter was only studied in the context of social media. Full article

In recent years, research reached a very high level of development and validation of augmented prototyping systems in support of collaborative design activities. However, there is still great scepticism in companies when it comes to integrating these new technologies within a consolidated working model. Among others, the main barrier to overcome concerns the lack of understanding of the impact of AR systems on the key objectives of a business, such as improving its efficiency and revenue. For this reason, this paper aims to quantify these indicators by observing the technological impact not on a single design session but on an entire product development process, during which the aspects related to its integration are also considered. Thanks to the collaboration with a design agency, it was possible to compare parameters such as the lead time, number of iterations, person-hours and costs between two similar and realistic projects in which only one was supported by projection-based AR technology. Full article

Although many of our interactions with technology nowadays take place in public places (e.g., using a mobile phone in public transportation), research and design on Human-Computer Interaction (HCI) has paid little attention to how this kind of technology usage affects others present—and vice versa. To illustrate the perspective of the attendant, i.e., a person who is not interacting with technology themselves but co-experiencing it as listener or viewer, we developed the so-called Attendant Card Set (ACS). In two studies, an expert survey and a student workshop, we tested its practical applicability and usefulness. It showed not only that experts assess the cards positively, i.e., helpful, informative, and relevant, but also that the cards can be used with laypersons for perspective-taking, creative ideation, and discussions. Thus, analyzing and/or comparing the experience of different types with the help of the ACS provides a unique approach to the consideration of the attendant perspective in the research and development process. Limitations of the present research and opportunities for future tool applications are discussed. In addition to establishing this concept in HCI, we also see potential in the transferability to other areas and contexts such as the design of public space or non-technological products. Full article

Interacting with virtual agents in immersive settings is becoming increasingly common thanks to the proliferation of dedicated media devices, such as consumer-grade virtual and augmented reality headsets. These technologies offer many advantages, e.g., in terms of presence and engagement, and can impact user behavior and attitudes toward virtual agents. Recognizing and understanding these effects is essential, especially in critical contexts involving the delegation of high-stake decisions to virtual agents. This article presents two experiments that explore users’ delegatory behavior toward virtual agents experienced via different media devices that vary in their technological immersion, i.e., a device’s technical capacity to deliver immersive experiences. The experiments’ results suggest that technological immersion is not a significant factor in users’ delegation decisions. Thus, for virtual agents designed to carry out critical tasks, developers may focus on other relevant factors, such as agents’ trustworthiness or performance. Full article

The relationship between workplace communication and affective outcomes, specifically connectedness at work and affective organizational commitment, is one that warrants further investigation for practical usage in the increasingly multimodal workplace. This study considers the frequency of use across five communication modalities, that being face-to-face, email, phone calls, instant messaging, and video calls, in relation to affective outcomes, as well as their relationships with communication meaningfulness. Employed participants (n = 516) completed an online survey in which they self-reported weekly communication tendencies, experienced connectedness, and affective organizational commitment. The final sample consisted of participants across 20 different industries in the United States. The most respondents worked in Health Care and Social Assistance or Professional, Scientific, and Technical Services (15% each), followed closely by respondents from Educational Services or Finance and Insurance (13% each). Data collection took place in between October 2021 and January 2022. Generally, participants who reported more frequent communication at work reported higher levels of connectedness and higher affective organizational commitment. Employees who found their communication more meaningful (irrespective of frequency) felt the most emotionally connected. Unique benefits of different communication modalities, as well as implications for hybrid and remote organizations, are discussed. Full article

This paper extends current research on Human–Robot Interaction (HRI) within Human–Computer Interaction (HCI), focusing on how future virtual agents and communication robots can support the temporal structures and routines within the home. We recruited representatives from 15 households with varied compositions, ranging from single inhabitants to full nest families. Drawing upon P.G. Wodehouse’s The Inimitable Jeeves as an inspiration, the methodology sought to imitate the relationship between a recruitment agency worker (researcher) and an employer (participant) seeking to hire a personal assistant (e.g., a virtual agent or communication robot device) for their home. A ‘household audit’ comprising a guided household tour and an architectural survey was conducted to ascertain the nuanced spatiotemporal routines within the home. The study analysed the responses of participants using the Labovian narrative schema, a traditional method in linguistics research. The findings were then examined through the lens of Reddy et al.’s temporal features of work to understand how domestic work unfolds within the home from a temporal perspective. We argue that the temporal concepts discussed by Reddy et al. provided valuable insights into the temporal dynamics of everyday activities and could inform the design of virtual agents and communication robotic devices to fulfil their roles as domesticated ‘personal assistants’. Full article

A chatbot, or else a conversational agent (CA), is a technology that is used in order to imitate the process of a conversation between a human being and a software application for supporting specific services. The utilization of this technology has been increasing considerably over the past five years, particularly in education where CAs are mostly utilized as teaching assistants that provide educational content. This paper aims to contribute to the existing body of knowledge by systematically reviewing the support provided by chatbots both to educational institutions and their students, investigating their capabilities in further detail, and highlighting the various ways that this technology could and should be used in order to maximize its benefits. Emphasis is given to analyzing and synthesizing the emerging roles of CAs, usage recommendations and suggestions, student’s desires, and challenges recorded in the literature. For this reason, a systematic literature review (SLR) was carried out using the PRISMA framework in order to minimize the common biases and limitations of SLRs. However, we must note that the SLR presented has specific limitations, namely using only Scopus as a search engine, utilizing a general search query, and selecting only journal articles published in English in the last five years. Full article

In this essay, we are interested in investigating some of the possible relations between design and digital humanities. In particular, we analyze the contribution that communication and interface design can bring to digital humanities. In a scene currently characterized by a heterogeneous set of activities and humanistic, technological, and cultural studies, the involvement of design seems confined to the development of digital instruments in accessing, exploring, and manipulating cultural data. How can design and the humanities work in an interdisciplinary way in order to shape new digital means to explore humanistic content? This essay presents four case studies (three of them developed by the authors), each of which suggests some methods and tools focused on the interdisciplinary relationships of scholars. The findings are both models of collaboration and models of digital architecture (data visualization) and showcase applied digital interactive platforms that present several paths to discovering different levels of content in the fields of art, psychology, literature, and history. In conclusion, this essay presents a manifesto focusing on ten points of virtuous relation between design humanities and the field of information visualization. Full article

This study explored drivers’ emotion-based impressions of car-borne central control platforms (CBCCPs) for personal-use vehicles. Thus, this preference-based study examined experts’ and drivers’ opinions regarding the appeal of CBCCPs from the perspective of Miryoku engineering. To this end, this study analyzed data via the EGM (evaluation grid method (EGM) and quantification theory type I. Results: Drivers’ preferences for specific CBCCP design characteristics were categorized into the factors “legible”, convenient”, and “tasteful”, which comprised the core of the EGM semantic hierarchical diagram. In addition, the importance of CBCCPs’ appeal factors and characteristics was assessed through quantification theory type I. The findings of this study provide valuable insights for designers, manufacturers, and researchers interested in the design of CBCCPs. Additionally, the results of this study can contribute to research on applied psychology, human–computer interactions, and car interface design. Full article

This research pursues the question of how the computer-generated analysis and visualization of communication can foster collaboration in teams that work together online. The audio data of regular online video meetings of three different teams were analyzed. Structural information regarding their communication was visualized in a communication report, and then, discussed with the teams in so-called digitally supported coaching (DSC) sessions. The aim of the DSC is to improve team collaboration by discerning helpful and less helpful patterns in the teams’ communication. This report allows us to recognize individual positions within the teams, as well as communication structures, such as conversational turn taking, that are relevant for group intelligence, as other research has shown. The findings pertaining to the team members during the DSC were gathered via questionnaires. These qualitative data were then matched with the quantitative data derived from the calls, particularly social network analysis (SNA). The SNA was inferred using the average number of interactions between the participants as measured in the calls. The qualitative findings of the teams were then cross-checked with the quantitative analysis. As a result, the assessment of team members’ roles was highly coherent with the SNA. Furthermore, all teams managed to derive concrete measures for improving their collaboration based on the reflection in the DSC. Full article

In this paper, we report a method of implementing a universal volumetric haptic actuation platform which can be adapted to fit a wide variety of visual displays with flat surfaces. This platform aims to enable the simulation of the 3D features of input interfaces. This goal is achieved using four readily available stepper motors in a diagonal cross configuration with which we can quickly change the position of a surface in a manner that can render these volumetric features. In our research, we use a Microsoft Surface Go tablet placed on the haptic enhancement actuation platform to replicate the exploratory features of virtual keyboard keycaps displayed on the touchscreen. We ask seven participants to explore the surface of a virtual keypad comprised of 12 keycaps. As a second task, random key positions are announced one at a time, which the participant is expected to locate. These experiments are used to understand how and with what fidelity the volumetric feedback could improve performance (detection time, track length, and error rate) of detecting the specific keycaps location with haptic feedback and in the absence of visual feedback. Participants complete the tasks with great success (p < 0.05). In addition, their ability to feel convex keycaps is confirmed within the subjective comments. Full article

Immersive Unit Visualization is an emergent form of visualization that arose from Immersive Analytics where, unlike traditional visualizations, each data point is represented by an individual visual mark in an immersive virtual environment. This practice has focused almost exclusively on virtual reality, excluding augmented reality (AR). This article develops and tests a prototype of an Immersive Unit Visualization (Floating Companies II) with two AR devices: head-mounted display (HMD) and hand-held display (HHD). Results from the testing sessions with 20 users were analyzed through qualitative research analysis and thematic coding indicating that, while the HHD enabled a first contact with AR visualization on a familiar device, HMD improved the perception of hybrid space by supporting greater stability of virtual content, wider field of view, improved spatial perception, increased sense of immersion, and more realistic simulation, which had an impact on information reading and sense-making. The materialization of abstract quantitative values into concrete reality through its simulation in the real environment and the ludic dimension stand out as important opportunities for this type of visualization. This paper investigates the aspects distinguishing two experiences regarding data visualization in hybrid space, and characterizes ways of seeing information with AR, identifying opportunities to advance information design research. Full article

Sign language (SL) avatar systems aid communication between the hearing and deaf communities. Despite technological progress, there is a lack of a standardized avatar development framework. This paper offers a systematic review of SL avatar systems spanning from 1982 to 2022. Using PRISMA guidelines, we shortlisted 47 papers from an initial 1765, focusing on sign synthesis techniques, corpora, design strategies, and facial expression methods. We also discuss both objective and subjective evaluation methodologies. Our findings highlight key trends and suggest new research avenues for improving SL avatars. Full article

Immersive journalism is a new form of media communication that uses extended reality systems to produce its content. Despite the possibilities it offers, its use is still limited in the media due to the lack of systematised and scientific knowledge regarding its application. This is a problem because it is a very powerful technology that changes the way audiences receive information and can be used both for new forms of storytelling that generate greater user engagement and for very sophisticated disinformation, which is why it is really important to study it. This study analyses articles published in the last 5 years that cover the use of extended technologies and the metaverse applied to immersive journalism. A systematic literature review applying PRISMA was carried out to identify literature within Web of Science, Scopus and Google Scholar (n = 61). Quantitative and qualitative analyses were conducted on the data collection techniques, the type of the data and the analysis techniques used. The results show a low level of methodological maturity, with research that is fundamentally descriptive and not very formalised, which limits the scope of its results and, therefore, the transfer of knowledge for its application in the configuration of new immersive journalistic products. The metaverse and extended technologies are considered independently and with distinct applications. It is concluded that research in this area is still in an initial exploratory and generalist stage that offers results that are not yet applicable to the promotion of this type of media format. Full article

The field of brain–computer interface (BCI) enables us to establish a pathway between the human brain and computers, with applications in the medical and nonmedical field. Brain computer interfaces can have a significant impact on the way humans interact with machines. In recent years, the surge in computational power has enabled deep learning algorithms to act as a robust avenue for leveraging BCIs. This paper provides an up-to-date review of deep and hybrid deep learning techniques utilized in the field of BCI through motor imagery. It delves into the adoption of deep learning techniques, including convolutional neural networks (CNNs), autoencoders (AEs), and recurrent structures such as long short-term memory (LSTM) networks. Moreover, hybrid approaches, such as combining CNNs with LSTMs or AEs and other techniques, are reviewed for their potential to enhance classification performance. Finally, we address challenges within motor imagery BCIs and highlight further research directions in this emerging field. Full article

Public warning systems are an essential element of safe cities. However, the functionality of neither traditional nor digital emergency warnings is understood well enough from the perspective of citizens. This study examines smart city development from the perspective of safety by exploring citizens’ viewpoints. It investigates people’s perceptions of the ways in which they obtain warnings and information about emergencies involving health risks. Data were collected in the form of focus group interviews and semi-structured interviews in Finland, Germany, and Greece. The results suggest that people place a lot of trust in their social network, receiving text messages, and their ability to use web-based search engines in order to obtain public warnings. The study discusses the challenges identified by citizens in the use of conventional radio and television transmissions and sirens for public warnings. Based on the results, citizens demonstrate informed ignorance about existing mobile emergency applications. Our results imply that it is not sufficient to build emergency communication infrastructure: the development of smart, safe cities requires continuous work and the integration of both hard and soft infrastructure-oriented strategies, i.e., technological infrastructure development including digitalisation and education, advancement of knowledge, and participation of people. Both strategic aspects are essential to enable people to take advantage of novel digital applications in emergency situations. Full article

Relational cues are extracts from actual verbal dialogues that help build the therapist–patient working alliance and stronger bond through the depiction of empathy, respect and openness. ECAs (Embodied conversational agents) are human-like virtual agents that exhibit verbal and non-verbal behaviours. In the digital health space, ECAs act as health coaches or experts. ECA dialogues have previously been designed to include relational cues to motivate patients to change their current behaviours and encourage adherence to a treatment plan. However, there is little understanding of who finds specific relational cues delivered by an ECA helpful or not. Drawing the literature together, we have categorised relational cues into empowering, working alliance, affirmative and social dialogue. In this study, we have embedded the dialogue of Alex, an ECA, to encourage healthy behaviours with all the relational cues (empathic Alex) or with none of the relational cues (neutral Alex). A total of 206 participants were randomly assigned to interact with either empathic or neutral Alex and were also asked to rate the helpfulness of selected relational cues. We explore if the perceived helpfulness of the relational cues is a good predictor of users’ intention to change the recommended health behaviours and/or development of a working alliance. Our models also investigate the impact of individual factors, including gender, age, culture and personality traits of the users. The idea is to establish whether a certain group of individuals having similarities in terms of individual factors found a particular cue or group of cues helpful. This will establish future versions of Alex and allow Alex to tailor its dialogue to specific groups, as well as help in building ECAs with multiple personalities and roles. Full article

Mobile learning has become increasingly important for higher education due to its numerous advantages and transformative potential. The aim of this study is to investigate how students perceive and utilize mobile learning (m-learning) services in universities. To achieve this objective, a conceptual model was developed, combining the TAM with additional new determinants, including perceived security, perceived trust, perceived risk, and service quality. The primary goal of this model is to assess the adoption of m-learning apps among users in university settings. To evaluate the proposed model, SEM was utilized to test the research model. The findings of the study highlight the critical roles of perceived security, perceived trust, and service quality in promoting the adoption of m-learning apps. Moreover, the results indicate that perceived risk negatively impacts both students’ trust and their attitudes towards using mobile learning services. The study reveals that the perceived trust, and service quality factors positively influence students’ attitudes towards adopting m-learning apps. These research findings hold significant implications for universities and academia, offering valuable insights to devise effective strategies for increasing the utilization of m- learning services among students. By gaining a deeper understanding of students’ perceptions and acceptance, universities can optimize their m-learning offerings to cater to students’ needs and preferences more effectively. Full article

The fast growth of data in the academic field has contributed to making recommendation systems for scientific papers more popular. Content-based filtering (CBF), a pivotal technique in recommender systems (RS), holds particular significance in the realm of scientific publication recommendations. In a content-based scientific publication RS, recommendations are composed by observing the features of users and papers. Content-based recommendation encompasses three primary steps, namely, item representation, user modeling, and recommendation generation. A crucial part of generating recommendations is the user modeling process. Nevertheless, this step is often neglected in existing content-based scientific publication RS. Moreover, most existing approaches do not capture the semantics of user models and papers. To address these limitations, in this paper we present a transparent Recommendation and Interest Modeling Application (RIMA), a content-based scientific publication RS that implicitly derives user interest models from their authored papers. To address the semantic issues, RIMA combines word embedding-based keyphrase extraction techniques with knowledge bases to generate semantically-enriched user interest models, and additionally leverages pretrained transformer sentence encoders to represent user models and papers and compute their similarities. The effectiveness of our approach was assessed through an offline evaluation by conducting extensive experiments on various datasets along with user study (N = 22), demonstrating that (a) combining SIFRank and SqueezeBERT as an embedding-based keyphrase extraction method with DBpedia as a knowledge base improved the quality of the user interest modeling step, and (b) using the msmarco-distilbert-base-tas-b sentence transformer model achieved better results in the recommendation generation step. Full article