Multimodal Technologies and Interaction

Mobile learning has become increasingly important for higher education due to its numerous advantages and transformative potential. The aim of this study is to investigate how students perceive and utilize mobile learning (m-learning) services in universities. To achieve this objective, a conceptual model was developed, combining the TAM with additional new determinants, including perceived security, perceived trust, perceived risk, and service quality. The primary goal of this model is to assess the adoption of m-learning apps among users in university settings. To evaluate the proposed model, SEM was utilized to test the research model. The findings of the study highlight the critical roles of perceived security, perceived trust, and service quality in promoting the adoption of m-learning apps. Moreover, the results indicate that perceived risk negatively impacts both students’ trust and their attitudes towards using mobile learning services. The study reveals that the perceived trust, and service quality factors positively influence students’ attitudes towards adopting m-learning apps. These research findings hold significant implications for universities and academia, offering valuable insights to devise effective strategies for increasing the utilization of m- learning services among students. By gaining a deeper understanding of students’ perceptions and acceptance, universities can optimize their m-learning offerings to cater to students’ needs and preferences more effectively. Full article

The fast growth of data in the academic field has contributed to making recommendation systems for scientific papers more popular. Content-based filtering (CBF), a pivotal technique in recommender systems (RS), holds particular significance in the realm of scientific publication recommendations. In a content-based scientific publication RS, recommendations are composed by observing the features of users and papers. Content-based recommendation encompasses three primary steps, namely, item representation, user modeling, and recommendation generation. A crucial part of generating recommendations is the user modeling process. Nevertheless, this step is often neglected in existing content-based scientific publication RS. Moreover, most existing approaches do not capture the semantics of user models and papers. To address these limitations, in this paper we present a transparent Recommendation and Interest Modeling Application (RIMA), a content-based scientific publication RS that implicitly derives user interest models from their authored papers. To address the semantic issues, RIMA combines word embedding-based keyphrase extraction techniques with knowledge bases to generate semantically-enriched user interest models, and additionally leverages pretrained transformer sentence encoders to represent user models and papers and compute their similarities. The effectiveness of our approach was assessed through an offline evaluation by conducting extensive experiments on various datasets along with user study (N = 22), demonstrating that (a) combining SIFRank and SqueezeBERT as an embedding-based keyphrase extraction method with DBpedia as a knowledge base improved the quality of the user interest modeling step, and (b) using the msmarco-distilbert-base-tas-b sentence transformer model achieved better results in the recommendation generation step. Full article

Humans are considered to be communicative, usually interacting in dyads or groups. In this paper, we investigate group interactions regarding performance in a rather formal gathering. In particular, a collection of ten performance indicators used in social group sciences is used to assess the outcomes of the meetings in this manuscript, in an automatic, machine learning-based way. For this, the Parking Lot Corpus, comprising 70 meetings in total, is analysed. At first, we obtain baseline results for the automatic prediction of performance results on the corpus. This is the first time the Parking Lot Corpus is tapped in this sense. Additionally, we compare baseline values to those obtained, utilising bidirectional long-short term memories. For multiple performance indicators, improvements in the baseline results are able to be achieved. Furthermore, the experiments showed a trend that the acoustic material of the remaining group should use for the prediction of team performance. Full article

Augmented reality offers many artistic possibilities when it comes to the creation of place-based public artworks. In this paper, we present a series of works around the topic of augmented reality (AR) art and place-based storytelling, including the use of walking as a creative method, a series of workshops with emerging artists, public AR art collaborations and a study to examine user experience when interacting with such artworks. Our findings from these works show the potential of integrating augmented reality with public physical artworks and offer guidance to artists and AR developers on how to expand this potential. For artists, we show the importance of the space in which the artwork will be placed and provide guidance on how to work with the space. For developers, we find that there is a need to create tools that work with artists’ existing practices and to investigate how to expand augmented reality past the limitations of site- or piece-specific apps. Full article

The contemporary geopolitical environment and strategic uncertainty shaped by asymmetric and hybrid threats urge the future development of hands-on training in realistic environments. Training in immersive, virtual environments is a promising approach. Immersive training can support training for contexts that are otherwise hard to access, dangerous, or have high costs. This paper discusses the challenges for virtual reality training in the CBRN (chemical, biological, radioactive, nuclear) domain. Based on initial considerations and a literature review, we conducted a survey and three workshops to gather requirements for CBRN training in virtual environments. We structured the gathered insights into four overarching themes—the future of CBRN training, ethical and safety requirements, evaluation and feedback, and tangible objects and tools. We provide insights on these four themes and discuss recommendations. Full article

Augmented Reality (AR) is increasingly present in several fields, including the museological space, where the challenges of presenting objects interactively and attractively are constant, especially with the sociocultural changes of recent decades. Although there are numerous studies on AR in museums, the perspective of museum professionals on the technology still needs to be explored. Thus, in this study, we use a qualitative design and conduct in-depth interviews with professionals from 10 Portuguese museums involved in creating or applying AR within these environments. Applying the grounded theory, the researchers propose a framework to understand Portuguese museum professionals’ practices, perceptions, and experiences with AR in museum environments. The findings allow the creation of a theoretical framework divided into four levels, namely the perceptions of museum professionals on the role and use of AR, the understanding of departments, museum teams, and digital strategies, the perceived challenges, limitations, and advantages in the use of augmented reality technologies, and the future perspectives of AR in museums. The theory resulting from this study may also contribute suggestions for the design and implementation of AR in museums, which both museum professionals and designers can use. Full article

This article presents a comparative analysis of four low or no-code location-based game (LBG) authoring tools, namely Taleblazer, Aris, Actionbound, and Locatify. Each tool is examined in detail, with an emphasis on the functions and capabilities it provides for the development of LBGs. The article builds on the history and purpose of LBGs, their characteristics, as well as basic concepts and previous applications, placing emphasis both on the technological and pedagogical dimensions of these games. The evaluation of the tools is based on certain criteria, or metrics, recorded in the literature and empirical data collected through the development of prototype games for each tool. The tools are comparatively analyzed in terms of the LBG’s constituent features they incorporate, the fundamental and additional functionality provided to the developer, as well as the existence or absence of features that captivate players in the game experience. Moreover, feedback is provided based on the practical use of the platforms for developing LBGs in order to support prospective developers in making an informed choice of an LBG platform for implementing a specific game. The games were created by taking advantage of as many features of the tools as possible in order to have a more fair and complete evaluation. This study aims to highlight the affordances and limitations of the investigated low or no-code LBG authoring tools, enabling anyone interested in developing an LBG to choose the most appropriate tool taking into account their needs and technological background or designing their own LBG authoring tools. Full article

This retrospective study presents and summarizes our long-term efforts in the popularization of robotics, engineering, and artificial intelligence (STEM) using the NAO humanoid robot. By a conservative estimate, over a span of 8 years, we engaged at least a couple of thousand participants: approximately 70% were preschool children, 15% were elementary school students, and 15% were teenagers and adults. We describe several robot applications that were developed specifically for this task and assess their qualitative performance outside a controlled research setting, catering to various demographics, including those with special needs (ASD, ADHD). Five groups of applications are presented: (1) motor development activities and games, (2) children’s games, (3) theatrical performances, (4) artificial intelligence applications, and (5) data harvesting applications. Different cases of human–robot interactions are considered and evaluated according to our experience, and we discuss their weak points and potential improvements. We examine the response of the audience when confronted with a humanoid robot featuring intelligent behavior, such as conversational intelligence and emotion recognition. We consider the importance of the robot’s physical appearance, the emotional dynamics of human–robot engagement across age groups, the relevance of non-verbal cues, and analyze drawings crafted by preschool children both before and after their interaction with the NAO robot. Full article

Integration of technology within problem-based learning curricula is expanding; however, information regarding student experiences and attitudes about the integration of such technologies is limited. This study aimed to evaluate pre-clinical medical student perceptions and use patterns of the “Road to Birth” (RtB) software, a novel program designed to support human maternal anatomy and physiology education. Second-year medical students at a large midwestern American university participated in a prospective, mixed-methods study. The RtB software is available as a mobile smartphone/tablet application and in immersive virtual reality. The program was integrated into problem-based learning activities across a three-week obstetrics teaching period. Student visuospatial ability, weekly program usage, weekly user satisfaction, and end-of-course focus group interview data were obtained. Survey data were analyzed and summarized using descriptive statistics. Focus group interview data were analyzed using inductive thematic analysis. Of the eligible students, 66% (19/29) consented to participate in the study with 4 students contributing to the focus group interview. Students reported incremental knowledge increases on weekly surveys (69.2% week one, 71.4% week two, and 78.6% week three). Qualitative results indicated the RtB software was perceived as a useful educational resource; however, its interactive nature could have been further optimized. Students reported increased use of portable devices over time and preferred convenient options when using technology incorporated into the curriculum. This study identifies opportunities to better integrate technology into problem-based learning practices in medical education. Further empirical research is warranted with larger and more diverse student samples. Full article

This research explores a novel Mexican Sign Language (MSL) lexicon video dataset containing the dynamic gestures most frequently used in MSL. Each gesture consists of a set of different versions of videos under uncontrolled conditions. The MX-ITESO-100 dataset is composed of a lexicon of 100 gestures and 5000 videos from three participants with different grammatical elements. Additionally, the dataset is evaluated in a two-step neural network model as having an accuracy greater than 99% and thus serves as a benchmark for future training of machine learning models in computer vision systems. Finally, this research provides an inclusive environment within society and organizations, in particular for people with hearing impairments. Full article

In this study, we propose the use of virtual urban field studies (VUFS) through context-based interface prototypes for evaluating the interaction design of auditory interfaces. Virtual field tests use mixed-reality technologies to combine the fidelity of real-world testing with the affordability and speed of testing in the lab. In this paper, we apply this concept to rapidly test sound designs for autonomous vehicle (AV)–pedestrian interaction with a high degree of realism and fidelity. We also propose the use of psychometrically validated measures of presence in validating the verisimilitude of VUFS. Using mixed qualitative and quantitative methods, we analysed users’ perceptions of presence in our VUFS prototype and the relationship to our prototype’s effectiveness. We also examined the use of higher-order ambisonic spatialised audio and its impact on presence. Our results provide insights into how VUFS can be designed to facilitate presence as well as design guidelines for how this can be leveraged. Full article

Educators and students have shown significant interest in the potential for generative artificial intelligence (AI) technologies to support student learning outcomes, for example, by offering personalized experiences, 24 h conversational assistance, text editing and help with problem-solving. We review contemporary perspectives on the value of AI as a tool in an educational context and describe our recent research with undergraduate students, discussing why and how we integrated OpenAI tools ChatGPT and Dall-E into the curriculum during the 2022–2023 academic year. A small cohort of games programming students in the School of Computing and Digital Media at London Metropolitan University was given a research and development assignment that explicitly required them to engage with OpenAI. They were tasked with evaluating OpenAI tools in the context of game development, demonstrating a working solution and reporting on their findings. We present five case studies that showcase some of the outputs from the students and we discuss their work. This mode of assessment was both productive and popular, mapping to students’ interests and helping to refine their skills in programming, problem-solving, critical reflection and exploratory design. Full article

As the global environmental crisis intensifies, there has been a significant interest in behavior change games (BCGs), as a viable venue to empower players’ pro-environmentalism. This pro-environmental empowerment is well-aligned with the notion of environmental citizenship (EC), which aims at transforming citizens into “environmental agents of change”, seeking to achieve more sustainable lifestyles. Despite these arguments, studies in this area are thinly spread and fragmented across various research domains. This article is grounded on a systematic review of empirical articles on BCGs for EC covering a time span of fifteen years and published in peer-reviewed journals and conference proceedings, in order to provide an understanding of the scope of empirical research in the field. In total, 44 articles were reviewed to shed light on their methodological underpinnings, the gaming elements and the persuasive strategies of the deployed BCGs, the EC actions facilitated by the BCGs, and the impact of BCGs on players’ EC competences. Our findings indicate that while BCGs seem to promote pro-environmental knowledge and attitudes, such an assertion is not fully warranted for pro-environmental behaviors. We reflect on our findings and provide future research directions to push forward the field of BCGs for EC. Full article

This paper presents a device that converts sound wave frequencies into colors to assist people with hearing problems in solving accessibility and communication problems in the hearing-impaired community. The device uses a precise mathematical apparatus and carefully selected hardware to achieve accurate conversion of sound to color, supported by specialized automatic processing software suitable for standardization. Experimental evaluation shows excellent performance for frequencies below 1000 Hz, although limitations are encountered at higher frequencies, requiring further investigation into advanced noise filtering and hardware optimization. The device shows promise for various applications, including education, art, and therapy. The study acknowledges its limitations and suggests future research to generalize the models for converting sound frequencies to color and improving usability for a broader range of hearing impairments. Feedback from the hearing-impaired community will play a critical role in further developing the device for practical use. Overall, this innovative device for converting sound to color represents a significant step toward improving accessibility and communication for people with hearing challenges. Continued research offers the potential to overcome challenges and extend the benefits of the device to a variety of areas, ultimately improving the quality of life for people with hearing impairments. Full article

For greater efficiency, human–machine and human–robot interactions must be designed with the idea of multimodality in mind. To allow the use of several interaction modalities, such as the use of voice, touch, gaze tracking, on several different devices (computer, smartphone, tablets, etc.) and to integrate possible connected objects, it is necessary to have an effective and secure means of communication between the different parts of the system. This is even more important with the use of a collaborative robot (cobot) sharing the same space and very close to the human during their tasks. This study present research work in the field of multimodal interaction for a cobot using the MQTT protocol, in virtual (Webots) and real worlds (ESP microcontrollers, Arduino, IOT2040). We show how MQTT can be used efficiently, with a common publish/subscribe mechanism for several entities of the system, in order to interact with connected objects (like LEDs and conveyor belts), robotic arms (like the Ned Niryo), or mobile robots. We compare the use of MQTT with that of the Firebase Realtime Database used in several of our previous research works. We show how a “pick–wait–choose–and place” task can be carried out jointly by a cobot and a human, and what this implies in terms of communication and ergonomic rules, via health or industrial concerns (people with disabilities, and teleoperation). Full article

Vision impairment affects an individual’s quality of life, posing challenges for visually impaired people (VIPs) in various aspects such as object recognition and daily tasks. Previous research has focused on developing visual navigation systems to assist VIPs, but there is a need for further improvements in accuracy, speed, and inclusion of a wider range of object categories that may obstruct VIPs’ daily lives. This study presents a modified version of YOLOv4_Resnet101 as backbone networks trained on multiple object classes to assist VIPs in navigating their surroundings. In comparison to the Darknet, with a backbone utilized in YOLOv4, the ResNet-101 backbone in YOLOv4_Resnet101 offers a deeper and more powerful feature extraction network. The ResNet-101’s greater capacity enables better representation of complex visual patterns, which increases the accuracy of object detection. The proposed model is validated using the Microsoft Common Objects in Context (MS COCO) dataset. Image pre-processing techniques are employed to enhance the training process, and manual annotation ensures accurate labeling of all images. The module incorporates text-to-speech conversion, providing VIPs with auditory information to assist in obstacle recognition. The model achieves an accuracy of 96.34% on the test images obtained from the dataset after 4000 iterations of training, with a loss error rate of 0.073%. Full article

What influences Information Communications and Technology (ICT) users’ privacy behavior? Several studies have shown that users state to care about their personal data. Contrary to that though, they perform unsafe privacy actions, such as ignoring to configure privacy settings. In this research, we present the results of an in-depth literature review on the factors affecting privacy behavior. We seek to investigate the underlying factors that influence individuals’ privacy-conscious behavior in the digital domain, as well as effective interventions to promote such behavior. Privacy decisions regarding the disclosure of personal information may have negative consequences on individuals’ lives, such as becoming a victim of identity theft, impersonation, etc. Moreover, third parties may exploit this information for their own benefit, such as targeted advertising practices. By identifying the factors that may affect SNS users’ privacy awareness, we can assist in creating methods for effective privacy protection and/or user-centered design. Examining the results of several research studies, we found evidence that privacy behavior is affected by a variety of factors, including individual ones (e.g., demographics) and contextual ones (e.g., financial exchanges). We synthesize a framework that aggregates the scattered factors that have been found in the literature to affect privacy behavior. Our framework can be beneficial to academics and practitioners in the private and public sectors. For example, academics can utilize our findings to create specialized information privacy courses and theoretical or laboratory modules. Full article

With the widespread of Monkeypox and increase in the weekly reported number of cases, it is observed that this outbreak continues to put the human beings in risk. The early detection and reporting of this disease will help monitoring and controlling the spread of it and hence, supporting international coordination for the same. For this purpose, the aim of this paper is to classify three diseases viz. Monkeypox, Chikenpox and Measles based on provided image dataset using trained standalone DL models (InceptionV3, EfficientNet, VGG16) and Squeeze and Excitation Network (SENet) Attention model. The first step to implement this approach is to search, collect and aggregate (if require) verified existing dataset(s). To the best of our knowledge, this is the first paper which has proposed the use of SENet based attention models in the classification task of Monkeypox and also targets to aggregate two different datasets from distinct sources in order to improve the performance parameters. The unexplored SENet attention architecture is incorporated with the trunk branch of InceptionV3 (SENet+InceptionV3), EfficientNet (SENet+EfficientNet) and VGG16 (SENet+VGG16) and these architectures improve the accuracy of the Monkeypox classification task significantly. Comprehensive experiments on three datasets depict that the proposed work achieves considerably high results with regard to accuracy, precision, recall and F1-score and hence, improving the overall performance of classification. Thus, the proposed research work is advantageous in enhanced diagnosis and classification of Monkeypox that can be utilized further by healthcare experts and researchers to confront its outspread. Full article

As technology has explosively and globally revolutionized the teaching and learning processes at educational institutions, enormous and innovative technological developments, along with their tools and applications, have recently invaded the education system. Using mobile learning (m-learning) employs wireless technologies for thinking, communicating, learning, and sharing to disseminate and exchange knowledge. Consequently, assessing the learning attitudes of students toward mobile learning is crucial, as learning attitudes impact their motivation, performance, and beliefs about mobile learning. However, mobile learning seems under-researched and may require additional efforts from researchers, especially in the context of the Middle East. Hence, this study’s contribution is enhancing our knowledge about students’ attitudes towards mobile-based learning. Therefore, the study goal was to investigate m-learning’s effect on the learning attitudes among technology education students. An explanatory sequential mixed approach was utilized to examine the attitudes of 50 students who took an educational technology class. A quasi-experiment was conducted and a phenomenological approach was adopted. Data from the experimental group and the control group were gathered. Focus group discussions with three groups and 25 semi-structured interviews were performed with students who experienced m-learning in their course. ANCOVA was conducted and revealed the impact of m-learning on the attitudes and their components. An inductive and deductive content analysis was conducted. Eleven subthemes stemmed out of three main themes. These subthemes included: personalized learning, visualization of learning motivation, less learning frustration, enhancing participation, learning on familiar devices, and social interaction, which emerged from the data. The researchers recommended that higher education institutions adhere to a set of guiding principles when creating m-learning policies. Additionally, they should customize the m-learning environment with higher levels of interactivity to meet students’ needs and learning styles to improve their attitudes towards m-learning. Full article