Multimodal Technol. Interact., Volume 4, Issue 2 (June 2020) – 24 articles

Access to performance data during matches and training sessions is important for coaches and players. Although there are many video tagging systems available which can provide such access, these systems require manual effort. Data from Inertial Measurement Units (IMU) could be used for automatically tagging video recordings in terms of players’ actions. However, the data gathered during volleyball sessions are generally very imbalanced, since for an individual player most time intervals can be classified as “non-actions” rather than “actions”. This makes automatic annotation of video recordings of volleyball matches a challenging machine-learning problem. To address this problem, we evaluated balanced and imbalanced learning methods with our newly proposed ‘super-bagging’ method for volleyball action modelling. All methods are evaluated using six classifiers and four sensors (i.e., accelerometer, magnetometer, gyroscope and barometer). We demonstrate that imbalanced learning provides better unweighted average recall, (UAR = 83.99%) for the non-dominant hand using a naive Bayes classifier than balanced learning, while balanced learning provides better performance (UAR = 84.18%) for the dominant hand using a tree bagger classifier than imbalanced learning. Our super-bagging method provides the best UAR (84.19%). It is also noted that the super-bagging method provides better averaged UAR than balanced and imbalanced methods in 8 out of 10 cases, hence demonstrating the potential of the super-bagging method for IMU’s sensor data. One of the potential applications of these novel models is fatigue and stamina estimation e.g., by keeping track of how many actions a player is performing and when these are being performed. Full article

There is an increasing concern about the trustworthiness of online reviews as there is no editorial process for verification of their authenticity. This study investigated the decision-making process of online consumers when reacting to a review, with the reputation score of the reviewer and the number of previous reviews incorporated along with anonymous and non-anonymous reviews. It recruited 200 participants and developed a 3 × 2 × 2 × 2 × 2 mixed experimental study, with the independent variables being the reaction to a review of a restaurant at 3 levels, the reputation score at 2 levels, the number of previous reviews at 2 levels, the valence of the reviews at 2 levels, and the level of anonymity at 2 levels. Five dependent variables were analyzed: level of trust, likelihood of going to the restaurant, a choice question of whether to go to the restaurant, confidence in the decision and the NASA-TLX workload. This study found that the reputation scores complemented the reaction to a review, improving the trust in the information and confidence in the decision made. The findings suggest that incorporating a user rating scale such as the reputation score of a user deters people from writing false or biased reviews and helps improve their accuracy. Although no significant effect of level of anonymity was found in this study, additional personal information about the users writing the review such as photos or other social media links may make a significant difference in the decision-making process. Full article

Interactive multimedia applications in museums generally aim at integrating into the exhibition complementary information delivered through engaging narratives. This article discusses a possible approach for effectively designing an interactive app for museum collections whose physical pieces are mutually related by multiple and articulated logical interconnections referring to elements of immaterial cultural heritage that would not be easy to bring to the public with traditional means. As proof of this concept, a specific case related to ancient Egyptian civilization has been developed. A collection of Egyptian artifacts such as mummies, coffins, and amulets, associated with symbols, divinities, and magic spells through the structured funerary ritual typical of that civilization, has been explained through a virtual application based on the concepts discussed in the methodological section. Full article

Increasingly, the web produces massive volumes of texts, alone or associated with images, videos, photographs, together with some metadata, indispensable for their finding and retrieval. Keywords/keyphrases that characterize the semantic content of documents should be, automatically or manually, extracted, and/or associated with them. The paper presents a novel method to address the problem of the automatic unsupervised extraction of keywords/phrases from texts, expressed both in English and in Italian. The main feature of this approach is the integration of two methods that have given interesting results: word embedding models, such as Word2Vec or GloVe able to capture the semantics of words and their context, and clustering algorithms, able to identify the essence of the terms and choose the more significant one(s), to represent the contents of a text. In the paper, the datasets used are presented, together with the method implemented and the results obtained. These results will be discussed, commented, and compared with those obtained in previous experimentations, using TextRank, Rapid Automatic Keyword Extraction (RAKE), and TF-IDF. Full article

In early 2020, as a countermeasure to mitigate the spread of the COVID-19 pandemic, governments issued limitations on the movements of their citizens, cancelling social events and advising people to stay home. As location-based games (LBGs) have been found to influence human movement, their role during COVID-19 deserves closer inspection. Under regular circumstances, the very aim of these games is to motivate people to go out, explore and meet other people. However, during COVID-19, people were advised to do the exact opposite. To study how LBG developers and players reacted to the situation, we used the netnography research method utilizing three types of data: (1) COVID-19 related in-game changes made by seven popular LBG developers during March 2020; (2) social media reactions on 20 posts across three popular Pokémon GO subreddits; and (3) the raiding activity (collaborative play) in Pokémon GO in a Finnish municipality during February–May 2020. All observed LBGs made in-game changes due to COVID-19. The social media reactions showed overwhelming appreciation towards these changes, and two central second order themes arose: (1) LBGs have the ability to influence human movement during pandemics; and (2) people should be able to self-regulate their behaviour during COVID-19 independent of LBG influence. Surprisingly, recorded Pokémon GO player activity in Finland was more influenced by offered in-game rewards than the COVID-19 pandemic. Our findings have implications on how games and gamification can be used to direct human movement in situations such as COVID-19 where population-level interventions are needed. Full article

This study explores if peripheral colored ambient light or colored light can be noticed and semantically associated with text while reading a novel from an Electronic Visual Display (EVD). Additionally, it is investigated if those colored signs are perceived as supplementary to the story and result in a more immersive experience. For that purpose, a user evaluation study was performed based on attention theory (selective attention, semantic priming, and divided attention), reading behavior, and semantics of colors. Participants received a reading assignment. While reading a novel from an e-book, dynamic colored ambient light or colored light was emitted. Throughout the reading assignment, the colors were adapted to the particular scenes the participant was reading. Findings suggest that attention can be divided between reading the novel from an EVD and the ambient color signs. Full article

After conversational agents have been made available to the broader public, we speculate that applying them as a mediator for adaptive environments reduces control complexity and increases user experience by providing a more natural interaction. We implemented and tested four agents, each of them differing in their system intelligence and input modality, as personal assistants for Mediated Atmospheres, an adaptive smart office prototype. They were evaluated in a user study ( $N=33$ ) to collect subjective and objective measures. Results showed that a smartphone application was the most favorable system, followed by conversational text and voice agents that were perceived as being more engaging and intelligent than a non-conversational voice agent. Significant differences were observed between native and non-native speakers in both subjective and objective measures. Our findings reveal the potential of conversational agents for the interaction with adaptive environments to reduce work and information overload. Full article

This research presents results from a study developing a smartphone app, Urban CoBuilder, in which citizens can collaboratively create designs for urban environments using augmented reality technology and game mechanics. Eight prototypes were developed to refine selected design criteria, including tracking strategies, design elements, user experience and the interface with game mechanics. The prototypes were developed through an iterative design process with assessments and incremental improvements. The tracking was especially challenging and using multiple bitonal markers combined with the smartphone’s gyroscope sensor to average the user position was identified as the most suitable strategy. Still, portability and stability linked to tracking need to be improved. Design elements, here building blocks with urban functions textures, were realistic enough to be recognizable and easy to understand for the users. Future studies will focus on usability tests with larger user groups. Full article

Facial thermal imaging is a non-contact technology which can be useful for ubiquitous deceptive anxiety recognition. To date, studies investigating this technology have produced equivocal results in classification accuracy and finding the most correlated regions on the face. This study was conducted using our dataset with 41 subjects using two different protocols and three modalities (thermal, GSR and PPG). We selected and tracked five regions of interest (ROI) on each facial thermal imprint including periorbital, forehead, cheek, perinasal and chin that were mostly used in previous papers. By employing six statistical features, four feature reduction techniques and three classifiers, we attempted to identify the ROIs which are mostly associated with activation of the sympathetic nervous system to increase the final classification accuracy rate. The results of linear classification models show significant improvement of classification accuracy by using ROC feature selection method. We achieved 90.1% and 74.7% accuracy rate for thermal features in mock crime and best friend scenarios, respectively. Our experimental results show that perinasal and cheek areas have greater discriminatory power in comparison with other ROIs on the face. Full article

Shift-by-wire technology enables more options concerning the design, placement and functions of gear shifters compared to traditional gear shifters with manual transmission. These variations can impact usability and driver performance. There is a lack of research regarding the potential advantages and disadvantages of different types of gear shifters. The purpose of this study was to investigate the efficiency and subjective ease-of-use of mono- and polystable joystick gear shifter types at different complexity levels and with full or limited visibility. An experimental study with 36 participants was conducted. The results showed that monostable joysticks, especially those with an I/J-shape, were overall less efficient and easy to use than polystable joysticks. The highest complexity level clearly affected the efficiency for the monostable joystick with an I/J-shape (mono I/J) compared with the other gear shifter types. The monostable joystick with an I/J-shape (mono I/J) was also most affected by reduced visibility at the highest level of complexity, indicating that it was more prone to causing users to take their eyes off the road. Full article

Digitalization in schools requires a rethinking of teaching materials and methods in all subjects. This upheaval also concerns traditional print media, like school atlases used in geography classes. In this work, we examine the cartographic technological feasibility of extending a printed school atlas with digital content by augmented reality (AR). While previous research rather focused on topographic three-dimensional (3D) maps, our prototypical application for Android tablets complements map sheets of the Swiss World Atlas with thematically related data. We follow a natural marker approach using the AR engine Vuforia and the game engine Unity. We compare two workflows to insert geo-data, being correctly aligned with the map images, into the game engine. Next, the imported data are transformed into partly animated 3D visualizations, such as a dot distribution map, curved lines, pie chart billboards, stacked cuboids, extruded bars, and polygons. Additionally, we implemented legends, elements for temporal and thematic navigation, a screen capture function, and a touch-based feature query for the user interface. We evaluated our prototype in a usability experiment, which showed that secondary school students are as effective, interested, and sustainable with printed as with augmented maps when solving geographic tasks. Full article

Usability tests serve as an insightful source of feedback for product teams that want to deliver user-centered solutions and enhance the User Experience (UX) of their products and services. However, in many cases, formative usability tests in particular may generate a large volume of qualitative and unstructured data that need to be analyzed for decision making and further actions. In this paper, we discuss a more formal method of analyzing empirical data, using a taxonomy, namely Engineering Usability Research Empirical Knowledge and Artifacts Taxonomy (EUREKATAX). We describe how it can provide guidance and openness for transforming fuzzy feedback statements into actionable items. The main aim of the proposed method is to facilitate a more holistic and standardized process to empirical data analysis while adapting on the solution or context. The main contributions of this work comprise the: (a) definition of the proposed taxonomy which represents an organization of information structured in a hierarchy of four main categories (discover, learn, act, and monitor), eight sub-categories, and 52 items (actions/operations with their respective properties); (b) description of a method, that is expressed through the taxonomy, and adheres to a systematic but modular approach for analyzing data collected from the usability studies for decision making and implementation; (c) formulation of the taxonomy’s theoretical framework based on meticulously selected principles like experiential learning, activity theory: learning by expanding, and metacognition, and (d) extended evaluation into two phases, with 80 UX experts and business professionals, showing on the one hand the strong reliability of the taxonomy and high perceived fit of the items in the various classifications, and on the other hand the high perceived usability, usefulness and acceptability of the taxonomy when put into practice in real-life conditions. These findings are really encouraging, in an attempt to generate comparable, generalizable and replicable results of usability tests’ qualitative data analysis, thereby improving the UX and impact of software solutions. Full article

In recent years, the employment of behavior models to motivate behavior change has become a global trend in fitness application design. However, there is hardly any large-scale study of these applications to understand users’ exercise-type preferences, their drivers and barriers, and the potential of employing them for gender-based tailoring. To bridge this gap, we conducted a mixed-method study among 669 participants to investigate users’ exercise-type preferences (their drivers and barriers) and how they and gender can impact users’ social-cognitive beliefs and projected performance of bodyweight exercises. Firstly, we presented to the study participants a behavior model performing push-up or squat bodyweight exercise in a fitness application and asked them to rate their perceived self-efficacy, self-regulation, outcome expectation, and projected (exercise) performance level as observers of the behavior model. Secondly, we presented the study participants with a preselected list of commonly employed exercise types in fitness applications and requested them to identify their most/least preferred, and the reasons behind their choices. Our results showed that there were differences between both genders in their exercise-type preferences, perceived self-efficacy and projected exercise performance level. Males prefer push-up, squat, crunch, plank, and chair dip the most, with effectiveness being the most important driver, followed by ease of performance and improvement of the physique, look, and appearance. On the other hand, females prefer squat, crunch, jumping jack, step up, and plank the most, with ease of performance being the most important driver, followed by improvement of the physique, look, appearance, and effectiveness. Moreover, males prefer running in place the least, while females prefer push-up the least, with perceived difficulty being the greatest barrier for both genders. Moreover, our analysis of variance supported the female’s least preference for a push-up. Females have a lower perceived self-efficacy and projected performance level for push-up than males. We discussed the implications of our findings and provided guidelines for tailoring fitness applications on the market to users’ preferences and gender. Full article

The paper argues for the importance and richness of gaze communication during orchestra and choir conduction, and presents three studies on this issue. First, an interview with five choir and orchestra conductors reveals that they are not so deeply aware of the potentialities of gaze to convey indications in music performance. A conductor who was utterly conscious of the importance of gaze communication, however, is Leonard Bernstein, who conducted a performance of Haydn’s Symphony No. 88 using his face and gaze only. Therefore, a fragment of this performance is analyzed in an observational study, where a qualitative analysis singles out the items of gaze exploited by Bernstein and their corresponding meanings. Finally, a perception study is presented in which three of these items are submitted to expert, non-expert, and amateur participants. The results show that while the signal for “start” is fairly recognized, the other two, “pay attention” and “crescendo and accelerando” are more difficult to interpret. Furthermore, significant differences in gaze item recognition emerge among participants: experts not only recognize them more, but they also take advantage of viewing the items with audio-visual vs. video-only presentation, while non-experts do not take advantage of audio in their recognition. Full article

Online shopping communities have emerged amid growing social shopping activities and involve user-centered online platforms that encourage user-generated content and interactions, such as reading and writing reviews, rating products, and sharing shopping experiences. However, similar to other online platforms and communities, online shopping communities face challenges to provide tailored content and support appropriate socialization to engage users and encourage individualized contribution within the communities. To provide unique, personalized support for each individual user, this study developed personas in online shopping communities based on their motivation for participation, as well as reading and posting behaviors. Based on the findings from 20 interviews and focus groups with 24 active online shopping community participants, we developed an online survey on MTurk to investigate the characteristics of the personas and received 194 valid responses. Four persona types emerge after the analysis of both the qualitative and quantitative data—Opportunists, Contributors, Explorers, and Followers. Full article

User Experience (UX) describes the holistic experience of a user before, during, and after interaction with a platform, product, or service. UX adds value and attraction to their sole functionality and is therefore highly relevant for firms. The increased interest in UX has produced a vast amount of scholarly research since 1983. The research field is, therefore, complex and scattered. Conducting a bibliometric analysis, we aim at structuring the field quantitatively and rather abstractly. We employed citation analyses, co-citation analyses, and content analyses to evaluate productivity and impact of extant research. We suggest that future research should focus more on business and management related topics. Full article

Fitness apps equipped with various persuasive features have become popular worldwide due to the physical inactivity crisis. However, there is a limited understanding of the most important persuasive features that drive their adoption and the moderating effect of age and gender. To bridge this gap, we designed storyboards illustrating six of the commonly employed persuasive strategies in persuasive health applications: Goal-Setting/Self-Monitoring, Reward, Social Learning, Social Comparison, Competition and Cooperation. We conducted an empirical study in which we asked the participants to evaluate their receptiveness to the six persuasive features and their intention to use a fitness app that features them. The result of our Partial Least Square Path Modeling (PLSPM) shows that, overall, Goal-Setting/Self-Monitoring is the strongest predictor of the intention to use a fitness app, followed by Reward and Competition, both of which are in second place. However, Social Learning and Social Comparison turn out to be non-predictors of intention to use. Based on these findings, we recommend that a minimally viable (one-size-fits-all) fitness app, in a personal setting, should support a Goal-Setting/Self-Monitoring feature, coupled with a Reward feature, to increase its appeal to a wide audience. Moreover, in a social setting, it should support a Competition feature to increase its appeal to a wide audience. We discuss these findings and the gender and age differences in the relationships between users’ receptiveness to the six persuasive features and their intention to use a fitness app that support them. Full article

One particular approach in the context of Animal Computer Interaction (ACI) is auditory enrichment for captive wild animals. Here we describe our research and the methodology used to design musical instruments and interfaces aimed at providing auditory enrichment for grey parrots living in captivity. The paper is divided into three main phases: a project review and classification, sonic experiments at the parrot shelter and the design of musical instruments. The overview of recent projects that involve animals in the interaction and music-generation process highlights the costs and benefits of projects of this kind and provides insights into current technologies in this field and the musical talents of animals. Furthermore, we document a series of sonic experiments conducted at a parrot shelter to develop acoustically enriched environments through the use of musical instruments. These investigations were intended to provide a better understanding of how grey parrots communicate through sound, perceive and respond to auditory stimuli and possibly generate sound and music through the usage of technological devices. Based on the cognitive, physiological, and auditory abilities of grey parrots, and their intrinsic interest in sonic and physical interactions, we finally developed and tested various interactive instrument prototypes and here we present our design results for auditory enrichment in the context of ACI and artistic research. Full article

In recent years, companies have been seeking communication skills from their employees. Increasingly more companies have adopted group discussions during their recruitment process to evaluate the applicants’ communication skills. However, the opportunity to improve communication skills in group discussions is limited because of the lack of partners. To solve this issue as a long-term goal, the aim of this study is to build an autonomous robot that can participate in group discussions, so that its users can repeatedly practice with it. This robot, therefore, has to perform humanlike behaviors with which the users can interact. In this study, the focus was on the generation of two of these behaviors regarding the head of the robot. One is directing its attention to either of the following targets: the other participants or the materials placed on the table. The second is to determine the timings of the robot’s nods. These generation models are considered in three situations: when the robot is speaking, when the robot is listening, and when no participant including the robot is speaking. The research question is: whether these behaviors can be generated end-to-end from and only from the features of peer participants. This work is based on a data corpus containing 2.5 h of the discussion sessions of 10 four-person groups. Multimodal features, including the attention of other participants, voice prosody, head movements, and speech turns extracted from the corpus, were used to train support vector machine models for the generation of the two behaviors. The performances of the generation models of attentional focus were in an F-measure range between 0.4 and 0.6. The nodding model had an accuracy of approximately 0.65. Both experiments were conducted in the setting of leave-one-subject-out cross validation. To measure the perceived naturalness of the generated behaviors, a subject experiment was conducted. In the experiment, the proposed models were compared. They were based on a data-driven method with two baselines: (1) a simple statistical model based on behavior frequency and (2) raw experimental data. The evaluation was based on the observation of video clips, in which one of the subjects was replaced by a robot performing head movements in the above-mentioned three conditions. The experimental results showed that there was no significant difference from original human behaviors in the data corpus and proved the effectiveness of the proposed models. Full article

In the field of motion analysis, the gold standard devices are marker-based tracking systems. Despite being very accurate, their cost, stringent working environments, and long preparation time make them unsuitable for small clinics as well as for other scenarios such as industrial application. Since human-centered approaches have been promoted even outside clinical environments, the need for easy-to-use solutions to track human motion is topical. In this context, cost-effective devices, such as RGB-Depth (RBG-D) cameras have been proposed, aiming at a user-centered evaluation in rehabilitation or of workers in industry environment. In this paper, we aimed at comparing marker-based systems and RGB-D cameras for tracking human motion. We used a Vicon system (Vicon Motion Systems, Oxford, UK) as a gold standard for the analysis of accuracy and reliability of the Kinect V2 (Microsoft, Redmond, WA, USA) in a variety of gestures in the upper limb workspace—targeting rehabilitation and working applications. The comparison was performed on a group of 15 adult healthy subjects. Each subject had to perform two types of upper-limb movements (point-to-point and exploration) in three workspace sectors (central, right, and left) that might be explored in rehabilitation and industrial working scenarios. The protocol was conceived to test a wide range of the field of view of the RGB-D device. Our results, detailed in the paper, suggest that RGB-D sensors are adequate to track the upper limb for biomechanical assessments, even though relevant limitations can be found in the assessment and reliability of some specific degrees of freedom and gestures with respect to marker-based systems. Full article

Some literature has already demonstrated the widespread influence of human personality on product design. Nevertheless, most of the existing user experience (UX) design methods and tools do not fully exploit knowledge about user personality in selecting the best participants to maximize the effectiveness of the design efforts. This research tries to fill the gap by introducing PERSEL, the ready-to-use PERsonality-based SELector. PERSEL is a Microsoft Excel workbook, free to download, which allows expression of the objectives (needs) and assessment of the user personality; in turn, PERSEL suggests the best users to be involved in UX redesign activities and in what way, in order to get solutions answering to the needs in the best possible way. A comparison of the solutions generated by the first adoption of PERSEL in the field with those coming from the involvement of users selected without obeying any specific criterion, begins validating the research results, mainly in terms of PERSEL functioning and effectiveness. Full article

Educational innovation is a reality that is present in learning spaces. The use of emerging methodologies such as gamification and flipped learning has shown great potential in improving the teaching and learning process. This study aims to analyze the effectiveness of innovative mixed practices, combining gamification and flipped learning in the subject of Spanish Language and Literature against the isolated use of flipped learning. For this, a quasi-experimental design of descriptive and correlational type, based on a quantitative methodology has been carried out. For its development, two study groups (control-experimental) have been set up. The selected sample is of an intentional nature and was composed of 60 students of the fourth year of Secondary Education of an educational center in Southern Spain. The data has been collected through a validated questionnaire. The results determine that the complement of gamification in flipped learning has led to improvements in various academic indicators. It is concluded that the development of gamified actions in the face-to-face phase of flipped learning improves the motivation, interaction with teachers, and interactions of students. Full article

The purpose of this research was to study the perceived readiness of higher education students for computer-supported collaborative learning (CSCL). Moreover, the role of important demographic variables, such as gender, major of study, and computer ownership, was examined in students’ perceived readiness and its sub-scales. The data was collected from 326 higher education students of four study groups from a state university in Iran. MANOVA analysis was conducted to explore the possible role of the demographic variables in students’ perceived readiness for CSCL. Most of the participants showed high readiness for CSCL. The male participants demonstrated more online learning aptitude compared to females. A statistically significant difference was found in the online learning aptitude of the respondents majoring in engineering and basic sciences with the rest of the participants. Furthermore, the students with a personal computer, laptop, or tablet demonstrated higher levels of readiness for CSCL and online learning aptitude. Full article

There is a worldwide concern for young children’s online safety and a growing necessity for e-safety skills to be taught to children from a young age as part of formal schooling. The purpose of this study was to design and evaluate the effectiveness and motivational capacity of an interactive web-based learning environment for improving children’s e-safety skills. A quasi-experimental pre-test post-test control group design was used with an experimental group of 48 sixth-grade primary school students, who used the web-based learning environment over two 80-min lessons, and a control group of 25 students who did not. Findings revealed a statistically significant difference (t(47) = −14.06, p < 0.01) in the experimental group students’ e-safety performance, when students’ pre-test scores (mean (Μ) = 41.13, SD = 10.47) were compared to their post-test scores (Μ = 56.69, SD = 9.38). The analysis of an attitudes questionnaire and of student interviews documented the experimental group students’ positive attitudes toward the learning environment. Findings provide evidence of the effectiveness and motivational capacity of the web-based learning environment, which can be used in either formal education or informal learning settings, for improving children’s e-safety skills. Full article