Multimodal Technologies and Interaction — Open Access Journal

The paper argues for the importance and richness of gaze communication during orchestra and choir conduction, and presents three studies on this issue. First, an interview with five choir and orchestra conductors reveals that they are not so deeply aware of the potentialities of gaze to convey indications in music performance. A conductor who was utterly conscious of the importance of gaze communication, however, is Leonard Bernstein, who conducted a performance of Haydn’s Symphony No. 88 using his face and gaze only. Therefore, a fragment of this performance is analyzed in an observational study, where a qualitative analysis singles out the items of gaze exploited by Bernstein and their corresponding meanings. Finally, a perception study is presented in which three of these items are submitted to expert, non-expert, and amateur participants. The results show that while the signal for “start” is fairly recognized, the other two, “pay attention” and “crescendo and accelerando” are more difficult to interpret. Furthermore, significant differences in gaze item recognition emerge among participants: experts not only recognize them more, but they also take advantage of viewing the items with audio-visual vs. video-only presentation, while non-experts do not take advantage of audio in their recognition. Full article

Online shopping communities have emerged amid growing social shopping activities and involve user-centered online platforms that encourage user-generated content and interactions, such as reading and writing reviews, rating products, and sharing shopping experiences. However, similar to other online platforms and communities, online shopping communities face challenges to provide tailored content and support appropriate socialization to engage users and encourage individualized contribution within the communities. To provide unique, personalized support for each individual user, this study developed personas in online shopping communities based on their motivation for participation, as well as reading and posting behaviors. Based on the findings from 20 interviews and focus groups with 24 active online shopping community participants, we developed an online survey on MTurk to investigate the characteristics of the personas and received 194 valid responses. Four persona types emerge after the analysis of both the qualitative and quantitative data—Opportunists, Contributors, Explorers, and Followers. Full article

User Experience (UX) describes the holistic experience of a user before, during, and after interaction with a platform, product, or service. UX adds value and attraction to their sole functionality and is therefore highly relevant for firms. The increased interest in UX has produced a vast amount of scholarly research since 1983. The research field is, therefore, complex and scattered. Conducting a bibliometric analysis, we aim at structuring the field quantitatively and rather abstractly. We employed citation analyses, co-citation analyses, and content analyses to evaluate productivity and impact of extant research. We suggest that future research should focus more on business and management related topics. Full article

Fitness apps equipped with various persuasive features have become popular worldwide due to the physical inactivity crisis. However, there is a limited understanding of the most important persuasive features that drive their adoption and the moderating effect of age and gender. To bridge this gap, we designed storyboards illustrating six of the commonly employed persuasive strategies in persuasive health applications: Goal-Setting/Self-Monitoring, Reward, Social Learning, Social Comparison, Competition and Cooperation. We conducted an empirical study in which we asked the participants to evaluate their receptiveness to the six persuasive features and their intention to use a fitness app that features them. The result of our Partial Least Square Path Modeling (PLSPM) shows that, overall, Goal-Setting/Self-Monitoring is the strongest predictor of the intention to use a fitness app, followed by Reward and Competition, both of which are in second place. However, Social Learning and Social Comparison turn out to be non-predictors of intention to use. Based on these findings, we recommend that a minimally viable (one-size-fits-all) fitness app, in a personal setting, should support a Goal-Setting/Self-Monitoring feature, coupled with a Reward feature, to increase its appeal to a wide audience. Moreover, in a social setting, it should support a Competition feature to increase its appeal to a wide audience. We discuss these findings and the gender and age differences in the relationships between users’ receptiveness to the six persuasive features and their intention to use a fitness app that support them. Full article

One particular approach in the context of Animal Computer Interaction (ACI) is auditory enrichment for captive wild animals. Here we describe our research and the methodology used to design musical instruments and interfaces aimed at providing auditory enrichment for grey parrots living in captivity. The paper is divided into three main phases: a project review and classification, sonic experiments at the parrot shelter and the design of musical instruments. The overview of recent projects that involve animals in the interaction and music-generation process highlights the costs and benefits of projects of this kind and provides insights into current technologies in this field and the musical talents of animals. Furthermore, we document a series of sonic experiments conducted at a parrot shelter to develop acoustically enriched environments through the use of musical instruments. These investigations were intended to provide a better understanding of how grey parrots communicate through sound, perceive and respond to auditory stimuli and possibly generate sound and music through the usage of technological devices. Based on the cognitive, physiological, and auditory abilities of grey parrots, and their intrinsic interest in sonic and physical interactions, we finally developed and tested various interactive instrument prototypes and here we present our design results for auditory enrichment in the context of ACI and artistic research. Full article

In recent years, companies have been seeking communication skills from their employees. Increasingly more companies have adopted group discussions during their recruitment process to evaluate the applicants’ communication skills. However, the opportunity to improve communication skills in group discussions is limited because of the lack of partners. To solve this issue as a long-term goal, the aim of this study is to build an autonomous robot that can participate in group discussions, so that its users can repeatedly practice with it. This robot, therefore, has to perform humanlike behaviors with which the users can interact. In this study, the focus was on the generation of two of these behaviors regarding the head of the robot. One is directing its attention to either of the following targets: the other participants or the materials placed on the table. The second is to determine the timings of the robot’s nods. These generation models are considered in three situations: when the robot is speaking, when the robot is listening, and when no participant including the robot is speaking. The research question is: whether these behaviors can be generated end-to-end from and only from the features of peer participants. This work is based on a data corpus containing 2.5 h of the discussion sessions of 10 four-person groups. Multimodal features, including the attention of other participants, voice prosody, head movements, and speech turns extracted from the corpus, were used to train support vector machine models for the generation of the two behaviors. The performances of the generation models of attentional focus were in an F-measure range between 0.4 and 0.6. The nodding model had an accuracy of approximately 0.65. Both experiments were conducted in the setting of leave-one-subject-out cross validation. To measure the perceived naturalness of the generated behaviors, a subject experiment was conducted. In the experiment, the proposed models were compared. They were based on a data-driven method with two baselines: (1) a simple statistical model based on behavior frequency and (2) raw experimental data. The evaluation was based on the observation of video clips, in which one of the subjects was replaced by a robot performing head movements in the above-mentioned three conditions. The experimental results showed that there was no significant difference from original human behaviors in the data corpus and proved the effectiveness of the proposed models. Full article

In the field of motion analysis, the gold standard devices are marker-based tracking systems. Despite being very accurate, their cost, stringent working environments, and long preparation time make them unsuitable for small clinics as well as for other scenarios such as industrial application. Since human-centered approaches have been promoted even outside clinical environments, the need for easy-to-use solutions to track human motion is topical. In this context, cost-effective devices, such as RGB-Depth (RBG-D) cameras have been proposed, aiming at a user-centered evaluation in rehabilitation or of workers in industry environment. In this paper, we aimed at comparing marker-based systems and RGB-D cameras for tracking human motion. We used a Vicon system (Vicon Motion Systems, Oxford, UK) as a gold standard for the analysis of accuracy and reliability of the Kinect V2 (Microsoft, Redmond, WA, USA) in a variety of gestures in the upper limb workspace—targeting rehabilitation and working applications. The comparison was performed on a group of 15 adult healthy subjects. Each subject had to perform two types of upper-limb movements (point-to-point and exploration) in three workspace sectors (central, right, and left) that might be explored in rehabilitation and industrial working scenarios. The protocol was conceived to test a wide range of the field of view of the RGB-D device. Our results, detailed in the paper, suggest that RGB-D sensors are adequate to track the upper limb for biomechanical assessments, even though relevant limitations can be found in the assessment and reliability of some specific degrees of freedom and gestures with respect to marker-based systems. Full article

Some literature has already demonstrated the widespread influence of human personality on product design. Nevertheless, most of the existing user experience (UX) design methods and tools do not fully exploit knowledge about user personality in selecting the best participants to maximize the effectiveness of the design efforts. This research tries to fill the gap by introducing PERSEL, the ready-to-use PERsonality-based SELector. PERSEL is a Microsoft Excel workbook, free to download, which allows expression of the objectives (needs) and assessment of the user personality; in turn, PERSEL suggests the best users to be involved in UX redesign activities and in what way, in order to get solutions answering to the needs in the best possible way. A comparison of the solutions generated by the first adoption of PERSEL in the field with those coming from the involvement of users selected without obeying any specific criterion, begins validating the research results, mainly in terms of PERSEL functioning and effectiveness. Full article

Educational innovation is a reality that is present in learning spaces. The use of emerging methodologies such as gamification and flipped learning has shown great potential in improving the teaching and learning process. This study aims to analyze the effectiveness of innovative mixed practices, combining gamification and flipped learning in the subject of Spanish Language and Literature against the isolated use of flipped learning. For this, a quasi-experimental design of descriptive and correlational type, based on a quantitative methodology has been carried out. For its development, two study groups (control-experimental) have been set up. The selected sample is of an intentional nature and was composed of 60 students of the fourth year of Secondary Education of an educational center in Southern Spain. The data has been collected through a validated questionnaire. The results determine that the complement of gamification in flipped learning has led to improvements in various academic indicators. It is concluded that the development of gamified actions in the face-to-face phase of flipped learning improves the motivation, interaction with teachers, and interactions of students. Full article

The purpose of this research was to study the perceived readiness of higher education students for computer-supported collaborative learning (CSCL). Moreover, the role of important demographic variables, such as gender, major of study, and computer ownership, was examined in students’ perceived readiness and its sub-scales. The data was collected from 326 higher education students of four study groups from a state university in Iran. MANOVA analysis was conducted to explore the possible role of the demographic variables in students’ perceived readiness for CSCL. Most of the participants showed high readiness for CSCL. The male participants demonstrated more online learning aptitude compared to females. A statistically significant difference was found in the online learning aptitude of the respondents majoring in engineering and basic sciences with the rest of the participants. Furthermore, the students with a personal computer, laptop, or tablet demonstrated higher levels of readiness for CSCL and online learning aptitude. Full article

There is a worldwide concern for young children’s online safety and a growing necessity for e-safety skills to be taught to children from a young age as part of formal schooling. The purpose of this study was to design and evaluate the effectiveness and motivational capacity of an interactive web-based learning environment for improving children’s e-safety skills. A quasi-experimental pre-test post-test control group design was used with an experimental group of 48 sixth-grade primary school students, who used the web-based learning environment over two 80-min lessons, and a control group of 25 students who did not. Findings revealed a statistically significant difference (t(47) = −14.06, p < 0.01) in the experimental group students’ e-safety performance, when students’ pre-test scores (mean (Μ) = 41.13, SD = 10.47) were compared to their post-test scores (Μ = 56.69, SD = 9.38). The analysis of an attitudes questionnaire and of student interviews documented the experimental group students’ positive attitudes toward the learning environment. Findings provide evidence of the effectiveness and motivational capacity of the web-based learning environment, which can be used in either formal education or informal learning settings, for improving children’s e-safety skills. Full article

Neuro Linguistic Programming (NLP) is a collection of techniques for personality development. Meta programmes, which are habitual ways of inputting, sorting and filtering the information found in the world around us, are a vital factor in NLP. Differences in meta programmes result in significant differences in behaviour from one person to another. Personality types can be recognized through utilizing and analysing meta programmes. There are different methods to predict personality types based on meta programmes. The Myers–Briggs Type Indicator^® (MBTI) is currently considered as one of the most popular and reliable methods. In this study, a new machine learning method has been developed for personality type prediction based on the MBTI. The performance of the new methodology presented in this study has been compared to other existing methods and the results show better accuracy and reliability. The results of this study can assist NLP practitioners and psychologists in regards to identification of personality types and associated cognitive processes. Full article

In e-commerce, the user interface design of a website is critical to its success. However, there is limited research on how colour and layout design elements influence the perception of e-commerce websites for mobile devices. To bridge this gap, we conducted an empirical study to investigate, how the layout of information and colour temperature of an e-commerce tourism website for mobile device influence essential Technology Acceptance Model (TAM) user experience (UX) design attributes and intention to use the website. The results of our Partial Least Square Path Modelling (PLSPM) showed that both interface design elements significantly influence perceived aesthetics, perceived enjoyment, perceived usefulness and intention to use. Specifically, layout (list = 0 and grid = 1) positively influences perceived aesthetics and perceived enjoyment, while colour temperature negatively influences perceived usefulness and intention to use. The first finding suggests that in tourism website design for mobile devices, a grid layout of products and services provides a better hedonic user experience than a list layout. Moreover, the second finding suggests that cooler-temperature (blue and green) tourism websites are viewed by users as more useful than warmer-temperature (orange and red) tourism websites. We discuss the implications of these findings in the context of website UX design for mobile devices in the tourism domain. Full article

Electronic health records (EHRs) can be used to make critical decisions, to study the effects of treatments, and to detect hidden patterns in patient histories. In this paper, we present a framework to identify and analyze EHR-data-driven tasks and activities in the context of interactive visualization tools (IVTs)—that is, all the activities, sub-activities, tasks, and sub-tasks that are and can be supported by EHR-based IVTs. A systematic literature survey was conducted to collect the research papers that describe the design, implementation, and/or evaluation of EHR-based IVTs that support clinical decision-making. Databases included PubMed, the ACM Digital Library, the IEEE Library, and Google Scholar. These sources were supplemented by gray literature searching and reference list reviews. Of the 946 initially identified articles, the survey analyzes 19 IVTs described in 24 articles that met the final selection criteria. The survey includes an overview of the goal of each IVT, a brief description of its visualization, and an analysis of how sub-activities, tasks, and sub-tasks blend and combine to accomplish the tool’s main higher-level activities of interpreting, predicting, and monitoring. Our proposed framework shows the gaps in support of higher-level activities supported by existing IVTs. It appears that almost all existing IVTs focus on the activity of interpreting, while only a few of them support predicting and monitoring—this despite the importance of these activities in assisting users in finding patients that are at high risk and tracking patients’ status after treatment. Full article

An estimated 70% of patients who have been in the Intensive Care Unit (ICU) experience some form of Post-Intensive Care Syndrome (PICS). As a stressful environment, the ICU can be traumatic for any patient; however, the disruption of sleep experienced by patients in ICU negatively impacts their mental status and recovery. One of the most significant contributors to sleep disruption is the constant blare of monitor alarms, many of which are false or redundant. Through multisensory approaches and procedural redesign, the hostile acoustic environment of the ICU that causes so many to suffer from PICS may be alleviated. In this paper, we present suggestions for improving the ICU acoustic environment to possibly reduce the incidence of post-ICU complications such as PICS. Full article

The evaluation of cultural websites is a complicated procedure that depends on several criteria. The main contribution of this paper is on showing how this process may be implemented. For this reason, different multi-criteria decision-making methods are combined and compared for processing the results of the evaluation. More specifically, the paper provides the criteria for evaluating a website of cultural heritage and presents the application of Analytical Hierarchy Process for estimating the weights of the criteria. Then, it compares two different multi-criteria decision-making models, fuzzy simple additive weighting and fuzzy weighted product model, for being combined with Analytical Hierarchy Process and processing the results of the evaluation. The evaluation involves 29 websites of the conservation labs of the international museums and useful conclusions are extracted about the application of all methods and the combination of Analytical Hierarchy Process with fuzzy simple additive weighting and fuzzy weighted product model. Full article

The electrification of vehicles is without a doubt one of the milestones of today’s automotive technology. Even though industry actors perceive it as a future standard, acceptance, and adoption of this kind of vehicles by the end user remain a huge challenge. One of the main issues is the range anxiety related to the electric vehicle’s remaining battery level. In the scope of the H2020 ADAS&ME project, we designed and developed an intelligent Human Machine Interface (HMI) to ease acceptance of Electric Vehicle (EV) technology. This HMI is mounted on a fake autonomous vehicle piloted by a hidden joystick (called Wizard of Oz (WoZ) driving). We examined 22 inexperienced EV drivers during a one-hour driving task tailored to generate range anxiety. According to our protocol, once the remaining battery level started to become critical after manual driving, the HMI proposed accurate coping techniques to inform the drivers how to reduce the power consumption of the vehicle. In the following steps of the protocol, the vehicle was totally out of battery, and the drivers had to experience an emergency stop. The first result of this paper was that an intelligent HMI could reduce the range anxiety of the driver by proposing adapted coping strategies (i.e., transmitting how to save energy when the vehicle approaches a traffic light). The second result was that such an HMI and automated driving to a safe spot could reduce the stress of the driver when an emergency stop is necessary. Full article

In manufacturing, augmented reality (AR)-based remote instruction systems, which enable workers to receive instructions from an avatar, are widely used. In this study, we developed such a system and investigated the effect of the body representation level of the avatar on the quality of AR-based remote instruction. Drawing on the avatar designs of previous works, three different avatar designs (“Hand only”, “Hand + Arm”, and “Body”), representing three body representation levels, were created. In the experiment with a within-participant design, the avatar pointed at blocks sequentially and participants touched each block as soon as they identified it. The results of the experiment indicate that an AR-based remote instruction system with a “Body” avatar exhibits higher usability and can enable the participants to have a lower workload and higher efficiency. Full article