Search Results (23)

This study proposes a process model for sharing ambiguous category concepts in virtual reality (VR) using an active inference framework. The model executes a dual-layer Bayesian update after observing both self and partner actions and predicts actions that minimize free energy. To incorporate agreement-seeking with others into active inference, we added disagreement in category judgments as a risk term in the free energy, weighted by gaze synchrony measured using Dynamic Time Warping (DTW), which is assumed to reflect joint attention. To validate the model, an object classification task in VR including ambiguous items was created. The experiment was conducted first under a bot avatar condition, in which ambiguous category judgments were always incorrect, and then under a human–human pair condition. This design allowed verification of the collaborative learning process by which human pairs reached agreement from the same degree of ambiguity. Analysis of experimental data from 14 participants showed that the model achieved high prediction accuracy for observed values as learning progressed. Introducing gaze synchrony weighting (${\gamma }_0\ge 0.5$) further improved prediction accuracy, yielding optimal performance. This approach provides a new framework for modeling socially shared cognition using active inference in human–robot interaction contexts. Full article

This paper presents the development and initial evaluation of a novel protocol for robot-assisted lower limb rehabilitation. It integrates dual-modal patient interaction, employing mirror therapy and an auto-adaptive EMG-driven control system, designed to enhance lower limb rehabilitation in patients with hemiparesis impairments. The system features a robotic platform specifically engineered for lower limb rehabilitation, which operates in conjunction with a virtual reality (VR) environment. This immersive environment comprises a digital twin of the robotic system alongside a human avatar representing the patient and a set of virtual targets to be reached by the patient. To implement mirror therapy, the proposed protocol utilizes a set of inertial sensors placed on the patient’s healthy limb to capture real-time motion data. The auto-adaptive protocol takes as input the EMG signals (if any) from sensors placed on the impaired limb and performs the required motions to reach the virtual targets in the VR application. By synchronizing the motions of the healthy limb with the digital twin in the VR space, the system aims to promote neuroplasticity, reduce pain perception, and encourage engagement in rehabilitation exercises. Initial laboratory trials demonstrate promising outcomes in terms of improved motor function and subject motivation. This research not only underscores the efficacy of integrating robotics and virtual reality in rehabilitation but also opens avenues for advanced personalized therapies in clinical settings. Future work will investigate the efficiency of the proposed solution using patients, thus demonstrating clinical usability, and explore the potential integration of additional feedback mechanisms to further enhance the therapeutic efficacy of the system. Full article

Human communication in daily life entails not only talking about what we are currently doing or will do, but also speculating about future possibilities that may (or may not) occur, i.e., “anticipatory speech”. Such conversations are central to social cooperation and social cohesion in humans. This suggests that such capabilities may also be critical for developing improved speech systems for artificial agents, e.g., human–agent interaction (HAI) and human–robot interaction (HRI). However, to do so successfully, it is imperative that we understand how anticipatory speech may affect the behavior of human users and, subsequently, the behavior of the agent/robot. Moreover, it is possible that such effects may vary across cultures and languages. To that end, we conducted an experiment where a human and autonomous 3D virtual avatar interacted in a cooperative gameplay environment. The experiment included 40 participants, comparing different languages (20 English, 20 Korean), where the artificial agent had anticipatory speech either enabled or disabled. The results showed that anticipatory speech significantly altered the speech patterns and turn-taking behavior of both the human and the agent, but those effects varied depending on the language spoken. We discuss how the use of such novel communication forms holds potential for enhancing HAI/HRI, as well as the development of mixed reality and virtual reality interactive systems for human users. Full article

With the rapid development of the metaverse, designing effective interfaces in virtual and augmented environments presents significant challenges. Additionally, keeping real-time sensory data flowing from users to their virtual avatars in a seamless and accurate manner is one of the biggest challenges in this domain. To this end, this article investigates a telepresence suit as an interface for interaction within the metaverse and its virtual avatars, aiming to address the complexities of signal generation, conversion, and transmission in real-time telepresence systems. We model a telepresence suit framework that systematically generates state data and transmits it to end-points, which can be either robotic avatars or virtual representations within a metaverse environment. Through a hand movement study, we successfully minimized the volume of transmitted information, reducing traffic by over 50%, which directly decreased channel load and packet delivery delay. For instance, as channel load decreases from 0.8 to 0.4, packet delivery delay is reduced by approximately half. This optimization not only enhances system responsiveness but also improves accuracy, particularly by reducing delays and errors in high-priority signal paths, enabling more precise and reliable telepresence interactions in metaverse settings. Full article

Background: As the Internet of Things (IoT) expands, it enables new forms of communication, including interactions mediated by teleoperated robots like avatars. While extensive research exists on the effects of these devices on communication partners, there is limited research on the impact on the operators themselves. This study aimed to objectively assess the psychological and physiological effects of operating a teleoperated robot, specifically Telenoid, on its human operator. Methods: Twelve healthy participants (2 women and 10 men, aged 18–23 years) were recruited from Osaka University. Participants engaged in two communication sessions with a first-time partner: face-to-face and Telenoid-mediated. Telenoid is a minimalist humanoid robot teleoperated by a participant. Blood samples were collected before and after each session to measure hormonal and oxidative markers, including cortisol, diacron reactive oxygen metabolites (d-ROMs), and the biological antioxidat activity of plasma (BAP). Psychological stress was assessed using validated questionnaires (POMS-2, HADS, and SRS-18). Results: A trend of a decrease in cortisol levels was observed during Telenoid-mediated communication, whereas face-to-face interactions showed no significant changes. Oxidative stress, measured by d-ROMs, significantly increased after face-to-face interactions but not in Telenoid-mediated sessions. Significant correlations were found between oxytocin and d-ROMs and psychological stress scores, particularly in terms of helplessness and total stress measures. However, no significant changes were observed in other biomarkers or between the two conditions for most psychological measures. Conclusions: These findings suggest that cortisol and d-ROMs may serve as objective biomarkers for assessing psychophysiological stress during robot-mediated communication. Telenoid’s minimalist design may help reduce social pressures and mitigate stress compared to face-to-face interactions. Further research with larger, more diverse samples and longitudinal designs is needed to validate these findings and explore the broader impacts of teleoperated robots. Full article

This paper presents a taxonomy of agents’ embodiment in physical and virtual environments. It categorizes embodiment based on five entities: the agent being embodied, the possible mediator of the embodiment, the environment in which sensing and acting take place, the degree of body, and the intertwining of body, mind, and environment. The taxonomy is applied to a wide range of embodiment of humans, artifacts, and programs, including recent technological and scientific innovations related to virtual reality, augmented reality, telepresence, the metaverse, digital twins, and large language models. The presented taxonomy is a powerful tool to analyze, clarify, and compare complex cases of embodiment. For example, it makes the choice between a dualistic and non-dualistic perspective of an agent’s embodiment explicit and clear. The taxonomy also aided us to formulate the term “embodiment by proxy” to denote how seemingly non-embodied agents may affect the world by using humans as “extended arms”. We also introduce the concept “off-line embodiment” to describe large language models’ ability to create an illusion of human perception. Full article

This study introduces a cybernetic control and architectural framework for a robotic fish avatar operated by a human. The behavior of the robot fish is influenced by the electromyographic (EMG) signals of the human operator, triggered by stimuli from the surrounding objects and scenery. A deep artificial neural network (ANN) with perceptrons classifies the EMG signals, discerning the type of muscular stimuli generated. The research unveils a fuzzy-based oscillation pattern generator (OPG) designed to emulate functions akin to a neural central pattern generator, producing coordinated fish undulations. The OPG generates swimming behavior as an oscillation function, decoupled into coordinated step signals, right and left, for a dual electromagnetic oscillator in the fish propulsion system. Furthermore, the research presents an underactuated biorobotic mechanism of the subcarangiform type comprising a two-solenoid electromagnetic oscillator, an antagonistic musculoskeletal elastic system of tendons, and a multi-link caudal spine composed of helical springs. The biomechanics dynamic model and control for swimming, as well as the ballasting system for submersion and buoyancy, are deduced. This study highlights the utilization of EMG measurements encompassing sampling time and $\mu$-volt signals for both hands and all fingers. The subsequent feature extraction resulted in three types of statistical patterns, namely, $\mathsf{\Omega },\gamma ,\lambda$, serving as inputs for a multilayer feedforward neural network of perceptrons. The experimental findings quantified controlled movements, specifically caudal fin undulations during forward, right, and left turns, with a particular emphasis on the dynamics of caudal fin undulations of a robot prototype. Full article

Recognizing the affective state of children with autism spectrum disorder (ASD) in real-world settings poses challenges due to the varying head poses, illumination levels, occlusion and a lack of datasets annotated with emotions in in-the-wild scenarios. Understanding the emotional state of children with ASD is crucial for providing personalized interventions and support. Existing methods often rely on controlled lab environments, limiting their applicability to real-world scenarios. Hence, a framework that enables the recognition of affective states in children with ASD in uncontrolled settings is needed. This paper presents a framework for recognizing the affective state of children with ASD in an in-the-wild setting using heart rate (HR) information. More specifically, an algorithm is developed that can classify a participant’s emotion as positive, negative, or neutral by analyzing the heart rate signal acquired from a smartwatch. The heart rate data are obtained in real time using a smartwatch application while the child learns to code a robot and interacts with an avatar. The avatar assists the child in developing communication skills and programming the robot. In this paper, we also present a semi-automated annotation technique based on facial expression recognition for the heart rate data. The HR signal is analyzed to extract features that capture the emotional state of the child. Additionally, in this paper, the performance of a raw HR-signal-based emotion classification algorithm is compared with a classification approach based on features extracted from HR signals using discrete wavelet transform (DWT). The experimental results demonstrate that the proposed method achieves comparable performance to state-of-the-art HR-based emotion recognition techniques, despite being conducted in an uncontrolled setting rather than a controlled lab environment. The framework presented in this paper contributes to the real-world affect analysis of children with ASD using HR information. By enabling emotion recognition in uncontrolled settings, this approach has the potential to improve the monitoring and understanding of the emotional well-being of children with ASD in their daily lives. Full article

Teleoperation technology combines the strength and accuracy of robots with the perception and cognition abilities of human experts, allowing the robots to work as an avatar of the operator in dangerous environments. The motion compatibility and intuitiveness of the human–machine interface directly affect the quality of teleoperation. However, many motion capture methods require special working environments or need bulky mechanisms. In this research, we proposed a wearable, lightweight, and passive upper limb exoskeleton, which takes intuitiveness and human-machine compatibility as a major concern. The upper limb pose estimation and teleoperation mapping control methods based on the exoskeleton are also discussed. Experimental results showed that by the help of the upper limb exoskeleton, people can achieve most areas of the normal range of motion. The proposed mapping control methods were verified on a 14-DOF anthropomorphic manipulator and showed good performance in teleoperation tasks. Full article

Background: This article presents the system architecture and validation of the NeuroSuitUp body–machine interface (BMI). The platform consists of wearable robotics jacket and gloves in combination with a serious game application for self-paced neurorehabilitation in spinal cord injury and chronic stroke. Methods: The wearable robotics implement a sensor layer, to approximate kinematic chain segment orientation, and an actuation layer. Sensors consist of commercial magnetic, angular rate and gravity (MARG), surface electromyography (sEMG), and flex sensors, while actuation is achieved through electrical muscle stimulation (EMS) and pneumatic actuators. On-board electronics connect to a Robot Operating System environment-based parser/controller and to a Unity-based live avatar representation game. BMI subsystems validation was performed using exercises through a Stereoscopic camera Computer Vision approach for the jacket and through multiple grip activities for the glove. Ten healthy subjects participated in system validation trials, performing three arm and three hand exercises (each 10 motor task trials) and completing user experience questionnaires. Results: Acceptable correlation was observed in 23/30 arm exercises performed with the jacket. No significant differences in glove sensor data during actuation state were observed. No difficulty to use, discomfort, or negative robotics perception were reported. Conclusions: Subsequent design improvements will implement additional absolute orientation sensors, MARG/EMG based biofeedback to the game, improved immersion through Augmented Reality and improvements towards system robustness. Full article

Geduldspiele cubes (also known as patience cubes in English) are interesting problems to solve with robotic systems on the basis of machine learning approaches. Generally, highly dexterous hand and finger movement is required to solve them. In this paper, we propose a reinforcement-learning-based approach to solve simple geduldspiele cubes of a flat plane, a convex plane, and a concave plane. The key idea of the proposed approach is that we adopt a sim-to-real framework in which a robotic agent is virtually trained in simulation environment based on reinforcement learning, then the virtually trained robotic agent is deployed into a physical robotic system and evaluated for tasks in the real world. We developed a test bed which consists of a dual-arm robot with a patience cube in a gripper and the virtual avatar system to be trained in the simulation world. The experimental results showed that the virtually trained robotic agent was able to solve simple patience cubes in the real world as well. Based on the results, we could expect to solve the more complex patience cubes by augmenting the proposed approach with versatile reinforcement learning algorithms. Full article

With continuing developments in artificial intelligence (AI) and robot technology, ethical issues related to digital humans, AI avatars, intelligent process automation, robots, cyborgs, and autonomous vehicles are emerging, and the need for cultural and social sustainability through AI ethics is increasing. Moreover, as the use of video conferencing and metaverse platforms has increased due to COVID-19, ethics concepts and boundaries related to information and communications technology, cyber etiquette, AI ethics, and robot ethics have become more ambiguous. Because the definitions of ethics domains may be confusing due to the various types of computing platforms available, this paper attempts to classify these ethics domains according to three main platforms: computing devices, intermediary platforms, and physical computing devices. This classification provides a conceptual ethics framework that encompasses computer ethics, information ethics, cyber ethics, robot ethics, and AI ethics. Several examples are provided to clarify the boundaries between the various ethics and platforms. The results of this study can be the educational basis for the sustainability of society on ethical issues according to the development of technology. Full article

We examined the influence of groups of agents and the type of avatar on movement interference. In addition, we studied the synchronization of the subject with the agent. For that, we conducted experiments utilizing human subjects to examine the influence of one, two, or three agents, as well as human or robot avatars, and finally, the agent moving biologically or linearly. We found the main effect on movement interference was the number of agents; namely, three agents had significantly more influence on movement interference than one agent. These results suggest that the number of agents is more influential on movement interference than other avatar characteristics. For the synchronization, the main effect of the type of the agent was revealed, showing that the human agent kept more synchronization compared to the robotic agent. In this experiment, we introduced an additional paradigm on the interference which we called synchronization, discovering that a group of agents is able to influence this behavioral level as well. Full article

The development of new approaches for creating more “life-like” artificial intelligence (AI) capable of natural social interaction is of interest to a number of scientific fields, from virtual reality to human–robot interaction to natural language speech systems. Yet how such “Social AI” agents might be manifested remains an open question. Previous research has shown that both behavioral factors related to the artificial agent itself as well as contextual factors beyond the agent (i.e., interaction context) play a critical role in how people perceive interactions with interactive technology. As such, there is a need for customizable agents and customizable environments that allow us to explore both sides in a simultaneous manner. To that end, we describe here the development of a cooperative game environment and Social AI using a data-driven approach, which allows us to simultaneously manipulate different components of the social interaction (both behavioral and contextual). We conducted multiple human–human and human–AI interaction experiments to better understand the components necessary for creation of a Social AI virtual avatar capable of autonomously speaking and interacting with humans in multiple languages during cooperative gameplay (in this case, a social survival video game) in context-relevant ways. Full article

A frequently used procedure to examine the relationship between categorical and dimensional descriptions of emotions is to ask subjects to place verbal expressions representing emotions in a continuous multidimensional emotional space. This work chooses a different approach. It aims at creating a system predicting the values of Activation and Valence (AV) directly from the sound of emotional speech utterances without the use of its semantic content or any other additional information. The system uses X-vectors to represent sound characteristics of the utterance and Support Vector Regressor for the estimation the AV values. The system is trained on a pool of three publicly available databases with dimensional annotation of emotions. The quality of regression is evaluated on the test sets of the same databases. Mapping of categorical emotions to the dimensional space is tested on another pool of eight categorically annotated databases. The aim of the work was to test whether in each unseen database the predicted values of Valence and Activation will place emotion-tagged utterances in the AV space in accordance with expectations based on Russell’s circumplex model of affective space. Due to the great variability of speech data, clusters of emotions create overlapping clouds. Their average location can be represented by centroids. A hypothesis on the position of these centroids is formulated and evaluated. The system’s ability to separate the emotions is evaluated by measuring the distance of the centroids. It can be concluded that the system works as expected and the positions of the clusters follow the hypothesized rules. Although the variance in individual measurements is still very high and the overlap of emotion clusters is large, it can be stated that the AV coordinates predicted by the system lead to an observable separation of the emotions in accordance with the hypothesis. Knowledge from training databases can therefore be used to predict AV coordinates of unseen data of various origins. This could be used to detect high levels of stress or depression. With the appearance of more dimensionally annotated training data, the systems predicting emotional dimensions from speech sound will become more robust and usable in practical applications in call-centers, avatars, robots, information-providing systems, security applications, and the like. Full article