Physical AI for Cross-Embodiment Human-to-Robot Motion Retargeting with Residual Deep Koopman Learning

Abstract

Cross-embodiment human-to-robot motion retargeting requires transferring human motion to robot-feasible actions despite differences in kinematic structure, reachable workspace, and sensing-to-control latency. This paper proposes Cross-Embodiment Motion Retargeting (CE-MR), a lightweight Physical AI framework that combines two wearable inertial measurement units (IMUs), a robot-independent canonical motion representation, residual Deep Koopman (RDK) prediction, and posture-aware inverse kinematics (IK). Upper-arm and forearm orientations are used to reconstruct shoulder-relative wrist motion and arm configuration without acceleration integration or direct human-to-robot joint correspondence. Human motion is represented by a 14-dimensional state and predicted over a 200-ms receding horizon using a constant-velocity prior corrected by a compact RDK model with 2270 trainable parameters. Using 100 human-arm demonstrations across five motion categories, RDK reduced canonical-position RMSE from 0.04506 to 0.03183 at 200 ms, corresponding to a 29.36% improvement over constant-velocity prediction, while reducing full-state normalized RMSE by 28.63%. Online evaluation using the official Gazebo simulation model of the ROBOTIS OpenMANIPULATOR-X showed 100% final-target IK success for all four ablation modes. In a representative single-operator Gazebo ablation, the complete CE-MR configuration showed a mean normalized human–robot posture discrepancy of 0.0708 compared with 0.1375 in the independently performed RDK trial, while maintaining a mean Cartesian IK residual of 0.728 mm. Stable integrated execution further demonstrated operation near 20 Hz under favorable IK conditions. These results support CE-MR as a compact proof-of-concept framework for predictive human-to-robot cross-embodiment motion transfer from wearable human sensing to online simulated robot execution.

1. Introduction

Physical artificial intelligence (Physical AI) extends data-driven intelligence beyond purely digital inference toward closed-loop interaction among sensing, learned models, robotic embodiment, and physical action [1]. Within this perspective, learning from demonstration (LfD) provides an intuitive means of transferring human knowledge to robots when desired behaviors are easier to demonstrate than to specify analytically [2,3]. Human-to-robot motion retargeting is therefore an important interface between human demonstration and robotic execution, requiring the robot to preserve task-relevant motion and temporal structure while satisfying its own kinematic, workspace, and actuation constraints.

A fundamental challenge is that human and robot embodiments are rarely kinematically equivalent. Differences in link lengths, degrees of freedom, joint limits, and end-effector structures make direct human-joint-to-robot-joint correspondence difficult or inappropriate. Previous studies have addressed this problem through constrained optimization, shared latent representations, pose-estimation-based retargeting, and motion adaptation [4,5,6,7]. However, many such systems rely on rich motion-capture measurements, camera-based human-pose estimation, or high-dimensional whole-body representations. Wearable inertial measurement units (IMUs) provide a more portable alternative and have been investigated for industrial robot teleoperation, arm–hand control, wearable robot interfaces, and human kinematic estimation [8,9,10,11,12,13]. Nevertheless, many existing wearable systems still employ multiple sensors distributed over the torso, arm, hand, or full body, motivating simpler sensing configurations that preserve task-space motion information.

At the same time, scalable and embodiment-flexible demonstration acquisition has become increasingly important in modern robot learning. UMI enables portable in-the-wild human demonstration collection without requiring every demonstration to be generated directly on the target robot [14], while DROID and BridgeData V2 emphasize the importance of collecting diverse large-scale robotic experience across tasks and environments [15,16]. Open X-Embodiment further demonstrates the potential of aggregating experience across heterogeneous robot platforms, and generalist policies such as RT-1 and Octo show that data diversity and cross-platform experience can substantially support robot-policy generalization [17,18,19]. Low-cost teleoperation systems such as Mobile ALOHA similarly illustrate the growing importance of practical data-acquisition interfaces for robot learning [20]. These developments suggest that lightweight sensing interfaces and robot-independent motion representations can be valuable not only for teleoperation but also as practical tools for scalable demonstration collection and future cross-embodiment reuse.

Motivated by this direction, the present paper uses only two IMUs, mounted on the upper arm and forearm, to reconstruct shoulder-relative wrist motion and arm configuration. Rather than integrating linear acceleration to estimate hand displacement, wrist motion is recovered through a calibrated two-link forward-kinematic model. More importantly, the resulting demonstration is not represented as robot-specific joint commands. Human motion is transformed into a canonical workspace representation and subsequently mapped relative to the target robot’s own reachable workspace and session anchor. Thus, the recorded human motion remains separated from the specific kinematics of the robot used during deployment. Although the present paper evaluates this principle using a single robot embodiment, the ROBOTIS OpenMANIPULATOR-X, through its official Gazebo simulation model, the representation is designed as a lightweight basis for future reuse of human demonstrations across heterogeneous robot embodiments. This perspective is also consistent with prior LfD and imitation-learning studies that seek compact and reusable representations of expert motion and trajectory-generation behavior [21,22].

A second challenge is temporal delay. In an online human-to-robot pipeline, sensing, filtering, state estimation, inverse kinematics, communication, and actuation introduce cumulative latency, causing a purely reactive robot to follow motion that has already occurred. Short-horizon human-motion prediction can partially compensate for this effect by providing a near-future target instead of relying only on the instantaneous measured state. Recent human-motion prediction research has increasingly employed recurrent, graph-based, transformer, and diffusion architectures [23,24], while lightweight neural architectures have also been explored when computational efficiency is important [25]. Receding-horizon generative control methods such as Diffusion Policy likewise demonstrate the effectiveness of predicting short action sequences for robotic manipulation [26]. For the low-dimensional wearable state considered in this paper, however, a substantially smaller predictive model is desirable for real-time integration.

Koopman operator theory provides a structured approach for representing nonlinear dynamics through approximately linear evolution in an appropriate observable space [27]. Deep neural networks can learn such nonlinear embeddings while retaining linear latent dynamics [28]. Learned Koopman representations have subsequently been applied to nonlinear control, demonstration-based manipulation, time-varying dynamics, and real-time manipulator control [29,30,31,32], with recent work further demonstrating their applicability to physically interacting robotic systems [33]. However, a learned predictor does not necessarily need to replace a simple analytical model when the latter already performs strongly over very short prediction horizons.

Accordingly, the proposed framework retains constant velocity (CV) as an analytical motion prior and trains a compact Deep Koopman model only to estimate the residual correction not captured by CV. At every 50-ms control update, the current measured human state is re-encoded, and a new 200-ms future state is predicted in a receding-horizon manner, avoiding unrestricted recursive rollout. The predicted canonical wrist motion is then transferred to the robot workspace and combined with a posture-aware inverse-kinematics (IK) objective. Instead of directly matching human and robot joint angles, the IK formulation transfers a normalized geometric arm-bending variable, enabling posture information to be incorporated across kinematically different embodiments while maintaining Cartesian end-effector tracking as the primary task.

The methodological contribution of CE-MR lies in a unified formulation that organizes sparse wearable sensing, canonical task-space representation, analytical short-horizon prediction, learned residual dynamics, and posture-aware retargeting around a common robot-independent human-motion state. This formulation separates demonstration representation and prediction from embodiment-specific execution, introducing the target robot’s kinematics only at the workspace-mapping and inverse-kinematics stages.

The main contributions of this work are summarized as follows:

• A unified human-motion interface that combines sparse two-IMU acquisition with a robot-independent canonical task-space representation. Shoulder-relative wrist motion and normalized arm configuration are reconstructed without acceleration integration or direct encoding in target-robot joint coordinates.
• A hybrid analytical–learned prediction formulation in which constant velocity provides the short-horizon motion prior and a compact residual Deep Koopman model learns only the correction not represented by that prior. This formulation retains the strong local behavior of the analytical model while introducing lightweight learned temporal structure over the finite 200-ms prediction horizon.
• A decoupled retargeting formulation in which canonical Cartesian motion and normalized geometric posture are transferred independently of direct human-to-robot joint correspondence, while robot-specific feasibility is handled at the workspace-mapping and inverse-kinematics stages.
• An integrated ROS 2 proof-of-concept combining real wearable human-motion acquisition with online OpenMANIPULATOR-X execution in its official Gazebo simulation model. On held-out complete motion sequences, RDK reduced canonical-position RMSE by 29.36% at 200 ms relative to CV, while the representative posture-aware CE-MR trial showed a substantially lower normalized posture discrepancy than the independently performed RDK trial.

The remainder of this paper is organized as follows. Section 2 reviews human-to-robot motion retargeting, wearable sensing, scalable demonstration acquisition, human-motion prediction, and Koopman-based learning. Section 3 presents the proposed cross-embodiment Physical AI framework. Section 4 describes the experimental setup and evaluation protocol, and presents the quantitative results. Finally, Section 5 discusses the implications and current limitations of the proposed approach and concludes the paper.

2. Related Work

2.1. Wearable Demonstration Acquisition and Cross-Embodiment Representation

Wearable inertial measurement units (IMUs) provide a compact alternative to optical motion-capture systems for acquiring human motion without external cameras or line-of-sight constraints. In human–robot interfaces, Škulj et al. developed an IMU-only wearable system for collaborative robot teleoperation with sensor-frame calibration and relative human-to-robot motion transformation [8]. Kurpath et al. combined IMUs and potentiometers to capture higher-dimensional arm–hand motion for robotic teleoperation [9], while Fornaro et al. demonstrated a ROS-based robot-arm interface using three wearable inertial sensors [10]. At a larger sensing scale, Zhou et al. used an IMU-based motion-capture suit for whole-body teleoperation of a legged manipulator [13]. Reliable reconstruction additionally depends on sensor calibration and human kinematic modeling; Białecka et al. evaluated IMU orientation accuracy using a robotic arm as a reference [12], and Dai et al. investigated simultaneous estimation of human motion and kinematic parameters from inertial measurements [11].

Recent robot-learning research has also emphasized portable and reusable demonstration acquisition. UMI enables in-the-wild human demonstrations to be collected without requiring every demonstration to be executed directly on the target robot [14], while BridgeData V2 and DROID highlight the importance of diverse and scalable demonstration datasets for real-world robot learning [15,16]. At a broader scale, Open X-Embodiment and Octo demonstrate the value of aggregating experience across heterogeneous robot platforms and learning from cross-embodiment data [17,19]. These studies indicate that the representation of collected demonstrations is increasingly important when data are intended to be reused beyond a single platform.

The present work combines these two perspectives through a deliberately sparse two-IMU interface and a robot-independent motion representation. Only upper-arm and forearm orientations are measured, and shoulder-relative wrist motion is reconstructed geometrically using a calibrated two-link forward-kinematic model rather than acceleration integration. The resulting demonstration is expressed in a normalized canonical workspace instead of robot-specific joint coordinates. The current experiments evaluate this representation using the ROBOTIS OpenMANIPULATOR-X and its official Gazebo simulation model, providing an initial validation of joint-independent human-to-robot transfer. The robot-independent formulation reduces dependence on a particular kinematic structure and provides a basis for extending the same representation to heterogeneous robot embodiments.

2.2. Human-Motion Prediction and Residual Deep Koopman Learning

Human-motion prediction can improve human–robot interaction by allowing robots to respond to expected future motion rather than purely reacting to delayed observations. Recent methods have progressed from recurrent and graph-based models toward transformer and diffusion architectures [23,24]. TransFusion, for example, combines transformer-based temporal modeling with diffusion for 3D human-motion prediction [24], whereas Zou showed that simplified neural architectures can retain useful predictive capability with reduced computational complexity [25]. In robotic manipulation, Diffusion Policy further demonstrated the effectiveness of repeatedly predicting short future action sequences in a receding-horizon manner [26]. These approaches motivate predictive robot control, but their complexity may be unnecessary when the state to be predicted is low-dimensional, and inference is embedded directly in a real-time control loop.

Koopman operator theory provides a structured alternative by representing nonlinear dynamics through approximately linear evolution in an appropriate observable space. Brunton et al. established finite-dimensional Koopman-invariant representations for nonlinear systems and control [27], and Lusch et al. showed that deep neural networks can learn nonlinear coordinates with approximately linear latent dynamics [28]. Deep Koopman representations have since been applied to nonlinear systems with control [29], demonstration-based dexterous manipulation [30], nonlinear time-varying dynamics [31], and real-time constrained manipulator control [32]. More recently, O’Neill et al. demonstrated Koopman-based global linearization for contact-rich robot locomotion and manipulation [33]. Most of these studies use Koopman representations primarily for robot dynamics modeling, control, or robot-skill learning.

In contrast, the present work applies Deep Koopman learning to the temporal evolution of the human-motion state before cross-embodiment transfer. Moreover, the network is not required to predict the complete future state from scratch. CV is retained as an analytical prior, and the learned model estimates only the residual correction $\Delta {\mathbf{x}}_{t+k}^{\mathrm{DK}}$, such that

$${\hat{\mathbf{x}}}_{t+k}={\mathbf{x}}_{t+k}^{\mathrm{CV}}+\Delta {\mathbf{x}}_{t+k}^{\mathrm{DK}}.$$

(1)

This residual formulation preserves the strong short-horizon behavior of CV while allowing learned nonlinear corrections to become more influential at longer prediction horizons. The model is further used only over a finite receding horizon, with the current measured human state reinitializing prediction at every control update.

The present CV–RDK comparison is designed to quantify the contribution of the complete residual predictor relative to its analytical CV prior. Within CE-MR, the Deep Koopman formulation provides a compact nonlinear encoding together with linear finite-horizon propagation in the latent space. The results therefore characterize the predictive benefit of the complete CV + RDK formulation over the evaluated horizons, while an architecture-controlled comparison with alternative residual predictors remains an important direction for further isolating the contribution of the Koopman parameterization.

2.3. Cross-Embodiment Motion Retargeting and Posture-Aware Inverse Kinematics

Human-to-robot motion retargeting must generate robot-feasible motion despite differences in morphology, degrees of freedom, joint limits, and reachable workspace. Penco et al. addressed this problem through constrained whole-body optimization for humanoid retargeting [4], while ImitationNet introduced an unsupervised shared latent space to reduce the need for paired human–robot demonstrations [5]. Baek et al. combined monocular human-pose estimation with motion retargeting for a full-scale humanoid robot [6], and Wen et al. formulated constrained motion adaptation to transform human commands into feasible references for collaborative bimanual manipulation [7]. Related LfD studies have also investigated reusable motion representations, including standardized expert harvesting motions learned through a kinematically regularized RNN-VAE [21] and imitation-based reproduction of expert trajectory-generation behavior [22]. Together, these studies demonstrate that embodiment mismatch must be addressed either through optimization, learned mappings, or transferable intermediate representations.

Inverse kinematics provides the final embodiment-specific transformation from task-space motion to robot joint configurations. In redundant or underconstrained tasks, multiple joint configurations may satisfy the same end-effector target, enabling secondary objectives such as joint continuity, joint-limit avoidance, or posture regulation to be incorporated into the IK solution. Recent work has addressed such redundancy through constrained optimization [34], while posture-aware retargeting studies have highlighted the trade-off between reproducing a human-like arm configuration and maintaining Cartesian end-effector accuracy [35]. The proposed method addresses this trade-off without directly equating human and robot joint angles. Human and robot arm configurations are instead described using normalized geometric bend variables, and the IK objective combines Cartesian tracking, joint continuity, and normalized posture correspondence. Thus, posture information is transferred as an embodiment-normalized geometric property while end-effector motion remains the primary retargeting objective.

CE-MR differs from these approaches primarily in where the embodiment transformation is performed. Constrained retargeting methods directly solve for robot configurations that reproduce a human pose under robot-specific constraints, whereas CE-MR first preserves the demonstration in a robot-independent canonical task-space representation and introduces the target-robot kinematics only at the final retargeting stage. In contrast to shared-latent-space methods, CE-MR does not learn a common human–robot latent alignment or require paired human–robot demonstrations. It also differs from imitation-learning approaches that learn a direct policy from demonstrations to robot actions; the present framework retains an explicit analytical mapping from canonical motion to the robot workspace and uses learning only for finite-horizon human-motion prediction. The resulting separation between representation, prediction, and embodiment-specific execution is the central design principle of CE-MR.

3. Cross-Embodiment Motion Retargeting Methodology

We refer to the complete proposed framework as Cross-Embodiment Motion Retargeting (CE-MR). CE-MR integrates sparse wearable sensing, finite-horizon human-motion prediction, canonical cross-embodiment representation, workspace-level retargeting, and posture-aware robot control within a unified human-to-robot motion-transfer framework. Two IMUs provide upper-arm and forearm orientations, from which a shoulder-relative human arm state is reconstructed and represented in a normalized canonical workspace. An RDK model predicts the near-future canonical motion, which is subsequently mapped to the robot workspace and executed through posture-aware inverse kinematics (IK). Importantly, CE-MR does not require direct human-to-robot joint correspondence or geometrically identical initial postures.

In this study, cross-embodiment refers to transferring motion from a human demonstrator to a kinematically distinct target robot without requiring direct joint-to-joint correspondence. The present validation focuses on this human-to-robot transfer setting using the OpenMANIPULATOR-X, while the robot-independent representation provides a basis for subsequent evaluation across heterogeneous robot platforms.

Figure 1 summarizes the complete CE-MR pipeline, from two-IMU human-state estimation and canonical workspace normalization to RDK-based motion prediction, cross-embodiment workspace mapping, and posture-aware robot execution.

Figure 1. Overview of the proposed CE-MR framework from two-IMU human-motion acquisition to predictive cross-embodiment robot execution.

3.1. Two-IMU Human-Motion Acquisition and Canonical Representation

After a T-pose calibration aligns each IMU with its corresponding body segment, let ${\mathbf{R}}_u\left(t\right)$ and ${\mathbf{R}}_f\left(t\right)$ denote the calibrated upper-arm and forearm rotations, and let $L_u$ and $L_f$ denote their segment lengths. With the shoulder frame as the origin, the elbow and wrist positions are

$$\begin{array}{c}\hfill {\mathbf{p}}_e\left(t\right)={\mathbf{R}}_u\left(t\right){\left[\begin{array}{ccc}{L}_u& 0& 0\end{array}\right]}^{\mathsf{T}},\end{array}$$

(2)

$$\begin{array}{c}\hfill {\mathbf{p}}_w\left(t\right)={\mathbf{p}}_e\left(t\right)+{\mathbf{R}}_f\left(t\right){\left[\begin{array}{ccc}{L}_f& 0& 0\end{array}\right]}^{\mathsf{T}}.\hfill \end{array}$$

(3)

Thus, wrist translation is reconstructed geometrically rather than by integrating IMU linear acceleration. The upper-arm and forearm unit vectors are denoted by ${\mathbf{u}}_t$ and ${\mathbf{f}}_t$, respectively, and the normalized human arm bend is

$${\beta }_t={\displaystyle \frac{1}{\pi }}{\cos }^{-1}\left(\mathrm{clip}\left[{\mathbf{u}}_t^{\mathsf{T}}{\mathbf{f}}_t,-1,1\right]\right).$$

(4)

To reduce dependence on subject-specific workspace scale, the shoulder-relative wrist position is transformed into a directional canonical coordinate. Let

$$r_t=\parallel {\mathbf{p}}_w\left(t\right)\parallel ,{\mathbf{d}}_t={\displaystyle \frac{{\mathbf{p}}_w\left(t\right)}{r_t+ϵ}}.$$

(5)

Using an inner radius $r_{\mathrm{in}}$ and a direction-dependent outer workspace radius $r_{\mathrm{out}}\left({\mathbf{d}}_t\right)$ obtained from the human workspace calibration, the normalized radial coordinate and canonical position are defined as

$${\rho }_t=\mathrm{clip}\left({\displaystyle \frac{r_t-r_{\mathrm{in}}}{r_{\mathrm{out}}\left({\mathbf{d}}_t\right)-r_{\mathrm{in}}+ϵ}},0,{\rho }_{\max }\right),{\mathbf{c}}_t={\rho }_t{\mathbf{d}}_t-{\mathbf{c}}_0,$$

(6)

where ${\mathbf{c}}_0$ is the median canonical position during the initial reference interval. The resulting 14-dimensional human state is

$${\mathbf{x}}_t={\left[{\mathbf{c}}_t^{\mathsf{T}},{\dot{\mathbf{c}}}_t^{\mathsf{T}},{\mathbf{u}}_t^{\mathsf{T}},{\mathbf{f}}_t^{\mathsf{T}},{\beta }_t,{\dot{\beta }}_t\right]}^{\mathsf{T}}\in {\mathbb{R}}^{14}.$$

(7)

This representation combines task-space motion with human arm-configuration information without introducing any target-robot joint coordinates. Accordingly, the human-motion representation remains independent of the specific joint structure of the target robot.

3.2. Residual Deep Koopman Motion Prediction

CE-MR employs a CV model as an analytical short-horizon motion prior. For prediction interval ${\tau }_k=k\Delta t$,

$${\mathbf{c}}_{t+k}^{\mathrm{CV}}={\mathbf{c}}_t+{\tau }_k{\dot{\mathbf{c}}}_t,{\beta }_{t+k}^{\mathrm{CV}}={\beta }_t+{\tau }_k{\dot{\beta }}_t,$$

(8)

while canonical velocity, arm directions, and bend velocity are propagated by persistence.

Rather than replacing the analytical CV prior, the RDK predictor learns the residual motion that is not captured by the constant-velocity approximation. Let the standardized human state be

$${\tilde{\mathbf{x}}}_t=({\mathbf{x}}_t-\mu )\oslash \sigma ,$$

(9)

where $\mu$ and $\sigma$ are computed exclusively from the training sequences. The encoder $E(·)$ maps the standardized state into a latent observable ${\mathbf{z}}_t$, which evolves through a trainable linear Koopman operator $\mathbf{K}$:

$${\mathbf{z}}_t=E\left({\tilde{\mathbf{x}}}_t\right),{\mathbf{z}}_{t+k}={\mathbf{K}}^k{\mathbf{z}}_t.$$

(10)

The residual decoder $D(·)$ predicts a correction in normalized feature space. The resulting RDK prediction in the original state space is

$${\hat{\mathbf{x}}}_{t+k}={\mathbf{x}}_{t+k}^{\mathrm{CV}}+\sigma \odot D\left({\mathbf{K}}^{k}E\left({\tilde{\mathbf{x}}}_t\right)\right).$$

(11)

The network uses a $14\to 32\to 16$ encoder and a $16\to 32\to 14$ residual decoder. The final residual layer is initialized to zero so that the untrained RDK model initially reduces to the analytical CV predictor. After each prediction, the predicted upper-arm and forearm direction vectors are re-normalized to unit length.

For a finite horizon of H steps, the model is trained using

$$\mathcal{L}={\mathcal{L}}_{\mathrm{pred}}+{\lambda }_{\mathrm{lat}}{\mathcal{L}}_{\mathrm{lat}}+{\lambda }_{\mathrm{res}}{\mathcal{L}}_{\mathrm{res}},$$

(12)

where

$$\begin{array}{cc}\hfill {\mathcal{L}}_{\mathrm{pred}}& ={\displaystyle \sum _{k=1}^H}{\overline{w}}_k{∥{\widehat{\tilde{\mathbf{x}}}}_{t+k}-{\tilde{\mathbf{x}}}_{t+k}∥}_2^2,\hfill \end{array}$$

(13)

$$\begin{array}{cc}\hfill {\mathcal{L}}_{\mathrm{lat}}& ={\displaystyle \frac{1}{H}}{\displaystyle \sum _{k=1}^H}{∥{\mathbf{K}}^{k}E\left({\tilde{\mathbf{x}}}_t\right)-E\left({\tilde{\mathbf{x}}}_{t+k}\right)∥}_2^2,\hfill \end{array}$$

(14)

$$\begin{array}{cc}\hfill {\mathcal{L}}_{\mathrm{res}}& ={\displaystyle \frac{1}{H}}{\displaystyle \sum _{k=1}^H}{∥D\left({\mathbf{K}}^{k}E\left({\tilde{\mathbf{x}}}_t\right)\right)∥}_2^2,\hfill \end{array}$$

(15)

and ${\overline{w}}_k=w_k/{\sum }_{j=1}^H{w}_j$. The detailed training hyperparameters are provided in Section 4.

During online CE-MR operation, prediction is performed in a receding-horizon manner. The measured human state reinitializes the RDK predictor at every control update, and only the selected finite-horizon prediction is transferred to the robot controller. Based on the validation results, the RDK prediction is used for canonical wrist motion, whereas the analytical CV prediction is retained for the elbow-bend reference.

3.3. Cross-Embodiment Retargeting and Posture-Aware Control

The predicted canonical human motion is transferred to the robot relative to its own reachable workspace rather than through direct human-to-robot joint correspondence. Let ${\mathbf{p}}_r^0$ denote the robot end-effector position measured during session initialization and let

$$\mathbf{G}=\mathrm{diag}(g_x,g_y,g_z)$$

(16)

denote the Cartesian workspace scaling. The desired robot end-effector position in CE-MR is defined as

$${\mathbf{p}}_r^d\left(t\right)={\mathbf{p}}_r^0+\mathbf{G}{\hat{\mathbf{c}}}_{t+\tau }.$$

(17)

Because ${\hat{\mathbf{c}}}_{t+\tau }$ represents session-relative canonical motion, the human and robot are not required to begin from geometrically identical postures. Instead, the demonstrated displacement is anchored to the robot’s own initial end-effector configuration.

Posture is likewise transferred geometrically rather than by associating specific human joints with specific robot joints. Let ${\theta }_r\left(\mathbf{q}\right)$ denote the geometric bend angle between the robot upper-arm-equivalent and forearm-equivalent link vectors. The normalized robot bend is

$${\beta }_r\left(\mathbf{q}\right)=\mathrm{clip}\left({\displaystyle \frac{{\theta }_r\left(\mathbf{q}\right)-{\theta }_{r,\min }}{{\theta }_{r,\max }-{\theta }_{r,\min }}},0,1\right),$$

(18)

where ${\theta }_{r,\min }$ and ${\theta }_{r,\max }$ are obtained from the robust reachable robot-posture range.

For a desired Cartesian position ${\mathbf{p}}^{*}$ and normalized human posture reference ${\beta }^{*}$, CE-MR obtains the robot configuration from

$${\mathbf{q}}^{*}=\arg \underset{\mathbf{q}}{\min }\left\{{∥{\displaystyle \frac{\mathbf{p}\left(\mathbf{q}\right)-{\mathbf{p}}^{*}}{s_p}}∥}_2^2+w_c{∥{\mathbf{W}}_q\mathrm{wrap}(\mathbf{q}-{\mathbf{q}}_{\mathrm{seed}})∥}_2^2+w_{\beta }{\left[{\beta }_r\left(\mathbf{q}\right)-{\beta }^{*}\right]}^2\right\},$$

(19)

subject to the robot joint limits. Feasibility with respect to the specified joint bounds is treated as a hard constraint. Within the feasible set, the Cartesian term represents the primary task objective, while joint continuity and normalized posture correspondence act as secondary penalties in the weighted objective. This weighting reflects the intended design preference for Cartesian task execution over posture resemblance; however, because Equation (19) is a weighted-sum formulation, it does not provide a strict lexicographic hierarchy between these objectives.

The complete CE-MR controller operates online using a stationary-motion gate, target smoothing, Cartesian and joint-step limiting, and workspace projection when a requested Cartesian target cannot be reliably solved by IK. These execution mechanisms do not modify the canonical human representation or the RDK prediction model; instead, they provide practical safeguards for stable robot operation. Their numerical settings are summarized in Section 4.

4. Experiments

The proposed CE-MR framework was evaluated from three complementary perspectives. First, the RDK predictor was evaluated offline on held-out human-motion sequences to determine whether the learned residual improves finite-horizon prediction beyond an analytical CV prior. Second, a representative online component ablation using the official Gazebo simulation model of the ROBOTIS OpenMANIPULATOR-X was conducted to examine the observed effects of analytical prediction, learned residual correction, and posture-aware inverse kinematics (IK). Finally, continuous execution of the complete CE-MR controller on the same simulated OpenMANIPULATOR-X platform was analyzed to evaluate computational feasibility and integrated robot operation.

4.1. Experimental Setup

Human arm motion was acquired online using two E2BOX EBIMU24GV5 (E2BOX, Hanam-si, Republic of Korea) inertial sensors mounted on the upper arm and forearm. The target robot embodiment was the ROBOTIS OpenMANIPULATOR-X. Robot execution was evaluated using the official Gazebo simulation model and ROS 2 control interface provided for the OpenMANIPULATOR-X platform. The complete CE-MR software pipeline was implemented using Ubuntu 24.04, Ros 2 Jazzy 24.04 LTS, Python3.8, and Gazebo. Human-state reconstruction, finite-horizon prediction, workspace retargeting, and robot control were integrated within the same ROS 2 framework.

Figure 2 shows the experimental sensing and robot-side configuration. Two wearable IMUs mounted on the upper arm and forearm provide the human-motion input, while RViz visualizes the reconstructed arm state and transferred pose. Robot execution is evaluated using the official Gazebo simulation model of the OpenMANIPULATOR-X.

Figure 2. Experimental system configuration for wearable human-motion acquisition and OpenMANIPULATOR-X retargeting. In the figure, orange represents the lower part of the human arm, while light blue represents the upper part of the arm; the overlapping text indicates the joint topic names.

After sensor-to-body T-pose calibration, the operator performed free three-dimensional arm motion to establish the human workspace. The robot workspace was estimated independently through forward-kinematic sampling within the OpenMANIPULATOR-X joint limits. The sampled robot configurations were used only to characterize reachable workspace geometry and to support projection of infeasible Cartesian targets; sampled joint configurations were never issued directly as robot commands.

The principal sensing, workspace-normalization, retargeting, and controller parameters used in the experiments are summarized in Table 1.

Table 1. System, retargeting, and controller configuration.

Category	Parameter	Configuration
Human sensing	IMU configuration	2 IMUs (upper arm/forearm)
	Segment lengths	0.30/0.27 m
	Workspace calibration	1897 samples over 38.4 s
Robot	Platform	ROBOTIS OpenMANIPULATOR-X
	Workspace sampling	100,000 FK configurations
	Joint-limit margin	$5^{\circ }$
Control	Nominal control period	0.05 s (20 Hz)
	Human/robot reference	0.5/0.8 s
	Prediction horizon	200 ms
	Maximum Cartesian/joint step	0.05 m/0.15 rad per update
	Command duration	0.06 s
Workspace mapping	$r_{\mathrm{in}}$	5th radial percentile
	$r_{\mathrm{out}}\left(\mathbf{d}\right)$	95th local radial percentile ($k=64$)
	${\rho }_{\max }$	2.0
	$\mathbf{G}$	$\mathrm{diag}(0.24472,0.24826,0.23234)$ m
Posture-aware IK	$s_p$	0.005 m
	$w_c/w_{\beta }$	0.10/1.0
	${\mathbf{W}}_q$	$\mathrm{diag}(2.0,1.0,1.0,1.2)$
	Solver budget	80 function evaluations
	Solver tolerances	${10}^{-6}$

The workspace-normalization parameters were obtained directly from the calibration data. The inner radius $r_{\mathrm{in}}$ was defined as the 5th percentile of the calibrated wrist radial distances. For each motion direction $\mathbf{d}$, $r_{\mathrm{out}}\left(\mathbf{d}\right)$ was estimated as the 95th percentile of the radial distances among the 64 calibration samples with the most similar directions, with a minimum radial margin of $r_{\mathrm{in}}+0.02$ m. The normalized radial coordinate was capped at ${\rho }_{\max }=2.0$.

The Cartesian scaling matrix was $\mathbf{G}=\mathrm{diag}(0.24472,0.24826,0.23234)$ m. Each diagonal component was computed as 0.80 times one half of the corresponding axis-wise 1st–99th percentile span of the sampled robot workspace. For posture-aware IK, the Cartesian residual was normalized by $s_p=0.005$ m, with $w_c=0.10$, $w_{\beta }=1.0$, and ${\mathbf{W}}_q=\mathrm{diag}(2.0,1.0,1.0,1.2)$. The bounded nonlinear least-squares solver used a maximum of 80 function evaluations with $\mathrm{xtol}=\mathrm{ftol}=\mathrm{gtol}={10}^{-6}$.

Five representative human-arm motion categories were considered: lateral, vertical, reach, circle, and pick-like motions. Twenty complete demonstrations were recorded for each category, resulting in 100 motion sequences in total. All 100 demonstrations were acquired from the same operator. All sequences were resampled to the fixed 20-Hz model rate and converted to the 14-dimensional human state defined in Equation (7). Consequently, the held-out evaluation characterizes within-operator generalization to unseen motion sequences, while broader cross-operator generalization remains to be examined through multi-subject evaluation.

Figure 3 provides representative qualitative examples of the acquired human motions and their corresponding CE-MR responses. For each example, the human demonstration is shown together with the reconstructed arm configuration in RViz and the corresponding OpenMANIPULATOR-X response in Gazebo. The displayed examples include T-pose calibration and rightward reach, forward reach, elbow flexion, leftward reach, downward reach, and upward reach.

Figure 3. Representative human motions with corresponding RViz reconstructions and OpenMANIPULATOR-X Gazebo responses.

The dataset was divided at the complete-sequence level using a stratified 80/20 training–validation split. For each motion category, 16 sequences were assigned to training and four sequences to validation, resulting in 80 training and 20 validation sequences overall. Temporal windows originating from the same continuous recording were never distributed across both sets, thereby preventing direct window-level leakage between training and validation.

With a four-step prediction horizon and unit stride, the complete dataset corresponded to approximately 49,728 training windows and 12,432 validation windows. All windows originating from an individual demonstration remained within the same subset. Therefore, the prediction results reported below represent performance on held-out complete demonstrations from the same operator rather than on randomly separated temporal windows. Accordingly, the prediction results characterize sequence-level generalization to held-out demonstrations from the same operator, providing the evaluation setting used in the present finite-horizon prediction study. Cross-operator performance will require subject-level separation in a multi-subject evaluation. The dataset and RDK training configuration are summarized in Table 2.

Table 2. Demonstration dataset and RDK training configuration.

Category	Parameter	Configuration
Dataset	Motion categories	Lateral, vertical, reach, circle, pick-like
	Demonstrations	100 sequences (20 per category)
	Train/validation split	80/20 complete sequences
	Train/validation per category	16/4 sequences
	Train/validation windows	≈49,728/12,432
	Model sampling rate	20 Hz
	Human state dimension	14
Network	Encoder	$14\to 32\to 16$
	Koopman latent dimension	16
	Residual decoder	$16\to 32\to 14$
	Trainable parameters	2270
	Prediction steps	$H=4$
	Prediction horizons	50, 100, 150, 200 ms
Training	Optimizer	Adam
	Learning rate	${10}^{-3}$
	Weight decay	${10}^{-5}$
	Batch size	128
	Maximum epochs	300 with early stopping
	LR scheduler	ReduceLROnPlateau
	Loss coefficients	${\lambda }_{\mathrm{lat}}=0.10$, ${\lambda }_{\mathrm{res}}=0.05$
	Horizon weights	$[1.0,1.2,1.5,2.0]$
	Random seed	1229

Feature-wise normalization statistics were computed exclusively from the training sequences, and model selection was based on the minimum validation objective. The selected checkpoint occurred at epoch 36 with a validation objective of 0.061761. Although the learned Koopman matrix had a spectral radius of 1.2908, the RDK predictor was used only over the fixed four-step horizon and was reinitialized from the measured human state at every control update. The model is therefore not interpreted as an unrestricted long-horizon autonomous rollout model.

The four-step receding-horizon design restricts latent propagation to the evaluated 200-ms interval and refreshes the predictor from the measured human state at every control update. The present analysis therefore focuses on finite-horizon predictive performance at the fixed 20-Hz model rate. Extending the formulation with explicit robustness or stability constraints under sensor noise, calibration uncertainty, and time-varying sampling constitutes an important direction for future Koopman-based prediction.

4.2. Finite-Horizon Human-Motion Prediction

The first experiment evaluated the prediction component of CE-MR. The analytical CV model assumes that the current motion velocity remains approximately constant over a short prediction interval. In contrast, the RDK predictor retains this CV estimate as an analytical prior and learns only the residual correction that cannot be represented by the constant-velocity approximation. Both predictors were initialized from the same measured human state and evaluated on the 20 held-out validation sequences at 50, 100, 150, and 200 ms.

Canonical-position RMSE was selected as the primary metric because the predicted canonical wrist displacement is subsequently transferred to the robot workspace by CE-MR. The normalized RMSE of the complete 14-dimensional human state was additionally evaluated to assess prediction of velocity, upper-arm and forearm directions, and arm-configuration information. For both metrics, lower values represent more accurate future-state prediction.

Table 3 and Figure 4 show a clear horizon-dependent behavior. At 50 ms, CV achieved a lower canonical-position RMSE than RDK, with values of 0.00697 and 0.01160, respectively. This indicates that a local constant-velocity approximation provides a strong analytical prior over a very short horizon. At 100 ms, RDK became slightly more accurate, reducing the canonical-position error by 3.75%. Its advantage increased substantially at 150 and 200 ms, where the corresponding RMSE reductions reached 25.70% and 29.36%, respectively.

Table 3. Finite-horizon prediction performance on the 20 held-out validation sequences.

	Canonical-Position RMSE			Normalized 14-D State RMSE
Horizon	CV	RDK	Reduction	CV	RDK	Reduction
50 ms	0.00697	0.01160	−66.52%	0.13405	0.12376	+7.68%
100 ms	0.01331	0.01281	+3.75%	0.24271	0.18702	+22.94%
150 ms	0.02730	0.02029	+25.70%	0.33464	0.24237	+27.57%
200 ms	0.04506	0.03183	+29.36%	0.41754	0.29798	+28.63%

Figure 4. Finite-horizon prediction performance of the CV and RDK predictors: (a) canonical-position RMSE and (b) normalized 14-dimensional state RMSE.

Figure 4a illustrates how the relative advantage of the two predictors changes with prediction horizon. CV is more accurate at 50 ms, whereas the RDK curve becomes lower from 100 ms onward and the difference increases toward 200 ms. This behavior indicates that the analytical prior is effective for nearly instantaneous motion, while the learned residual correction becomes increasingly beneficial as the prediction interval grows.

The full-state results in Figure 4b provide additional evidence that RDK captures temporal motion structure beyond wrist displacement alone. Unlike the canonical-position metric, RDK reduced the normalized 14-dimensional state RMSE at all four horizons. The reduction increased from 7.68% at 50 ms to 28.63% at 200 ms. At the 200-ms horizon, upper-arm and forearm direction errors were also reduced by 38.74% and 30.49%, respectively.

Elbow-bend prediction was an exception to this general tendency. At 200 ms, CV produced an elbow-angle error of 1.145°, whereas RDK produced 1.166°. Although the difference was small, the analytical estimate remained slightly more accurate for the bend component. Consequently, CE-MR uses RDK for canonical wrist-motion prediction while retaining the CV estimate for the elbow-bend reference.

Taken together, Table 3 and Figure 4 indicate that the advantage of RDK is primarily observed at the longer prediction horizons. The 200-ms horizon was therefore selected for the subsequent online CE-MR retargeting experiments.

4.3. Cross-Embodiment Gazebo Ablation

The second experiment evaluated the contribution of the individual CE-MR components during online cross-embodiment retargeting. Four controller configurations were implemented within the same Gazebo pipeline, as summarized in Table 4.

Table 4. Gazebo component-ablation configurations.

Mode	Canonical Target	Horizon	Posture-Aware IK
Baseline	Current measured state	0 ms	No
CV	Constant velocity	200 ms	No
RDK	CV + learned residual	200 ms	No
CE-MR (Ours)	CV + learned residual	200 ms	Yes

The baseline used the currently measured canonical state without future prediction. CV used a 200-ms analytical constant-velocity prediction. The RDK mode used the same 200-ms CV prior together with the learned Deep Koopman residual correction. Finally, CE-MR, representing the complete proposed configuration, used the same RDK prediction while additionally activating the posture-aware IK objective. Thus, the comparison between RDK and CE-MR was designed to examine the effect of activating the posture-aware objective while retaining the same finite-horizon prediction architecture. Because the human motion was performed independently for each configuration rather than identically replayed, this comparison does not constitute a controlled paired trial.

All four modes used identical human-state representation, workspace calibration, session anchoring, Cartesian scaling, target smoothing, motion limits, IK tolerances, and workspace-fallback logic. One representative lateral-motion trial was performed independently for each mode by the same operator. Because the human motion was independently performed for each configuration, the ablation is used as a representative component-level comparison of the four controller settings rather than as a paired repeated-trajectory experiment.

The principal quantitative results are summarized in Table 5. Workspace fallback and IK success quantify different stages of the controller. The fallback rate denotes the fraction of originally requested Cartesian targets that required workspace projection or replacement before IK because they were not considered reliably solvable. IK success denotes successful solution of the final target presented to the IK solver, including projected fallback targets when fallback was activated. Under this definition, all four configurations maintained 100% final-target IK success despite nonzero fallback rates. P95 denotes the 95th percentile of the frame-wise IK residual. For example, the RDK P95 value of 3.219 mm indicates that 95% of the logged IK solutions in that trial had a Cartesian residual of 3.219 mm or less. Thus, P95 characterizes the upper portion of the observed error distribution and should not be interpreted as a statistical confidence interval.

Table 5. Representative four-method Gazebo cross-embodiment ablation.

Method	Final-Target IK Success	Fallback	Mean IK Error	P95 IK Error	Posture MAE	Effective Rate
Baseline	100%	25.15%	1.357 mm	5.184 mm	0.1291	19.95 Hz
CV	100%	28.35%	0.724 mm	3.804 mm	0.1438	20.00 Hz
RDK	100%	38.73%	0.740 mm	3.219 mm	0.1375	19.97 Hz
CE-MR (Ours)	100%	30.64%	0.728 mm	4.588 mm	0.0708	16.74 Hz

Posture correspondence was evaluated using the normalized geometric bend discrepancy

$$e_{\beta }\left(t\right)=\left|{\beta }_t-{\beta }_r\left({\mathbf{q}}_t\right)\right|,$$

(20)

where smaller values indicate closer correspondence between the normalized human and robot arm configurations.

Figure 5 visualizes the frame-level posture and Cartesian IK-error distributions using empirical cumulative distribution functions (ECDFs). For a horizontal-axis value x, the corresponding vertical value represents the fraction of logged control samples whose error is less than or equal to x. Therefore, for an error metric, a curve located further toward the upper-left corresponds to a larger fraction of samples with relatively small error.

Figure 5. Empirical cumulative distributions of (a) normalized human–robot posture error and (b) Cartesian IK residual for the Gazebo ablation.

The effect of posture-aware retargeting is illustrated by the representative RDK and CE-MR trials. Both configurations used the same RDK prediction architecture, whereas only CE-MR activated the normalized posture objective. The independently performed CE-MR trial had a mean normalized posture error of 0.0708, compared with 0.1375 in the RDK trial, corresponding to an observed difference of approximately 48.5%. Because the human motions were not identically replayed, and only one online trial was available for each configuration, this value is descriptive and should not be interpreted as a statistically significant paired effect.

Figure 5a provides a distribution-level view of this result. The CE-MR curve is shifted toward smaller posture-error values over a substantial portion of the representative trial. This indicates that the reduction is reflected across the frame-level error distribution rather than being determined only by a small number of isolated samples. Relative to Baseline and CV, the mean posture-error reductions were approximately 45.2% and 50.8%, respectively.

In contrast, Figure 5b shows broadly comparable Cartesian IK residual distributions among the predictive configurations. The mean residuals were 0.724 mm for CV, 0.740 mm for RDK, and 0.728 mm for CE-MR. The purpose of posture-aware IK is therefore not to minimize the Cartesian IK residual beyond all other configurations. Rather, Table 5 and Figure 5 together show that CE-MR substantially improves normalized human–robot posture correspondence while preserving a similar level of Cartesian IK accuracy.

Figure 6 provides a temporal interpretation of the aggregate posture-error result. In Figure 6a, the human demonstration is compared with the robot response generated by CE-MR. The shaded region between the two trajectories represents their instantaneous normalized geometric discrepancy.

Figure 6. Time-domain posture correspondence during the representative CE-MR trial: (a) human and robot arm configurations and (b) corresponding normalized posture error.

The CE-MR robot response follows the general direction and relative magnitude of the demonstrated arm configuration, and the discrepancy remains relatively small during several quasi-steady posture intervals. Larger deviations occur mainly when the demonstrated arm configuration changes rapidly. These transitions correspond to the localized peaks in Figure 6b, whereas lower-error intervals are observed when the human posture varies more gradually.

The mean normalized posture error for the representative CE-MR trial was approximately 0.071. The displayed trajectories were lightly smoothed to improve visual readability; however, all quantitative values reported in Table 5 and throughout the text were calculated from the original logged signals.

The workspace-fallback results reveal an additional trade-off. The RDK mode exhibited the highest fallback rate at 38.73%, while CE-MR reduced this value to 30.64% but did not outperform Baseline or CV on this metric. Accordingly, CE-MR should not be interpreted as uniformly superior across all controller objectives. The present design gives operational precedence to feasibility, treats Cartesian end-effector motion as the primary retargeting objective, and uses posture correspondence as an additional secondary objective. The current weighted-sum IK does not strictly enforce this priority ordering. More principled alternatives include hierarchical IK, explicit Cartesian-accuracy constraints, or adaptive posture weighting that decreases as the requested target approaches workspace or joint-limit boundaries.

4.4. Real-Time Integrated Execution

The final evaluation examined the computational feasibility of the complete CE-MR processing pipeline. The nominal controller rate was 20 Hz, corresponding to a 50-ms control period. Mean prediction time was approximately 0.03 ms for CV and 0.14 ms for RDK. Relative to the available 50-ms period, the learned RDK prediction stage therefore introduced only a small computational overhead.

The principal additional computational cost was associated with posture-aware IK. Mean IK solve time increased from approximately 5.92 ms in the RDK mode to 13.85 ms in CE-MR, while the CE-MR P95 solve time was approximately 30.75 ms. To examine the complete timing behavior rather than prediction inference alone, Figure 7 presents the ECDF of the total controller computation time.

Figure 7. Empirical cumulative distribution of controller computation time. The dashed line indicates the 50-ms nominal control budget.

The vertical reference in Figure 7 corresponds to the 50-ms computation budget required by the nominal 20-Hz controller. Baseline, CV, and RDK completed approximately 96.3%, 97.0%, and 96.4% of their logged cycles within this budget, respectively. The similar timing distributions of CV and RDK indicate that the RDK prediction itself is not the dominant real-time bottleneck.

For CE-MR, 76.7% of the logged cycles were completed within 50 ms. The longer computational tail is consistent with the additional nonlinear optimization required by posture-aware IK and with the 16.74-Hz effective rate observed during the representative CE-MR ablation in Table 5.

A tail-latency analysis of the same logged controller computation times showed that the CE-MR P95, P99, and maximum observed cycle computation times were 108.32 ms, 151.94 ms, and 198.76 ms, respectively. For comparison, the corresponding P99 values for Baseline, CV, and RDK were 61.23 ms, 55.92 ms, and 60.79 ms, and their maximum observed computation times were 69.93 ms, 67.62 ms, and 70.21 ms, respectively. These results further indicate that the long computational tail is associated with the complete posture-aware CE-MR configuration rather than with the RDK prediction stage itself.

The measured timing profile characterizes the current CE-MR implementation as a soft real-time system. Most control cycles were completed near the nominal period, whereas the long computational tail introduced by posture-aware IK prevented deterministic 50-ms deadline compliance. Accordingly, safety-critical deployment would require explicit control of the IK computation budget and a bounded-time fallback mechanism.

To distinguish the representative ablation trial from continuous execution under more favorable IK conditions, two additional runs were recorded using the frozen complete CE-MR controller. Both runs maintained 100% final-target IK success, as shown in Table 6. Run 2 achieved an effective rate of 20.02 Hz without invoking workspace fallback, while maintaining a mean Cartesian IK residual of 0.444 mm and a P95 residual of 3.009 mm.

Table 6. Stable integrated execution of the complete CE-MR controller.

Run	Rate	Final-Target IK Success	Fallback	Mean IK Error	P95 IK Error
Run 1	18.93 Hz	100%	8.47%	1.672 mm	4.852 mm
Run 2	20.02 Hz	100%	0%	0.444 mm	3.009 mm

Taken together, Figure 7 and Table 6 show that the complete CE-MR sensing, prediction, retargeting, and posture-aware IK pipeline can operate near the intended 20-Hz rate under favorable IK conditions. However, more demanding posture-aware optimization can produce occasional deadline overruns. The principal computational trade-off is therefore associated with the posture-aware IK stage rather than with RDK inference.

A hard real-time implementation would require explicit control of the IK computation budget. Practical options include imposing a fixed maximum number of solver iterations or an explicit time-out, retaining the previous feasible command when the computation budget is exceeded, and reformulating the posture term using a hierarchical or quadratic-programming IK solver with bounded iteration limits. Another option is to update the posture objective at a slower rate than the primary Cartesian command. These strategies provide a clear path toward bounded-time CE-MR execution and will be considered in future physical-system implementations.

Finally, the reported Cartesian IK residual should not be interpreted as a time-aligned end-to-end trajectory-tracking error. Within the current logging architecture, the robot pose stored in each control cycle precedes execution of the newly generated future target. Consequently, the target and robot pose stored in the same CSV row do not correspond to the same physical instant. Same-row target–robot differences were therefore excluded from end-to-end trajectory-tracking claims.

5. Discussion and Conclusions

This paper presented CE-MR, a lightweight Physical AI framework that transfers human arm motion to a robot without requiring direct human-to-robot joint correspondence. CE-MR combines sparse two-IMU motion acquisition, a normalized canonical human-motion representation, RDK prediction, workspace-level retargeting, and posture-aware inverse kinematics within a single online control framework.

The prediction results clarify the role of the RDK component. At a very short prediction horizon of 50 ms, the analytical CV model provided a stronger canonical-position estimate. However, the benefit of the learned residual increased with prediction horizon, and RDK reduced canonical-position RMSE by 29.36% and normalized 14-dimensional state RMSE by 28.63% at 200 ms relative to CV. These results support the residual formulation used in CE-MR: the analytical CV model provides a simple short-horizon prior, while RDK learns the nonlinear motion variations that become increasingly important at longer horizons. Because the measured human state reinitializes the predictor at every control update, the model is used as a finite-horizon receding predictor rather than as an unrestricted long-horizon dynamical simulator.

The present comparison therefore establishes the predictive benefit of the complete residual CV + RDK formulation over CV within the evaluated finite horizons. A matched comparison with compact non-Koopman residual predictors would further isolate the contribution of the Koopman latent-propagation structure and is included in the planned extension of this work.

A second contribution of CE-MR is the separation of human-motion representation from robot joint coordinates. Human wrist motion is expressed in a normalized canonical workspace and transferred relative to the target robot’s own reachable workspace, while arm configuration is represented through a normalized geometric bend rather than through direct joint-angle matching. In the representative Gazebo ablation, the observed mean normalized human–robot posture discrepancy was 0.0708 in the CE-MR trial compared with 0.1375 in the independently performed RDK trial, corresponding to a 48.5% lower observed value while the mean Cartesian IK residual remained below 1 mm. This result supports posture-aware transfer in the evaluated human-to-OpenMANIPULATOR-X setting and motivates broader validation across robot morphologies with different kinematic structures.

The complete CE-MR pipeline also demonstrated practical online feasibility on the OpenMANIPULATOR-X embodiment. Although posture-aware IK introduced additional computation compared with the RDK-only configuration, stable integrated execution achieved 100% final-target IK success and operation at close to the nominal 20-Hz control rate under favorable IK conditions. Robot execution was evaluated using the official Gazebo simulation model of the OpenMANIPULATOR-X, while the human-motion input was acquired online from physical wearable IMUs. Therefore, the evaluation combines real human-motion acquisition with online control of the target OpenMANIPULATOR-X embodiment within the manufacturer’s simulation framework.

The present study also identifies several directions for extending the validation of CE-MR. First, the current experiments establish joint-independent human-to-robot transfer using the OpenMANIPULATOR-X embodiment. Broader platform-level evaluation can apply the same recorded canonical trajectories to robots with different degrees of freedom, link proportions, joint limits, and end-effector structures, using original-target feasibility, workspace-fallback rate, final-target IK success, Cartesian residual, posture discrepancy, and joint-limit violations as common evaluation criteria.

Second, the current human-motion dataset was collected from a single operator, and the held-out sequence evaluation therefore characterizes within-operator generalization. A broader population-level study should include multiple operators and multiple sessions per operator, together with subject-level train/test separation such as leave-one-subject-out evaluation. Prospective power analysis and subject-level confidence intervals, effect sizes, and repeated-measures statistics would provide a stronger basis for evaluating inter-subject variability.

Third, the timing analysis demonstrates soft real-time execution while also showing that posture-aware IK introduces a long computational tail. Deterministic real-time deployment will require bounded solver iterations or time limits, a deterministic fallback mechanism, or a hierarchical or constrained IK formulation with a predictable computation budget.

Finally, the present RDK evaluation focuses on finite-horizon prediction using a fixed architecture and training configuration. The four-step receding-horizon implementation limits latent propagation and refreshes the prediction from measured human motion at every control update. Future work will examine stability-constrained Koopman formulations, robustness to sensor and calibration uncertainty, hyperparameter sensitivity, and architecture-controlled comparisons with alternative residual predictors. Overall, the results demonstrate that combining an analytical short-horizon motion prior with lightweight residual dynamics learning and embodiment-normalized posture constraints provides a practical approach to predictive human-to-robot motion transfer. CE-MR enables human motion to be acquired online using only two wearable IMUs, represented independently of robot joint structure, predicted over a useful short horizon, and transferred to the OpenMANIPULATOR-X embodiment for online execution in its official Gazebo simulation environment. These properties provide a compact basis for scalable cross-embodiment demonstration acquisition and Physical AI-based robot learning.