Scaling Functional Electrical Stimulation Control for Diverse Users Through Offline Distributional Reinforcement Learning

Abstract

Functional Electrical Stimulation (FES) can restore motor function; however, achieving precise multi-joint control remains challenging due to nonlinear muscle dynamics and fatigue. Reinforcement Learning (RL) offers a promising solution, but practical deployment is hindered by the need for patient-specific calibration. This study investigates offline RL approaches for controlling planar arm movements using heterogeneous datasets, aiming to enable zero-shot transfer to new users. We develop a biomechanical arm model in MuJoCo and evaluate four RL algorithms coupled with three offline techniques: conservative Q learning (SAC-CQL and QBR-CQL), Randomized Ensemble (QBR-REM), and distributional RL (IQNBR). Across all conditions, IQNBR demonstrates robust learning and superior control performance, achieving an average RMSE of $3.8\pm 0.6$ cm, even when trained on mixed-quality data. These results highlight the potential of distributional RL as a base learning method to build generic FES controllers that can operate without exhaustive calibration, with broader implications for controlling robots with human-like actuation systems.

1. Introduction

Upper-limb impairment is one of the most prevalent motor deficits following stroke and spinal cord injury [1,2]. This impairment severely restricts the ability to perform daily activities [3,4]. Functional Electrical Stimulation (FES) is a technique that utilizes low-energy electrical pulses to stimulate muscles and artificially induce muscle contractions, providing a solution to restore arm movement. FES has been applied for rehabilitation purposes [5,6,7,8,9,10] as well as for directly inducing functional movements [11,12,13,14,15]. The latter approach, commonly referred to as movement restoration, is the focus of this study, in which the paralyzed arm is considered as a bio-robotic system.

Inducing functional or goal-directed movements using FES requires precise control of stimulation sequences to guide the arm toward arbitrary targets. From a robotics perspective, this problem is analogous to controlling a redundant tendon-driven robotic manipulator with nonlinear, time-varying actuation. In particular, this control problem entails the coordinated activation of multiple muscles during the movement. Early research in this domain focuses predominantly on single-joint control, such as the elbow joint [16,17,18,19,20]. This is partly because single-joint movements require relatively simple controllers, which can be managed using model-free, error-based control strategies, such as proportional-integral-derivative (PID) controllers. Although these studies demonstrate the successes of single-joint FES control in real-world settings, their applicability to daily activities of the upper limb remains limited [21,22,23].

In contrast, multi-joint controls require more complex control schemes that incorporate dynamical models to account for muscle synergies and joint coordination [24]. Various approaches have been explored in simulation, including open-loop control [25], PID control [26], PID control with neural networks [27], adaptive cooperative control [28], and model predictive control (MPC) [29]. Several approaches demonstrate successes in real-world settings; these include PID control with a Gaussian Process model [14], muscle synergy-based feedback control [30,31], state-machine-based control [15,32], MPC [33,34], trajectory optimization with learned model [23], and iterative learning control [12,35,36,37]. Among these control schemes, iterative learning control achieves the most accurate path tracking [12]. However, it requires repetitive learning of a specific task or trajectory and may not perform tasks that differ from those used for training [37,38]. A muscle synergy-based feedback control produces good reaching movement toward arbitrary targets in a planar setting, with an average error of less than 2 cm [31]. However, it was tested only on able-bodied individuals and required synergy force data from experimental sessions, hindering its applicability outside the laboratory. Several studies improve control accuracy and expand workspace by employing hybrid systems that combine exoskeletal devices with FES, often using MPC-based controllers [39,40].

Recently, reinforcement learning (RL), a machine-learning framework with an agent learning an optimal policy through interaction with the environment to maximize a cumulative reward [41], has gained significant attention for FES control as well as for robotic manipulation. RL has demonstrated a number of promising results in controlling multi-joint arm movements in simulation [21,22,38,42,43,44,45,46,47], but its success in real-world applications remains mostly limited to single-joint movements [20,48,49,50,51]. This is in part due to the low data efficiency of RL, as the large number of interactions required can be impractical in human-in-the-loop systems. Our previous work introduced a physics-informed method that improves sample efficiency, enabling RL to control planar reaching movement toward arbitrary targets in a real-world setting [52]. Despite these advances, several fundamental questions remain: How much data does RL truly need for FES control? Which algorithms are the most suitable? Can data from multiple individuals be used to enable zero-shot transfer without patient-specific calibration?

This study investigates these questions in a base scenario of FES-induced planar arm movements, with the primary objective of investigating zero-shot transfer through offline RL trained on heterogeneous data—a promising yet underexplored approach for FES control. This is important because it demonstrates the feasibility of leveraging data collected from multiple users to learn a controller that can operate without patient-specific calibration, thereby reducing risk, time, and complexity in clinical and home use. The secondary objective is to quantify the data requirements of vanilla RL and to establish the level of control performance that can be expected. This provides baseline capability of vanilla RL in this application. Our contributions are as follows. First, we present a detailed biomechanical model with fatigable muscles developed in MuJoCo [53], allowing fast simulation and reproducible benchmarking. Second, we introduce a configuration that allows the distributional RL—originally designed for discrete actions—to control muscle stimulation in a continuous space. Third, we systematically evaluate several notable RL algorithms in terms of data efficiency and control accuracy. Finally, we analyze the performance of these algorithms when trained only on static, heterogeneous datasets. Since a paralyzed arm can be viewed as a redundant manipulator with nonlinear actuators, these contributions also have broader implications for soft robotics and tendon-driven systems.

2. Biomechanical Model

This section introduces a biomechanical model of planar arm movement which is used to investigate RL algorithms in this study. The model is developed using MuJoCo (v3.3.0) [53], an open-source physics simulator with advanced capabilities, and is based on our previous Arm-Planar model [45] created in OpenSim (v4.5) [54]. The new model, named Arm26-Planar, represents a detailed musculoskeletal linkage similar to those used in prior FES simulation studies [21,22,43,44]. Compared to the OpenSim-based model, the MuJoCo implementation provides approximately a tenfold increase in simulation speed, consistent with the runtime comparison reported in [55]. This improvement enables faster RL training and opens possibilities for real-time applications such as motion planning and model predictive control. The model consists of two main components: a musculoskeletal model and a muscle fatigue model, which are described in the following subsections.

2.1. Musculoskeletal Model of Planar Arm Movement

Figure 1 shows Arm26-Planar, which has the right arm with six muscles connected to a fixed upper body. The shoulder and elbow joints are modeled as pin joints that rotate around vertical axes. These joints include damping and movement limits to prevent unnatural motion. The shoulder and elbow rotation angles (${\theta }_s$ and ${\theta }_e$) are measured relative to the back plane and the upper arm, respectively.

Figure 1. Arm26-Planar model developed in MuJoCo. The left panel shows the back view and the right panel shows the isometric view of the model. The zoomed-in images highlight muscle wrapping around the shoulder and elbow joints. Yellow markers indicate how shoulder (${\theta }_s$) and elbow (${\theta }_e$) angles are measured.

The movement of Arm26-Planar is controlled through a 4-dimensional control vector ($\mathbf{u}\in {\mathbb{R}}^4$), where each element ($u_i\in [0,1]$) represents a normalized stimulation signal. These signals directly influence the activation ($a_m\in [0,1]$) of four primary muscles: Brachialis, Triceps Short Head, Pectoralis Major Clavicular, and Deltoid Posterior. Additionally, The Biceps Long Head and Triceps Long Head receive 50% of the stimulation applied to the Brachialis and Triceps Short Head, respectively. This spillover effect is based on our real-world experiment [52], where surface electrodes placed over the bellies of Biceps and Triceps muscles cover around half of the bellies’ areas, thereby partially triggering Biceps and Triceps Long Head, whose bundles span the entire length of the upper arm. The stimulation causes muscles to activate and generate force. Muscle force production follows the Force–Length–Velocity (FLV) relationship based on Millard et al.’s muscle model [56], expressed as:

$$F(l_m,v_m,u)=(F_l\left(l_m\right)·F_v\left(v_m\right)·a_m+F_p\left(l_m\right))·S_m,$$

(1)

where $F_l(·)$ and $F_v(·)$ are active forces as a function of muscle length $l_m$ and contraction velocity $v_m$, respectively; $F_p(·)$ is a passive muscle force as a function of muscle length. $S_m$ is a peak force scaling constant which can be varied to represent muscle strength. Muscle activation $a_m$ follows a first-order excitation-activation dynamics, described by:

$${\displaystyle \frac{d{a}_m}{dt}}={\displaystyle \frac{u-a_m}{\tau (u,a_m)}},$$

(2)

where $\tau (u,a_m)$ is an effective time constant computed as:

$$\tau (u,a_m)=\left\{\begin{array}{cc}\hfill & {\tau }_{\mathrm{act}}(0.5+1.5a_m)\mathrm{if}u>a_m\hfill \\ \hfill & {\tau }_{\mathrm{deact}}/(0.5+1.5a_m)\mathrm{if}u\le a_m\hfill \end{array}\right.$$

(3)

The default values of the time constant parameters ${\tau }_{\mathrm{act}}$ and ${\tau }_{\mathrm{deact}}$ are $0.01$ s and $0.04$ s, respectively. We changed the value of ${\tau }_{\mathrm{act}}$ to $0.1$ s to capture the delay of FES-induced muscle activation, which is longer than that of naturally-induced activation [57,58].

2.2. Muscle Fatigue Model

To incorporate fatigue behavior into muscle force generation, we introduce a fatigue factor ($f_f\in [0,1]$) into Equation (1), resulting in the modified formulation:

$$F(l_m,v_m,u)=(F_l\left(l_m\right)·F_v\left(v_m\right)·a_m·f_f+F_p\left(l_m\right))·S_m.$$

(4)

In simple terms, when $f_f<1$, the active component of muscle force is reduced. The value of $f_f$ changes dynamically over time, depending on both the duration and intensity of the stimulation. We adopt the three-compartment muscle fatigue model proposed by the Xia and Frey Law [59], which was experimentally validated in [60] and adopted in several FES-related studies [21,46,61].

As illustrated in Figure 2, the Xia and Frey Law’s model conceptualizes a muscle as a combination of fibers in three states: resting, activated, and fatigued. The fractions of muscle fibers in each state are represented by variables $M_R,M_A,M_F\in [0,1]$, with the constraint $M_R+M_A+M_F=1$. The dynamics of each compartment are described by the following first-order differential equations:

$$\begin{array}{cc}\hfill {\displaystyle \frac{d{M}_R}{dt}}& =-C_t+R\times M_F,\hfill \end{array}$$

(5)

$$\begin{array}{cc}\hfill {\displaystyle \frac{d{M}_A}{dt}}& =C_t-F\times M_A,\hfill \end{array}$$

(6)

$$\begin{array}{cc}\hfill {\displaystyle \frac{d{M}_F}{dt}}& =F\times M_A-R\times M_F,\hfill \end{array}$$

(7)

where $C_t$ represents the recruitment of muscle fibers, defined as:

$$C_t=\left\{\begin{array}{cc}u-M_A,\hfill & \mathrm{if}u\ge a\mathrm{and}u-M_A\le M_R\hfill \\ M_R,\hfill & \mathrm{if}u\ge a\mathrm{and}u-M_A>M_R\hfill \\ u-M_A,\hfill & \mathrm{if}u<a.\hfill \end{array}\right.$$

(8)

The transition from the activated to the fatigued states is determined by fatigue rate F and the transition from the fatigued to the resting states is determined by recovery rate R. The values of F and R vary across individuals, depending on their muscle conditions and tolerance to FES. In this study, we use $F=0.217$ and $R=0.0034$, which reflect the muscle of SCI individuals who regularly perform FES-based exercise.

Figure 2. Three-compartment muscle fatigue model introduced by the Xia and Frey Law [59]. The model represents muscle fibers in three states, with transitions governed by recruitment ($C_t$), fatigue (F), and recovery (R) rates. Arrows indicate the direction of state transitions.

3. Reinforcement Learning Controllers

The RL-based FES control system aims to generate stimulation signals that move the arm toward a desired pose or along a specified path. As illustrated in Figure 3, the RL controller acts as the decision-making unit of the closed-loop control, determining control commands (stimulation levels) based on the current arm state and the assigned target. These commands are sent to a stimulator, represented abstractly in this simulation study, which produces electrical pulses and delivers them to the arm via surface electrodes placed over the target muscles. The pulses cause muscle contractions that drive the arm toward the desired position or along the specified path.

Figure 3. Overview of generalized RL-based FES controller for arm movement restoration. The control target pose or path is defined through shoulder and elbow target angles (${\vartheta }_s,{\vartheta }_e$). The RL controller, parameterized by a neural network, receives the target and current arm state, which includes the angles and angular velocity of the shoulder and elbow $({\theta }_s,{\theta }_e,{\dot{\theta }}_s,{\dot{\theta }}_e)$, and determines stimulation intensity command to the stimulator which applies stimulation pulses to the muscles. The RL controller is trained in an offline setting using heterogeneous data, represented by the dots with different colors, that are collected from multiple environments.

In this study, specifically, the control task is to apply a 4-dimensional control signal $\mathbf{u}\in {\mathbb{R}}^4$ ($u_i\in [0,1]$), which contains the muscle stimulation of 4 muscles, to move the arm to a desired pose specified by a target vector $\mathit{\vartheta }\in {\mathbb{R}}^2$ containing shoulder ${\vartheta }_s$ and elbow ${\vartheta }_e$ target angles. The observed state vector of the system comprises the angles and angular velocity of the shoulder and elbow, $\mathbf{o}=[{\theta }_s,{\theta }_e,{\dot{\theta }}_s,{\dot{\theta }}_e]$. This setup focuses on performing reaching toward arbitrary targets that could allow paralyzed individuals to perform motor functions such as reaching toward an object on a table. In this configuration, the number of control dimensions is limited by the space available to place surface electrodes as well as the resolution of surface FES. The examples of real-world implementations of similar configurations can be found in [31,52]. We consider two categories of model-free RL algorithms: those with native continuous action spaces and those with discretized action spaces. The following subsections describe the RL setup of each algorithm category for performing this planar arm control task.

3.1. Reinforcement Learning Setup with Continuous Action

The continuous-action setup is used for actor–critic algorithms, which can naturally handle tasks in continuous spaces without architectural changes. This setup is straightforward. As illustrated in Figure 4, the agents take 6-dimensional state vectors whose elements are the angles and angular velocity of shoulder and elbow joints and target joint angles, expressed as $\mathit{s}=[{\theta }_s,{\theta }_e,{\dot{\theta }}_s,{\dot{\theta }}_e,{\vartheta }_s,{\vartheta }_e]$. The agent computes a 4-dimensional action vector $\mathbf{a}$ with each component squashed into the range [0, 1] using Sigmoid activation, through a neural network that has two hidden layers. This network models the learnable policy ${\pi }_{\varphi }\left(s\right)$ that maps system states to control actions. The calculation is formally expressed as

$$\mathbf{a}=Sigmoid(ReLU(ReLU(\mathbf{s}{\mathbf{w}}_1+{\mathbf{b}}_1){\mathbf{w}}_2+{\mathbf{b}}_2){\mathbf{w}}_3+{\mathbf{b}}_3),$$

(9)

where ${\mathbf{w}}_1,{\mathbf{w}}_2,{\mathbf{w}}_3,{\mathbf{b}}_1,{\mathbf{b}}_2,{\mathbf{b}}_3$ are the weight and bias parameters of the network, collectively denoted as $\varphi$; and $ReLU\left(x\right)=max(x,0)$ is a rectified linear unit used as activation functions of the hidden layers.

Figure 4. A schematic diagram illustrating the continuous action setup for controlling the movement of Arm26-Planar. The policy network, parameterized by a fully-connected neural network that have 2 hidden layers with ReLU activation function, takes a 6-dimensional state vector comprising joint angles, angular velocities, and target angles, and outputs a 4-dimensional stimulation vector using Sigmoid activation. The stimulation vector is applied to 4 muscles on the right arm.

To learn ${\pi }_{\theta }$, we consider two notable actor-critic algorithms: Soft Actor-Critic (SAC) [62] and Twin Delayed Deep Deterministic Policy Gradient (TD3) [63]. These algorithms are used in several RL-controlled FES studies. Specifically, TD3 is used in [21,22,43,44], which are closely related studies that apply RL to control planar arm movements. SAC is the algorithm used in our previous studies [45,46,64] that, based on our empirical experiments, demonstrates robustness in a wide range of tasks and environments. SAC was also successfully applied to control elbow movement in the real world [50,65].

The learning procedure of SAC and TD3 follows their standard implementations, in which the agent comprises an actor (policy) and a critic parameterized by neural networks ${\pi }_{\varphi }\left(s\right)$ and $Q_{\theta }(s,a)$, with learnable parameters $\varphi$ and $\theta$, respectively. For TD3, the parameters $\theta$ are optimized via gradient descent to minimize temporal difference (TD) error:

$${\delta }_{TD3}=r+\gamma Q_{\theta }(s^{\prime },a^{\prime })-Q(s,a),a^{\prime }\sim {\pi }_{\varphi }\left(s^{\prime }\right),$$

(10)

where r is an immediate reward; and $\gamma \in [0,1)$ is a discount factor. The policy parameters $\varphi$ are updated using gradient assent as

$$\varphi \leftarrow \varphi +l_r{\nabla }_{\varphi }{Q}_{\theta }(s,{\pi }_{\varphi }\left(s\right)),$$

(11)

where $l_r$ is a learning rate. For SAC, the parameters $\theta$ are optimized to minimize TD error:

$${\delta }_{SAC}=r+\gamma (Q_{\theta }(s^{\prime },a^{\prime })-\alpha log{\pi }_{\varphi }\left({\tilde{a}}^{\prime }\right|s^{\prime }))-Q_{\theta }(s,a),{\tilde{a}}^{\prime }\sim {\pi }_{\varphi }(·|s^{\prime }),$$

(12)

where $\alpha =1$ is a temperature coefficient. SAC’s policy parameters are then updated through gradient assent as:

$$\varphi \leftarrow \varphi +l_r{\nabla }_{\varphi }(Q_{\theta }(s,\tilde{a})-\alpha log{\pi }_{\varphi }\left(\tilde{a}\right|s)),\tilde{a}\sim {\pi }_{\varphi }(·|s).$$

(13)

3.2. Reinforcement Learning Setup with Discretized Action

Vanilla Q-learning and distributional RL are designed for discrete action spaces. To adapt them to the continuous action space required for this task, we use an action branching architecture [66]. This approach organizes Q network’s outputs into D branches, where the number of branches is equal to the dimension of action. Each branch has n sub-actions which represent the discretization of continuous action. The details of Q-learning and distributional RL with action branching are described as follows.

3.2.1. Q-Learning with Action Branching

The learning mechanism of Q-learning with action branching, denoted as QBR, is described as follows. Conventional Q-learning learns the action-value function, also known as the Q-value, defined as $Q(s,a)=\mathbb{E}\left[{\sum }_{t=0}^{\infty }{\gamma }^{t}r(s_t,a_t)\right]$, where $r(s_t,a_t)$ is the immediate reward corresponding to the state $s_t$ and the action $a_t$ at time t. The dueling architecture [67] introduces an advantage function defined as $A(s,a)=Q(s,a)-V\left(s\right)$, where $V\left(s\right)$ is a value function which is also known as the expected return. The Q-value can be expressed with the advantage function as

$$Q(s,a)=V\left(s\right)+A(s,a).$$

(14)

The value of $Q(s,a)$ can be approximated by a parameterized function such as a neural network that is trained by minimizing TD error:

$$\delta ={[\underset{\mathrm{TD}\mathrm{target}}{\underbrace{r+\gamma \widehat{Q}(s^{\prime },\underset{a^{\prime }\in A}{argmax}Q(s^{\prime },a^{\prime }))}}-Q(s,a)]}^2,$$

(15)

where $\widehat{Q}$ denotes a target network. The action branching architecture simply arranges the action-value function into D branches. Formally, for an action dimension $d\in \{1,\dots ,D\}$ and sub-action $a_d\in A_d$ ($|A_d|=n$), the action-value function (Equation (14)) can be expressed as

$$Q_d(s,a_d)=V\left(s\right)+A_d(s,a_d),$$

(16)

where $Q_d$ denotes the branch d of the Q network and $a_d$ denotes sub-action in the branch. Alternatively, we can locally subtract each branch’s mean advantage from its sub-action advantages which, based on [66], gives better performance than naively adding the sub-action advantages to the state value. This alternative can be expressed as

$$Q_d(\mathbf{s},a_d)=V\left(\mathbf{s}\right)+(A_d(\mathbf{s},a_d)-{\displaystyle \frac{1}{n}}\sum _{a_d^{\prime }\in A_d}{A}_d(\mathbf{s},a_d^{\prime })),$$

(17)

The state and advantage value functions are parameterized by fully-connected neural networks which are optimized in a typical Q learning fashion to minimize the temporal difference loss averaged over the branches [66] as:

$$\mathcal{L}={\mathbb{E}}_{(s,a,r,s^{\prime })\sim \mathcal{D}}\left[{\displaystyle \frac{1}{D}}\sum _d{(y_d-Q_d(s,a_d))}^2\right],$$

(18)

where a denotes joint-action tuple $(a_1,\dots ,a_D)$. y is a temporal difference target expressed as

$$y_d=r+\gamma {\widehat{Q}}_d(s^{\prime },\underset{a_d^{\prime }\in A_d}{argmax}{Q}_d(s^{\prime },a_d^{\prime })),$$

(19)

where ${\widehat{Q}}_d$ denotes the branch d of the target network $\widehat{Q}$. In practice, the temporal difference loss (Equation (18)) is approximated using mini-batch $\mathcal{B}$ sampled from $\mathcal{D}$, expressed as:

$${\mathcal{L}}_{\mathrm{batch}}={\displaystyle \frac{1}{\left|\mathcal{B}\right|}}\sum _{(s,a,r,s^{\prime })\in \mathcal{B}}\left[{\displaystyle \frac{1}{D}}\sum _d{(y_d-Q_d(s,a_d))}^2\right].$$

(20)

The QBR setup for FES control uses the same 6-dimensional input as the continuous-action approach, consisting of joint angles, angular velocities, and target angles. On the output side, the Q-network is organized into four branches—one for each muscle. Each branch has 11 sub-actions that discretize $a_i\in [0,1]$ at 0.1 increments: $A_d=\{0.0,0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9,1.0\}$. Note that finer discretization can provide finer control but may slow the learning. The action vector is formed by putting together the sub-actions with the highest Q-values in each branch. This gives a 4-dimensional control vector, which is the stimulation applied in each time step.

3.2.2. Distributional Reinforcement Learning with Action Branching

For distributional reinforcement learning, we implement Implicit Quantile Network (IQN) [68], one of the most widely adopted distributional RL algorithms whose raw performance is comparable to that of RAINBOW [69]–the combined advancement in deep Q-learning. Similar to Q-learning, IQN originally operates in discrete action spaces and therefore requires an action branching architecture to perform this FES control task. This configuration is denoted as IQNBR, and its learning mechanism is described as follows.

Distributional RL considers the distribution over returns $Z(s,a)$ instead of the scalar action-value function used in conventional Q-learning. In IQN [68], the distribution is approximated by a parameterized function, a neural network, that models the state-action quantile function $Z_{\tau }(s,a)$ at quantile $\tau$. The network takes the element-wise product of state $\mathbf{s}$ and an embedding quantile $\varphi \left(\tau \right)$. This is expressed as

$$Z_{\tau }(s,a)=f{(\mathbf{s}\odot \varphi \left(\tau \right))}_a,$$

(21)

where $\varphi \left(\tau \right)$ is a single layer neural network of size e ($e=64$ [68]) with ReLU activation functions that transform e cosine basis functions ($cos\left(\pi i\tau \right)$, $i\in \{1,2,\dots ,e\}$) into a vector with the same dimension as $\mathbf{s}$. IQNBR simply arranges $Z_{\tau }(s,a)$ into D branches, denoted as $Z_{\tau ,d}(s,a_d)=f{(\mathbf{s}\odot \varphi \left(\tau \right))}_{a_d}$. The parameters of both $\varphi (·)$ and $f(·)$ are trained by gradient descent to minimize the following loss function:

$$\mathcal{L}={\mathbb{E}}_{(s,a,r,s^{\prime })\sim \mathcal{D}}\left[{\displaystyle \frac{1}{D}}\sum _d\left({\displaystyle \frac{1}{N^{\prime }}}\sum _{i=1}^N\sum _{j=1}^{N^{\prime }}{\rho }_{{\tau }_i}^{\kappa }\left({\delta }_d^{{\tau }_i,{\tau }_j^{\prime }}\right)\right)\right],$$

(22)

where N and $N^{\prime }$ denote the number of samples ${\tau }_i,{\tau }_j^{\prime }\sim U\left([0,1]\right)$. ${\delta }_d^{\tau ,{\tau }^{\prime }}$ is temporal difference (TD) error computed as

$${\delta }_d^{\tau ,{\tau }^{\prime }}=r+\gamma {\widehat{Z}}_{{\tau }^{\prime },d}(s^{\prime },\underset{a_d^{\prime }\in A_d}{argmax}{Q}_d(s^{\prime },a_d^{\prime }))-Z_{\tau ,d}(s,a_d),$$

(23)

where $\widehat{Z}$ denotes a target network and the action-value function is computed as

$$Q_d(s,a_d)=\underset{\tau \sim U\left(\right[0,1\left]\right)}{\mathbb{E}}\left[Z_{\tau ,d}(s,a_d)\right].$$

(24)

${\rho }_{\tau }^{\kappa }(·)$ is the Huber quantile regression loss with threshold $\kappa$,

$$\begin{array}{cc}\hfill {\rho }_{\tau }^{\kappa }\left(\delta \right)& =|\tau -\mathbb{I}\{\delta <0\}|{\displaystyle \frac{{\mathcal{L}}_{\kappa }\left(\delta \right)}{\kappa }},\mathrm{with}\hfill \\ \hfill {\mathcal{L}}_{\kappa }\left(\delta \right)& =\left\{\begin{array}{c}{\displaystyle \frac{1}{2}}{\delta }^2,\mathrm{if}\delta \le \kappa \hfill \\ \kappa (|\delta |-{\displaystyle \frac{1}{2}}\kappa ),\mathrm{otherwise}.\hfill \end{array}\right.\hfill \end{array}$$

Similarly to QBR, in practice, the loss expression (Equation (22)) is approximated using a mini-batch $\mathcal{B}$, expressed as:

$${\mathcal{L}}_{\mathrm{batch}}={\displaystyle \frac{1}{\left|\mathcal{B}\right|}}\sum _{(s,a,r,s^{\prime })\in \mathcal{B}}\left[{\displaystyle \frac{1}{D}}\sum _d\left({\displaystyle \frac{1}{N^{\prime }}}\sum _{i=1}^N\sum _{j=1}^{N^{\prime }}{\rho }_{{\tau }_i}^{\kappa }({\delta }_d^{{\tau }_i,{\tau }_j^{\prime }})\right)\right],$$

(25)

We also apply the dueling architecture in the implementation. The setup of IQNBR is therefore similar to that of QBR (Figure 5), where the advantage value is $Z_{\tau ,d}(s,a_d)$, and the Q-value is calculated using Equation (24).

Figure 5. A schematic diagram illustrating the discretized action setup for controlling Arm26-Planar. The output layer is organized into four branches, each containing 11 discrete sub-actions representing stimulation levels from 0.0 to 1.0 in 0.1 increments. The final 4-dimensional control vector is formed by selecting the highest-valued sub-action from each branch.

3.3. Offline Reinforcement Learning

This study also explores offline training which poses challenges on many RL algorithms. The performance of an offline-trained agent can vary significantly depending both on the choice of algorithm and the characteristics of the dataset [70,71,72]. For instance, standard algorithms such as SAC and TD3 tend to perform well when trained on datasets containing a diverse range of random actions. However, their performance may degrade when trained on datasets collected exclusively from expert demonstrations [70].

Several strategies have been proposed to enable standard RL agents to be effectively trained in offline settings, including the use of personalized simulators [73] and federated learning frameworks [74]. In this study, we focus on prominent approaches that are closely aligned with the previously discussed algorithms and require minimal architectural modifications. Specifically, we examine conservative Q-learning (CQL) [72] and Q-learning with Randomized Ensembles (REM) [71] which demonstrated robust performance across diverse environments [73]. These two algorithms use different approaches to mitigate overestimations of Q values at unseen actions which subsequently lead to bootstrapping errors and the failure of offline learning [70]. In simple terms, CQL mitigates the overestimations by penalizing Q values at unseen actions. In contrast, REM uses an ensemble of Q networks whose outputs are randomly combined to estimate Q values. This is expressed as

$$Q(s,a)={\displaystyle \frac{1}{K}}\sum _k{\alpha }_k{Q}_k(s,a),$$

(26)

where $Q_k(s,a)$ is the output of the k-th Q network in an ensemble of K networks and ${\alpha }_k\sim U\left([0,1]\right)$, with the constraint ${\sum }_k{\alpha }_k=1$. Intuitively, this strategy results in the estimations of Q values at unsupported state-action pairs to be merely random values, which are unlikely to be the highest Q values that become TD targets (Equation (19)). In addition, as demonstrated in several studies, distributional reinforcement learning (RL) algorithms have shown strong performance in offline settings [71,72]. Therefore, we include IQNBR among the candidate methods to evaluate its effectiveness under offline training conditions.

3.4. Training Setups and Scenarios

We explore the behaviors and performances of the RL algorithms in two scenarios: online and offline training. The former is a conventional RL training which provides a baseline performance that RL agents can achieve. The offline training scenario explores the potential of developing a generic agent capable of operating immediately in a new environment without additional data collection. Both scenarios are described as follows.

3.4.1. Online Training

Online training follows a standard RL training paradigm, where an RL agent interacts with an environment, collects experience tuples, and learns to control the system. In this study, the training is episodic; each episode begins with a random arm pose and a randomly selected target, where initial shoulder and elbow angles are sampled from uniform distributions: ${\theta }_s,{\theta }_e\sim U\left([0^{\circ },{10}^{\circ }]\right)$. Each episode consists of 30 time steps, with a time-step duration of 100 ms, resulting in a total movement duration of 3 s. Note that the time-step size of 100 ms is to ensure that the system remains Markovian. This consideration is crucial because FES induces muscle activation with a delay of approximately 100 ms [57,58], which is characterized by the value of $\tau$ in Equation (3). If the time steps were shorter, the effects of stimulation might not be fully reflected in the resulting movement.

At each time step, the agent receives an immediate reward that penalizes the squared error in the joint space and the magnitude of the action taken. This is formalized as:

$$r_t=-{({\mathit{\theta }}_{t+1}-{\mathit{\vartheta }}_t)}^2-{\displaystyle \frac{{\sum }_{i=0}^n{a}_i^2}{n}},$$

(27)

where $n=4$ is the number of stimulated muscles. This reward function, which was also used in our previous work [45,52], encourages a policy ${\pi }^{*}$ that brings the joint angles to the targets with minimal stimulation. This function differs from the sparse rewards used in related studies [21,43,44] where the agent only perceives whether it reaches the goal. In contrast, dense rewards give continuous feedback at every step, helping agents learn faster. For this reason, we adopt a dense reward function to examine the minimal amount of data needed to learn the arm control task.

The online training procedure is summarized in Algorithm 1. The training comprises 200 episodes ($N_T=200$) in which the agent is assigned to reach a random pair of target joint angles. Each episode is divided into 2 phases: interaction and update. The interaction phase involves the agent observing the current state of the environment and performing action at each time step. The interaction data are collected in the replay buffer $\mathcal{D}$. The update phase begins by sampling a mini-batch of $N_b$ experience tuples $(\mathbf{s},\mathbf{a},r,{\mathbf{s}}^{\prime })$ from the replay buffer $\mathcal{D}$ and computing the average loss over the samples. The network parameters of the agent’s networks are then updated via a gradient descent technique.

After each episode, the agent’s performance is evaluated on two tasks: circular tracking and a 100-target reaching task. In the tracking task, the agent is assigned to track a circular path whose center is located in the middle of the workspace. This task includes 150 time steps (15 s of movement), with the first 30 steps dedicated to moving the hand to the starting point of the path. During tracking, the agent receives continuously changing target angles of the shoulder and elbow (${\vartheta }_s,{\vartheta }_e$), which guide its movement along the path. In the reaching task, the agent is assigned to perform 100 reaching trials toward 100 different targets, which are defined by a set of target shoulder and elbow angles obtained from the Cartesian product of ${\vartheta }_s,{\vartheta }_e\in \{10,\dots ,100\}$. Performance is evaluated in terms of hand location error in Cartesian coordinates, calculated over the last second of each trial.

Algorithm 1 Online training of RL-based FES controller for planar arm movement

INPUT: episode length: $L_e$, # training episode: $N_T$, mini-batch size: $N_b$, Environment: $Env$, Initialized replay buffer: $\mathcal{D}$, RL Agent: $Agent$ for $episode\leftarrow 1$ to $N_T$ do:  $\mathbf{o}\leftarrow Env.reset\left(\right)$ \\reset the environment with initial $f_f\sim U[0.5,1.0]$  ${\vartheta }_s,{\vartheta }_e\sim U[0,\pi /3]$ \\get random target joint angles  \\Interaction (Data collection) phase  for $step\leftarrow 1$ to $L_e$ do:   $\mathbf{s}=[\mathbf{o};{\vartheta }_s;{\vartheta }_e]$ \\make state vector   $\mathbf{a}\leftarrow Agent.get\_action\left(\mathbf{s}\right)$\\$ϵ$-greedy policy   $\mathbf{o},r\leftarrow Env.step\left(\mathbf{a}\right)$ \\apply the action to the environment   ${\mathbf{s}}^{\prime }=[\mathbf{o};{\vartheta }_s;{\vartheta }_e]$ \\make state vector   $\mathcal{D}.collect(\mathbf{s},\mathbf{a},r,{\mathbf{s}}^{\prime })$ \\collect transition data in the replay buffer   $\mathbf{s}\leftarrow {\mathbf{s}}^{\prime }$ \\prepare for the next step  \\Update phase  for $grad\_step\leftarrow 1$ to $L_e$ do   Sample mini-batch of $N_b$ transitions $(\mathbf{s},\mathbf{a},r,{\mathbf{s}}^{\prime })$ from $\mathcal{D}$   Compute loss over the mini-batch \\(Equation (20) for QBR; and Equation (25) for IQNBR)   Update agent’s networks using gradient descent  Perform performance evaluation (reaching and tracking tasks)

All agents are implemented using PyTorch (v2.4.1) library, which provides automatic differentiation for efficient gradient computation. All neural networks’ parameters are trained using Adam optimizer with a learning rate ($l_r$) of 0.0003 and a mini-batch size ($\left|\mathcal{B}\right|$) of 256, following the implementation in [62].

3.4.2. Offline Training

Offline training simulates a scenario in which an RL agent is trained using datasets collected from multiple individuals and then transferred to a new user. To emulate such a scenario, we create multiple variants of Arm26-Planar, each with distinct muscle properties representing individuals with different muscle conditions and responses. Specifically, we create 7 Arm26-Planar variants with the following properties: typical, strong, very strong (v-strong), weak (weak-all), weak Pectoralis (weak-pect.), weak Biceps (weak-bic.), and no spillage muscles. The no spillage variant simulates a condition in which stimulation of monoarticular muscles does not affect nearby biarticular muscles. In other words, the stimulation targeting Brachilis and Triceps Short Head does not affect Biceps and Triceps Long Head. These variations are inspired by the scenarios described in [21], where the v-strong and weak-all profiles represent healthy individuals and those with chronic spinal cord injury (SCI), respectively. Additionally, strong and typical profiles correspond to SCI individuals with and without regular physical exercise.

We collect heterogeneous datasets from all variants through online training (Algorithm 1). As a result, the datasets contain a wide range of movement trajectories and stimulation patterns. Early-stage data consist primarily of failed or poorly-executed trajectories with random or inefficient stimulation. In contrast, the data from later stages have more well-performed movements and near-optimal stimulation strategies. This diversity enables a detailed investigation into how the characteristics of the datasets influence the performance of RL agents trained in an offline setting. Note that the online and offline training scenarios are carried out independently. Online training is used only to generate aggregated datasets and no trained agent is transferred between scenarios.

4. Results

We evaluated RL agents in two scenarios: online training and offline training. Online training investigates the amount of interaction data required to learn the task from scratch and establishes a performance baseline. Offline training assesses the ability of agents to learn generalized policies from heterogeneous datasets without additional interaction.

4.1. Online Training in a Single Environment

This experiment has SAC, TD3, QBR, and IQNBR learn the control task on three Arm26-Planar variants, following the setup described in Section 3.4.1. Figure 6 shows the learning curves of all algorithms in the reaching and tracking tasks. SAC has the fastest improvement in the early period (episode $<30$) in both tasks, particularly in typical and strong profiles. TD3 is the worst performer in the early period, showing large fluctuations and delayed convergence. QBR has moderate learning and final performance but lacks consistency, performing poorly in the weak muscle profile. IQNBR has steady progress and strong robustness in all profiles, achieving the lowest RMSE (≈3 cm) and the standard deviation (≈0.6 cm) in both tasks. This RMSE is roughly $0.5$ cm higher than the fatigue-free baselines (dashed lines). Regarding data requirement, all agents reach their performance plateau within 80–100 episodes. This amounts to 4–5 min of interaction data.

Figure 6. Learning curves of SAC, TD3, QBR, and IQNBR under fatigable muscle conditions compared to fatigue-free baselines. Panels (a–c) show reaching RMSE and panels (d–f) show tracking RMSE across 200 episodes on Arm26-Planar with (a,d) typical, (b,e) strong, and (c,f) weak muscles. Solid lines and shaded regions represent the mean standard deviation over 5 runs.

Reaching accuracy toward 100 targets is shown in Figure 7. Each dot represents a target, color-coded by RMSE during the last second of movement, with dot size indicating variability across five runs. SAC has relatively uniform performance with moderate errors, with relatively high RMSE in the right region. QBR exhibits uneven performance with clusters of high error and variability and has the highest average standard deviation of 3.5 cm. TD3 performs slightly better than QBR, particularly in the central and left regions. IQNBR achieves the most consistent accuracy across the workspace, maintaining RMSE below 5 cm for most targets with the lowest average standard deviation of under 2.2 cm.

Figure 7. Reaching performance maps of (a) SAC, (b) TD3, (c) QBR, and (d) IQNBR after 200 online training episodes on typical Arm26-Planar. Each dot represents a target location, color-coded by RMSE during the last second of movement. Dot size indicates variability across five runs. The origin $(0,0)$ corresponds to the shoulder position.

Example tracking behaviors of the four algorithms are shown in Figure 8. The top row illustrates hand trajectories compared to the circular target path. The bottom row shows the stimulation applied to four muscles during tracking. SAC has reasonable tracking with smooth trajectories. However, it suffers from overshoot and prolonged deviations, indicating limited adaptation to fatigue. TD3 produces shakier and less stable trajectories, while QBR struggles to apply appropriate stimulation, resulting in irregular and unstable paths. In contrast, IQNBR delivers the most consistent and precise trajectories, closely following the target circle. Regarding stimulation strategies, SAC applied minimal stimulation compared to the others, which leads to under-correction during deviations. TD3 has inconsistent stimulation levels, further explaining its unstable trajectories. QBR and IQNBR apply frequent and abrupt stimulation changes; however, IQNBR’s stimulation is more effective, enabling accurate tracking with the lowest RMSE of below 2.5 cm.

Figure 8. Example tracking behaviors of (a) SAC, (b) TD3, (c) QBR, and (d) IQNBR after 200 training episodes. The blue curves in the top-row figures show 5 example trajectories of the hand, and the black circles show tracking target trajectories. The bottom-row figures show the stimulation applied to four muscles along the tracking.

4.2. Offline Training with Heterogeneous Datasets

In this scenario, we evaluate the performance of four algorithms: SAC with conservative Q-learning (SAC-CQL), QBR with conservative Q-learning (QBR-CQL), QBR with randomized ensemble (QBR-REM), and vanilla IQNBR. Note that TD3 is excluded in this scenario due to its poor performance in online training. Offline learning depends on both the size and the quality of the dataset. To systematically assess performance and robustness, we train RL agents on multiple datasets of various sizes and characteristics, which are described in the following subsections.

4.2.1. Datasets and Learning Behaviors

Heterogeneous datasets are generated from multiple variants of the Arm26-Planar model, each representing distinct muscle conditions and stimulation responses. Primary data were collected by training SAC, QBR, and IQNBR in online settings, with three repetitions to reduce bias. As a result, the datasets contain a wide range of movement trajectories and stimulation patterns, enabling a comprehensive evaluation of offline RL algorithms. However, in real-world cases, collecting large datasets through online training is impractical. One reason is that having an untrained agent randomly stimulate a human arm might raise an ethical and safety issue. Another reason is that there is no guarantee that the control system can deliver precise control, as seen after extensive training (e.g., 150 episodes in Figure 6). Considering these factors, we organized the primary data into five datasets, focusing on the data generated between episodes 50 and 100, where the agents have decent performance. This range reflects a realistic scenario in which pre-trained agents are deployed in the real world. The datasets are defined as follows:

• Small: Episodes 50–55 (5 trials per variant; 1050 transitions),
• Medium: Episodes 50–75 (25 trials per variant; 5250 transitions),
• Large: Episodes 50–100 (10,500 transitions),
• Medium-Expert: Episodes 175–200 (5250 near-optimal transitions),
• Very Large: Episodes 50–200 (31,500 transitions).

Note that the medium dataset aligns with the real-world experiments in our previous work [52], where fine-tuning data were collected through 25 reaching trials, each lasting 3 s. These datasets enable systematic evaluation of how dataset size and quality affect offline RL performance and generalization across diverse muscle profiles.

Figure 9 shows principal component analysis (PCA) projections of the five datasets, illustrating the distribution of 8-dimensional state–action pairs (excluding target joint angles ${\vartheta }_s,{\vartheta }_e$) compared to the entire primary data. The small dataset has the narrowest distribution due to its size (Figure 9a), while the very large dataset almost completely overlaps the primary data, indicating a broad coverage (Figure 9e). The medium and large datasets (Figure 9b,c) progressively expand the coverage from the small dataset, capturing more diversity in state–action pairs. In contrast, the medium-expert dataset (Figure 9d) has slightly smaller coverage than the medium dataset because it mainly consists of near-optimal trajectories rather than diverse transitions.

Figure 9. Principal component analysis (PCA) projections of state–action pairs in five datasets compared to the whole primary data (black). Blue points represent (a) small, (b) medium, (c) large, (d) medium-expert, and (e) very large datasets.

Offline learning experiments follow a leave-one-out approach, where data from an Arm26-Planar variant are excluded when that variant is used for evaluation. This simulates the transfer of the trained agent to an unseen environment. Figure 10 shows the learning curves of four offline algorithms evaluated in typical Arm26-Planar. When trained on the small dataset (Figure 10a), all algorithms converge slowly and exhibit high variability, indicating insufficient coverage of transitions. IQNBR and QBR-CQL have stable and robust progress. In contrast, QBR-REM fails to learn the task with the small dataset. As the dataset size increases to medium and large (Figure 10b,c), the learning improves substantially for all algorithms except SAC-CQL, which stagnates around an RMSE of 10 cm. QBR-REM benefits from larger datasets, eventually matching IQNBR’s accuracy and even slightly surpassing it in the reaching task when trained on the very large dataset (Figure 10e). SAC-CQL and QBR-CQL, however, continue to show limited improvement. An interesting observation occurs with the medium-expert dataset (Figure 10d). Here, QBR-REM underperforms other algorithms, while SAC-CQL and QBR-CQL learn quickly and overcome the stagnation seen in other cases, converging to performance levels similar to IQNBR. Overall, IQNBR exhibits stable learning and the strongest final performance across datasets, indicating robustness to both dataset size and quality. When trained on all datasets except the small one, IQNBR converges to an RMSE between 3.5 cm and 3.8 cm, which is approximately 0.8 cm higher than its performance under conventional online training. This gap illustrates a limitation of offline training and zero-shot transfer without additional personalized calibration.

Figure 10. Offline learning curves of SAC-CQL, QBR-CQL, QBR-REM, and IQNBR trained on (a) small, (b) medium, (c) large, (d) medium-expert, and (e) very large datasets. The curves show RMSE in (top row) reaching and (bottom row) tracking tasks, evaluated on the typical Arm26-Planar with fatigable muscles, over gradient steps.

4.2.2. Performance at Convergence

This section examines the accuracy achieved across the workspace and the stimulation strategies learned by each algorithm in the offline setting. Figure 11 compares reaching accuracy maps of the four algorithms when trained on the medium dataset. IQNBR again demonstrates the most uniform and precise control, maintaining RMSE below 5 cm for most targets with minimal variability, similar to its strong performance in online training (Figure 7). In contrast, SAC-CQL has widespread inaccuracies, particularly on the left side of the workspace. QBR-REM achieves moderate performance, with noticeable improvements over QBR-CQL in the central region. However, both QBR variants exhibit high average error and variability, particularly near workspace boundaries. These observations highlight IQNBR’s superior consistency and accuracy compared to other methods, while SAC-CQL fails to maintain the good performance it achieves in online learning.

Figure 11. Performance maps, evaluated on a typical Arm26-Planar, of (a) SAC-CQL, (b) QBR-CQL, (c) QBR-REM, and (d) IQNBR, which are trained with the medium dataset in the offline setting. The (0,0) coordinate represents the shoulder location. The last-second reaching RMSEs are color-coded. The dots are sized based on their standard deviation across 5 runs.

Figure 12 illustrates how the four algorithms perform reaching movements. IQNBR and SAC-CQL produce a smooth and accurate movement. SAC-CQL benefits from its continuous action space, allowing fine adjustments in stimulation and resulting in a very smooth bell-shaped velocity profile (Figure 12a, middle row). IQNBR also produces a smooth bell-shaped profile, but with a longer tail caused by overshoot correction following initial overstimulation of the Biceps and Pectoralis during the first 0.5 s. In contrast, QBR-CQL and QBR-REM produce less smooth and less accurate movements. QBR-CQL exhibits frequent abrupt changes in stimulation, resulting in a long and shaky tail in its velocity profile. QBR-REM performs slightly better, with improved trajectory continuity compared to QBR-CQL, but its velocity profile remains steep and unnatural.

Figure 12. Example reaching behaviors on a typical Arm26-Planar with fatigable muscles of (a) SAC-CQL, (b) QBR-CQL, (c) QBR-REM, and (d) IQNBR trained with heterogeneous data in an offline setting. The top-row figures show example arm reaching movement toward targets indicated by the red circles. The bottom-row figures show the stimulation applied to 4 muscles along the reaching movements. The middle-row figures show hand speed along the movement.

Figure 13 shows example tracking behaviors of the four algorithms. SAC-CQL has poor tracking performance, with trajectories deviating significantly from the target path. QBR-CQL also produces poor tracking with large deviations and rough trajectories. QBR-REM performs noticeably better than QBR-CQL, achieving lower RMSE and smoother trajectories. IQNBR delivers the best tracking performance, closely following the target path with only a small deviation on the left side of the circle. Considering the stimulation, IQNBR appears to achieve this accuracy through forceful and frequent abrupt stimulation changes. In contrast, SAC-CQL tends to understimulate the Triceps and Deltoid, which leads to prolonged deviations.

Figure 13. Example tracking behaviors on a typical Arm26-Planar with fatigable muscles of (a) SAC-CQL, (b) QBR-CQL, (c) QBR-REM, and (d) IQNBR trained in an offline setting. The blue curves in the top-row figures show 5 example trajectories of the hand, and the black circles show target paths. The bottom-row figures show the stimulation applied to 4 muscles along the tracking movements.

4.2.3. Robustness Across Datasets and Environments

This section compares the robustness and generalization of four algorithms across different environments and datasets. Figure 14 summarizes their reaching and tracking performance in seven Arm26-Planar variants. The results align with the trends observed in the learning curves (Figure 14). IQNBR consistently achieves the best performance, with overall RMSE of 3.8 cm on the reaching task. This represents an increase of 0.8 cm compared to online training. An exception occurs in the reaching task in the weak Pectoralis variant, where QBR-REM slightly outperforms IQNBR, and QBR-CQL falls behind SAC-CQL. Although the average performance gaps between IQNBR and QBR-REM appear small in these bar charts, the detailed results in Figure 11 and Figure 13 highlight the qualitative advantages of IQNBR over QBR-REM.

Figure 14. (a) Reaching and (b) tracking performance across seven Arm26-Planar variants of SAC-CQL, QBR-CQL, QBR-REM, and IQNBR trained on the medium dataset. The training and evaluation in each variant are repeated 5 times, whose standard deviation is represented by the error bars.

A comprehensive comparison of algorithm performance across multiple datasets and muscle profiles is shown in Figure 15. Larger and more diverse datasets generally improve the performance of most algorithms, but the degree of improvement varies (Figure 15a,b). IQNBR consistently achieves the lowest errors and smallest variability overall. It has a median RMSE of approximately 3 cm in both tasks, which is close to that obtained through online learning under fatigable muscle conditions (Figure 6). This suggests IQNBR’s robustness to both the sizes and the quality of the datasets. QBR-REM improves significantly with very large datasets, occasionally approaching IQNBR’s accuracy, while SAC-CQL and QBR-CQL show limited gains even with very large data. A twist occurs in the expert dataset: SAC-CQL and QBR-CQL have substantial improvement, while QBR-REM perform poorly. Figure 15c,d show the breakdown of the overall results by individual variants, highlighting the superior robustness and generalization of IQNBR.

Figure 15. Reaching and tracking performances of SAC-CQL, QBR-CQL, QBR-REM, and IQNBR trained and evaluated on different datasets and variants of Arm26-Planar. (a,b) show the averaged performances over seven Arm26-Planar variants of the RL algorithms trained with different datasets. (c,d) show the overall performance by variant. The training and evaluation in each variant are repeated 5 times, and their standard deviation is represented by the error bars.

4.3. Performance Comparison with Other Studies

Table 1 compares our results with closely related studies. Most studies that employ RL-based controllers were conducted in simulation. In [21,38,44], performance is evaluated by whether the hand can stay within a circular target of 7.5 cm radius, with reported success rates ranging from 95% to 100%. Note that these studies use simpler arm models than those implemented in OpenSim or MuJoCo. The lower success rate reported in [21] is attributed to the inclusion of muscle fatigue in the model, which increases the difficulty of the control task. Razavian et al. [31] use feedback control that utilizes person-specific muscle synergy data and conduct their experiments on real able-bodied participants, reporting an average reaching error of 1–3 cm. Note that their evaluation is limited to only nine target points around the center of the workspace.

Table 1. Performance comparison to related studies.

Study	Controller	Environment	Performance Measures
Jagodnik et al. [38]	Actor-critic (RL)	Simple Arm26	99% reaching success (target radius: 7.5 cm)
Razavian et al. [31]	Feedback control	Real human	Reaching RMSE: 1–3 cm
Cowder et al. [44]	TD3 (RL)	Simple Arm26	100% reaching success (target radius: 7.5 cm)
Abreu et al. [21]	TD3 (RL)	Simple Arm26	95% reaching success (target radius: 7.5 cm)
Wolf et al. [23]	MPC	Real human	Reaching RMSE: 5 cm
Wannawas et al. [52]	SAC (RL)	Real human	Reaching RMSE: 5.5 cm
Wannawas et al. [47]	Model-based RL	OpenSim	Reaching RMSE: 3 cm
Present study	Distributional RL	MuJoCo	Reaching RMSE online training: 3.0 cm offline training: 3.8 cm

Wolf et al. [23] use MPC with a learned dynamics model to control arm movement in a 3-dimensional space, achieving an average error of 5 cm. However, this result is obtained within a relatively small workspace. Our previous work [52] applies SAC to control planar arm movements in a real-world setting. The controller is evaluated on 25 targets distributed across the reachable workspace and achieves an average reaching error of 5.5 cm. This approach, however, relies on person-specific calibration data for RL training. More recently, our work [47] uses a physics-informed, model-based RL method on an OpenSim arm model and achieves an average reaching error of 3 cm. However, its heavy computation may limit its feasibility for real-time control. The distributional RL approach presented in this work achieves average RMSEs of 3.0 cm and 3.8 cm through online and offline training, respectively. This indicates that distributional RL can reach performance comparable to the physics-informed, model-based RL when operating in a single environment, while remaining considerably more computationally lightweight. The performance achieved through offline training can be translated into 100% success rate in the performance measure used in [21,38,44], indicating strong control capability. When applied in real-world scenarios, the performance of distributional RL can be expected to be comparable to that reported in [52], but without the need for additional calibration.

5. Discussion

The online training results align with the results in [21], which report the reduction of TD3’s success rate in performing reaching tasks when the muscles are fatigued. This study extends the investigation by evaluating multiple algorithms across both reaching and tracking tasks and across different muscle conditions, offering a broader view of RL performance in FES control. The results illustrate that fatigue affects RL algorithms differently. Distributional RL (IQNBR) is the most robust, maintaining lower errors and variability across tasks and muscle profiles. The tracking task (Figure 8) shows that fatigue hampers precise muscle coordination, resulting in deviated movements. In terms of data requirements, our previous fatigue-free study [45] shows that SAC reaches its performance plateau after about 1.5 min of interaction. The results here indicate that fatigue causes the data requirement to increase by approximately twofold, reflecting the complexity that the uncertainty caused by the time-varying muscle force adds to the control system.

The offline training experiment shows superior performance of the distributional RL over other offline algorithms when trained with heterogeneous datasets. Conservative approaches, SAC-CQL and QBR-CQL, perform well when the data come from expert demonstrators. However, they struggle with mixed-quality data, failing to improve beyond a modest accuracy level. In contrast, the ensemble method, QBR-REM, struggles when data lack diversity, as seen with small and expert datasets, and benefits mainly from broad coverage. IQNBR can leverage heterogeneous datasets efficiently and deliver good results even in cases where other algorithms fail. In addition, IQNBR scales with data volume, continuing to improve as the dataset grows, and remains consistent across different muscle profiles and task types. These behaviors suggest that modeling the full return distribution helps the agent remain stable under data heterogeneity and out-of-distribution transitions. Together, the results imply the feasibility of a general controller that does not require user-specific calibration, thereby reducing clinician workload and accelerating the deployment of FES systems in clinical and at-home settings.

Despite these promising results, several limitations remain. First, IQNBR achieves high tracking accuracy by applying forceful stimulation, which produces accurate movements but also causes oscillations that may feel unnatural or uncomfortable for users. Second, all evaluations were conducted in simulation, which cannot fully capture real-world variability such as electrode placement differences, skin impedance changes, or user-specific fatigue behaviors. Third, although offline datasets are heterogeneous and diverse, they may still fail to represent rare clinical conditions or unusual transitions. These gaps introduce risks when deploying the learned controller directly on human users.

6. Conclusions

This study primarily investigates the use of offline RL and heterogeneous data to develop generalized FES controllers for arm movement. We built a biomechanical model with fatigable muscles in MuJoCo and presented RL configurations for controlling planar arm movement. The baselines established in online training illustrate IQNBR’s superior control accuracy and consistency across environments. IQNBR takes approximately 4 min of interaction data to achieve the RMSE of below 3 cm. In offline training, IQNBR demonstrates robust learning and strong performance across muscle profiles, achieving an average RMSE of 3.8 cm and consistently outperforming Conservative Q-Learning (CQL) and Randomized Ensemble (REM) method when trained on heterogeneous data of various sizes and quality. While this performance remains inferior to that achieved through online training, the results demonstrate the promise of distributional RL as a foundational learner for developing FES controllers that do not require patient-specific calibration.

Future work should address the limitations to enable real-world deployment. First, the reward should include terms that penalizes jerks or abrupt stimulation changes to promote smoother and more natural movements. Configurations that control the rate of stimulation change, rather than the absolute intensity, can also improve safety by preventing stimulation spikes. Second, real-world experiments are also needed to assess robustness under electrode placement variability and user-specific muscle responses. Finally, constructing mixed-quality datasets that include sufficient expert demonstrations–along with incorporating uncertainty-aware mechanisms into the policy–could help the controller avoid risky actions in unfamiliar situations. Together, these developments are essential for creating deployable FES controllers. These developments not only advance FES control but also have broader implications for controlling robots with human-like actuation systems.