FRMA: Four-Phase Rapid Motor Adaptation Framework

Abstract

In many real-world control tasks, agents operate under partial observability, where access to complete state information is limited or corrupted by noise. This poses significant challenges for reinforcement learning algorithms, as methods relying on full states or long observation histories can be computationally expensive and less robust. Four-Phase Rapid Motor Adaptation (FRMA) is a reinforcement learning framework designed to address these challenges in high-frequency control tasks under partial observability. FRMA proceeds through four sequential stages: (i) full-state pretraining to establish a strong initial policy, (ii) auxiliary hidden-state prediction for LSTM memory initialization, (iii) aligned latent representation learning to bridge partial observations with full-state dynamics, and (iv) latent-state policy fine-tuning for robust deployment. Notably, FRMA leverages full-state information ($s_t$) only during training to supervise latent representation learning, while at deployment it requires only short sequences of recent observations and actions. This allows agents to infer compact and informative latent states, achieving performance comparable to policies with full-state access. Extensive experiments on continuous control benchmarks show that FRMA attains near-optimal performance even with minimal observation–action histories, reducing reliance on long-term memory and computational resources. Moreover, FRMA demonstrates strong robustness to observation noise, maintaining high control accuracy under substantial sensory corruption. These results indicate that FRMA provides an effective and generalizable solution for partially observable control tasks, enabling efficient and reliable agent operation when full state information is unavailable or noisy.

1. Introduction

Reinforcement learning (RL) is a computational framework in which agents learn to make decisions by interacting with an environment and receiving feedback in the form of rewards [1,2]. Unlike supervised learning, RL does not require labeled datasets; instead, agents discover optimal behavior through direct exploration. This makes RL highly suitable for sequential decision-making problems where explicit models are unavailable or inaccurate.

Over the past decade, RL has achieved significant milestones in domains such as game playing [3], robotic control [4,5], and continuous control environments like MuJoCo [6,7]. These successes are largely attributed to the ability of RL algorithms to learn policies in high-dimensional, nonlinear environments directly from experience. However, many real-world or physics-based simulated environments are only partially observable—the agent does not have direct access to the full system state and must instead infer latent dynamics from limited observations, which may be noisy or partially missing.

Such settings are commonly modeled as Partially Observable Markov Decision Processes (POMDPs) [8,9], which introduce additional complexity to the learning problem. In POMDPs, the agent does not have direct access to the full environment state; instead, it must infer latent states from partial observations. To make informed decisions, agents typically maintain a memory of past observations and actions, which is especially important in environments with long-term dependencies or delayed rewards.

To address partial observability, the Rapid Motor Adaptation (RMA) framework [10] has been proposed. RMA operates in two stages: first, it learns a base policy with full-state access $\pi (a_t\mid s_t)$; second, it trains an encoder to approximate a latent state representation $z_t$ from sequences of past observations and actions $(o_{t-T:t},a_{t-T:t})$. This allows the policy to be deployed under partial observability as $\pi (a_t\mid z_t)$, leveraging historical information to infer hidden states.

While RMA has demonstrated strong performance in low-frequency robotic locomotion tasks, its reliance on fixed-length input windows and infrequent inference steps limits its applicability in high-speed control tasks or environments with rapidly changing latent dynamics.

To overcome these challenges, this paper proposes a general framework called Four-Phase Rapid Motor Adaptation (FRMA). FRMA extends the RMA design with a four-phase training pipeline, enabling high-frequency decision-making under partial observability. The four phases are designed to progressively transfer knowledge from full-state information to compact latent representations that can be used for robust control with only partial observations:

• Full-State Pretraining: A base policy is first trained using full access to the ground-truth system state. This policy captures essential control behaviors without concern for observability constraints, providing a strong performance ceiling and serving as a foundation for subsequent learning stages.
• Auxiliary Hidden-State Prediction: An auxiliary network is trained to predict the initial hidden and cell states $(h_0,c_0)$ of the recurrent encoder (LSTM) from recent observation–action sequences. These predicted states initialize the LSTM memory, enabling immediate inference of latent dynamics, even in partially observable conditions.
• Aligned Latent Representation Learning: A recurrent encoder is trained to map sequences of partial observations and actions to latent states $z_t$ that are aligned with those learned during full-state training. This phase bridges the gap between limited observations and the underlying full latent states, ensuring that the encoder produces informative representations for policy execution.
• Latent-State Policy Fine-Tuning: Finally, the policy is fine-tuned using only the estimated latent encoding $z_t$, allowing it to operate under realistic partially observable conditions. This phase ensures that the agent can perform fast, reliable control without requiring full-state information at deployment.

FRMA introduces several core innovations that make it particularly suitable for high-frequency, partially observable control tasks:

• High-Frequency, Single-Step Adaptation: During training, FRMA leverages multi-step sequence regression to learn latent-state inference. At deployment, it switches to single-step LSTM updates, continuously updating the hidden and cell states $(h_t,c_t)$ at each timestep. This allows the latent encoding $z_t$ to be refreshed at high frequency, enabling rapid and precise control.
• Extended Temporal Coverage: By propagating the LSTM’s hidden and cell states across timesteps, FRMA avoids fixed-horizon limitations (e.g., 50-step windows). It effectively captures long-term temporal dependencies without the computational burden of long input sequences, reducing GPU memory usage and accelerating both training and inference.
• Policy-Invariant Latent Representation: Observation–action histories contain both environmental dynamics and policy-dependent information. FRMA uses supervised learning to ensure that the LSTM hidden and cell states $(h_t,c_t)$ primarily encode environment dynamics, effectively minimizing the influence of the sampling policy $\pi$. This design enhances generalization and robustness when the policy changes or under different deployment conditions.

The remainder of this paper is organized as follows. Section 2 reviews related work on recurrent neural networks, reinforcement learning, and the RMA framework, providing the technical background for FRMA. Section 3 presents the FRMA framework in detail, describing each of the four sequential stages. Section 4 details the experimental setup and results, demonstrating FRMA’s performance on continuous control tasks in the MuJoCo environment. Finally, Section 5 concludes the paper and discusses potential avenues for extending the FRMA framework.

2. Related Work

2.1. Artificial Neural Networks

Artificial neural networks (ANNs) have achieved remarkable success across a wide range of machine learning tasks [11,12]. Feedforward neural networks (FNNs), such as multilayer perceptrons, can approximate arbitrary functions given sufficient data and model capacity, and are commonly applied to static input–output mappings [13]. However, in dynamic environments where temporal dependencies are critical, FNNs are limited by their inability to retain historical context. Recurrent neural networks (RNNs) and their variants, such as long short-term memory (LSTM) networks [14,15,16], address this limitation by incorporating memory through recurrent connections. This allows information to persist over time, enabling the modeling of temporal dependencies essential for tasks such as speech recognition, language modeling, and time-series prediction [17,18]. Despite these advances, traditional neural networks—including LSTM-based architectures—are typically trained in a supervised learning setting, where each input x is paired with a target output y. While effective when labeled data is abundant and the input–output mapping is relatively static, supervised learning alone struggles in complex, high-dimensional control tasks. In particular, continuous control environments with partial observability and delayed rewards—common in robotics and embodied AI—require sequential decision-making over time, which exceeds the capabilities of standard supervised approaches.

2.2. Classical Control Methods

Classical control methods have been widely applied to complex systems, including Active Disturbance Rejection Control (ADRC) [19], multilayer neurocontrol for high-order uncertain nonlinear systems [20], Robust Integral of the Sign of the Error (RISE) [21], observer-based RISE controllers [22], adaptive ADRC with recursive parameter estimation [23], and Auxiliary RISE (ARISE) controllers for nonsmooth switched systems [24]. These methods have demonstrated strong robustness and effective disturbance rejection across various engineered systems. However, classical controllers typically require accurate system modeling, and their performance may deteriorate in high-dimensional, partially observable, or stochastic environments. In contrast, neural network-based reinforcement learning approaches can learn control policies directly from data, enabling adaptive behavior in complex scenarios and offering greater flexibility and potential performance in modern control tasks.

2.3. Reinforcement Learning

Classical reinforcement learning (RL) methods can be broadly categorized into three types: tabular methods, value-based methods, and policy gradient methods. Tabular approaches, such as Q-learning and SARSA [25], maintain a lookup table estimating action values for each discrete state–action pair. While effective in low-dimensional environments, tabular methods become infeasible in continuous or high-dimensional spaces due to the curse of dimensionality. Value-based methods address this limitation by approximating the value function with neural networks, enabling generalization across large or continuous state spaces. Deep Q-Networks (DQN) [26] achieved notable success in discrete-action tasks, such as Atari games, but exhibit instability and overestimation bias in continuous domains. Policy gradient methods directly optimize parameterized policies, supporting stochastic and continuous actions. Representative examples include REINFORCE [27], Proximal Policy Optimization (PPO) [28], and Soft Actor-Critic (SAC) [29], which combines policy gradients with entropy regularization and an actor–critic architecture. Despite their success, these algorithms typically assume full observability of the environment state $s_t$. In practice, many control tasks provide only partial observations $o_t$, causing performance degradation when the agent cannot access complete state information. This motivates the integration of architectures capable of temporal memory and latent state inference, such as recurrent networks or encoder–decoder models, to handle partially observable environments effectively.

2.4. Observation–Action Sequence-Based Reinforcement Learning

Many reinforcement learning methods for partially observable environments rely on short histories of observations and actions to infer latent states. Rapid Motor Adaptation (RMA) and its variants are representative examples of such observation–action (oa) sequence-based RL approaches. The original RMA framework first trains a base policy using full-state trajectories and then employs a supervised adaptation module to infer a latent context $z_t$ from recent observation–action sequences $(o_{t-T:t},a_{t-T:t})$. However, RMA is constrained by a fixed input window (typically 50 steps) and a low adaptation frequency on the order of 10 Hz [10]. Extensions of RMA address specific limitations. A-RMA explicitly fine-tunes the base policy with an imperfect extrinsic estimator to improve performance in biped locomotion, but it retains the same adaptation module and fixed-horizon inference [30]. RMA² targets manipulation tasks by combining depth perception and symbolic embeddings to estimate latent object parameters, yet it still operates over a fixed-length OA segment and lacks high-frequency adaptation [31]. Other sequence-based RL approaches include Deep Recurrent Q-Networks (DRQN) and Action-Conditional DRQN (ADRQN), which incorporate recurrent layers over observations or observation–action streams to implicitly encode belief states, trained end-to-end via Q-learning [32,33]. Recurrent Predictive State Policy (RPSP) networks similarly learn a predictive latent belief state using supervised prediction alongside policy gradients [34]. While these methods demonstrate the utility of sequence-based encoding, they generally train the encoder jointly with the policy and maintain memory only over a truncated horizon, which can limit adaptation speed and efficiency. Overall, these OA sequence-based RL methods highlight the importance of latent-state inference from recent history, but they also reveal limitations in adaptation frequency, memory length, and policy–encoder decoupling—challenges that FRMA aims to overcome.

3. FRMA Details

This section introduces the FRMA framework in detail, including its four-phase training strategy and the inference procedure used during deployment.

Table 1. Symbol definitions and network components in FRMA.

Symbol	Description
$o_t$	Observation at time step t
$s_t$	State at time step t
$a_t$	Action at time step t
$o_{t-T:t}$	Observation sequence from time $t-T$ to t: $o_{t-T:t}=[o_{t-T},\dots ,o_t]$
$a_{t-T:t}$	Action sequence from time $t-T$ to t: $a_{t-T:t}=[a_{t-T},\dots ,a_t]$
$z_t$	Latent state encoding: $z_t={\mathbf{e}}_{\mathbf{s}}\left(s_t\right)$
$h_t,c_t$	LSTM hidden and cell states at time t
${\pi }_1,{\pi }_2$	Primary and secondary policy networks: $z_t\mapsto a_t$
${\mathbf{e}}_{\mathbf{s}}$	State encoder: $s_t\mapsto z_t\in {\mathbb{R}}^k$
${\mathbf{e}}_{\mathbf{o}}$	LSTM-based encoder: $(o_{t-T:t},a_{t-T:t},h_{t-T},c_{t-T})\mapsto (h_t,c_t)$
$\mathbf{h}$	Hidden network: $s_t\mapsto (h_t,c_t)\in {\mathbb{R}}^d\times {\mathbb{R}}^d$
$\mathbf{f}$	Mapping network: $(h_t,c_t)\mapsto z_t\in {\mathbb{R}}^k$
T	Length of observation–action history used by LSTM
d	LSTM hidden size
k	Dimension of latent state encoding

summarizes all key variables and network components, and Figure 1 illustrates the sequential workflow. Detailed proofs are provided in Appendix C and Appendix D, and comprehensive pseudocode for both training and deployment is included in Appendix A.

The FRMA training proceeds through four sequential phases, each targeting a distinct network module and progressively bridging the gap between full-state learning and deployment under partial observability. This stagewise design ensures that FRMA can leverage full-state information during training while requiring only partial observations at deployment, achieving high-frequency adaptation, robust latent-state inference, and stable policy performance.

3.1. Phase I: Full-State Policy Learning

Phase I focuses on training the foundational policy network, ${\mathit{\pi }}_1$, together with the state encoder, ${\mathbf{e}}_{\mathbf{s}}$, using the complete environment state $s_t$ as input. This stage assumes full observability, meaning the agent has access to all relevant features of the underlying Markov Decision Process (MDP). By removing partial observability and temporal dependencies, the learning process is simplified, allowing the model to concentrate on extracting meaningful latent representations and learning basic control strategies.

The state encoder ${\mathbf{e}}_{\mathbf{s}}$ transforms the high-dimensional raw state $s_t$ into a compact latent vector $z_t$, which serves as the input to the policy ${\mathit{\pi }}_1$. The policy is optimized using the Proximal Policy Optimization (PPO) algorithm, which balances exploration and exploitation while ensuring stable updates via a clipped surrogate objective. In this phase, standard policy gradient methods are applied without recurrent layers or memory modules, avoiding the complexity of long-term temporal credit assignment and enabling faster convergence.

Formally, the forward pass in Phase I is expressed as:

$$z_t={\mathbf{e}}_{\mathbf{s}}\left(s_t\right),a_t\sim {\mathit{\pi }}_1\left(z_t\right).$$

(1)

By the end of this phase, the agent acquires a well-initialized encoder and policy capable of operating under full observability, providing a strong foundation for adaptation to partially observable environments in subsequent phases.

3.2. Phase II: Supervised Training of Hidden Network

Long short-term memory (LSTM) networks extract and integrate temporal information from observation–action sequences, encoding this knowledge in their hidden and cell states $(h_t,c_t)$. In existing approaches, such as RMA and ARMA, this temporal integration is achieved by repeatedly processing the most recent T timesteps at each control step. While effective, such repeated sequence processing incurs significant computational cost, inherently limiting the achievable control frequency.

To address this limitation, FRMA adopts a single-step inference paradigm during deployment. Instead of recomputing over the entire T-step history at each timestep, the system maintains and incrementally updates $(h_t,c_t)$ throughout the episode. This design preserves long-term temporal information while minimizing computational latency. However, a mismatch arises between training and deployment: during training, LSTMs are typically optimized over short, fixed-length sequences, which may limit their ability to capture dependencies spanning the full episode.

To resolve this, we introduce an auxiliary hidden-state inference network $\mathbf{h}$, which maps the current full state $s_t$ directly to the corresponding $(h_t,c_t)$. This network is trained in a supervised manner to minimize the mean squared error (MSE) between its predictions and the hidden states generated by the LSTM during full-episode rollouts. By enabling reconstruction of the LSTM’s internal memory from a single state input, this approach ensures consistency between training and deployment while retaining the computational efficiency of single-step inference.

Formally, the hidden-state inference process and training loss are defined as:

$$({\widehat{h}}_t,{\widehat{c}}_t)=\mathbf{h}\left(s_t\right),(h_t,c_t)={\mathbf{e}}_{\mathbf{o}}\left(o_{t-T:t},a_{t-T:t},\mathbf{h}\left(s_{t-T}\right)\right),$$

(2)

$${\mathcal{L}}_{\mathrm{hidden}}=\parallel {\widehat{h}}_t-h_t{\parallel }_2^2+{\parallel {\widehat{c}}_t-c_t\parallel }_2^2,$$

(3)

where $({\widehat{h}}_t,{\widehat{c}}_t)$ denotes the target hidden state derived from the full state $s_t$, and $(h_t,c_t)$ is the LSTM output given the partial observation–action sequence ${\tau }_{o,t-T:t}$ and the initialized memory at time $t-T$.

This supervised training approach follows prior work on deep LSTM supervision [14,35]. Minimizing ${\mathcal{L}}_{\mathrm{hidden}}$ allows the LSTM to be accurately initialized at any point in an episode, enabling high-frequency control while preserving long-term temporal reasoning.

3.3. Phase III: Learning Latent Representations from Partial Observations

In realistic control settings, agents rarely have access to the full environment state $s_t$, relying instead on partial and potentially noisy observations $o_t$. To enable robust decision-making under such partial observability, Phase III trains the agent to reconstruct the latent representation $z_t$—originally learned from full-state information in Phase I—using only past observations and executed actions. This approach follows the core principle of RMA, where a history encoder infers latent task variables from recent experience.

The LSTM-based observation encoder ${\mathbf{e}}_{\mathbf{o}}$ is initialized at the start of each sequence with hidden and cell states $(h_{t-T},c_{t-T})$ predicted by the hidden-state inference network $\mathbf{h}$ from Phase II. It then processes a fixed-length window of past observations and actions $\left(o_{t-T:t},a_{t-T:t}\right)$ to update its internal memory $(h_t,c_t)$. These updated states are passed through a mapping network $\mathbf{f}$ to produce the estimated latent representation ${\widehat{z}}_t$. The target latent vector $z_t={\mathbf{e}}_{\mathbf{s}}\left(s_t\right)$, computed using the state encoder ${\mathbf{e}}_{\mathbf{s}}$ trained in Phase I, serves as the ground truth.

Training minimizes the mean squared error (MSE) between ${\widehat{z}}_t$ and $z_t$, aligning the partial-observation pathway with the full-state pathway. Through this supervised alignment, the observation encoder learns to capture temporally extended dependencies and infer task-relevant latent states without direct access to $s_t$, a capability essential for real-world deployment.

Formally, the encoding and mapping process is expressed as:

$$(h_t,c_t)={\mathbf{e}}_{\mathbf{o}}\left(o_{t-T:t},a_{t-T:t},\mathbf{h}\left(s_{t-T}\right)\right),{\widehat{z}}_t=\mathbf{f}(h_t,c_t),$$

(4)

$${\mathcal{L}}_{\mathrm{latent}}={\parallel {\widehat{z}}_t-{\mathbf{e}}_{\mathbf{s}}\left(s_t\right)\parallel }_2^2.$$

(5)

By the end of this phase, the agent can reliably transform sequences of partial observations into compact latent representations that retain the task-relevant information needed for downstream policy execution.

3.4. Phase IV: Policy Fine-Tuning on Estimated Latent States

In the final phase, the control policy transitions from relying on ground-truth latent states $z_t$ to operating entirely on their estimated counterparts ${\widehat{z}}_t$ derived from partial observations. This step is essential for bridging the sim-to-real gap and ensuring robust decision-making despite reconstruction errors introduced in Phase III.

Following the ARMA-inspired design, the base policy ${\mathit{\pi }}_1$ learned in Phase I is duplicated to initialize a new policy ${\mathit{\pi }}_2$, which is then fine-tuned using ${\widehat{z}}_t$ as input. This duplication preserves the well-optimized structure of ${\mathit{\pi }}_1$ while enabling ${\mathit{\pi }}_2$ to adapt its action selection to the shifted latent distribution.

Fine-tuning employs the Proximal Policy Optimization (PPO) algorithm, providing stable gradient updates under the new input distribution. The process allows the policy to compensate for systematic biases or noise in ${\widehat{z}}_t$, effectively learning to act reliably using only estimated latent states. By the end of this phase, the agent achieves fully observation-based control without any dependence on privileged full-state information, completing the FRMA pipeline from full-state pretraining to deployable inference.

Formally, the fine-tuning process is expressed as:

$$(h_t,c_t)={\mathbf{e}}_{\mathbf{o}}\left(o_{t-T:t},a_{t-T:t},\mathbf{h}\left(s_{t-T}\right)\right),{\widehat{z}}_t=\mathbf{f}(h_t,c_t),$$

(6)

$$a_t\sim {\mathit{\pi }}_2\left({\widehat{z}}_t\right),$$

(7)

where ${\mathbf{e}}_{\mathbf{o}}$ and $\mathbf{f}$ are fixed from Phase III, and ${\mathit{\pi }}_2$ is optimized to maximize expected returns using ${\widehat{z}}_t$ as the sole input.

3.5. Training Procedure

The training of FRMA is organized into four distinct phases, each executed for a fixed number of iterations specified by the hyperparameter train_step. The second important hyperparameter, batch_size, controls the number of samples used in each parameter update. The design explicitly avoids end-to-end training from scratch; instead, each phase focuses on a targeted subset of networks to ensure stable convergence and avoid detrimental interference between modules. The procedure is as follows:

• Phase I: Full-State Policy Learning. For train_step iterations, sample batch_size state–action–reward transitions from full-state rollouts. The state encoder ${\mathbf{e}}_{\mathbf{s}}$ and the base policy ${\mathit{\pi }}_1$ are jointly optimized using Proximal Policy Optimization (PPO) under full observability. This phase establishes a high-quality latent state representation $z_t$ and a competent control policy, providing a strong foundation for subsequent modules.
• Phase II & III: Hidden-Network Pretraining and Joint LSTM–Mapping Training. These two phases are interleaved within each iteration to maintain a balanced learning pace between the hidden state predictor $\mathbf{h}$ and the observation-based latent inference pipeline $({\mathbf{e}}_{\mathbf{o}},\mathbf{f})$. For train_step iterations:
  • Sample 5 independent batches to update $\mathbf{h}$ via the hidden-state MSE loss ${\mathcal{L}}_{\mathrm{hidden}}$, using full-state trajectories as supervision.
  • Sample 1 fresh, non-overlapping batch to jointly optimize ${\mathbf{e}}_{\mathbf{o}}$ and $\mathbf{f}$ by minimizing the mapping loss ${\mathcal{L}}_{\mathrm{map}}$, which aligns the predicted latent ${\widehat{z}}_t$ with the ground-truth latent $z_t$ from Phase I.
  This strict batch independence is critical: $\mathbf{h}$ is trained on $s_t$, whereas ${\mathbf{e}}_{\mathbf{o}}$ relies on $\mathbf{h}\left(s_{t-T}\right)$. Sharing identical samples would bias $\mathbf{h}$ toward the short-horizon regime, degrading its generalization to long-term dependencies.
• Phase IV: Policy Fine-Tuning. For train_step iterations, sample batch_size trajectories of estimated latent states ${\widehat{z}}_t$ from the Phase III pipeline. The policy ${\mathit{\pi }}_2$, initialized from ${\mathit{\pi }}_1$, is fine-tuned using PPO to adapt to the noisy and potentially biased distribution of ${\widehat{z}}_t$. This step completes the transition from privileged, full-state decision-making to deployable, observation-driven control.

The deliberate phase separation ensures that each module is trained under optimal supervision, minimizing interference and preventing unstable dynamics that often occur in fully joint optimization.

3.6. Deployment

After completing all four training phases, FRMA is deployed in an online control loop that operates exclusively on partial observations. The deployment strategy emphasizes high control frequency and low latency, eliminating the need to reprocess long observation–action histories at each timestep. Instead, the LSTM encoder ${\mathbf{e}}_{\mathbf{o}}$ maintains its hidden and cell states $(h_t,c_t)$ throughout the episode, updating them incrementally with each new observation-action pair.

At the start of each episode, the LSTM memory is zero-initialized:

$$h_0=\mathbf{0},c_0=\mathbf{0}.$$

(8)

This zero-initialization ensures fair comparison with baseline methods (ARMA, RMA, ADRQN, RPSP), which do not include a learned initializer. In principle, FRMA can reconstruct $(h_0,c_0)$ from an initial environment state $s_0$ using the hidden network $\mathbf{h}\left(s_0\right)$, enabling “hot-start” inference and mitigating cold-start degradation; this feature is omitted here solely for comparability.

At each timestep t, the agent executes the following sequence:

• Receives the latest observation $o_{t-1}$ and executed action $a_{t-1}$ from the previous step.
• Updates the LSTM memory via a single forward step:

  $$(h_t,c_t)={\mathbf{e}}_{\mathbf{o}}\left(o_{t-1},a_{t-1},h_{t-1},c_{t-1}\right).$$

  (9)
• Computes the current latent state estimate through the feature extraction network:

  $${\widehat{z}}_t=\mathbf{f}(h_t,c_t).$$

  (10)
• Feeds ${\widehat{z}}_t$ into the fine-tuned policy ${\mathit{\pi }}_2$ to determine the next action:

  $$a_t={\mathit{\pi }}_2\left({\widehat{z}}_t\right).$$

  (11)

This approach minimizes per-step computation since the LSTM state is propagated incrementally rather than recomputed from scratch. On standard hardware (Intel i9-13900K CPU or RTX 4070 GPU), the combined MLP-LSTM forward pass executes in under 1 ms per step, enabling kilohertz-range control frequencies even with batch size 1.

The deployment procedure relies on the following assumptions:

• Zero-initialization. Used here for fair comparison; FRMA inherently supports hot-start initialization via $\mathbf{h}\left(s_0\right)$ when privileged or learned initial state information is available.
• Policy optimization. Phase IV uses on-policy PPO for stability in MuJoCo, but FRMA is compatible with off-policy methods such as SAC for enhanced generalization.
• Full-state supervision during training. Ground-truth state information shapes the encoder only; deployment relies solely on partial observations. When a simulator is unavailable, FRMA can use data-driven environment models or other reconstruction methods to provide training supervision.
• Staged training. The multi-phase schedule decouples encoder learning from policy learning to improve stability and fairness; end-to-end training remains feasible within the same framework.

By aligning deployment-time inference with Phase IV fine-tuning under these assumptions, FRMA minimizes train–deploy mismatch and achieves high-frequency, low-latency control, while retaining flexibility to incorporate learned initializers, dynamic objectives, off-policy training, or alternative state reconstruction in future extensions.

4. Experiments

All experiments were conducted on a workstation equipped with an AMD EPYC 9654 96-Core Processor and an NVIDIA RTX 4090 GPU. In the figures presented below, each training curve visualizes the episode-level cumulative reward as a function of timesteps, with a maximum of 1000 timesteps per episode. Each experiment was repeated across five random seeds, and the results were averaged to ensure statistical reliability. Solid lines indicate the mean reward over sampled episodes, while shaded regions represent the variance. The full implementation, including configuration files and scripts for reproducing all figures and ablation studies, is publicly available on GitHub: https://github.com/guanyimu/FRMA.git (accessed on 20 September 2025).

The MuJoCo suite [36] provides a standard benchmark for evaluating reinforcement learning agents on continuous control tasks. Common environments such as HalfCheetah, Walker2d, and Hopper are modeled as fully observable Markov Decision Processes (MDPs), where the agent has access to the complete system state at each timestep.

To evaluate performance under partial observability, the underlying MuJoCo MDPs are converted into partially observable MDPs (POMDPs) by restricting the agent’s access to the full state. Specifically, only the first few dimensions of the state vector are retained as observations to simulate limited sensing, as summarized in

Table 2. Observation dimensionality used in each MuJoCo environment.

Environment	State Dim.	Action Dim.	Observation Dim.
InvertedPendulum-v4	4	1	2
HalfCheetah-v4	17	6	2
Walker2d-v4	17	6	10
InvertedDoublePendulum-v4	9	1	5
Hopper-v4	11	3	5
Humanoid-v4	348	17	25

. In these environments, the retained dimensions typically correspond to directly measurable quantities such as positions, distances, and angles, while the omitted dimensions include variables that are more difficult to measure, such as velocities, angular velocities, or forces. This setup reflects realistic scenarios in which some physical quantities are directly accessible, whereas others must be inferred indirectly.

The following methods are evaluated in our experiments:

• FullState (baseline) [28]: A PPO agent trained with full-state access, representing an upper bound on achievable performance.
• FRMA (ours): The proposed Four-phase Rapid Motor Adaptation framework, designed for learning in partially observable environments.
• ARMA [30]: An extension of RMA that fine-tunes the base policy using full-state supervision after adaptation.
• RMA [10]: Rapid Motor Adaptation with a fixed-length encoder over recent observation–action sequences.
• ADRQN [33]: An adaptation of Deep Recurrent Q-Networks employing LSTM encoders to process partial observation–action histories.
• RPSP [34]: A recurrent predictive state policy method that explicitly models predictive latent states for control.
• TrXL [37]: A Transformer-XL-based agent using multi-head attention to encode long-term dependencies in observation–action sequences.

Since the core novelty of FRMA resides in its four-phase training procedure, the chosen baselines correspond naturally to variants of FRMA with fewer phases. Specifically, when trained in a single phase, the framework reduces to a recurrent PPO agent akin to ADRQN, directly learning from partial observation–action histories. A two-phase configuration—comprising separate encoder adaptation and policy learning—corresponds to RMA. Adding a third phase of full-state fine-tuning yields ARMA. Our full FRMA implementation completes the four-phase design. Therefore, comparisons among ADRQN, RMA, ARMA, and FRMA can be interpreted as an implicit ablation study on the number of training phases, illustrating how each successive phase contributes to stability and performance.

Detailed network architectures, including both encoder and policy designs for all methods, are provided in Appendix B to support reproducibility.

As shown in Figure 2, the FullState agent achieves the highest returns, as expected, due to access to complete environment information. Among the observation-based methods, FRMA attains performance nearly indistinguishable from FullState, maintaining comparable return values across all six environments. Furthermore, FRMA consistently outperforms RPSP, ADRQN, RMA, and ARMA in every tested scenario. Within these baselines, RPSP shows performance close to ARMA in some environments, indicating that leveraging observations rather than full states for auxiliary training can help mitigate the ${\pi }_1$–${\pi }_2$ mismatch. Nevertheless, state information still provides substantial guidance, so ARMA and especially FRMA ultimately achieve higher asymptotic performance.

Among the three weaker baselines, ARMA generally performs better than RMA; however, both suffer from slower convergence and lower performance ceilings. By contrast, the TrXL algorithm yields the poorest results overall. This is likely because its attention-based encoder, while capable of modeling complex, variable-length dependencies, introduces unnecessary computational overhead in the MuJoCo tasks studied here, which involve short sequences with well-defined temporal relationships. These results demonstrate that FRMA effectively reconstructs rich and informative latent representations from partial observations, enabling robust decision-making in high-dimensional continuous control tasks.

FRMA is organized as a four-phase framework, and several baseline algorithms correspond to natural stage-wise ablations of this design. Specifically, ADRQN represents the single-phase baseline that trains a recurrent policy directly from observation–action (OA) sequences without auxiliary supervision. RMA extends this by using encoder supervision derived from the simulator state encoding, i.e., the OA encoder is trained to reproduce information obtainable from $s_t$. ARMA further incorporates a third phase intended to mitigate the mismatch between the initial auxiliary policy ${\pi }_1$ and the refined policy ${\pi }_2$ (policy-mismatch correction). FRMA completes the pipeline by adding a hidden-network phase that learns a mapping from the true state $s_t$ to the LSTM memory $(h_t,c_t)$, thereby removing the strict dependence of the LSTM on long OA input windows.

Partially Observable Markov Decision Processes (POMDPs) require extracting informative latent state representations from histories of observation–action pairs. The key innovation of Phase II in FRMA is to model the LSTM hidden state $(h_t,c_t)$ as a function of the underlying true state $s_t$, enabling supervised training that decouples the LSTM’s representational capacity from the raw number of observation timesteps. By leveraging this hidden-state network, the LSTM can encode much longer effective temporal context even when provided only a short OA segment, which reduces sensitivity to the sequence length T.

Figure 3 presents the ablation results across six MuJoCo environments, comparing FRMA and ARMA under observation–action sequence lengths $T=1,3,5$. In the figure, solid lines denote FRMA and dashed lines denote ARMA. Curves of the same color correspond to the same sequence length T, allowing direct comparison between methods at each temporal window. As expected, increasing T generally improves the performance of both FRMA and ARMA, reflecting the benefit of longer sequences that provide richer temporal context. Notably, FRMA with $T=1$ often exceeds the performance of ARMA with $T=5$, highlighting FRMA’s ability to effectively encode longer temporal dependencies even from very short observation–action segments. The marginal improvement of FRMA from $T=3$ to $T=5$ further demonstrates its low sensitivity to T, whereas ARMA continues to gain substantially from longer sequences. These results confirm that the Phase II hidden-network design allows the LSTM to maintain high representational efficiency with smaller input windows, reducing memory usage and enabling larger batch sizes without compromising performance.

To further evaluate the robustness of the proposed FRMA framework in partially observable settings, Gaussian noise $\mathcal{N}(0,{\sigma }^2)$ is added during testing either to the full state vector (for FullState) or to the partial observation vector (for all other methods). The horizontal axis represents the noise standard deviation $\sigma$, with larger values corresponding to stronger corruption of the input signals.

All policies are kept fixed during evaluation, so the results directly reflect the robustness of each method’s learned latent representations and decision policies under noisy inputs. For each environment and algorithm, five independent trials with different random seeds are conducted. The mean episode return is shown with solid lines, while the shaded regions indicate the standard deviation across seeds.

As shown in Figure 4, all algorithms experience performance degradation as noise levels increase, reflecting reduced fidelity of environment information. Among observation-based methods, FRMA exhibits the smallest performance drop across all environments, often approaching the robustness level of FullState. In contrast, ADRQN, RMA, and ARMA show more pronounced declines, especially at higher noise magnitudes and in high-dimensional environments such as Humanoid and HalfCheetah, where corrupted inputs can destabilize dynamics estimation in models with less temporal integration.

The robustness advantage of FRMA stems from its ability to reconstruct latent states over extended temporal windows and maintain LSTM memory across timesteps. This temporal integration effectively filters transient observation noise, enabling stable control despite heavy corruption of individual observations.

Additional robustness evaluations are provided in Appendix E, where five separate tables summarize the results across different perturbation scenarios:

• standard evaluation without perturbations,
• observation corrupted by Gaussian noise,
• observation dropouts with a fixed probability,
• action corrupted by Gaussian noise,
• action dropouts with a fixed probability.

Each table reports both the mean and standard deviation of episode returns for all seven algorithms (FullState, FRMA, ARMA, RMA, ADRQN, RPSP, TrXL) across six MuJoCo environments (InvertedPendulum-v4, Hopper-v4, HalfCheetah-v, Walker2d-v4, InvertedDoublePendulum-v4, Humanoid-v4).

These tables collectively demonstrate that FRMA consistently maintains higher mean returns and lower variance under all tested perturbations compared with baseline methods, confirming the framework’s superior robustness and reliability in partially observable and noisy environments. The detailed numerical results complement the qualitative trends illustrated in Figure 4 and provide a comprehensive view of performance under diverse operational conditions.

5. Conclusions and Future Work

This paper presented Four-Phase Rapid Motor Adaptation (FRMA), a modular framework that decouples latent-state inference from policy optimization to address high-frequency, partially observable control tasks. By combining supervised pretraining of a hidden-memory network with joint training of an LSTM encoder and a mapping network, FRMA enables single-step LSTM inference at deployment. This design allows the latent-state encoding $z_t$ to be updated at high frequency, thereby supporting rapid and robust control. Empirical results on the MuJoCo benchmark suite demonstrate that FRMA matches the performance of a full-state PPO agent while significantly outperforming existing RMA-based and recurrent policies when operating solely on observation–action histories.

Despite these promising results, several limitations and opportunities remain. First, although FRMA exhibits strong performance even with very short observation–action histories (e.g., $T=1,3,5$), indicating minimal cold-start effects, the initial hidden and cell states $(h_0,c_0)$ are currently zero-initialized. Using the hidden network to reconstruct $h\left(s_0\right)$ from an initial environment state could enable hot-start initialization, further improving first-step accuracy. Second, in our experiments, the fine-tuning of ${\pi }_2$ is performed with on-policy PPO, exposing the LSTM encoder only to trajectories generated by this policy. If deployment encounters observation–action sequences that deviate substantially from this distribution, inference errors may exceed those of standard PPO. However, FRMA is agnostic to the choice of policy optimization algorithm; ${\pi }_2$ could be replaced with off-policy methods such as SAC or other policy learners to enhance generalization and robustness to out-of-distribution sequences. Third, FRMA currently relies on full-state information from a simulator to train the hidden network. While real-time simulators may not always be available due to high-frequency requirements, data-driven or learned neural-network models could serve as surrogates for $s_t$, enabling FRMA training without a full simulator. Finally, FRMA currently tracks only static physical quantities or poses. Incorporating dynamic control targets as additional network inputs could enable tracking of time-varying objectives. Addressing these challenges will further enhance the robustness, versatility, and applicability of FRMA to high-frequency, safety-critical control domains.

Moreover, several recent works in power electronics and motor drives provide concrete evidence that the challenges addressed by FRMA—such as partial observability, parameter drift, nonlinear thermal–electrical dynamics, and high control rates—are active research concerns in industrial settings. For example, Dini et al. [38] demonstrates an adaptive model-predictive control scheme for six-phase synchronous motor drives that operates under sensor latency and load variations at high switching/control frequencies. Similarly, Dini et al. [39] present a predictive control design for resonant converters that models both thermal and electrical dynamics and operates with low computational complexity suitable for real-time use. Dini et al. [40] further leverage digital-twin simulations to train predictive models under diverse conditions, mirroring FRMA’s strategy of pretraining in simulated environments before adapting to sparse or noisy observations. These examples suggest that if FRMA’s latent-state reconstruction layer is deployed as an auxiliary context estimator or integrated with predictive controllers, the method could translate effectively to high-frequency industrial control applications, extending its impact beyond MuJoCo benchmarks.