Multi-Slot Attention with State Guidance for Egocentric Robotic Manipulation

Abstract

Visual perception is fundamental to robotic manipulation for recognizing objects, goals, and contextual details. Third-person cameras provide global views but can miss contact-rich interactions and require calibration. Wrist-mounted egocentric cameras reduce these limitations but introduce occlusion, motion blur, and partial observability, which complicate visuomotor learning. Furthermore, existing perception modules that rely solely on pixels or fuse imagery with proprioception as flat vectors do not explicitly model structured scene representations in dynamic egocentric views. To address these challenges, a multi-slot attention fusion encoder for egocentric manipulation is introduced. Learnable slot queries extract localized visual features from image tokens, and Feature-wise Linear Modulation (FiLM) conditions each slot on the robot’s joint states, producing a structured slot-based latent representation that adapts to viewpoint and configuration changes without requiring object labels or external camera priors. The resulting structured slot-based latent representation is used as input to a Soft Actor–Critic (SAC) agent, which achieves a higher mean cumulative return than pixel-only CNN/DrQ and state-only baselines on a ManiSkill3 egocentric manipulation task. Probing experiments and real-camera evaluation further show that the learned representation remains stable under egocentric viewpoint shifts and partial occlusions, indicating robustness in practical manipulation settings.

1. Introduction

Learning directly from pixels has driven major advances in reinforcement learning (RL), from human-level performance in Atari [1] to continuous-control policies in simulated environments [2]. In robotics, visual RL has enabled agents to acquire manipulation skills such as grasping [3], pushing and stacking [2], and large-scale goal-directed control [4]. These achievements highlight the potential of end-to-end visuomotor learning, but also reveal that challenges remain in how visual input is represented and aligned with control, particularly in embodied robotic settings.

One fundamental reason for this misalignment is the choice of viewpoint. Many visuomotor policies rely on external third-person cameras, which provide broad workspace coverage but are inherently decoupled from the robot’s action space [5]. In such views, the same motor command can produce different pixel changes depending on where the object lies in the external frame. Moreover, maintaining alignment with the robot’s workspace requires careful camera calibration and consistent placement. This may suffice in static or simulated environments but becomes challenging in real-world systems, particularly for mobile or multi-arm platforms, where maintaining stable external camera calibration and synchronization is difficult [6,7].

By contrast, egocentric vision, where the wrist camera offers a perspective naturally aligned with robot motion [8], grounds visual consequences in the manipulator’s workspace [9]. However, egocentric views also introduce challenges including viewpoint shifts, narrow fields of view, and partial observability as objects enter and leave the frame [10,11]. These issues are difficult to solve by vision alone, since the system cannot distinguish between changes caused by self-motion and those caused by object motion. Proprioceptive state provides this missing context, enabling the encoder to interpret visual changes relative to the robot’s configuration.

However, most visual RL pipelines were developed for pixel-only benchmarks, where proprioceptive state was deliberately excluded to test the limits of image-based learning. Methods such as DrQ [12], CURL [13], and RAD learn data-efficient visual representations but do not incorporate robot states. Furthermore, sequence models, such as Decision Transformer [14] process states and actions as separate tokens but still rely on globally pooled image embeddings rather than conditioning visual representations on the robot’s configuration. When state information is included, as in world-model approaches like Dreamer [15,16,17], it is generally fused with visual latents through simple concatenation, without structured cross-modal interaction. Related efforts in object-centric RL [18] highlight the importance of compositional representations but again lack explicit coupling between proprioception and attention. Insights from human sensorimotor coordination suggest a path forward and reflect the principle of embodied perception, where perception is shaped by the agent’s body configuration rather than processed independently. During reaching, the human gaze follows and adapts to hand motion, prioritizing visual features relevant to the current motor plan [19,20].

This perception–action alignment suggests that visuomotor systems should emphasize object regions and interactions that matter for the current control strategy rather than processing the entire visual scene uniformly. Following this principle, we propose a multi-slot attention architecture that couples perception and proprioception so that the robot’s body state shapes visual representation, and the resulting visual interpretation directly informs action selection, forming a closed perception–action loop.

The module first learns a set of spatially localized query vectors (slots) [21,22] that attend to image regions corresponding to task-relevant entities through cross-attention. To stabilize slot–object binding under egocentric viewpoint shifts, the visual slot encoder is initialized with a supervised perception pre-training phase. During this phase, the simulator provides ground-truth object poses, which are projected into 2D pixel coordinates to generate supervision targets. The encoder receives only RGB observations as input. The projected 2D coordinates are used solely as training targets to optimize slot attention and position prediction so that each slot aligns with a corresponding object region in the image. The policy and value networks are not trained during this phase. During subsequent Soft Actor–Critic (SAC) [23] training, the encoder receives RGB observations and the robot’s proprioceptive state. The encoder then computes slot attention over image tokens and applies Feature-wise Linear Modulation (FiLM) layers [24] to condition each slot on the robot’s proprioceptive state, producing modulated features. These modulated features serve as inputs to the policy and value networks. Gradients from the critic loss update the trainable encoder components, while the visual backbone remains frozen. This design integrates perception with control context and replaces flat pixel fusion with a slot-based representation. Slot attention and FiLM modulation maintain alignment between slots and object regions under egocentric viewpoint changes, improving robustness to viewpoint shifts and partial occlusions.

The proposed method was evaluated in the ManiSkill simulator [25] using an egocentric manipulation setup and compared against vanilla SAC (state-only), SAC with CNN, and SAC with DrQ-style image augmentation.

The main contributions are as follows:

• A state-guided multi-slot attention encoder that produces slot-based visual features from egocentric RGB observations.
• Proprioceptive conditioning of slot features via Feature-wise Linear Modulation (FiLM).
• An SAC training formulation in which slot-based visual features are modulated by proprioception and updated through actor–critic losses while keeping the visual backbone frozen.
• Empirical evaluation against SAC-based visual baselines in an egocentric ManiSkill manipulation setting.

2. Related Works

Recent visual RL research has focused on stabilizing training and improving sample efficiency in pixel-only domains. Data augmentation methods such as RAD and DrQ-v2 [12,26] improve stability by injecting invariances into pixel-level learning, while contrastive approaches such as CURL [13] shape visual encoders by aligning augmented views. Latent dynamics models, including SLAC [27] and the Dreamer family [15,16,17], learn compact latent spaces to model temporal dynamics, with Dreamer in particular using latent imagination rollouts for long-horizon control. These works established strong benchmarks for image-based reinforcement learning, but their encoders are typically trained from pixel observations alone and are not explicitly conditioned on the robot’s proprioceptive configuration. In many vision-based control frameworks, perception is processed through flattened convolutional features or globally pooled visual tokens, and any low-dimensional state is concatenated only in later MLP layers of the policy or value network, leaving the visual encoder itself blind to the robot’s configuration [2,4,13].

Before deep RL, alignment between vision and control was achieved through calibration-based pipelines. [28] remains a canonical approach to hand–eye calibration, computing the rigid transformation between a robot’s end-effector and its camera. Visual servoing [29,30] extended this concept by iteratively adjusting actions to minimize image space error. Although effective in structured settings, these methods rely on engineered geometric features and brittle calibration procedures, which limit scalability to dynamic, visually rich environments.

Learning-based visuomotor systems seek to overcome these constraints. QT-Opt [4] demonstrated large-scale off-policy learning for robotic grasping using multiple fixed third-person cameras. Transporter Networks [31] introduced spatial transport maps for pick-and-place actions but assumed static overhead views, avoiding viewpoint-shift challenges inherent to egocentric vision. More recent transformer-based architectures, including RT-1 [32], Gato [33], Decision Transformer [14], and PerAct [34], process visual observations as token sequences and condition policies on pooled embeddings. While these models scale effectively with data, they still rely on globally pooled visual representations or late-fused proprioception, rather than using it to structure the visual encoding around the robot’s embodiment. Related closed-loop manipulation frameworks further highlight the need for structured perception–control coupling in cluttered scenes, often introducing hierarchical controllers or auxiliary imitation signals to stabilize visuomotor learning [35].

To move beyond monolithic feature encoders, recent work in representation learning has explored structured visual representations, including object-centric approaches. Methods such as MONet [36], IODINE [37], and SCALOR [38] decompose images into latent slots that capture entities and spatial structure. However, these models are trained with reconstruction or prediction losses under primarily static or slowly varying camera assumptions. Relational approaches such as VIN [39], C-SWM [40], and Relational RL [18] model interactions between pre-segmented objects but do not address the problem of extracting structured representations directly from pixels under egocentric viewpoint shifts. Slot Attention [21] provides a flexible slot-based representation mechanism that can be integrated into downstream learning pipelines without requiring a dedicated reconstruction decoder. The proposed method adopts this mechanism and introduces state-guided slot attention that conditions visual feature organization on the proprioceptive state, enabling consistent perception under self-motion and dynamic views for object manipulation tasks.

3. Problem Formulation

We formulate vision-based robotic manipulation as a partially observable Markov decision process (POMDP) framework, defined by the tuple $\mathcal{M}=(\mathcal{S},\mathcal{O},\mathcal{A},\mathcal{T},r,\gamma )$, where $\mathcal{S}$ is the latent physical state (joint configuration, object poses), $\mathcal{O}$ is the observation space, $\mathcal{A}$ is the continuous action space, $\mathcal{T}(s_{t+1}\mid s_t,a_t)$ is the transition dynamics, $r:\mathcal{S}\times \mathcal{A}\to \mathbb{R}$ is the reward function, and $\gamma \in [0,1)$ is the discount factor.

At each timestep t, the agent receives a partial observation $o_t=(o_t^{\mathrm{img}},o_t^{\mathrm{state}})\sim \Omega (·\mid s_t)$, generated from the latent state $s_t$ through an observation function $\Omega$, where $o_t^{\mathrm{img}}\in {\mathbb{R}}^{H\times W\times 3}$ is an RGB image captured from an end-effector-mounted egocentric camera, and a low-dimensional proprioceptive state $o_t^{\mathrm{state}}\in {\mathbb{R}}^d$ including joint angles, velocities, and the end-effector pose. The agent samples an action $a_t\sim \pi (a\mid o_t)$, receives a scalar reward $r_t=r(s_t,a_t)$ and the process evolves according to the transition dynamics $s_{t+1}\sim \mathcal{T}(s_t,a_t)$. The learning objective is to find a policy that maximizes the expected discounted return:

$${\pi }^{*}=\arg \underset{\pi }{\max } {\mathbb{E}}_{\pi }\left[\sum _{t=0}^{\infty }{\gamma }^t r(s_t,a_t)\right],$$

where ${\pi }^{*}$ is the optimal policy, $\gamma$ is the discount factor, and the expectation is taken over trajectories induced by the policy $\pi$ under the environment dynamics. Since the egocentric camera provides only a partial and viewpoint-dependent observation of the workspace, the environment is inherently non-stationary from the robot’s perspective. The same motor command may produce different pixel changes depending on viewpoint, occlusion, or arm configuration. This coupling between motion and perception motivates a state-conditioned perception model, in which the encoder explicitly integrates proprioceptive state to interpret visual input relative to the robot’s configuration.

In the following sections, we introduce a multi-slot attention encoder modulated by proprioceptive state, enabling visual features to remain consistent under egocentric motion and occlusion. We then describe how this encoder is incorporated into Soft Actor–Critic to learn control-aligned representations for egocentric manipulation.

4. Proposed Method

The proposed method learns a control-oriented visual representation for egocentric robotic manipulation in which visual features extracted from the wrist-mounted camera are modulated by the robot’s proprioceptive state. The architecture consists of a visual encoder that extracts spatial features from the image observation, a slot attention module that produces structured visual tokens, and an FiLM modulation mechanism that adapts these features according to the robot’s current configuration. The resulting state-modulated representation is used as the observation input to the Soft Actor–Critic (SAC) policy. This design forms a bidirectional perception–action loop where the robot state modulates visual features and the policy acts on the resulting representation.

Formally, at each timestep t, the agent receives an egocentric RGB observation $o_t^{img}\in {\mathbb{R}}^{H\times W\times 3}$ and a proprioceptive state vector $o_t^{state}\in {\mathbb{R}}^d$. After standard preprocessing, we denote the image input as $I_t$. As shown in Figure 1, $I_t$ is encoded by a ResNet-18 backbone followed by a $1\times 1$ projection into a fixed channel dimension, producing a spatial feature map that is flattened into a sequence of visual tokens with positional encoding. The proprioceptive state is projected to a feature vector through a fully connected layer and used later for FiLM conditioning.

A small set of learnable slot queries attends to the visual tokens through multihead cross-attention and produces K spatially localized slot embeddings. If an additional goal token is available during RL, it is mixed with the pooled slot features using a lightweight two-way softmax gate. FiLM conditioning is then applied to the slot features to incorporate state information, producing the final latent representation passed into SAC. During RL, the ResNet backbone and $1\times 1$ conv remain frozen for stability, and gradients flow only through the trainable perception modules (beige background in Figure 1). This configuration allows the trainable fusion layers to adapt to the task reward, while the frozen backbone preserves low-level visual features learned during pre-training.

4.1. Learning Slot-Based Representations from Egocentric Vision

We begin by constructing a structured perceptual representation from egocentric images using a slot-based attention mechanism. Each slot specializes in a distinct scene region, providing the policy with separate visual features for each task-relevant object rather than a single global image embedding. The slot representation is trained in a supervised manner using 2D projections of task-relevant 3D keypoints, allowing the model to associate each slot with a consistent visual concept.

• Visual geometry and observation model: The hand-mounted camera provides egocentric visual observations. Slot supervision labels are generated by projecting simulator object positions onto the image plane using a standard pinhole camera model:

  $$(u_t^{\left(k\right)},v_t^{\left(k\right)})=\mathsf{\Pi }\left(P_t{X}_t^{\left(k\right)}\right),$$

  where $P_t=K[R_t\mid t_t]$ combines fixed camera intrinsics K and time-varying extrinsic $R_t,t_t$, and $\mathsf{\Pi }$ denotes homogeneous division with visibility masking. The resulting 2D coordinates serve as annotation-free supervision targets for slot position regression, requiring no segmentation masks or manual labels.

• Image encoder: The RGB image $I_t$ is first processed by a truncated ResNet-18 backbone, preserving the feature map up to the third convolutional block. This produces a spatial tensor $F_t=h_{\psi }\left(I_t\right)\in {\mathbb{R}}^{C^{\prime }\times H_f\times W_f},$ which is linearly projected to a feature space ${\mathbb{R}}^D$ through a $1\times 1$ convolution: ${\tilde{F}}_t=W_{\mathrm{proj}}\ast F_t.$ Flattening along spatial dimensions results in a sequence of $S=H_f{W}_f$ image tokens $X_t^{\mathrm{img}}=\mathrm{flatten}\left({\tilde{F}}_t\right)\in {\mathbb{R}}^{S\times D}.$ A learned 2D positional encoding $P^{2\mathrm{D}}\in {\mathbb{R}}^{S\times D}$, constructed from row and column embeddings, is added to preserve spatial locality: $X_t^{\mathrm{img}}=X_t^{\mathrm{img}}+\lambda P^{2\mathrm{D}},$ where $\lambda =0.05$ is empirically tuned to prevent the positional bias from dominating the learned features.

4.1.1. Learnable Slot Queries

To extract structured visual features, the model uses a set of learnable slot queries. The token sequence $X_t^{\mathrm{img}}\in {\mathbb{R}}^{S\times D}$ is obtained by flattening the spatial feature map produced by the visual encoder, where $S=H_f{W}_f$ is the number of spatial locations. Each token corresponds to a local region in the image. Instead of collapsing these tokens into a single global embedding, the model maintains a set of $N_s$ learnable slot vectors:

$$Q^{\mathrm{slot}}={[q^{\left(1\right)},q^{\left(2\right)},\dots ,q^{\left(N_s\right)}]}^{\top }\in {\mathbb{R}}^{N_s\times D},$$

which serve as queries in a multihead cross-attention module. Each slot query competes to attend to different regions of the image token grid, and auxiliary supervision during pre-training encourages each slot to specialize in a distinct task-relevant object. During RL, when a goal pixel $(u_t^{goal},v_t^{goal})$ is available, an additional goal query is constructed by sampling its corresponding feature vector from the spatial feature map ${\tilde{F}}_t$. This reuses the frozen encoder’s spatial features to represent the goal location without introducing additional learned parameters.

4.1.2. Cross-Attention

The slot queries use cross-attention to attend to the visual tokens. The query set is defined as:

$$Q_t=\left\{\begin{array}{cc}{Q}^{\mathrm{slot}},\hfill & \mathrm{pre}\text{-}\mathrm{training},\hfill \\ \left[ Q^{\mathrm{slot}}, Q_t^{\mathrm{goal}} \right],\hfill & \mathrm{RL}.\hfill \end{array}\right.$$

Given the sequence of image tokens $X_t^{\mathrm{img}}$, cross-attention is computed using multihead attention

$$Y_t=\mathrm{MHA}(Q_t, K=X_t^{\mathrm{img}}, V=X_t^{\mathrm{img}}).$$

Each query attends to the image tokens by computing similarity scores in the feature space and retrieving a weighted combination of spatial features. The resulting attention matrix $A_t\in {[0,1]}^{|Q_t|\times S}$ satisfies ${\sum }_j{A}_t[i,j]=1$ for each query i, representing a soft spatial assignment over the image tokens. Each output $y_t^{\left(i\right)}\in {\mathbb{R}}^D$ represents the visual features aggregated for one object location.

4.1.3. Permutation-Invariant Aggregation and Gating

The perception module is trained in two stages: pre-training on geometric supervision followed by RL fine-tuning through critic gradients. During pre-training, supervision uses a best-of-N matching scheme that selects the minimum-cost assignment between slots and objects at each step, allowing slots to reassign based on currently visible objects rather than maintaining a fixed slot-to-object mapping. This design improves pre-training stability under occlusion, where a fixed assignment would leave a slot without a valid supervision signal when its assigned object is out of view.

During RL fine-tuning, slot attention weights are updated by critic gradients to emphasize task-relevant features, causing attention patterns to drift from pre-training assignments and making slot swaps inevitable. Mean pooling is used as the slot aggregation function, producing a permutation-invariant representation that remains stable regardless of slot ordering:

$$z_t^{\mathrm{slot}}={\displaystyle \frac{1}{N_s}}\sum _{i=1}^{N_s}{y}_t^{\left(i\right)}.$$

When a goal query is available, the aggregated slot feature $z_t^{\mathrm{slot}}$ and the goal feature $y_t^{\mathrm{goal}}$ are combined through a learned soft gating mechanism:

$$z_t={\alpha }_t^{\left(s\right)} z_t^{\mathrm{slot}}+{\alpha }_t^{\left(g\right)} y_t^{\mathrm{goal}}, [{\alpha }_t^{\left(s\right)}, {\alpha }_t^{\left(g\right)}]=\mathrm{softmax}\left([{\mathsf{\ell }}_s, {\mathsf{\ell }}_g+b\left({\nu }_t^{\mathrm{goal}}\right)]\right),$$

where ${\mathsf{\ell }}_s,{\mathsf{\ell }}_g$ are learnable logits, and the bias term $b\left({\nu }_t^{\mathrm{goal}}\right)$ assigns a large negative value when the goal lies outside the camera’s field of view. Softmax gating keeps contributions normalized and suppresses the goal feature when it is not visible.

4.2. State-Conditioned Modulation

In egocentric manipulation, slot features encode spatially structured visual information in image space, but the relevance of those locations for control depends on the robot’s current configuration. The same visual feature can therefore have different control significance depending on the relative pose between the arm and the object. The slot representation $z_t$ must therefore be interpreted in the context of the robot’s current state before being passed to the policy. Rather than concatenating $o_t^{\mathrm{state}}$ with $z_t$, which treats state as an additional policy input rather than a signal that shapes visual interpretation, Feature-wise Linear Modulation (FiLM) conditions the visual features directly on the proprioceptive state. This allows the robot state to gate the visual features before policy inference so that control decisions operate on state-aware perceptual representations rather than raw visual embeddings.

Given visual feature $z_t\in {\mathbb{R}}^D$ and the proprioceptive observation $o_t^{\mathrm{state}}\in {\mathbb{R}}^{d_s}$, FiLM computes channel-wise scaling and shift parameters from the proprioceptive state and applies them to the slot features:

$${\widehat{o}}_t=W_s o_t^{\mathrm{state}}+b_s, {\gamma }_t=tanh(W_{\gamma }{\widehat{o}}_t+b_{\gamma }), {\beta }_t=W_{\beta }{\widehat{o}}_t+b_{\beta },$$

$${\tilde{z}}_t=(1+{\gamma }_t)\odot z_t+{\beta }_t,$$

where ⊙ denotes element-wise multiplication.

The tanh activation bounds ${\gamma }_t\in [-1,1]$, which results in the effective scaling factor $1+{\gamma }_t\in [0,2]$. This bounded modulation allows robot states to amplify or attenuate individual visual channels while preserving the sign and relative structure of the pre-trained slot representation. As a result, proprioceptive conditioning can adjust the relative importance of visual features according to the current arm-object configuration without distorting the learned slot representation.

The multiplicative term ${\gamma }_t$ performs channel-wise scaling, while ${\beta }_t$ applies a channel-wise shift, resulting in a feature-wise affine modulation of the slot representation. An ablation comparing FiLM variants is provided in Section 7.

4.3. Auxiliary Supervision and Pre-Training

Before policy learning, the perception module is pre-trained to establish spatial correspondence between egocentric image observations and physical object locations. This stage provides dense pixel-level supervision that would be difficult to derive from sparse task reward alone. During pre-training, the agent observes simulated rollouts generated by a random policy. For each timestep t, we collect the following:

$$\{ I_t, o_t^{\mathrm{state}}, I_{t+1}, o_{t+1}^{\mathrm{state}}, u_t, v_t, u_{t+1}, v_{t+1} \}.$$

Here $I_t$ and $I_{t+1}$ are egocentric RGB images, and $o_t^{\mathrm{state}}$ and $o_{t+1}^{\mathrm{state}}$ are the corresponding proprioceptive observations. The vectors $u_t,v_t\in {\mathbb{R}}^{N_o}$ contain the projected pixel coordinates of each object at time t, obtained from the calibrated camera model $P_t=K_t\left[R_t\right|t_t]$.

All supervision labels are derived from the simulator state via camera projection. The encoder is optimized using three auxiliary losses targeting slot position, object visibility, and attention alignment.

• Position regression: Each attention slot output $y_t^{\left(i\right)}$ predicts a pixel coordinate $({\widehat{u}}_t^{\left(i\right)},{\widehat{v}}_t^{\left(i\right)})$ through a small regression head. Because slots are exchangeable, we adopt a best-of-N matching scheme that aligns predicted coordinates with object labels via the minimal assignment:

  $${\mathcal{L}}_{\mathrm{pos}}=\underset{\sigma \in S_{N_s,N_o}}{min}{\displaystyle \frac{1}{N_o}}\sum _{k=1}^{N_o}{m}_t^{\left(k\right)}\parallel ({\widehat{u}}_t^{\left(\sigma \right(k\left)\right)},{\widehat{v}}_t^{\left(\sigma \right(k\left)\right)})-(u_t^{\left(k\right)},v_t^{\left(k\right)}){\parallel }_2^2.$$

  (1)

  Here $\sigma$ denotes a permutation (one-to-one assignment) from objects to slots, and $\sigma \left(k\right)$ selects the slot matched to object k under that assignment. $N_s$ is the number of slots, and $N_o$ is the number of tracked objects (or keypoints) for which pixel labels are available. The set $S_{N_s,N_o}$ contains all assignments $\sigma :\{1,\dots ,N_o\}\to \{1,\dots ,N_s\}$. In our implementation, we set $N_s=N_o=2$; this minimization reduces to evaluating the two possible matchings (identity vs. swapped) and taking the lower error, while $m_t^{\left(k\right)}\in \{0,1\}$ indicates whether object k is visible in frame t, masks out objects that are not visible and the loss is normalized by the number of valid objects in the batch. This encourages each slot to specialize consistently toward one object without enforcing a fixed ordering.
• Visibility prediction: A visibility head predicts the probability that each object lies within the camera’s field of view. A small visibility head takes the attention-derived features and outputs ${\widehat{m}}_t^{\left(k\right)}\in (0,1)$ for every object k, indicating whether its projected pixel location lies on-screen. The corresponding supervision signal is the geometric validity mask $m_t^{\left(k\right)}\in \{0,1\}$, derived directly from camera projection. We optimize a per-object binary cross-entropy loss (BCE):

  $${\mathcal{L}}_{\mathrm{vis}}=-{\displaystyle \frac{1}{N_o}}\sum _{k=1}^{N_o}\left[m_t^{(k)}log{\widehat{m}}_t^{(k)}+(1-m_t^{(k)})log(1-{\widehat{m}}_t^{(k)})\right].$$

  (2)

This loss encourages the encoder to recognize when an object’s projected location is valid and to avoid allocating attention to regions that fall outside the field of view.

• Attention alignment: Attention alignment supervises the attention mechanism to place probability mass at each object’s projected token location. Given the projected pixel location of object k, $\mathrm{idx}(u_t^{(k)},v_t^{(k)})$ provides the corresponding token index on the $H_f\times W_f$ feature grid. For each object, the summed attention probability across all slots at each object’s token index is:

  $$p_t^{\left(k\right)}=\sum _{i=1}^{N_s}{\mathcal{A}}_t[i, \mathrm{idx}(u_t^{\left(k\right)},v_t^{\left(k\right)})],$$

  and we define the alignment loss as follows

  $${\mathcal{L}}_{\mathrm{attn}}=-{\displaystyle \frac{1}{N_o}}\sum _{k=1}^{N_o}{m}_t^{\left(k\right)} log{p}_t^{\left(k\right)}.$$

  (3)

  In this formulation, object-level supervision is applied to the sum of slot-wise attention weights rather than a specific slot.
• The total pre-training loss is:

  $${\mathcal{L}}_{\mathrm{pre}}={\lambda }_{\mathrm{pos}}{\mathcal{L}}_{\mathrm{pos}}+{\lambda }_{\mathrm{vis}}{\mathcal{L}}_{\mathrm{vis}}+{\lambda }_{\mathrm{attn}}{\mathcal{L}}_{\mathrm{attn}}.$$

  (4)

  During this stage, all components of the perception module, including the ResNet18 backbone, slot attention, FiLM layers, and auxiliary prediction heads, are trained jointly. After pre-training, gradients to the backbone are disabled, and the encoder produces state-conditioned embeddings ${\tilde{z}}_t$ that are used as input to SAC.

4.4. Integration with Reinforcement Learning

After pre-training, the fused representation ${\tilde{z}}_t$ is used as the input to the SAC actor and critic networks. During RL optimization, gradients from the critic losses propagate through the trainable encoder layers, allowing the visual representation to adapt to the control objective. At each timestep t, the agent receives a partial observation

$$o_t=(I_t, o_t^{\mathrm{state}}),$$

where $I_t$ is the egocentric RGB image, and $o_t^{\mathrm{state}}$ is the proprioceptive vector containing joint angles, velocities, and gripper pose. These are mapped by the pre-trained fusion encoder $f_{\psi }$ into a structured latent representation

$${\tilde{z}}_t=f_{\psi }(I_t, o_t^{\mathrm{state}}, g_t),$$

where $g_t$ denotes a goal input when provided by the environment and is omitted otherwise. The encoder applies slot cross-attention to the egocentric image and modulates the resulting features through FiLM using the proprioceptive state.

This latent representation serves as the sole input to the SAC components. The policy is a stochastic actor ${\pi }_{\theta }(a_t\mid {\tilde{z}}_t)$ that outputs a Gaussian distribution over continuous actions $a_t\in {\mathbb{R}}^n$. Two critic networks $Q_{{\varphi }_1}({\tilde{z}}_t,a_t)$ and $Q_{{\varphi }_2}({\tilde{z}}_t,a_t)$, estimate soft Q-values, and a separate value network $V_{\eta }\left({\tilde{z}}_t\right)$ provides the bootstrap target used for temporal-difference updates through a slowly updated target network $V_{\overline{\eta }}\left({\tilde{z}}_t\right)$. Transitions $\{o_t,a_t,r_t,o_{t+1},{\mathtt{done}}_t\}$ sampled from the replay buffer $\mathcal{D}$ are used for the SAC updates. The reward function is defined in Section 4.5.

The critic target is computed as the soft Bellman backup:

$$y_t=r_t+\gamma (1-{\mathtt{done}}_t) V_{\overline{\eta }}\left({\tilde{z}}_{t+1}\right),$$

(5)

where $\gamma \in [0,1)$ is the discount factor, and $V_{\overline{\eta }}$ denotes a slowly updated target network. The critics minimize the temporal-difference loss

$${\mathcal{L}}_Q\left({\varphi }_i\right)=\frac{1}{2} {\mathbb{E}}_{\mathcal{D}} {\left[Q_{{\varphi }_i}({\tilde{z}}_t,a_t)-y_t\right]}^2, i\in \{1,2\}.$$

(6)

For the value and policy objectives, actions are re-sampled from the current actor, $a_t^{\prime }\sim {\pi }_{\theta }(·\mid {\tilde{z}}_t)$. The value network is then trained to regress toward the expected critic output under the current policy:

$${\mathcal{L}}_V\left(\eta \right)=\frac{1}{2} {\mathbb{E}}_{\mathcal{D}} {\left[V_{\eta }\left({\tilde{z}}_t\right)-\left(\underset{i}{\min }{Q}_{{\varphi }_i}({\tilde{z}}_t,a_t^{\prime })-\alpha \log {\pi }_{\theta }(a_t^{\prime }\mid {\tilde{z}}_t)\right)\right]}^2,$$

(7)

where $\alpha >0$ is the entropy temperature that controls the trade-off between reward maximization and policy stochasticity. The policy parameters are optimized by minimizing the entropy-regularized objective

$${\mathcal{L}}_{\pi }\left(\theta \right)={\mathbb{E}}_{\mathcal{D}} \left[\alpha \log {\pi }_{\theta }(a_t^{\prime }\mid {\tilde{z}}_t)-\underset{i}{\min }{Q}_{{\varphi }_i}({\tilde{z}}_t,a_t^{\prime })\right],$$

(8)

which encourages the actor to select actions with both high expected value and high entropy. Finally, the target parameters are updated via Polyak averaging:

$$\overline{\eta }\leftarrow \tau \eta +(1-\tau )\overline{\eta },$$

(9)

with $\tau \in (0,1)$ typically set to $0.005$. All components are optimized with the Adam optimizer using learning rates shown in

Table A1. Training hyperparameters.

Parameter	Value	Description
SAC
$\alpha$	0.0001	Actor network learning rate
$\beta$	0.001	Critic and value learning rate
$\gamma$	0.99	Discount factor
$\tau$	0.005	Target network soft update
Reward Scale	10	Adjusts reward magnitude
Hidden Layers	256, 256	Neural network layer sizes
Batch Size	128	Training batch size
Buffer Size	1 M	Maximum replay buffer size
Encoder Pre-training
Learning Rate	0.0001	Encoder learning rate
Feature Dim	42	Slot and FiLM feature dimension
Attention Heads	3	Multihead cross-attention heads
Pre-training Steps	150,000	Gradient steps for encoder pre-training
RL Encoder Fine-tuning
Fusion layers lr	0.0001	Learning rate for slot attention and FiLM
Goal gate lr	0.00003	Learning rate for goal gating logits
Optimizer	Adam	Both pre-training and RL fine-tuning
RL reward
$\alpha$	10	Scales obj-TCP distance
$\varpi$	5	Adjusts stability term
${\delta }_1,{\delta }_2$	0.025	Proximity threshold (environment set)

.

Algorithm 1 summarizes the overall training pipeline. At each iteration, a minibatch of transitions is sampled from the replay buffer and encoded into ${\tilde{z}}_t$ and ${\tilde{z}}_{t+1}$ by the fusion encoder. The soft Bellman backup (5) is computed using the target value network, and the twin critics are updated by minimizing the TD loss (6). The reparameterized actions $a_t^{\prime }\sim {\pi }_{\theta }(·\mid {\tilde{z}}_t)$ are then used to update the value network (7) and the actor (8). Finally, the target value parameters are updated by Polyak averaging.

Algorithm 1 State-guided slot attention with SAC.

Require: Replay buffer $\mathcal{D}$, encoder $f_{\psi }$, actor ${\pi }_{\theta }$, critics $Q_{{\varphi }_1},Q_{{\varphi }_2}$, value $V_{\eta }$, target value $V_{\overline{\eta }}$, batch size B, discount $\gamma$   1: Initialize network parameters $\psi ,\theta ,{\varphi }_1,{\varphi }_2,\eta ,\overline{\eta }$   2: for each training iteration do   3:     Sample batch ${\left\{(o_t,a_t,r_t,o_{t+1},{\mathtt{done}}_t)\right\}}_{i=1}^B\sim \mathcal{D}$   4:     Encode ${\tilde{z}}_t=f_{\psi }\left(o_t\right)$ and ${\tilde{z}}_{t+1}=f_{\psi }\left(o_{t+1}\right)$   5:     Compute critic target Equation (5)   6:     Update critics $Q_{{\varphi }_1},Q_{{\varphi }_2}$ by minimizing Equation (6)   7:     Backpropagate critic gradients into trainable encoder parameters $\psi$   8:     Sample reparameterized action $a_t^{\prime }\sim {\pi }_{\theta }(·\mid {\tilde{z}}_t)$   9:     Update value network $V_{\eta }$ by minimizing Equation (7) using $a_t^{\prime }$ 10:      Update actor ${\pi }_{\theta }$ by minimizing Equation (8) using $a_t^{\prime }$ 11:      Update target value network: Equation (9) 12: end for

The ResNet-18 backbone and 1 × 1 projection remain frozen throughout RL, while the slot attention, FiLM, and goal gate are updated only through critic gradients. Actor and value updates treat ${\tilde{z}}_t)$ as fixed to ensure the visual representation is refined toward task-relevant features without destabilizing the policy.

4.5. Reward Function

In egocentric manipulation, the wrist camera moves with the arm, and the distance between the end-effector and the object changes continuously during interaction. Effective control therefore requires reward signals that guide large movements across the workspace while also supporting precise alignment near the target. To capture these different interaction phases, the reward uses a multiscale shaping design that combines exponential and inverse-distance terms. These complementary components produce reward sensitivity across both large movements in the workspace and small positional adjustments near the target. The formulation is implemented through a piecewise design for reaching and placement with gated components.

• Reaching:

  $$r_{\mathrm{reach}}=\left\{\begin{array}{cc}exp(-\alpha \parallel {\mathbf{x}}_{\mathrm{obj}}-{\mathbf{x}}_{\mathrm{TCP}}\parallel ),\hfill & d>{\delta }_1\hfill \\ \frac{1}{\parallel {\mathbf{x}}_{\mathrm{obj}}-{\mathbf{x}}_{\mathrm{TCP}}\parallel +ϵ},\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

  where $d=\parallel {\mathbf{x}}_{obj}-{\mathbf{x}}_{TCP}\parallel$.
  The exponential term produces a smooth distance-based signal for global motion toward the object, while the inverse-distance term increases sensitivity when the end-effector is close to the object to support precise alignment.
• Grasping: A binary reward marks the transition from reaching to transport upon successful grasp.
• Placing:

  $$r_{\mathrm{place}}=\left\{\begin{array}{cc}exp(-\alpha \parallel {\mathbf{x}}_{\mathrm{obj}}-{\mathbf{x}}_{\mathrm{goal}}\parallel ),\hfill & \mathrm{if} d>{\delta }_2\hfill \\ \frac{1}{\parallel {\mathbf{x}}_{\mathrm{obj}}-{\mathbf{x}}_{\mathrm{goal}}\parallel +ϵ},\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$
• Post-grasp guidance: Once the object is grasped, an additional shaping term encourages movement toward the placement objective during transport. The term is defined as

  $$r_{\mathrm{pg}}=exp(-d_{\mathrm{goal}})·{⊮}_{\mathrm{grasp}}$$

  where $d_{\mathrm{goal}}=\parallel {\mathbf{x}}_{\mathrm{obj}}-{\mathbf{x}}_{\mathrm{goal}}\parallel$ and ${⊮}_{\mathrm{grasp}}$ indicates that the term is active only when the object is grasped. The exponential factor reduces the influence of this term as the object approaches the placement region so that the reward transitions smoothly to the placement objective.
• Stability: Once placement has been achieved, this stability term discourages residual joint motion so that the robot settles into a stable configuration before termination. It is defined as:

  $$r_{\mathrm{static}}=1-tanh(\varpi ·|{\dot{\mathbf{q}}}_{\mathrm{robot}}\left|\right)$$

  where $\varpi$ controls the sensitivity of the stabilization term.

The switching threshold ${\delta }_1,{\delta }_2=0.025$ corresponds to the environment-defined threshold indicating proximity to the object, and $ϵ$ is a small constant (${10}^{-5}$) to avoid division by zero. The placement and stability components activate only when the object is grasped and placed, respectively, enforcing sequential phase ordering. The contribution of each component across training is analyzed in Section 6.3, with further ablations provided in Section 7.2.

5. Experimental Setup

We evaluate the proposed method in a simulated robotic manipulation environment using a PickBanana task in ManiSkill [25]. The scene contains a seven-DoF robotic arm, a banana object, and a bowl positioned near a marked target region shown as a green point in Figure 2. In each episode, the agent must reach, grasp, and place the banana at the target location. The bowl serves as a movable object, introducing contact-rich interaction into the task.

Table 1. Environment and observation specifications for the PickBanana task.

Component	Specification
Robot model	7-DOF Franka Emika Panda
Camera setup	Egocentric RGB camera mounted on the end-effector (Pandawrist)
Observation vector $o_t^{\mathrm{state}}$	35-D proprioceptive state (joint positions, velocities, gripper pose)
Action space $\mathcal{A}$	7-D continuous action space
Objects	Banana, target location, and a bowl
Simulator	SAPIEN engine via ManiSkill framework (panda_wristcam agent)

summarizes the main environment specifications. The wrist-mounted camera follows the default pinhole perspective model in ManiSkill. Images were rendered at $128\times 128$ resolution synchronously at a control frequency of 20 Hz. No sensor noise or lens distortion was modeled during training. At evaluation, the robustness of the learned representations was assessed under three types of visual perturbation: random occlusion ($15\%$ of the image), additive Gaussian noise ($\sigma =0.05$), and camera-like affine perturbation ($3^{°}$ yaw rotation, $3\%$ translation, $1.05\times$ scale), applied independently to the input images.

Camera intrinsics and extrinsics were obtained directly from the ManiSkill simulator at each episode. The intrinsic matrix K defines the focal lengths ($f_x,f_y$) and principal point ($c_x,c_y$) at $128\times 128$ resolution. The extrinsic matrix $R_t$ ($3\times 4$) encodes the rigid transformation from world coordinates to the wrist camera frame, updated at every environment step to reflect the camera’s pose as the arm moves. These parameters are used to project 3D object positions into pixel-space for UV supervision only during perception pre-training. Complete camera specifications are provided in

Table A4. Comparison between the simulation wrist camera and mobile device camera for egocentric viewpoint approximation (Samsung Galaxy Note 20, main wide-angle lens).

	Simulation Wrist Camera	Samsung Galaxy Note 20
FOV	90°	∼79° (main wide-angle lens at 1080p video mode)
Resolution	128 × 128 at 20 fps	1920 × 1080 at 25 fps (streamed via IP Webcam app, downsampled to 128 × 128 for inference)
Model	Pinhole (no distortion during training)	Real lens with distortion
Near/Far	0.01 m/100 m	N/A
Mount	Wrist-mounted on camera_link (end-effector link) with identity relative pose	Handheld to approximate egocentric wrist viewpoint

.

5.1. Implementation Detail

All neural network components were implemented in PyTorch (v2.3.0, Python 3.9). The SAC agent was implemented following the original formulation by [23], which includes a stochastic actor network, two critic networks, and a value network. All SAC-based methods (state-only, CNN pixel, DrQ-v1 [41], and the proposed variants) share identical SAC hyperparameters, actor and critic architecture, and reward function. Only the visual encoder and input representation differ across methods. Three pixel-based baselines are included: a four-layer CNN with ReLU activations and linear projection; DrQ-v1 [41] following the original architecture with minor adaptations for the ManiSkill3 observation space and image resolution; and DrQ-v2 [12], following its original DDPG-based formulation and included as an additional reference baseline. Details are provided in Appendix A.5. The state-only baseline uses the proprioceptive state as input to the actor and critics without visual input. Hyperparameters are summarized in Table A1, and hardware specifications are provided in

Table A2. Computational setup (primary setup).

Component	Details
Operating System	Ubuntu 22.04.2 (jammy)
GPU Model	NVIDIA RTX A6000
CPU	AMD EPYC 7313P (16-Core)
CUDA Version	12.6

.

Visual observations are processed by a ResNet-18 backbone initialized with ImageNet-pre-trained weights. The backbone serves as a spatial feature extractor, providing low-level visual features over an $8\times 8$ spatial grid for $128\times 128$ inputs, and is therefore truncated after the third residual stage. The slot attention module uses $N_s=2$ slots, matching the number of task-relevant objects in the scene. FiLM scale and shift networks are zero-initialized, ensuring exact identity modulation at the start of RL training. The backbone is frozen during SAC optimization, while slot attention and FiLM layers remain trainable. Training is executed as a single run of 3000 epochs with 400 environment steps per epoch. During the initial data collection phase, the agent interacts with the environment for 100,000 steps to populate the replay buffer without policy optimization. The perception encoder is then pre-trained for $\mathrm{150,000}$ gradient steps using samples from this replay buffer. SAC optimization begins after pre-training and continues for the remaining epochs, with one gradient update per environment step. The encoder pre-training required approximately 100 min, and the full run required approximately $32.1$ h of wall-clock time.

5.2. Evaluation Objectives and Metrics

Evaluation targets two aspects of the method. First, the fusion encoder should improve reinforcement learning performance from egocentric RGB compared to pixel-only and state-only baselines. Second, the slot-based representation should remain stable over time and specialize in task-relevant objects. Both are evaluated through policy-level and perception-level metrics.

• Perception-level evaluation: Encoder quality is assessed during pre-training by reporting pixel-space position regression error between predicted UV coordinates and simulator-projected object locations, with visibility masking and swap-based matching, together with attention alignment NLL and visibility BCE loss. Furthermore, slot specialization is assessed qualitatively by visualizing attention maps across viewpoint changes and occlusion events.
• Policy-level evaluation: Policy performance is measured by final return and success rate, defined as the fraction of episodes in which the banana is placed within the target region. With two slots matching two foreground objects, position and visibility losses directly reflect whether each slot consistently tracks its assigned object.

6. Results and Discussions

6.1. Perception-Level Results and Discussion

Figure 3 shows the pre-training curves for all three losses. The attention alignment loss decreases sharply early in training and approaches near-zero values, indicating that slot queries learn to consistently select object-relevant regions rather than diffuse background features.

The position regression loss drops rapidly within the first portion of training and remains low, confirming that the UV heads recover accurate pixel-space object locations. The visibility loss follows a similar downward trend, stabilizing at low value, indicating that the encoder reliably distinguishes whether each object lies within the egocentric field of view.

Furthermore, Figure 4 shows that one slot persistently focuses on the banana while the other tracks the bowl across camera motion and contact-driven object displacement. When neither object is visible, attention concentrates on the end-effector, which remains within the field of view throughout the episode. This is a consistent agent-centric fallback in egocentric perception in which the end-effector is always in field of view. The auxiliary heads remain consistent with these attention patterns. As shown in Figure 5a–d, both objects are visible, and the predicted UV markers lie close to the corresponding ground-truth projections, demonstrating accurate localization across varied spatial configurations and scales. Figure 5e shows a harder boundary case where objects are near or outside the camera view due to ground-truth projections falling outside the valid region, and the visibility mask appropriately down-weights their contribution.

To examine whether slot attention generalizes beyond simulation rendering, printed images of a banana and a bowl were held in front of a Samsung Galaxy Note 20 smartphone and live-streamed through the encoder in real time without fine-tuning. The smartphone camera differs substantially from the simulation camera in resolution, FOV, and lens distortion (Table A4), with additional MJPEG compression artifacts absent during training.

As shown in Figure 6, slot attention maps remained localized on the printed targets, and position and visibility predictions were consistent with the observed objects across consecutive frames. Furthermore, when the camera approached one object closely, attention shifted toward it proportionally, redistributing as the viewpoint re-centered. This observation does not involve a robotic platform or closed-loop control and is not intended as a quantitative sim-to-real benchmark. However, it confirms that the learned representation remains coherent under real visual noise and viewpoint variation beyond simulation rendering.

6.2. Policy-Level Results and Discussion

Figure 7 shows the return curves for all evaluated methods. Among the baselines, the state-only SAC exhibits a delayed but steady rise after the mid-training phase, stabilizing at moderate returns, indicating that proprioception alone supports partial task completion but saturates below methods that incorporate visual features. The DrQ-v1 baseline achieves modest returns, typically stabilizing in the 300 range, while the CNN pixel baseline fails to learn. Both results indicate that raw pixel encoders struggle to form stable representations under wrist-mounted camera motion and occlusion.

The proposed method without goal conditioning achieves the highest return, stabilizing near 3.7–3.9 k, and converges fastest at around ∼1.1–1.2 k episodes. The proposed method with goal conditioning reaches the second-highest return, plateauing around ∼3.1–3.2 k. The lower return of the goal-conditioned variant reflects the added input complexity of goal conditioning, which slows early optimization while still producing a strong final policy.

Replacing FiLM with simple concatenation results in slower convergence and a lower final return, suggesting that channel-wise state modulation contributes beyond simple feature fusion. The variant w/o state (attention fusion output used alone) rises much later, peaks around ∼2.6–2.7 k, and then declines toward the end of training. This late collapse indicates that visual slots alone are insufficient to resolve the partial observability in egocentric views and that proprioceptive context is needed for stable control.

Figure 7 indicates that pixel-only baselines saturate at low returns, while state-guided slot fusion with FiLM conditioning achieves faster convergence and higher final performance. Figure 8 shows the full manipulation sequence: approach, grasp, transport, and placement, executed successfully. Furthermore, evaluating the final checkpoint over 50 deterministic rollouts results in a $92\%$ success rate, confirming that the return gains correspond to reliable task completion. Beyond task success, these results highlight practical aspects for robotic manipulation. Freezing the backbone after pre-training reduces computation during policy inference and makes the method easier to deploy on resource-limited platforms. The wrist-mounted camera provides an egocentric view that moves with the arm, so the policy observes objects from a consistent reference frame. Conditioning the representation on the robot’s proprioceptive state further aligns the policy input with the current arm configuration and supports stable control throughout the manipulation sequence.

6.3. Reward Component Analysis

The reward function defines three primary components: reach, post-grasp guidance, and place, each corresponding to a distinct stage of the pick-and-place task. To analyze how reward contributions evolved over training, the logged environment steps were divided into three equal windows: early (0–$0.4\times {10}^6$ steps), mid ($0.4$–$0.8\times {10}^6$ steps), and late ($0.8$–$1.2\times {10}^6$ steps). For each phase, the mean and standard deviation of each component’s fractional contribution to the total reward were computed over all recorded environment steps.

As shown in Figure 9 and

Table 2. Percentage contribution of reward components across training phases (mean ± standard deviation over environment steps).

Phase	Reach	Post-Grasp	Place
Early	$79.1\pm 13.4$	$0.3\pm 1.1$	$20.6\pm 13.4$
Mid	$72.9\pm 18.2$	$3.5\pm 4.1$	$23.6\pm 18.3$
Late	$31.9\pm 24.4$	$4.5\pm 5.0$	$63.6\pm 25.9$

, the reach reward dominates early training ($79.1\pm 13.4\%$), indicating that the agent initially focuses on approaching the object before acquiring grasping or placement behavior. The post-grasp guidance component remains near-zero during the early and mid phases, increasing only after approximately $0.6\times {10}^6$ steps ($4.1\pm 5.0\%$ in the late phase), consistent with the agent learning to maintain a stable grasp during object transport. The place reward shows a clear upward trend, increasing from $20.6\pm 13.4\%$ in the early phase to $63.6\pm 25.9\%$ in the late phase, indicating that successful object placement becomes the dominant reward signal as training progresses. The variance in the late phase reflects the step-level distribution of reward contributions across different task phases within each episode.

6.4. Visuomotor Robustness and Efficiency Evaluation

To examine what information is encoded in the learned latent representation $\widetilde{z_t}$, linear probes are trained on frozen encoder outputs collected from 10 rollout episodes across all ablations. Since the encoder is frozen and the probes are linear, strong probe performance indicates that the relevant signals are linearly decodable from $\widetilde{z_t}$ and directly accessible to the policy. The same probes are applied identically across all ablations (FiLM vs. concatenation, with vs. without proprioceptive state, and image-only and state-only variants) on identical rollout frames to isolate the effect of the fusion mechanism.

• Interaction alignment: Egocentric manipulation hinges on detecting when the robot has reached, contacted, or grasped an object. To test whether the latent representation separates interaction from non-interaction states, a linear classifier is trained to predict a binary contact label from $\widetilde{z_t}$, and performance is reported as AUROC. As shown in

  Table 3. Representation probing on PickBanana latents (mean ± std over three seeds). Contact AUROC measures interaction alignment. Δdist $R^2$ measures progress encoding. ΔObj $R^2$ is averaged over $x,y,z$ object displacement probes. Occ/Noise ΔAUROC reports the change in Contact AUROC under occlusion and noise (closer to 0 indicates higher robustness).

  Variant	Contact AUROC ↑	$\Delta$dist ${\mathit{R}}^2$↑	$\Delta$Obj ${\mathit{R}}^2$ (avg) ↑	Occ $\Delta$AUROC $\to 0$	Noise $\Delta$AUROC $\to 0$
  FiLM (full)	0.879 ± 0.019	0.841 ± 0.011	0.552 ± 0.029	−0.002 ± 0.026	−0.001 ± 0.006
  State-only	0.852 ± 0.009	0.813 ± 0.008	0.570 ± 0.025	$0.000\pm 0.000$	$0.000\pm 0.000$
  Image-only	0.638 ± 0.142	0.821 ± 0.022	0.529 ± 0.024	$-0.038\pm 0.101$	$-0.020\pm 0.101$
  Concat (no FiLM)	0.531 ± 0.206	0.846 ± 0.014	0.577 ± 0.035	$-0.058\pm 0.131$	$+0.042\pm 0.088$

  Note: ↑ indicates higher values are better; $\to 0$ indicates values closer to zero are better. Bold values indicate the best results.

  , the full FiLM variant achieves the highest contact AUROC $(0.879\pm 0.019)$, indicating that the latent representation separates contact from non-contact states more cleanly than other variants. The state-only variant also achieves high AUROC, consistent with the fact that contact correlates with proprioceptive cues.
• Progress and object-dynamics encoding: Contact detection alone does not capture how the end-effector approaches the object or how agent actions displace it. Two additional probes target these aspects directly. One-step TCP-to-object distance change $\mathsf{\Delta }{d}_t=d_{t+1}-d_t$ and object displacement $\mathsf{\Delta }ob{j}_t=ob{j}_{t+1}-ob{j}_t$ are regressed from $\widetilde{z_t}$, with performance reported as $R^2$.

As shown in Table 3, all variants achieve strong $R^2$ on both probes, indicating that TCP-to-object distance change and object displacement are linearly decodable across all fusion variants. The FiLM variant maintains these high scores while also achieving the strongest contact AUROC, whereas the concatenation variant achieves comparable Δdist/ΔR² but substantially lower contact AUROC. This indicates that concatenation preserves motion-related information but produces weak interaction alignment than FiLM.

• Egocentric robustness: Egocentric cameras introduce self-motion, occlusion, and sensor noise that can degrade learned representations. The trained contact probe is evaluated on perturbed latents ${\widetilde{z_t}}^{occ}$ and ${\widetilde{z_t}}^{noise}$ with robustness measured as AUROC drops: $\mathsf{\Delta }{AUROC}_{occ}=AUROC\left({\tilde{z}}^{occ}\right)-AUROC\left(\tilde{z}\right)$, and $\mathsf{\Delta }{AUROC}_{noise}=AUROC\left({\tilde{z}}^{noise}\right)-AUROC\left(\tilde{z}\right)$.

As shown in Table 3, the FiLM variant produces near-zero $\mathsf{\Delta }AUROC$ under both occlusion and noise, indicating that the contact signal remains stable under egocentric perturbations. The state-only variant produces $\mathsf{\Delta }AUROC=0$ by construction since it receives no image input. The image-only and concatenation variants show larger AUROC drops, indicating that pixel-based representations without proprioceptive state modulation are more sensitive to egocentric perturbations.

7. Ablation Studies

7.1. FiLM Conditioning Variants

Three state conditioning variants are evaluated against the proposed method to examine the role of FiLM modulation along two dimensions: where the state enters the pipeline and how it conditions the visual representation. The proposed method, FiLM with tanh (proposed), applies channel-wise multiplicative modulation to slot features using $(1+\tanh \left({\gamma }_t\right))\odot z_t+{\beta }_t,$ where ${\gamma }_t$ and ${\beta }_t$ are predicted from the proprioceptive state. In this formulation, the state conditions the visual representation directly at the slot feature level multiplicatively.

The first variant, FiLM linear+clamp, replaces the tanh scaling with a linear scale factor clamped to $[-1, 1]$. The conditioning mechanism and entry point remain identical to the proposed method. This variant isolates the contribution of smooth bounded modulation from the FiLM mechanism itself.

The second variant, concat+projection, replaces FiLM with a linear projection applied to the concatenation of the state and slot features, projecting the result back to the original feature dimension. Here the state still enters early at the slot feature level but interacts with the visual features additively rather than multiplicatively. This tests whether the state must actively reshape the visual feature space channel-wise or whether simple additive mixing is sufficient.

The third variant, proposed with concat (no FiLM), appends the state to the aggregated slot features only at the policy input stage after all visual processing is complete. In this case, the state never influences the visual representation itself and is provided only as an additional input to the policy. This variant tests whether passive perception with state as a side input is sufficient without explicit visual feature conditioning.

As shown in Figure 10, the concat+projection variant fails to learn a meaningful policy, indicating that additive fusion of state and visual features at the slot level is insufficient. The late concat variant learns the task but converges to a substantially lower return than the proposed method, showing that providing the state only at the policy input without conditioning the visual features is also inadequate. In contrast, both FiLM variants successfully learn the task. The tanh-modulated FiLM converges faster and reaches a higher stable return than the clamp variant. These results indicate that proprioceptive state must condition visual features early and multiplicatively, consistent with the embodied perception principle underlying the proposed method.

7.2. Reward Function Ablations

To evaluate the contribution and robustness of the reward design, we perform ablation experiments examining both the structural components and the sensitivity to key parameters.

• Stability sensitivity: The scaling coefficient $\varpi$ controls the sharpness of the velocity penalty in the stability term. We vary the stabilization parameter $\varpi \in \{1,5,10\}$ to test the sensitivity of the bounded velocity penalty. As shown in Figure 11, all three values eventually learn the task, confirming that the method is robust to this parameter. However, $\varpi =5$ achieves the fastest convergence and highest stable return. $\varpi =1$ produces the slowest and most unstable learning, while $\varpi =10$ delays convergence relative to the proposed value. These results confirm that $\varpi =5$ provides the most effective balance between penalizing residual motion and preserving the fine control needed during placement.

• Stability term removal: We remove the stabilization component to evaluate whether stable placement behavior emerges without explicitly penalizing residual joint motion.

Figure 12 compares the proposed method against a variant with the stability term removed entirely. Removing the stability term results in faster initial convergence but a lower final return, with the agent saturating around 3700 compared to 3900 for the proposed method. Both variants follow a similar learning trajectory until approximately episode 1250, after which the proposed method pulls ahead and stabilizes at a higher return.

• Reward-scale robustness: Ablation is done on the distance shaping parameter $\alpha \in \{5,10,15\}$ to examine the robustness of the multiscale reward shaping formulation.

The scaling coefficient $\alpha$ controls the rate of exponential decay in the reaching, post-grasp, and placement reward terms. As shown in Figure 13, all three values eventually learn the task, indicating that the method is robust to this parameter. However, $\alpha =10$ achieves the fastest convergence and the highest stable return. Reducing $\alpha$ to 5 slows convergence and introduces instability during the learning transition, while increasing $\alpha$ to 15 delays the onset of learning.

7.3. Surplus Slot Analysis

In the proposed method, the number of slots was set equal to the number of task-relevant objects $N_o=2$ in the main experiments. This ablation examines what happens when surplus slots are introduced, $N_s=4$ with $N_o=2$. During pre-training, only two slots receive pixel-space supervision targets corresponding to the task-relevant objects. The remaining slots have no explicit supervision and are free to attend to any image region. This setting tests whether surplus unsupervised slots destabilize slot binding. Figure 14 and Figure 15 show representative attention maps from the two slots that produce the strongest activation responses. The remaining slots exhibit weaker diffuse responses and are not selected by the position prediction head. Additionally, object localization remains highly accurate.

7.4. End-to-End Training Without Pre-Training

The main method initializes the visual encoder through a supervised pre-training stage before RL begins. This ablation evaluates the contribution of pre-training by training the full encoder end-to-end with SAC from random initialization, with no pre-training stage. The ResNet backbone, slot attention, and FiLM layers are all updated jointly through the critic loss from the start of training. This isolates the contribution of the pre-training stage and investigates whether the learned visual representation it provides is necessary for task performance, or whether critic gradients alone are sufficient to learn effective slot-based visual features from scratch.

Figure 16 shows the learning curve for an end-to-end variant. The return increases, indicating that the agent is learning. However, the policy never reaches successful task completion, and the return saturates below the performance achieved by the proposed approach. This behavior arises because the visual encoder must simultaneously learn object localization, stable feature binding, and control-relevant representations from the reinforcement learning objective alone. The RL signal is task-oriented and provides only indirect supervision for pixel-level representation learning, resulting in high-variance gradients that are weakly coupled to object localization and spatial reasoning. Pre-training the encoder therefore improves training efficiency by providing a stable visual representation before policy optimization begins.

8. Conclusions

This work introduced a state-guided multi-slot attention fusion architecture for egocentric robotic manipulation and integrated it with Soft Actor–Critic for end-to-end visuomotor control. In the PickBanana manipulation task, the proposed approach achieved the fastest learning and the highest final return among all compared methods. Ablations showed that the fusion framework remains strong with or without explicit goal input, FiLM-based state modulation is critical for sample-efficient and stable learning compared to naive concatenation, and proprioceptive conditioning is necessary for reliable control in partially observed egocentric views. Qualitative rollouts further confirmed that the learned representation supports coherent, temporally structured manipulation behaviors.

Overall, the findings demonstrate that slot-based representations, when anchored to the agent’s state, provide an effective and practical route to egocentric manipulation policies in simulation. This establishes a foundation for future work on broader task suites and transfer to real-robot egocentric settings, where occlusion and viewpoint changes are unavoidable. However, the method has a limitation in that it uses simulator supervision, and tasks that require more complex interactions have not been considered. Future work could explore replacing oracle simulator supervision with perception-only signals to enable real-robot transfer and experiment on more complex interactions.