Skip to Content
BiomimeticsBiomimetics
  • Article
  • Open Access

2 September 2026

37 Pages

Bio-Inspired Phase-Aware Skill Graph for Robust Long-Horizon Robotic Manipulation with Promptable Control

,
,
,
,
,
and
1
College of Information Science and Engineering, Northeastern University, Shenyang 110819, China
2
Shenyang Institute of Automation, Chinese Academy of Sciences, Shenyang 110016, China
3
University of Chinese Academy of Sciences, Beijing 101408, China
*
Authors to whom correspondence should be addressed.

Abstract

Long-horizon robotic manipulation requires coordinating multiple motor primitives under uncertainty, especially in contact-rich and changing environments. Existing end-to-end visuomotor policies often lack explicit temporal structure, causing brittle execution and limited recovery after disturbances. Inspired by biological motor control, where reusable primitives are organized through phase decomposition and feedback-dependent transitions, we propose a bio-inspired phase-aware framework that represents execution as structured transitions over perception-grounded motor phases. The framework integrates three modules: a Multimodal Phase-and-Primitive Detector that extracts semantically and physically consistent phases from visual, proprioceptive, and force–torque signals; a Multimodal Perception Skill Graph (MPSG) that encodes feasible phase transitions and supports skipping, rollback, and recovery; and Promptable Phase Control, which converts language instructions into graph-level ordering constraints for task reordering without retraining low-level policies. Experiments on four multi-stage tasks show improved robustness, increasing the average disturbed-condition success rate from 25.0% to 73.8% relative to the monolithic Action Chunking with Transformers (ACT) baseline.

1. Introduction

Biological motor control systems demonstrate remarkable robustness and adaptability when performing complex manipulation and locomotion tasks in uncertain environments. Rather than relying on monolithic execution strategies, biological systems organize behavior into hierarchically structured motor primitives such as reaching, grasping, lifting, and releasing. These primitives are coordinated through phase transitions, enabling flexible adaptation to perturbations and changes in environmental conditions. Neurophysiological studies suggest that such structured control is distributed across cortical and subcortical circuits, where phase-level organization plays a central role in stabilizing motor execution and reducing error propagation across time [1,2,3,4,5,6,7,8].
In contrast, contemporary robotic manipulation systems, particularly end-to-end visuomotor policies [9,10,11,12] and large vision–language–action models [13,14,15], typically lack explicit phase-level representations. Although these methods have achieved strong performance in short-horizon tasks, they often entangle perception, planning, and control within a single policy representation, making them vulnerable to compounding errors in long-horizon execution. As a result, small perturbations in contact-rich environments can propagate through the entire trajectory, leading to cascading failures and limited recovery capability.
To address these limitations, hierarchical imitation learning and skill-based decomposition methods have been proposed [16,17,18,19,20,21,22,23,24]. While these approaches improve compositionality, they generally rely on predefined skill boundaries or fixed execution graphs determined offline. Consequently, they cannot adapt the execution structure during runtime in response to unexpected physical feedback or environmental changes.
Existing approaches primarily treat temporal structure either as a latent variable or as a fixed planning artifact. However, in real-world manipulation, task feasibility is not static; it depends on evolving physical interactions such as contact states, force transitions, and geometric uncertainty [25,26,27,28]. This reveals a fundamental gap: current systems lack a perception-grounded, phase-level execution representation that supports dynamic reconfiguration during execution. This observation motivates a shift in perspective: robotic manipulation should be modeled as a phase-organized motor system with adaptive transitions between reusable primitives, rather than a fixed trajectory execution problem.
The term “bio-inspired” in this work refers to a functional abstraction of biological motor organization rather than a direct reproduction of neural circuitry. Specifically, the proposed framework adopts four principles: hierarchical separation between intention and motor execution, composition through reusable primitives, sensory-feedback-dependent phase transitions, and localized correction following disturbances. These principles directly motivate the separation between PPC and low-level policies, the node-based organization of motor primitives, multimodal perceptual gating, and rollback-based recovery, respectively.
In this work, we propose a biomimetic phase-aware manipulation framework that explicitly incorporates phase structure into long-horizon robotic execution.
First, a Multimodal Phase-and-Primitive Detector (PPD) segments demonstrations into semantically meaningful motor phases by leveraging visual, proprioceptive, and force–torque modalities, enabling physically grounded boundary detection under uncertainty.
Second, MPSG organizes these primitives into a structured execution topology, where nodes represent perception-conditioned motor skills and edges encode feasible transitions. Unlike static skill graphs, MPSG supports runtime reconfiguration, enabling phase skipping, rollback, and recovery based on real-time perception.
Third, Promptable Phase Control (PPC) provides a language-to-graph interface for execution-time task-order reconfiguration. PPC converts language instructions into deterministic graph-level constraints that reorder or activate previously learned primitives without retraining their low-level policies. The scope of this capability is limited to new execution orders composed from the existing primitive library and validated graph nodes; it does not imply zero-shot acquisition of unseen skills, objects, or environments.

3. Method

3.1. Biological Principles and Computational Abstraction

The proposed framework does not attempt to reproduce the detailed neural circuits, muscle synergies, or physiological dynamics of biological motor control. Instead, it computationally abstracts four functional principles that are consistently observed in biological motor organization: hierarchical control, reusable motor primitives, sensory-feedback-dependent phase transitions, and localized corrective responses. Details are shown in Figure 1, and the mapping between biological principles and computational abstractions is summarized in Table 2.
Figure 1. Bio-inspired phase-aware framework and its functional mapping to biological motor-control principles. The framework does not reproduce detailed neural circuitry; instead, it abstracts hierarchical intention, reusable motor primitives, sensory-feedback-dependent phase transitions, and localized corrective responses into PPC, node-level skill policies, multimodal phase perception, and graph-based recovery, respectively.
Table 2. Mapping between biological motor-control principles and their computational abstractions in the proposed framework. The framework abstracts functional organization principles rather than reproducing detailed neural circuitry or physiological dynamics.
First, biological motor behavior is hierarchically organized, with high-level intention and task organization separated from lower-level motor execution. This principle motivates the separation between Promptable Phase Control (PPC), the phase-level execution graph, and the node-specific visuomotor policies. PPC modifies task-level constraints, whereas the underlying ACT policies remain responsible for local motor execution.
Second, complex behavior is composed from reusable motor primitives rather than generated as a single monolithic trajectory. In the proposed framework, each MPSG node encapsulates a locally executable motor primitive together with its multimodal perceptual anchor. The same primitive can therefore be reused across different execution sequences without retraining its low-level policy.
Third, biological phase transitions depend on sensory evidence rather than on elapsed time alone. This principle is abstracted by the Multimodal Phase-and-Primitive Detector (PPD) and the perception-gated graph traversal mechanism. Visual, proprioceptive, and force–torque observations jointly determine phase boundaries and phase feasibility, thereby preventing purely time-driven progression.
Fourth, biological motor systems tend to contain disturbances through local corrective responses instead of restarting an entire behavior. MPSG abstracts this principle through skipping, rollback, and recovery transitions. These graph operations do not model physiological reflexes directly; rather, they provide a functional analogue of localized feedback-based correction at the phase level.

3.2. Multimodal Phase-and-Primitive Detector (PPD)

To clarify the task decomposition, we distinguish a phase from a primitive. A phase denotes a temporally contiguous interval characterized by locally consistent sensory–motor interaction patterns, whereas a primitive denotes the semantic action category assigned to that interval, such as Reach, Grasp, Move, Place, or Release. In other words, the phase specifies when a consistent interaction mode occurs, while the primitive specifies what action that segment represents.
To improve downstream execution robustness, particularly in contact-rich manipulation tasks, the extracted segments are rigorously required to satisfy dual criteria: semantic and physical consistency. Semantic consistency requires that frames within a phase share the same dominant primitive semantics, such as Reach, Grasp, Move, Place, or Release. Physical consistency requires that proprioceptive and force–torque patterns remain locally coherent within the segment, while phase boundaries are aligned with salient interaction changes, such as contact onset, gripper closure, or force release, rather than superficial visual fluctuations.
Driven by these criteria, long-horizon demonstrations are decomposed by the Multimodal Phase-and-Primitive Detector (PPD) into a structured sequence of semantically and physically consistent phase segments rather than unstructured frame-level labels, as illustrated in Figure 2. Each observation consists of multi-view RGB images, robot proprioceptive states, and force–torque signals. Language prompts are not used to determine temporal boundaries. Instead, they are used only after boundary detection to assign semantic primitive names to the segmented phases. Visual features are extracted via a pre-trained CLIP encoder [31], while a compact five-dimensional physical descriptor derived from the binary gripper state and external joint-torque signals is encoded using a lightweight MLP. Joint positions and velocities remain part of the complete robot observation and node-level policy input but are not directly concatenated into the reported PPD physical branch. These modality-specific features are fused into a unified representation for change-point detection. Phase boundaries are identified by detecting multimodal consistency shifts in the fused trajectory, ensuring alignment with physical interaction events rather than visual noise. The resulting segments are refined using an MS-TCN [32,33] to produce temporally consistent semantic primitives.
Figure 2. Overview of the Multimodal Phase-and-Primitive Detector (PPD). (a) Multi-view images and the complete robot observation are synchronized at 30 Hz. For phase segmentation, the physical branch uses a compact five-dimensional contact-oriented representation derived from the binary gripper state and external joint torques, while joint positions and velocities remain available to the robot-control and node-level policy components. The two visual embeddings and the encoded physical feature are fused before temporal modeling. (b) Candidate phase boundaries are generated from the fused features and refined by MS-TCN to produce temporally consistent semantic phase segments. Ellipses in the state sequence indicate omitted intermediate manipulation steps that are not the focus of this figure.
Multimodal Observation Representation: At each time step t , the robot observes a set of synchronized multimodal signals:
O t = I t ( r ) r = 1 N v , q t , q ˙ t , τ t ext , g t ,
where I t ( r ) ∈ ℝ H × W × 3 is the RGB image from the r -th camera, N v is the number of camera views, q t ∈ ℝ n q and q . t ∈ ℝ n q denote the robot joint positions and velocities, respectively, and g t denotes the gripper state. τ t ext ∈ ℝ n q denotes the estimated external joint torque. Equation (1) describes the complete observation available to the robotic system. Joint positions and velocities are used by the robot-control interface and node-level visuomotor policies; the reported phase-segmentation branch uses the compact contact-oriented representation defined below rather than directly concatenating the complete raw proprioceptive vector.
p t = g t , m t , Δ m t , a t , c t ⊤ ∈ ℝ 5 , m t = τ t S 2 ,         Δ m t = τ t S 2 − τ t − 1 S 2 , u t = τ t S τ t S 2 + ϵ ,         a t = 1 − u t ⊤ u t − 1 , c t = 1 | W t | ∑ j ∈ W t u j 2 ,         ϵ = 10 − 6 ,         | W t | = 4 .
Here, g t is the binary gripper state, τ t S contains the selected contact-sensitive external-torque channels, m t is the torque magnitude, Δ m t is the frame-to-frame torque-magnitude change, a t is the torque-direction change, and c t is the local torque-direction consistency computed over a four-frame window. Therefore, the physical representation consists of one gripper-state feature and four torque-derived contact features.
Semantic-Visual and Physical Encoding: Each camera image is encoded using OpenCLIP ViT-B/32 with the laion2b_s34b_b79k pretrained weights. The resulting image embedding is L 2 -normalized:
z t v , r = f CLIP I t ( r ) f CLIP I t ( r ) 2 + ϵ ∈ ℝ 512 .
The embeddings from the head and right cameras are concatenated:
z t v = z t v , h ; z t v , r ∈ ℝ 1024 .
Before physical encoding, the five channels of p t are standardized using channel-wise means and standard deviations computed exclusively from the training set. The standardized physical vector is processed by a two-layer physical encoder:
z t p = E p p t ∈ ℝ 64 ,
where E p consists of
E p : Linear ( 5 , 32 ) → ReLU → LayerNorm ( 32 ) → Dropout ( 0.1 ) → Linear ( 32 , 64 ) → ReLU → LayerNorm ( 64 ) → Dropout ( 0.1 )
Multimodal Feature Fusion: The modality-specific features are computed as
h t = E f [ z t v ; z t p ] ∈ ℝ 256 ,
where the concatenated feature has 1024 + 64 = 1088 dimensions. The fusion network E f is implemented as
L i n e a r ( 1088 , 512 ) → G E L U → D r o p o u t ( 0.2 ) → L i n e a r ( 512 , 256 ) → G E L U → D r o p o u t ( 0.1 ) .
The resulting 256-dimensional fused feature sequence { h t } t = 1 T is used for change-point detection and subsequently provided to the MS-TCN.
Physically Grounded Segmentation: Candidate phase boundaries are generated from the fused feature sequence h t t = 1 T using binary segmentation with a squared-Euclidean segment cost. For a temporal interval [ a , b ] , the segment cost is
C ( a , b ) = ∑ t = a b h t − h ¯ a : b 2 2 ,      h ¯ a : b = 1 b − a + 1 ∑ t = a b h t .
In the reported implementation, binary segmentation uses a penalty of 2.0, a minimum segment size of 3 frames, and a jump interval of 5 frames. The resulting pseudo-phase labels are temporally smoothed using an 11-frame window. During post-processing, a minimum phase duration of 15 frames (0.5 s at 30 Hz) is enforced before the pseudo-label sequence is used to supervise the MS-TCN.
Temporal Phase-and-Primitive Modeling: The candidate boundaries obtained from change-point detection are further refined using a Multi-Stage Temporal Convolutional Network (MS-TCN) to enforce global temporal consistency and eliminate boundary jitter. Let C p denote the number of primitive classes, p t = [ p t , 1 , … , p t , C p ] ⊤ denote the MS-TCN softmax output, and y t ∈ { 0 , 1 } C p denote the one-hot training label.
During training, the MS-TCN is supervised using frame-wise pseudo-labels generated by the physically grounded change-point detector. For each task, 20 development demonstrations were partitioned at the episode level into 16 training episodes and 4 validation episodes before temporal-window extraction. Across the four tasks, this resulted in 64 training episodes and 16 validation episodes. The training partition was used for gradient-based optimization, whereas the validation partition was used for early stopping, checkpoint selection, and hyperparameter selection.
An additional set of 20 successful demonstrations per task was independently collected and manually annotated, resulting in an 80-episode manual test benchmark. This benchmark was disjoint from the training and validation partitions and was used exclusively for the final quantitative evaluation of phase-segmentation performance after all model configurations had been fixed. The reported MS-TCN is optimized using frame-wise cross-entropy loss:
L s e g = − 1 T ∑ t = 1 T ∑ c = 1 C p y t , c log ( p t , c + ϵ ) .
Following the inference of frame-wise probabilities, continuous frames sharing identical maximum probability labels are aggregated into a structured set of discrete phase segments:
S = { s m } m = 1 M ,      s m = ( l m , t m s , t m e , ϕ m )
where M denotes the total number of partitioned phases within a single demonstration trajectory. For the m -th phase segment s m , l m represents the assigned primitive label, t m s and t m e signify the starting and ending time frames respectively, and ϕ m constitutes the multimodal perceptual anchor extracted within the temporal extent [ t m s , t m e ] . This segment-level formulation provides the necessary structural primitives and sensory anchors to directly initialize the downstream MPSG.
The detailed architecture, preprocessing procedure, and training configuration of PPD are provided in Appendix A, Table A1. All configurations were fixed before final evaluation and kept unchanged across the four evaluated tasks.

3.3. Multimodal Perception Skill Graph (MPSG)

While PPD decomposes demonstrations into semantically labeled and physically grounded phase segments, robust long-horizon execution requires an additional structure to model how these phases can be selected, skipped, repeated, or recovered during runtime. A fixed sequential execution order is insufficient in disturbed environments, because the physical preconditions of a phase may become invalid, already satisfied, or temporarily ambiguous. To address this problem, we introduce the MPSG, a perception-conditioned execution topology that represents manipulation as graph traversal over reusable phase-level skill units.
In MPSG, each node corresponds to a locally executable skill primitive equipped with a multimodal perceptual anchor, and each directed edge represents a feasible transition observed in demonstrations or validated through successful recovery traces. Therefore, the graph does not merely encode the nominal task order; it also stores alternative transitions that support runtime branching, phase skipping, rollback, and failure recovery. This formulation allows long-horizon manipulation to be executed as perception-grounded phase navigation rather than as a rigid temporal sequence.
Incremental Graph Construction: Unlike rigid state machines with manually predefined connections, the topology of MPSG is built dynamically. The initial directed graph G = ( V , E ) , V = { ν i } i = 1 N , E ⊆ V × V . is constructed by aggregating the PPD-segmented trajectories.
Nodes ( V ): Each node ν i = l i , π i , ϕ i ∈ V , where l i ∈ L is the semantic primitive label, π i is the node-specific ACT policy, and ϕ i is the multimodal perceptual anchor associated with the physical context of that node. The symbols i and j index graph-node instances, whereas t denotes time and k is reserved for indexing phase segments. Semantic names such as Open, Place, and Close are elements of the primitive-label set L rather than node indices. Multiple nodes may share the same semantic label, l i = l j , while retaining different perceptual anchors or execution contexts. Conversely, segments with sufficiently similar semantic and multimodal contexts may be merged into the same graph node.
Edges ( E ): Directed edges e i j = ν i , ν j , n i j , r i j ∈ E , represent feasible transitions explicitly observed and logged by the system. r i j ∈ { n o m i n a l , s k i p , r o l l b a c k , r e c o v e r y } , n i j denotes the total number of observed transitions from phase i to phase j across both the expert demonstrations and the validated recovery traces.
Experience-driven recovery behaviors are explicitly encoded in the graph through an incremental self-expansion mechanism. During graph construction and pre-evaluation development, validated recovery trajectories may be incorporated as recovery edges. For all quantitative experiments reported in this paper, graph construction is completed before evaluation and the resulting graph is frozen. No node, edge, edge count, perceptual anchor, or node-level policy is modified during evaluation.
Phase-Aware Graph Traversal: During execution, MPSG acts as a probabilistic state estimator and navigator. At each step t , the system determines the next skill node by evaluating the Perceptual Gating Score:
s j ( t ) = ω v cos z t v , ϕ j v + ω p cos z t p , ϕ j p ,
where ω v , ω p ≥ 0 , ω v + ω p = 1 .
Transition prior:
P E ( j ∣ i ) = n i j + ϵ E ∑ m ∈ N + ( i ) n i m + ϵ E ,
where N + ( i ) denotes the outgoing neighbors of node ν i and ϵ E > 0 is a smoothing constant.
Equation (10) represents a static prior obtained from demonstrations and validated recovery traces. At execution time t , this prior is reweighted using the observation-dependent perceptual score s j ( t ) of each candidate node:
P ¯ t ( j ∣ i t ) = exp s j ( t ) / τ g P E ( j ∣ i t ) λ E ∑ m ∈ N + ( i t ) exp s m ( t ) / τ g P E ( m ∣ i t ) λ E ,
where τ g > 0 is the softmax temperature controlling the sharpness of the normalized candidate scores; λ E ≥ 0 denotes the weight assigned to the empirical transition prior; and i t represents the index of the currently active graph node at time step t .
This mechanism enables dynamic execution:
Automatic Skipping: Suppose that the currently active node v i has the primitive label l i = O p e n , while a candidate successor v j represents a post-opening phase, such as l j = P l a c e . If the drawer has already been opened externally, the current observation becomes more consistent with the perceptual anchor ϕ j than with ϕ i . Consequently, s j ( t ) increases and the graph pointer may advance to the context-consistent successor, provided that the graph-connectivity and feasibility conditions are satisfied.
Robust Transitioning: Transitions are not blindly triggered by time; they are gated by physical reality (e.g., verifying a torque spike before transitioning from Grasp to Lift), ensuring that the robot never proceeds to the next stage prematurely.
Constraint Injection: Each parsed action a i is deterministically mapped to a graph operator: Node masking: ( v i ) ⇒ α i = 0 ; Pointer update: ( v j ) ⇒ v t ← v j ; Edge rewiring: ( v i , v j ) ⇒ e i j ∈ E .
Constraint-Aware Graph Execution: Let v i denote the currently active node, E t the active edge set, and M t the node-mask set at time t . In the absence of prompt-driven constraints, E t = E and M t = ∅ . Based on the perceptual score s j ( t ) and the normalized transition confidence P t ( j ∣ i ) defined above, the admissible candidate set is constructed as
A t = j | ( ν i t , ν j ) ∈ E t ,   j ∉ M t ,   s j ( t ) ≥ η ϕ ,
where η ϕ is the perceptual-feasibility threshold.
P t ( j ∣ i t , M t , E t ) = I [ j ∈ A t ] P ¯ t ( j ∣ i t ) ∑ m I [ m ∈ A t ] P ¯ t ( m ∣ i t )
The next active node is selected via
i t + 1 = argmax j ∈ A t   P t ( j ∣ i t , M t , E t )
Threshold Application: Four fixed thresholds are used in graph traversal. A candidate node first satisfies the perceptual-feasibility condition s j ( t ) ≥ η p , with η p = 0.70 . After combining the perceptual score with the empirical transition prior, node activation additionally requires P t ( j ∣ i ) ≥ η t , where η t = 0.80 . A nominal forward transition is enabled only after the current phase satisfies its completion criterion with confidence at least η c = 0.85 . Nonadjacent forward skipping uses the stricter condition s j ( t ) ≥ η s = 0.90 . All thresholds are fixed before evaluation and are identical across trials.
To guarantee robust execution and resolve edge cases during long-horizon manipulation, the graph transition logic enforces the following deterministic safeguards:
Tie-Breaking: If multiple nodes yield identical maximum probabilities, tie-breaking prioritizes the node with the highest empirical occurrence n i t j to favor historically reliable transitions.
Admissible Set Depletion ( A t = ∅ ): If no valid transitions exist, the framework triggers a localized rollback to the most recent stable phase. or when no feasible nominal or skipping successor remains. A stable rollback target is defined as a previously executed node that: (1) was successfully completed earlier in the current episode; (2) is connected to the current node by a validated rollback or recovery edge; and (3) satisfies the current perceptual-feasibility condition s j ( t ) ≥ η p . If multiple rollback targets satisfy these conditions, the most recently completed feasible node is selected. The graph pointer is reassigned to this node, and its associated node-level policy is reactivated.
Recovery Loop Prevention: Because rollback and recovery edges may introduce cycles, the executor maintains a trial-level recovery counter N r e c max . A recovery attempt is counted only when a rollback or recovery edge is actually activated. Multiple control cycles spent executing the same recovery node do not constitute additional attempts. The counter is initialized to zero at the beginning of every trial and incremented once per activated rollback/recovery transition. Once n r e c = N r e c max = 3 . Further recovery transitions are disabled when n r e c > N r e c max . If no feasible forward candidate is available at that point, the system performs a safe stop and records a recovery failure. In all experiments, N r e c max = 3 , as summarized in Appendix A, Table A3.
Feasibility-Gated Reconfiguration: Graph-edit directives, including FORCE_JUMP, remain subordinate to the perception-grounded execution constraints of MPSG. A requested target node is activated only when it exists in the current graph, is reachable under the active graph constraints, and satisfies the perceptual-feasibility condition s j ( t ) ≥ η p . If any of these conditions is violated, the command is rejected, the current graph pointer and active graph constraints remain unchanged, and the low-level policy associated with the infeasible target is not executed. This mechanism prevents language-derived graph commands from overriding the physical admissibility conditions of the robot execution process.
The numerical thresholds and deterministic execution parameters used by MPSG are summarized in Appendix A, Table A3. All values were fixed before the final evaluation and kept unchanged across tasks and baselines.

3.4. Promptable Phase Control (PPC)

To expose the MPSG as a controllable phase-level topology, Promptable Phase Control (PPC) is introduced. Rather than functioning as a conventional task planner that autoregressively generates action sequences, PPC is strictly formulated as a language-to-graph-constraint compiler coupled with a feasibility-aware graph edit interface. Unlike language-conditioned visuomotor policies that treat natural language as a soft, latent conditioning signal for end-to-end control, PPC translates language into deterministic structural constraints applied directly to the execution graph. The main notation used throughout this paper is summarized in Table 3.
Table 3. Main notation used in the proposed framework.
Semantic-to-Topological Projection: Let p ∈ P denote a natural-language instruction and Σ G denote the predefined graph-operation grammar. The constrained language parser is defined as
Φ LLM : ( p , Σ G ) ↦ a = ( a 1 , … , a R ) ,
where R is the number of parsed graph operations. Every operation is defined as
a r = o r , u r , σ r ,
where o r ∈ O G is the operator type, u r is an ordered tuple of node arguments, and σ r specifies the operation scope:
O G = { M a s k , U n m a s k , S e t P o i n t e r , A d d E d g e , R e m o v e E d g e , O r d e r } .
In the reported implementation, PPC is used to impose an ordered sequence over existing graph nodes. It neither creates a previously unseen node nor synthesizes a new low-level skill. The parsed sequence is converted into a temporary node mask and effective edge set, which are subsequently executed using the perception-gated MPSG traversal rule. If the requested sequence references an unavailable node, violates graph connectivity, or fails the current perceptual-feasibility check, the constraint is rejected. The current graph pointer and active graph constraints remain unchanged, and no low-level policy associated with the infeasible target is activated.
The LLM is bounded by a fixed instruction schema to emit only valid graph-edit operations (e.g., mask, set_pointer, rewire, force_jump), guaranteeing that natural language is compiled into a deterministic grammar rather than interpreted as free-form commands. These operations reconfigure the admissible phase-transition space while leaving the underlying low-level policies strictly frozen, thereby establishing a robust interface for runtime intervention. By manipulating the graph topology rather than altering policy weights, the system can enforce temporal constraints, bypass irrelevant phases, and proactively redirect execution toward feasible recovery nodes under external disturbances. Furthermore, to ensure physical stability during reconfiguration, the graph-edit interface is strictly feasibility-aware. A generated operation a k is accepted if and only if the target node v k satisfies the perceptual feasibility check governed by multimodal anchors; If the requested target node does not satisfy the perceptual-feasibility condition, the graph-edit command is rejected and the current graph pointer is retained. No low-level policy associated with the infeasible target is activated. Ultimately, this paradigm allows language to act as a hard, physically grounded topological constraint, fundamentally decoupling high-level human intervention from low-level motor generation.
Prompt-Driven Primitive-Order Reconfiguration: Because PPC modifies graph-level constraints rather than low-level policy parameters, it can impose a previously unevaluated execution order over the existing primitive library without policy retraining. For example, a nominal sequence A → B → C can be constrained to C → B → A , provided that the requested primitives and their corresponding graph nodes have already been learned and that the resulting transitions pass the perceptual-feasibility checks. This capability is restricted to recombination and reordering of previously trained primitives within the evaluated task domain. PPC does not acquire a new motor skill, infer a policy for an unseen object, construct an unobserved primitive, or guarantee transfer to a previously unseen environment.
PPC uses Kimi K2.6, accessed through an API, as a high-level semantic parser. The parser is invoked on demand once per new language instruction. The observed parsing latency is approximately 1 s and receives the user instruction together with the currently available primitive nodes in the MPSG. It converts the instruction into an ordering constraint over existing graph nodes and does not directly generate robot actions or low-level control commands. The parsed ordering constraint is subsequently processed by the MPSG executor, which searches for an executable graph route according to graph connectivity and the current perceptual-feasibility scores. Therefore, the language model specifies the intended task order, whereas the MPSG determines how that order can be physically executed.
Language-to-Graph Interface: Given the currently available primitive labels and a free-form natural-language instruction, the semantic parser returns a JSON-formatted suggested graph containing an ordered node list and corresponding directed edges. The parser is restricted to the supplied primitive vocabulary and cannot introduce unseen primitives or low-level actions. The suggested graph is not executed directly. It first undergoes deterministic syntax, schema, node-existence, and graph-validity checks. Valid outputs are converted into temporary MPSG constraints, while invalid outputs are rejected and leave the active graph unchanged. All accepted targets remain subject to the perceptual-feasibility checks of MPSG during execution.
The PPC execution flow is summarized in Algorithm 1.
Algorithm 1 Promptable Phase-Aware Execution (PPC)
Input: MPSG G = ( V , E ) , node policies {πi}, perceptual anchors {Φi}, optional language instruction p
1: Initialize the active node v i
2: E t ← E , M t ← ∅ , n r e c ← 0
3: While the task is not terminated do
4:        Acquire the synchronized observation O t
5:        if a new instruction p is received then
6:              Parse p into an ordered primitive sequence Q = l 1 , … , l L
7:              Resolve each label in Q to an existing graph node
8:              if all labels are resolved, the ordering is graph-consistent then
9:                       Construct tentative constraints M ′ t , E ′ t ← InjectOrder G , Q
10:                     if the requested target is perceptually feasible then
11:                               M t , E t ← M ′ t , E ′ t
12:                     else
13:                              Reject the instruction and retain M t , E t
14:                     end if
15:               else
16:                     Reject the instruction and retain the current graph
17:               end if
18:         end if
19:         Compute the perceptual scores s j t for enabled outgoing nodes
20:         Construct the admissible set A t i
21:         if  A t i ≠ ∅  then
22:               Select j * = arg max j ∈ A t ( i ) P t ( j ∣ i , M t , E t )
23:               Update the active node: i ← j *
24:          else if a feasible rollback target exists n r e c < N r e c m a x   π i O t   i
25:                 Set i to the most recent feasible stable node
26:                 n rec ← n rec + 1
27:          else execute a safe stop
28:                 record the trial as a recovery failure
29:                 break
30:          end if
31:          Execute the node-level policy π i ( O t )
32: end while

4. Results

We evaluate the proposed framework on four real-world long-horizon manipulation tasks involving sequential pick-and-place, contact-rich insertion/removal, multi-object rearrangement, and ambiguous-state manipulation. The experiments are designed to verify three hypotheses: (i) sensory-feedback-dependent phase organization should produce more reliable phase boundaries than visual observation alone, particularly under occlusion and contact ambiguity; (ii) localized phase-level correction should reduce premature transitions and improve recovery compared with monolithic or fixed sequential execution; and (iii) hierarchical separation between task-level intention and low-level motor execution should enable task-order reconfiguration without retraining the underlying skill policies.
All methods are evaluated using task success rate (TSR), premature transition rate (PTR), order compliance, and phase-segmentation metrics. For all skill-based methods, the same node-level ACT policies are used. The monolithic ACT baseline is trained with the same demonstrations and comparable backbone capacity but without explicit phase decomposition or graph-level execution control.

4.1. Experimental Setup and Evaluation Protocol

Experimental Setup: As shown in Figure 3, the experimental platform supports real-time, multi-view perception and teleoperated skill demonstration. It integrates a 7-DoF Franka Emika Panda robotic arm, a Gello isomorphic teleoperation device, and multiple depth cameras for synchronized RGB-D sensing. Computation is distributed across an NVIDIA Jetson AGX Orin (NVIDIA Corporation, Santa Clara, CA, USA) for model inference, skill perception, and high-level reasoning, and an Intel NUC for time-critical control. The system adopts a distributed runtime architecture. Model inference, multimodal skill perception, and MPSG traversal are executed on an NVIDIA Jetson AGX Orin, whereas the time-critical robot-control loop runs on an Intel NUC under real-time Linux. The two computers communicate through Gigabit Ethernet.
Figure 3. Overview of the robotic system architecture. (a) Experimental platform with a torque-controlled Franka Emika Panda robot and multi-view RGB-D perception. (b) Isomorphic teleoperation via Gello, mapping operator joint motions to robot joint commands. (c) Software architecture and data flow. Model inference, multimodal skill perception, and local graph execution run on an NVIDIA Jetson AGX Orin, whereas time-critical motion control is executed on an Intel NUC with real-time Linux. The optional PPC language parser is accessed through an external API.
The runtime system follows an asynchronous multi-rate architecture. The time-critical robot controller operates at 1 kHz on the real-time control computer, while the node-level ACT policies generate action references at 30 Hz on the Jetson AGX Orin. Online phase recognition is executed by an independent asynchronous monitoring process, with an average update latency of approximately 60 ms. The latest valid phase estimate is shared with the MPSG executor without blocking either the 30 Hz policy loop or the 1 kHz robot-control loop. Graph-pointer and constraint updates are event triggered and require approximately 10 ms after a valid phase or language constraint becomes available. Kimi K2.6 is accessed through an external API only when a new language instruction is issued. In the reported prompt-driven experiment, the language instruction was parsed before robot execution and therefore did not enter the continuous motor-control cycle. Detailed deployment modes and timing measurements are provided in Appendix B, Table A7.
For each of the four tasks, 20 demonstration episodes were manually annotated, resulting in a total of 80 annotated episodes. These episodes were excluded from model training and were used exclusively as reference annotations for evaluating the phase boundaries and primitive labels generated by the automatic phase-segmentation procedure. The annotations were produced by one of the authors. A phase boundary was assigned when a persistent change in the dominant sensory–motor interaction pattern was observed. Each resulting temporal segment was then assigned a primitive label from the predefined vocabulary. During quantitative evaluation, these manual annotations were treated as the reference labels for computing frame-wise accuracy, tolerance-based Boundary-F1, mean intersection over union, and boundary-detection metrics. Because the annotations were produced by a single annotator, inter-annotator agreement was not available. This limitation is considered when interpreting the reported segmentation results.
The task-specific completion criteria are event based rather than defined by manually specified metric thresholds. The target configuration of each primitive is determined from successful demonstrations and represented by the corresponding semantic phase and multimodal perceptual anchor. Therefore, quantities such as drawer displacement, insertion depth, block position, or knob rotation angle are not assigned additional manually tuned numerical thresholds for primitive completion. Small geometric deviations are tolerated as long as the intended semantic state is reached and the subsequent task can be completed successfully.
We evaluate four execution strategies for phase-aware execution and failure recovery:
ACT: A monolithic end-to-end visuomotor policy trained through imitation learning on complete long-horizon demonstrations.
Flat Primitive Sequence: A skill-based executor that invokes a predefined sequence of primitive policies without perception-conditioned graph traversal, phase skipping, or rollback.
SayCan-inspired Planner: A score-based high-level primitive selector that combines language compatibility and multimodal perceptual compatibility. For each candidate primitive j ,
S j SC ( t ) = 0.6 S j perc   ( t ) + 0.4 S j lang   ( p ) ,
where S j perc   uses the same multimodal perceptual scoring function as MPSG-Control and S j lang measures compatibility between the task instruction and the candidate primitive. The highest-scoring primitive is selected. The baseline uses the same frozen node-level ACT policies but maintains no explicit transition graph, rollback edges, or recovery history.
MPSG-Control: The proposed method, which performs perception-grounded phase execution using multimodal feasibility gating, graph-based transition selection, phase skipping, and rollback.
All experimental settings are summarized in Table 4. All methods are trained from the same demonstration corpus and evaluated using the same visual observations, robot states, low-level controller, hardware platform, disturbance conditions, and termination criteria. For all skill-based methods, the same node-level ACT policies and checkpoints are used. Therefore, the comparison isolates the effect of the high-level execution structure rather than differences in low-level policy capacity. The monolithic ACT baseline is trained on the original unsegmented trajectories, whereas the skill-based methods use phase-specific subsets derived from the same demonstrations. No additional task demonstrations are provided to any baseline. All methods were executed using the same robot controller, control frequency, camera streams, computing platform, disturbance protocol, and termination criteria. Evaluation trials were initialized using the same randomization ranges, and the execution order of the evaluated methods was randomized to reduce systematic bias. For all binary execution-level outcomes, we report the observed event counts and corresponding proportions. Two-sided 95% Wilson score confidence intervals are provided in Appendix D, Table A9, Table A10 and Table A11.
Table 4. Implementation settings and fairness controls for the evaluated methods. “Same” indicates that the corresponding configuration was shared with the proposed method.

4.2. Semantic Phase Segmentation Ablation

To evaluate the contribution of multimodal cues to semantic phase perception, we conduct ablation experiments on long-horizon manipulation tasks characterized by smooth transitions and visual ambiguity. Four representative real-world tasks are selected to assess robustness as illustrated in Figure 4:
Figure 4. Experimental tasks for long-horizon manipulation evaluation. (a) Sequential pick-and-place: objects are grasped and placed in a target sequence; (b) Contact-rich insertion/removal under occlusion: a peg is inserted into and removed from a socket with partial visual obstruction; (c) Multi-object rearrangement with spatial variability: multiple objects are repositioned with varying initial and target configurations; (d) Ambiguous-state manipulation with symmetric configurations: objects with rotational symmetry are manipulated under perceptually ambiguous orientations.
Task 1 involves sequential pick-and-place operations (open –> place –> close), requiring modeling of multi-stage dependencies.
Task 2 is a contact-rich charger insertion/removal task with severe visual occlusion, where key transition events are primarily reflected in force–torque signals rather than visual appearance.
Task 3 consists of multi-block stacking under randomized initial configurations, testing robustness to geometric variability and accumulated temporal drift.
Task 4 involves knob rotation and self-resetting switch manipulation, where identical initial and terminal states introduce ambiguity in purely visual segmentation.
For each task, 20 multimodal demonstrations were collected and manually annotated to provide ground-truth temporal boundaries. We compare three input configurations for phase segmentation: (i) Vision only, which uses multi-view RGB features. (ii) Vision + Proprioception, which additionally incorporates robot joint states and gripper states. (iii) Vision + Proprioception + Torque, which further includes external joint torque signals as physical interaction cues.
Phase-boundary F1: Phase-boundary localization is evaluated using a tolerance-based boundary F1 metric rather than the segmental temporal-IoU F1 metric. Predicted and groundtruth segmentations are first converted into frame-wise phase-label sequences. A boundary is extracted at frame t when the phase label changes between frames t − 1 and t . A predicted boundary b i pred   is counted as a true positive if it can be matched one-to-one with an unmatched ground-truth boundary b j gt satisfying
b i pred − b j gt ≤ δ .
Unmatched predicted boundaries are counted as false positives, while unmatched ground-truth boundaries are counted as false negatives. Boundary precision, recall, and F1 are then computed as
P B = N TP N TP + N FP ,      R B = N TP N TP + N FN , F 1 B = 2 P B R B P B + R B .
We report δ = 25 and 50 frames, corresponding to approximately 0.83 and 1.67 s at 30 Hz. A larger tolerance is less restrictive and may therefore produce a higher Boundary-F1 score.
Figure 5, Figure 6 and Table 5 evaluate temporal phase-segmentation performance rather than robot-control success. All three modality configurations are evaluated offline on the same recorded demonstration trajectories and against the same manually annotated reference boundaries. The robot motions, inverse-kinematics solutions, low-level controller, and physical execution data are therefore identical across the three configurations; only the observation channels supplied to the segmentation pipeline are changed. Under this controlled comparison, V + Prop + T achieves the highest observed Boundary-F1, frame-wise accuracy, and mIoU values among the evaluated configurations. These results indicate that proprioceptive and external joint-torque signals provide complementary temporal cues for phase-boundary localization in the evaluated demonstrations. They should not be interpreted as evidence that force feedback is uniquely necessary for successful robot control or that a conventional controller without force feedback must fail.
Figure 5. Qualitative phase segmentation in the contact-rich charger insertion/removal task. Representative keyframes show the principal task phases. The colored bars compare vision-only segmentation with the proposed Multimodal Phase-and-Primitive Detector (PPD). External joint-torque trajectories from joints J2 and J4 provide physical anchors for contact transitions. Dashed lines denote the detected phase boundaries, while the shaded regions indicate visually ambiguous contact intervals.
Figure 6. Quantitative phase-segmentation performance under different modality configurations. V denotes vision-only input, V + Prop denotes vision with robot proprioception, and V + Prop + T additionally includes external joint torque. Performance is evaluated using tolerance-based boundary F1 scores at tolerances of ±25 and ±50 frames, frame-wise accuracy, and mean intersection over union (mIoU). Error bars indicate standard deviations across the four evaluated tasks.
Table 5. Quantitative phase-segmentation results under different modality configurations.

4.3. Phase-Aware Execution and Failure Recovery

To evaluate phase-aware graph reasoning under long-horizon disturbances, we compare structured execution with monolithic and sequential baselines. All skill-based methods share identical node-level ACT policies, whereas the monolithic ACT baseline is trained on the same demonstrations using a comparable backbone capacity.
Using the baselines and fairness protocol defined in Section 4.1, we evaluate whether explicit phase-aware graph execution improves robustness under long-horizon disturbances. The comparison includes monolithic ACT, Flat Primitive Sequence, the SayCan-inspired planner, and the proposed MPSG-Control. All skill-based methods share identical node-level ACT policies.
Evaluation is conducted under two conditions: D1 denotes nominal execution without external intervention. D2 introduces runtime disturbances, including object displacement, externally completed subtasks, state reset after partial execution, and visual occlusion. These disturbances are designed to test whether the policy blindly advances according to a fixed temporal order or adapts according to perceived phase feasibility. External subtask completion denotes a condition in which an experimenter physically completes an upcoming subtask before the corresponding policy is executed, such as opening an already scheduled drawer or placing an object at its intended intermediate location. This disturbance tests whether the executor can recognize that the nominal phase has already been satisfied and skip the redundant phase. Partial-execution state reset denotes a condition in which, after one or more phases have been completed, the environment is returned to a valid state associated with an earlier phase. This disturbance tests whether the executor can redirect the graph pointer and resume from a physically consistent phase rather than continuing the original temporal sequence. Object displacement denotes a controlled change in the position of the task-relevant object during execution. Visual occlusion denotes temporary blocking of one or more task-relevant visual observations during a phase-transition interval. Grasp failures were not manually injected as a predefined disturbance category but could occur naturally during autonomous execution. Such failures could trigger the recovery mechanism of methods supporting rollback. Because recovery opportunities were not prospectively logged as an independent event-level variable with a complete denominator, we do not report a standalone recovery-success rate. Instead, the effect of recovery is evaluated through task-level success and premature-transition behavior under D2, together with the rollback ablation in Section 4.3.
For each task and each evaluated method, 20 independent execution trials were conducted under the nominal condition D1 and another 20 trials were conducted under the disturbed condition D2. Thus, each method was evaluated in 40 trials per task. Trials were reset independently, and no execution trajectory was reused across repetitions. At the beginning of each trial, the environment, graph pointer, recovery counter, and temporary graph constraints were reset to their predefined initial states. The frozen MPSG, perceptual anchors, and policy checkpoints were identical across repetitions; therefore, no trial could influence any subsequent trial. For each task, the 20 D2 trials followed a predefined disturbance composition: eight trials involved external completion of a subtask, eight trials involved resetting the environment to a previously encountered state after partial execution, two trials involved displacement of the manipulated object, and two trials involved temporary visual occlusion. The order of these disturbance instances was randomized across trials. The same disturbance composition and initialization protocol were used for all evaluated methods.
As shown in Figure 7, MPSG enables structured responses to environmental inconsistencies by reassigning execution pointers under state resets and suppressing redundant phases when subtasks are externally completed. These adjustments are achieved through graph-level reasoning without modifying low-level policies.
Figure 7. Perception-grounded graph reconfiguration under structured runtime disturbances. (a) After the environment is reset to an earlier valid state, MPSG detects the phase–state inconsistency and rolls the graph pointer back from Move Home to Close. (b) When the Close subtask is completed externally, MPSG recognizes that the corresponding phase is already satisfied and skips the redundant node. The node-level skill policies remain unchanged in both cases. The triangle indicates the current active node, and dashed lines represent robust execution paths.
A trial was considered successful when all required subtasks were completed, the final environment state satisfied the task-specific success criteria, and no manual intervention, safety termination, or execution timeout occurred. The task success rate is defined as (TSR)
TSR = N succ N trial ,
where N succ and N trial denote the numbers of successful and total execution trials, respectively.
A premature transition was recorded when the executor activated the next primitive before the task-specific completion condition of the current primitive had been satisfied. Under D2, the trial-level premature transition rate is defined as
PTR D 2 = N trial PT N trial D 2 ,
where N trial PT is the number of D2 trials containing at least one premature transition and N trial D 2 = 20 . Multiple premature transitions within the same trial are counted only once. Unlike frame-wise segmentation accuracy, PTR is an online execution-level metric. Under D2, the physical environment is modified after robot execution has begun, and the executor must retain or update the graph pointer according to the newly observed phase feasibility. A premature transition indicates that the execution structure failed to respond before an inconsistent successor primitive was activated. Therefore, a lower PTR reflects a stronger ability to suppress state-inconsistent phase progression under runtime disturbances. PTR does not measure the wall-clock delay of a successful graph update. Instead, it evaluates whether the online response occurs before an incorrect phase transition is executed. The online MPSG traversal and node-level policy operate at 30 Hz, while the low-level robot controller operates at 1 kHz.
As reported in Table 6, MPSG consistently outperforms baselines, achieving higher success rates and reduced PTR under D2 disturbances. Task 4 introduces a state-symmetric manipulation setting where visually indistinguishable configurations lead to ambiguity in phase identification. Such ambiguity causes both ACT and Flat Primitive Sequence policies to suffer from incorrect phase progression due to the absence of explicit structural disambiguation. MPSG mitigates this issue through phase-level consistency tracking and graph-constrained execution, resulting in improved robustness under state aliasing conditions.
Table 6. Long-Horizon Execution Robustness Across Structured Conditions.
PTR is not reported for Task4 because its interaction structure does not provide a unique ordered successor relation for defining premature activation. Therefore, the trial-level PTR analysis is restricted to Tasks 1–3, whereas TSR is reported for all four tasks. For Task1, Flat Primitive Sequence exhibits a trial-level PTR of 100% under D2, meaning that every disturbed trial contains at least one activation of a successor primitive before the completion condition of the current primitive is satisfied. Nevertheless, its D2 TSR remains 35% (7/20), because seven trials eventually reach the final task goal despite the premature transition. This result shows that a premature transition does not necessarily cause immediate terminal failure: in some cases, the downstream low-level policy can tolerate a moderate phase–state mismatch. However, consistently premature progression reduces execution reliability. TSR and PTR therefore characterize different aspects of execution: TSR measures the terminal task outcome, whereas PTR measures the correctness of phase progression under runtime disturbances.
To isolate the structural contributions of MPSG, we conduct an ablation study under the disturbed condition D2. No language instruction is issued during this experiment, and PPC is excluded from the ablation factors. All variants use identical node-level ACT policies and differ only in their high-level execution structures. We compare four variants: (1) MPSG (Full), which retains perceptual gating, graph-based forward transitions, skipping, and rollback; (2) w/o Rollback, which removes backward recovery transitions while retaining perception-conditioned node selection and forward skipping; (3) w/o Perceptual Gating, which retains the graph topology and rollback edges but removes observation-conditioned feasibility assessment; and (4) Flat Primitive Sequence, which executes the primitive policies in a fixed order without graph-based selection, skipping, or rollback.
As shown in Table 7, removing rollback substantially reduces TSR because local failures can no longer be redirected to a feasible predecessor phase. Removing perceptual gating markedly increases PTR, indicating that graph connectivity alone is insufficient when candidate nodes are not verified against the current multimodal state. Flat Primitive Sequence produces the weakest disturbance adaptation because it lacks both state-conditioned node selection and recovery transitions. These results show that perceptual gating and rollback provide complementary functions: the former prevents infeasible forward progression, whereas the latter supports localized recovery after execution inconsistencies.
Table 7. Structural ablation under D2. PTR is averaged over Tasks 1–3, for which premature-transition events are explicitly defined, whereas TSR is averaged over Tasks 1–4.

4.4. Structural Controllability via Hard Topological Constraints

PPC is evaluated separately from the MPSG structural ablation because it serves a different functional role. Perceptual gating and rollback determine whether the execution graph can progress and recover consistently under physical disturbances, whereas PPC converts language instructions into graph-level ordering constraints. Accordingly, PPC is evaluated using prompt-conditioned order compliance and task success. Recovery and rollback robustness are evaluated separately in Section 4.3 through disturbed-condition execution and structural ablation.
To evaluate whether language instructions can reliably induce execution-time reconfiguration, we study Promptable Phase Control (PPC) as a mechanism for mapping semantic prompts into graph-level constraints over the MPSG execution structure. Unlike approaches that treat language as a soft conditioning signal, PPC operates directly on the phase graph by applying deterministic topological masks. This enables controllable modification of execution order and activation constraints without the need to retrain low-level policies. To isolate the impact of the execution structure from control capability, we compare PPC against two baselines sharing identical low-level skill primitives:
LC-ACT: A stage-conditioned ACT baseline inspired by the stage-conditioning formulation of StageACT. The prompt is mapped to the existing semantic primitive vocabulary, and the currently requested primitive is encoded as a one-hot stage vector. This vector is projected to the ACT hidden dimension and introduced as an additional Transformer conditioning token. All other ACT components and training hyperparameters are unchanged.
Flat Planner: The instruction is parsed once into a fixed primitive sequence before execution. The corresponding frozen node-level ACT policies are invoked sequentially without online candidate re-scoring, perceptual graph gating, skipping, rollback, or recovery.
The robot-level PPC evaluation uses Open–Close–Open as a representative reordering case. This experiment is intended to evaluate whether a natural-language-specified order over existing primitives can be compiled into a valid MPSG constraint and executed without retraining the underlying policies; it is not intended to establish unrestricted natural-language or arbitrary graph-generation capability.
We evaluate prompt-driven task-order reconfiguration by specifying an execution order that was not used to train the low-level primitive policies. All requested actions are drawn from the same previously trained primitive library, and the same node-level ACT policies remain frozen throughout evaluation. Thus, the experiment evaluates whether graph-level constraints can reorganize known primitives at execution time rather than whether the system can acquire unseen motor skills. LC-ACT tends to follow its learned temporal priors and may deviate from the requested ordering under ambiguity. In contrast, PPC compiles the instruction into deterministic graph-level ordering constraints, resulting in higher order compliance while preserving the recovery edges of the underlying MPSG. These results demonstrate training-free primitive-order reconfiguration within the evaluated skill library and task domain. PPC is evaluated through prompt-driven task reordering.
The quantitative results, summarized in Table 8, validate the structural advantages of our approach. LC-ACT achieves only 30% (6/20) order compliance and 45% (9/20) task success, indicating that semantic conditioning alone does not reliably enforce the requested execution order. Flat Planner achieves perfect order compliance (20/20) but only 55% (11/20) task success, showing that explicitly specifying the desired primitive order does not by itself ensure robust task execution. MPSG-Control achieves both 100% (20/20) order compliance and the highest observed task-success rate of 75% (15/20), supporting the benefit of combining hard ordering constraints with perception-grounded graph execution in the evaluated reordering case. These results indicate that separating semantic instruction parsing from perception-grounded graph execution can preserve strict order control while improving disturbance recovery in the tested setting. This experiment is intended to evaluate execution-time reconfiguration of previously learned primitives rather than the general language-understanding capability of the API model. Perception-triggered skipping, pointer reassignment, and rollback-based recovery are evaluated separately in Figure 8.
Table 8. Quantitative Comparison of Execution Controllability and Robustness.
Figure 8. Performance comparison under prompt-driven task-order reconfiguration (Open -> Close -> Open). (Left) Qualitative execution flows. (Right) Quantitative results highlighting the trade-off between order compliance and task success across LC-ACT, Flat Planner, and the proposed PPC framework.

5. Discussion

The experimental results support the central hypothesis that explicit phase-level organization can improve the robustness and controllability of long-horizon robotic manipulation. In contrast to monolithic visuomotor policies, which implicitly couple perception, task progression, and motor execution within a single representation, the proposed framework introduces an intermediate phase abstraction between low-level skill execution and high-level task control. This abstraction allows local motor behavior to remain stable within each phase, while the global execution structure can be adapted through perception-grounded phase transitions.
In the evaluated contact-rich tasks, vision-only observations provided insufficient temporal cues for reliable phase-boundary localization. In contact-rich or partially occluded manipulation, critical phase transitions are often reflected in proprioceptive and force–torque variations rather than in visually salient image changes. By incorporating physical interaction cues into phase boundary detection, PPD produces more temporally consistent phase segments and reduces visually induced over-segmentation. This supports the biological motivation of the proposed framework, in which phase transitions are driven by sensory feedback rather than by a fixed temporal schedule.
The MPSG further demonstrates that graph-structured execution provides advantages over fixed sequential skill execution. A flat sequence can only advance according to a predefined order, making it vulnerable when a subtask fails, is externally completed, or becomes temporarily infeasible. In contrast, MPSG evaluates phase feasibility through multimodal perceptual anchors and transition priors. This enables the robot to suppress premature transitions, skip redundant phases, or roll back to a recoverable predecessor when the current phase becomes inconsistent with the observed state. As a result, local disturbances are handled at the phase level rather than propagating through the entire long-horizon task.
Promptable Phase Control (PPC) provides a complementary mechanism for high-level controllability. Its role is not to improve low-level motor execution directly, but to expose the phase graph as a structured interface for semantic intervention. In this paper, PPC is evaluated through prompt-driven task reordering, where language instructions specify a desired execution order that differs from the nominal sequence. By compiling such instructions into deterministic graph-level ordering constraints, PPC achieves strict order compliance while keeping the underlying low-level skill policies unchanged. Within the evaluated Open–Close–Open setting, representing the language instruction as a hard graph-level ordering constraint produced higher observed order compliance than the evaluated soft-conditioning baseline.
Scalability is primarily determined by the local branching structure of MPSG rather than by the total number of graph nodes. At each graph decision, the executor evaluates only the nominal, skipping, and validated recovery candidates connected to the active node, instead of performing a global search over the complete graph. For a fixed-dimensional perceptual representation, the resulting online traversal cost is linear in the number of local candidates. Nevertheless, graph storage, context-specific node duplication, and node-level policy storage increase as the task library expands. A detailed analysis of traversal complexity, graph expansion, memory growth, and primitive transferability is provided in Appendix C. Several limitations remain. First, the current graph topology depends on phase segments and recovery traces observed in demonstrations or validated during execution; therefore, recovery behaviors outside the collected experience may not be available unless they are discovered through additional interaction. Second, the primitive vocabulary remains predefined, which may limit scalability to more open-ended manipulation scenarios. Third, although PPC uses a constrained graph-edit grammar, the present experiments focus on task-order reconfiguration rather than more complex operations such as new skill insertion or open-ended subgoal generation. Finally, the current graph traversal relies on relatively simple perceptual matching and transition priors; more expressive uncertainty modeling may further improve robustness under severe ambiguity or partial observability.
Future work will explore automatic primitive discovery, uncertainty-aware graph traversal, and broader forms of promptable graph editing beyond task reordering. Another promising direction is to combine phase-aware graph execution with compliant low-level control, so that structural recovery at the task level and physical adaptation at the contact level can be jointly optimized for more complex real-world manipulation tasks.

6. Conclusions

This paper presented MPSG-Control, a functionally bio-inspired, phase-aware framework for structured execution in multi-stage robotic manipulation. The framework represents a demonstrated task as transitions among perception-grounded motor phases rather than as a single monolithic visuomotor trajectory. The Multimodal Phase-and-Primitive Detector uses visual, proprioceptive, and force–torque observations to identify phase segments; the Multimodal Perception Skill Graph organizes the resulting segments into node-level skill policies with nominal, skipping, rollback, and recovery transitions; and Promptable Phase Control converts a language instruction into an ordering constraint over previously trained graph nodes. Within the four evaluated manipulation tasks, the tested robot platform, and the predefined disturbance protocol, the experimental results indicate that multimodal phase perception can improve phase-segmentation accuracy relative to the evaluated reduced-modality variants. Under the tested runtime disturbances, perception-gated graph execution produced fewer premature phase transitions and higher observed task-success rates than the evaluated monolithic and fixed-sequence baselines. The prompt-driven experiment further showed that the existing Open and Close primitives could be executed in the requested open –> close –> open order without retraining their node-level policies. These results support the usefulness of explicit phase-level organization for the evaluated tasks, but they should not be interpreted as evidence of general robustness across arbitrary long-horizon manipulation problems. Likewise, the language experiment demonstrates task-order reconfiguration over an existing primitive library rather than open-ended language understanding, unseen-skill acquisition, or generalization to new objects, task structures, and environments. See Supplementary Materials for additional experimental videos.
Future work will evaluate the framework on larger task families, more diverse objects and environments, and broader prompt variations. Additional directions include automatic primitive discovery, uncertainty-aware graph traversal, scalable graph maintenance, and more general graph operations beyond task-order reconfiguration.

Supplementary Materials

The following supporting information can be downloaded at https://www.mdpi.com/article/10.3390/biomimetics11090622/s1. The following videos are provided as qualitative demonstrations of the proposed framework. Each video shows a representative execution example and is intended to illustrate the corresponding graph-transition or execution mechanism. The videos are not used to compute or independently validate the aggregate quantitative results reported in Section 4. Video S1: Long-horizon robustness under runtime disturbances (mpsg.mp4). This video presents representative long-horizon executions of MPSG-Control under manually introduced runtime disturbances. The demonstrated cases illustrate how the executor responds to changes in the physical task state through perception-grounded graph adaptation, including phase retention, pointer reassignment, skipping, and rollback when applicable; Video S2: Nominal execution of Task 1: Drawer manipulation. This video shows successful nominal execution of the complete drawer-manipulation sequence, including drawer opening, cookie grasping, transfer and placement into the drawer, drawer closing, and return to the initial configuration. No external disturbance is introduced. The video illustrates normal MPSG node transitions and nominal task completion; Video S3: Nominal execution of Task 2: Charger manipulation. This video shows successful nominal execution of the charger task, including approach, charger removal, transfer to the target location, insertion, release, and return. No external disturbance is introduced. The video illustrates normal graph progression through contact-rich manipulation phases; Video S4: Nominal execution of Task 3: Three-block stacking. This video shows successful nominal stacking of three blocks in the predefined sequence. Each block undergoes approach, grasp, transfer, and placement. No external disturbance is introduced. The video illustrates repeated traversal of phase-level manipulation primitives during a long-horizon task; Video S5: Nominal execution of Task 4: Button and rotary-switch manipulation. This video shows successful nominal execution of the button/rotary-switch task. No external disturbance is introduced. The video illustrates MPSG phase transitions in a manipulation sequence containing discrete interaction-state changes; Video S6: Recovery by graph-pointer reassignment (recover). This drawer-manipulation example corresponds to Figure 8 (left). After the robot has progressed to a later stage of the task, the environment is manually returned to an earlier incomplete state. MPSG detects that the current physical state is inconsistent with the active graph phase and redirects the graph pointer to a feasible predecessor phase, after which execution resumes from the restored state; Video S7: Skipping an externally completed subtask (skip). This drawer-manipulation example corresponds to Figure 8 (right). Before the corresponding robot primitive is executed, the experimenter manually completes the required subtask. MPSG recognizes that the associated phase has already been satisfied and skips the redundant primitive, allowing execution to continue from the next physically consistent graph node. These supplementary videos are selected representative trials and are provided solely to visualize the behaviors described in the manuscript. They should not be interpreted as additional experimental repetitions or as validation of the aggregate success rates. All quantitative results are derived from the independent evaluation trials reported in Section 4.

Author Contributions

Conceptualization, J.S., Y.L., X.Z. and Y.Z.; methodology, J.S., Y.L. and Y.C.; software, J.S. and X.W.; validation, J.S., X.W. and Y.J.; formal analysis, J.S. and Y.C.; investigation, J.S. and X.W.; resources, Y.L., X.Z. and Y.Z.; data curation, J.S. and X.W.; writing—original draft preparation, J.S.; writing—review and editing, Y.L., Y.C., Y.J., X.Z. and Y.Z.; visualization, J.S. and X.W.; supervision, Y.L., X.Z. and Y.Z.; project administration, Y.L., X.Z. and Y.Z.; funding acquisition, X.Z. and Y.Z. All authors have read and agreed to the published version of the manuscript.

Funding

Research supported by National Key R&D Program of China (Grant No. 2024YFB4708900), Key R&D Program of Liaoning Province (Grant No. 2024JH2/102400032), National Key R&D Program of China (Grant No. 2024YFB4709800), Young Scientists Fund of the National Natural Science Foundation of China (Grant No. 52405045), State Key Laboratory Independent Research Project (Grant No. E31 s0107), and in part by the National Natural Science Foundation of China under Grant U22A2067. (Corresponding author: Yang Luo and Yiwen Zhao).

Institutional Review Board Statement

Not applicable.

Data Availability Statement

The source code, trained model checkpoints, configuration files, manual phase annotations, task definitions, disturbance protocols, and evaluation scripts will be made publicly available upon publication at [repository (https://github.com/cheng9911/mpsg_control) (accessed on 11 August 2026)]. The complete raw multimodal dataset will be available from the corresponding author upon reasonable request. MPSG-Control—Bio-Inspired Phase-Aware Skill Graph (https://cheng9911.github.io/PhaseAct-Website/index.html) (accessed on 11 August 2026).

Conflicts of Interest

The authors declare no conflict of interest.

Abbreviations

PPD Multimodal Phase-and-Primitive Detector
MPSGMultimodal Perception Skill Graph
PPCPromptable Phase Control
ACTAction Chunking Transformer
LC-ACTLanguage-Conditioned Action Chunking Transformer
DoFDegrees of Freedom
CLIPContrastive Language–Image Pre-training
MS-TCNMulti-Stage Temporal Convolutional Network
TSRTask Success Rate
PTRPremature Transition Rate
mIoUMean Intersection over Union

Appendix A. Implementation and Reproducibility Details

Table A1. Implementation details and hyperparameters of the PPD phase-segmentation pipeline. All settings were fixed using the development training/validation partitions before final evaluation.
Table A2. Architecture and training configuration of the shared ACT policies used by the skill-based methods and the monolithic ACT baseline.
Table A3. Numerical thresholds and deterministic traversal and recovery parameters of MPSG. All values were fixed before final evaluation and kept unchanged across the evaluated tasks and baselines.
Table A4. Baseline implementation details S perc denotes the same multimodal perceptual compatibility score used by MPSG-Control. All ACT-based methods use the architecture and optimization settings reported in Table A2. No baseline receives additional robot demonstrations.
Table A5. PPC language-parser configuration.
Table A6. Task-specific primitive-completion and trial-success criteria used in the robot experiments.

Appendix B. Runtime Deployment and Computational Timing

Table A7. Summarizes the deployment modes and timing characteristics of the principal system components. The reported system uses an asynchronous multi-rate architecture rather than executing all modules sequentially within a single control cycle. Consequently, the individual latencies in the table should not be directly summed to obtain the 1 kHz robot-control period.

Appendix C. Scalability, Computational Complexity, and Transferability of MPSG

This appendix analyzes the computational and storage complexity of MPSG and clarifies the scope of the transferability claims. Let G = V , E denote the directed skill graph, where N = | V | and M = | E | are the numbers of graph nodes and edges, respectively. For the currently active node v i , let d i denote the number of enabled nominal and skipping successors, and let r i denote the number of validated rollback or recovery candidates. The dimensionality of each multimodal perceptual anchor is denoted by D .

Appendix C.1. Online Graph-Traversal Complexity

MPSG does not perform a global graph search at every policy step. Given the active node v i , the executor retrieves only its locally connected nominal, skipping, and recovery candidates. Candidate retrieval therefore requires O d i + r i operations when the graph is represented using adjacency lists.
For each candidate, the executor computes a multimodal perceptual matching score, combines this score with the corresponding edge-transition prior, checks the feasibility conditions, and identifies the candidate with the largest valid transition confidence. Computing a D -dimensional perceptual similarity for all local candidates requires
  O d i + r i D
time. Candidate masking, score normalization, and maximum-confidence selection additionally require O d i + r i time. The overall graph decision complexity is therefore
  O d i + r i D
Because D is fixed during deployment, the practical decision cost is linear in the local candidate count:
  O d i + r i
This local-search formulation means that increasing the total number of nodes does not directly increase the cost of every graph decision, provided that the local node degree remains bounded. In the worst case of a densely connected graph, d i + r i may grow proportionally to N , resulting in an upper-bound complexity of O N D per graph decision.

Appendix C.2. Graph-Expansion Complexity

Graph expansion may involve adding a validated transition edge or inserting a new context-specific node. With an adjacency-list representation, adding an edge requires constant structural insertion time once the source and target nodes have been identified. Any additional feasibility validation or duplicate-edge checking depends on the implementation and may require searching the corresponding local neighborhood.
Adding a new graph node requires storing its multimodal perceptual anchor and graph metadata, resulting in O D additional graph-level storage. If the new node is associated with k incoming and outgoing edges, its structural insertion requires O k additional edge-storage operations. When an independent node-specific policy with P parameters is assigned to the new node, an additional O P model-storage cost is introduced.
The present implementation expands the graph only through validated demonstration transitions and successful recovery traces. It does not perform unconstrained online creation of arbitrary new skills or graph nodes.

Appendix C.3. Memory Complexity

Using adjacency lists, the graph topology requires O N + M storage. The perceptual anchors require O N D storage. Excluding the low-level policy parameters, the complete graph-level representation therefore requires memory.
  O N D + M
If every graph node maintains an independent policy containing P parameters, the upper-bound storage requirement of the complete framework becomes
  O N P + D + M
Thus, storage increases linearly with the number of explicitly retained nodes and policies and with the number of graph edges. For sparse phase-transition graphs, M generally grows approximately linearly with N . In a theoretical densely connected graph, however, M can approach O N 2 . Context-specific duplication of semantically similar nodes and independent storage of node-level policy checkpoints may therefore become important limitations for substantially larger task libraries.

Appendix C.4. Transferability and Scope

The current experiments use one robot platform, a predefined primitive vocabulary, and relatively small task graphs. Accordingly, the demonstrated primitive reuse refers to recombination of previously trained primitives within the evaluated robot, object, sensing, and task domain. The present results do not establish direct transfer of the perceptual anchors or node-level ACT policies to unseen robot embodiments, new object categories, or substantially different environments.
Scaling MPSG to longer or more open-ended tasks may require hierarchical graph decomposition, clustering or merging of contextually redundant nodes, graph pruning, shared or parameter-efficient primitive policies, and uncertainty-aware retrieval of candidate nodes. These extensions are identified as future work rather than capabilities demonstrated by the current experiments.
Table A8. Computational and storage complexity of the principal MPSG operations.

Appendix D. Wilson Confidence Intervals for Execution-Level Binary Outcomes

The execution-level task success rate (TSR), premature transition rate (PTR) and order compliance are proportions estimated from binary trial outcomes. Because several evaluated conditions contain only 20 independent trials and include boundary cases such as 0/20 and 20/20, two-sided 95% Wilson score confidence intervals are reported instead of normal-approximation intervals. For an observed event count k from n independent trials, let p ^ = k / n . The two-sided 95% Wilson score interval is computed as
CI 95 % = p ^ + z 2 2 n ± z p ^ ( 1 − p ^ ) n + z 2 4 n 2 1 + z 2 n ,      z = 1.96 .
All intervals are reported as percentages and rounded to one decimal place. The intervals quantify the sampling uncertainty of each observed proportion; they are not interpreted as formal pairwise significance tests.
Table A9. Two-sided 95% Wilson score confidence intervals for the task-wise TSR and trial-level PTR results reported in Table 6. Each condition contains 20 independent trials. PTR is not reported for Task 4 because a unique premature-successor criterion is not defined for its state-symmetric interaction structure.
Table A10. Two-sided 95% Wilson score confidence intervals for the structural ablation results reported in Table 7. TSR is computed over 80 trials across Tasks 1–4, whereas PTR is computed over 60 disturbed trials across Tasks 1–3.
Table A11. Two-sided 95% Wilson score confidence intervals for the prompt-driven task-order reconfiguration results reported in Table 8.

References

  1. Flash, T.; Hogan, N. The coordination of arm movements: An experimentally confirmed mathematical model. J. Neurosci. 1985, 5, 1688–1703. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  2. Bizzi, E.; Mussa-Ivaldi, F.A.; Giszter, S. Computations underlying the execution of movement. Science 1991, 253, 287–291. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  3. Todorov, E.; Jordan, M.I. Optimal feedback control as a theory of motor coordination. Nat. Neurosci. 2002, 5, 1226–1235. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  4. Wolpert, D.M.; Ghahramani, Z. Computational principles of movement neuroscience. Nat. Neurosci. 2000, 3, 1212–1217. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  5. Flash, T.; Hochner, B. Motor primitives in vertebrates and invertebrates. Curr. Opin. Neurobiol. 2005, 15, 660–666. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  6. Mussa-Ivaldi, F.A. Modular features of motor control and learning. Curr. Opin. Neurobiol. 1999, 9, 713–717. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  7. d’Avella, A.; Bizzi, E. Shared and specific muscle synergies in natural motor behaviors. Proc. Natl. Acad. Sci. USA 2005, 102, 3076–3081. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  8. Scott, S.H. The computational and neural basis of voluntary motor control. Nat. Rev. Neurosci. 2004, 5, 532–546. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  9. Zhao, T.Z.; Kumar, V.; Levine, S.; Finn, C. Learning fine-grained bimanual manipulation with low-cost hardware. arXiv 2023, arXiv:2304.13705. [Google Scholar] [CrossRef] [Scilit]
  10. Zeng, A.; Florence, P.; Tompson, J.; Welker, S.; Chien, J.; Attarian, M.; Armstrong, T.; Krasin, I.; Duong, D.; Sindhwani, V.; et al. Transporter networks: Rearranging the visual world for robotic manipulation. Proc. Mach. Learn. Res. 2021, 155, 726–747. [Google Scholar]
  11. Brohan, A.; Brown, N.; Carbajal, J.; Chebotar, Y.; Dabis, J.; Finn, C.; Gopalakrishnan, K.; Hausman, K.; Herzog, A.; Hsu, J.; et al. RT-1: Robotics transformer for real-world control at scale. arXiv 2023, arXiv:2212.06817. [Google Scholar] [CrossRef] [Scilit]
  12. Chi, C.; Xu, Z.; Feng, S.; Cousineau, E.; Du, Y.; Burchfiel, B.; Tedrake, R.; Song, S. Diffusion policy: Visuomotor policy learning via action diffusion. Int. J. Robot. Res. 2025, 44, 1684–1704. [Google Scholar] [CrossRef] [Scilit]
  13. Zitkovich, B.; Yu, T.; Xu, S.; Xu, P.; Xiao, T.; Xia, F.; Wu, J.; Wohlhart, P.; Welker, S.; Wahid, A.; et al. RT-2: Vision-language-action models transfer web knowledge to robotic control. Proc. Mach. Learn. Res. 2023, 229, 2165–2183. [Google Scholar]
  14. Driess, D.; Xia, F.; Sajjadi, M.S.M.; Lynch, C.; Chowdhery, A.; Ichter, B.; Wahid, A.; Tompson, J.; Vuong, Q.; Yu, T.; et al. PaLM-E: An embodied multimodal language model. Proc. Mach. Learn. Res. 2023, 202, 8469–8488. [Google Scholar]
  15. Kim, M.J.; Pertsch, K.; Karamcheti, S.; Xiao, T.; Balakrishna, A.; Nair, S.; Rafailov, R.; Foster, E.P.; Sanketi, P.R.; Vuong, Q.; et al. OpenVLA: An open-source vision-language-action model. Proc. Mach. Learn. Res. 2025, 270, 2679–2713. [Google Scholar]
  16. Ross, S.; Gordon, G.; Bagnell, D. A reduction of imitation learning and structured prediction to no-regret online learning. Proc. Mach. Learn. Res. 2011, 15, 627–635. [Google Scholar]
  17. Osa, T.; Pajarinen, J.; Neumann, G.; Bagnell, J.A.; Abbeel, P.; Peters, J. An algorithmic perspective on imitation learning. Found. Trends Robot. 2018, 7, 1–179. [Google Scholar] [CrossRef] [Scilit]
  18. Sutton, R.S.; Precup, D.; Singh, S. Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning. Artif. Intell. 1999, 112, 181–211. [Google Scholar] [CrossRef] [Scilit]
  19. Bacon, P.-L.; Harb, J.; Precup, D. The option-critic architecture. Proc. AAAI Conf. Artif. Intell. 2017, 31, 1726–1734. [Google Scholar] [CrossRef] [Scilit]
  20. Eysenbach, B.; Gupta, A.; Ibarz, J.; Levine, S. Diversity is all you need: Learning skills without a reward function. arXiv 2018, arXiv:1802.06070. [Google Scholar] [CrossRef] [Scilit]
  21. Lynch, C.; Khansari, M.; Xiao, T.; Kumar, V.; Tompson, J.; Levine, S.; Sermanet, P. Learning latent plans from play. Proc. Mach. Learn. Res. 2020, 100, 1113–1132. [Google Scholar]
  22. Pertsch, K.; Rybkin, O.; Ebert, F.; Finn, C.; Jayaraman, D.; Levine, S. Long-horizon visual planning with goal-conditioned hierarchical predictors. Adv. Neural Inf. Process. Syst. 2020, 33, 17321–17332. [Google Scholar]
  23. Mishra, U.A.; Xue, S.; Chen, Y.; Xu, D. Generative skill chaining: Long-horizon skill planning with diffusion models. Proc. Mach. Learn. Res. 2023, 229, 2905–2925. [Google Scholar]
  24. Ijspeert, A.J.; Nakanishi, J.; Hoffmann, H.; Pastor, P.; Schaal, S. Dynamical movement primitives: Learning attractor models for motor behaviors. Neural Comput. 2013, 25, 328–373. [Google Scholar] [CrossRef] [Scilit] [PubMed]
  25. Hogan, N. Impedance control: An approach to manipulation: Part I—Theory. J. Dyn. Syst. Meas. Control 1985, 107, 1–7. [Google Scholar] [CrossRef] [Scilit]
  26. Suomalainen, M.; Karayiannidis, Y.; Kyrki, V. A survey of robot manipulation in contact. Robot. Auton. Syst. 2022, 156, 104224. [Google Scholar] [CrossRef] [Scilit]
  27. Lee, M.A.; Zhu, Y.; Srinivasan, K.; Shah, P.; Savarese, S.; Fei-Fei, L.; Garg, A.; Bohg, J. Making sense of vision and touch: Self-supervised learning of multimodal representations for contact-rich tasks. In Proceedings of the 2019 International Conference on Robotics and Automation; IEEE: New York, NY, USA, 2019; pp. 8943–8950. [Google Scholar] [CrossRef] [Scilit]
  28. He, Z.; Fang, H.; Chen, J.; Fang, H.-S.; Lu, C. FoAR: Force-aware reactive policy for contact-rich robotic manipulation. IEEE Robot. Autom. Lett. 2025, 10, 5625–5632. [Google Scholar] [CrossRef] [Scilit]
  29. Huang, W.; Mordatch, I.; Pathak, D.; Abbeel, P. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. In Proceedings of the International Conference on Machine Learning; PMLR: Cambridge, MA, USA, 2022. [Google Scholar]
  30. Ahn, M.; Brohan, A.; Brown, N.; Chebotar, Y.; Cortes, O.; David, B.; Finn, C.; Fu, C.; Gopalakrishnan, K.; Hausman, K.; et al. Do as I can, not as I say: Grounding language in robotic affordances. Proc. Mach. Learn. Res. 2023, 205, 287–318. [Google Scholar]
  31. Radford, A.; Kim, J.W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; et al. Learning transferable visual models from natural language supervision. Proc. Mach. Learn. Res. 2021, 139, 8748–8763. [Google Scholar]
  32. Abu Farha, Y.; Gall, J. MS-TCN: Multi-stage temporal convolutional network for action segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition; IEEE: New York, NY, USA, 2019; pp. 3575–3584. [Google Scholar]
  33. Li, S.-J.; Abu Farha, Y.; Liu, Y.; Cheng, M.-M.; Gall, J. MS-TCN++: Multi-stage temporal convolutional network for action segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 2023, 45, 6647–6658. [Google Scholar] [CrossRef] [PubMed]
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Article Metrics

Citations

Article Access Statistics

Multiple requests from the same IP address are counted as one view.