BioEgo-HSG: A Memory-Augmented Hierarchical Skill-Graph Framework for Long-Horizon Biomedical Laboratory Automation from Egocentric Videos

Abstract

Learning procedural knowledge from human egocentric videos offers a source of demonstrations for long-horizon biomedical laboratory automation. Such workflows span functional zones, depend on persistent object and workspace states, and include recovery from procedural deviations. This paper formulates BioEgo-HSG as a memory-augmented hierarchical skill-graph framework for this setting. Laboratory Interaction-Centric Tokens combine hand–object relations, object-state cues, zone information, task progress, and confidence. A multi-scale temporal parser organizes demonstrations into procedure phases, functional subtasks, and embodiment-agnostic atomic skills. Episodic procedural memory carries object, zone, task-progress, and failure-recovery information across changes in visibility. A hierarchical skill graph represents temporal, prerequisite, dependency, transition, alternative, and recovery relations. A graph-constrained planner uses this representation and the memory state to select semantic skills and handle postcondition mismatches through a local execution interface. The contribution is the framework formulation and its procedural representation, memory mechanism, and skill-graph planning interface. The reported offline summaries and physical demonstrations provide preliminary feasibility observations within the studied laboratory setting.

1. Introduction

Biomedical laboratories are increasingly adopting robotic workstations, automated instruments, and artificial-intelligence-assisted experimental platforms to improve reproducibility, throughput, and operator safety. Representative systems have shown that mobile robots can autonomously operate laboratory equipment, while self-driving laboratories can close the loop between experiment planning, execution, and analysis [1,2,3]. However, most successful systems rely on highly engineered workcells, standardized interfaces, and predefined protocols. Many routine biomedical procedures remain difficult to automate because they involve heterogeneous objects, deformable or transparent consumables, changing container states, strict cleanliness requirements, and movement among storage areas, inventory shelves, open benches, and enclosed operation spaces. Their automation therefore requires more than the accurate execution of isolated pick-and-place actions; it requires a persistent understanding of procedural progress and the ability to recover from deviations.

Human egocentric video provides a scalable source of such procedural knowledge. Unlike third-person recordings, first-person video naturally captures the operator’s visual attention, hand–object interactions, local object geometry, and the order in which task-relevant states become observable. Large-scale datasets, including EPIC-KITCHENS-100, Ego4D, and Ego-Exo4D, have substantially advanced egocentric activity recognition and the fine-grained understanding of skilled human behavior [4,5,6]. These developments suggest that wearable-camera demonstrations could reduce the need for extensive robot-specific data collection. Recent human-to-robot methods further support this possibility. R + X retrieves task-relevant clips from long, unlabeled human videos for immediate robot execution [7], while HumanEgo introduces an entity-level interaction representation to reduce visual and kinematic embodiment gaps [8]. Nevertheless, these approaches primarily address local or short-horizon skills and do not explicitly represent the persistent procedural state required by multi-zone laboratory workflows.

Long-horizon robot learning has commonly been addressed through hierarchical decomposition. Neural Task Graphs represent demonstrated procedures as reusable graph structures [9], Play-LMP learns latent plans from unstructured play data [10], and MimicPlay combines human play with a low-level visuomotor controller for long-horizon manipulation [11]. Large-scale demonstration synthesis, such as MimicGen, can further improve data efficiency and scene diversity [12]. At the execution level, Diffusion Policy provides expressive action generation for complex manipulation [13], whereas SayCan, PaLM-E, and vision–language–action models demonstrate how semantic reasoning can be connected to embodied skills [14,15,16]. Despite this progress, sequence prediction or language-based planning alone remains vulnerable to compounding errors when an object leaves the field of view, a prerequisite is silently violated, or a locally plausible action conflicts with the global experimental procedure.

This limitation is particularly critical in biomedical laboratories. A container may be retrieved from cold storage, verified at an inventory station, opened and processed on a bench, transferred to an enclosed workspace, and finally sealed, recorded, and returned. The meaning of a local action therefore depends on earlier operations and latent states that may no longer be visually observable. A system must remember whether an identity check has been completed, whether a sample has already been processed, whether a lid is open or closed, whether a workspace requires cleaning, and whether a failed action has introduced an unresolved risk. Existing egocentric recognition, robot imitation, and laboratory automation studies generally treat these problems separately. What remains missing is an intelligent information-processing framework that learns structured procedures from natural human demonstrations, preserves task-relevant states over long temporal gaps, and converts the inferred structure into executable and recoverable robot skill sequences.

For this setting, we propose BioEgo-HSG, a memory-augmented hierarchical skill-graph framework for long-horizon biomedical laboratory automation from egocentric videos. The design separates high-level procedural representation from embodiment-specific motion generation. Laboratory Interaction-Centric Tokens combine the visual appearance, object geometry, hand–object relations, discrete object states, functional-zone identity, and task-progress cues. A multi-scale temporal parser organizes demonstrations into procedure phases, functional subtasks, and embodiment-agnostic atomic skills. A hierarchical skill graph then represents temporal, prerequisite, object-dependency, zone-transition, alternative, and recovery relations. The framework is studied through a preliminary offline-to-online feasibility protocol.

BioEgo-HSG further introduces episodic procedural memory to retain object, zone, task-progress, and failure-recovery states after relevant entities leave the current view. During execution, a graph-constrained planner selects only skills whose preconditions are consistent with the current memory state. When a failure or state conflict is detected, the planner updates the memory, follows an appropriate recovery transition, and resumes from a valid graph state instead of restarting the entire procedure. The selected semantic skill, object arguments, target zone, and desired postcondition are finally passed to a local manipulation policy, allowing the high-level model to interface with imitation learning, vision–language–action policies, motion planning, or conventional robot controllers.

The contributions are fourfold. First, we formulate long-horizon laboratory automation as procedural learning with multi-zone transitions, persistent states, and delayed error propagation. Second, we introduce an interaction-centric hierarchical representation of phases, subtasks, and atomic skills from untrimmed egocentric videos. Third, we formulate episodic procedural memory and a hierarchical skill graph that organize preconditions, state updates, and recovery transitions for skill selection. Fourth, we provide an offline-to-online feasibility protocol linking procedure parsing, state tracking, graph construction, planning, and physical execution. These contributions concern the design and instantiation of the framework.

2. Related Work

2.1. Egocentric Human-to-Robot Learning and Long-Horizon Skill Modeling

Egocentric video provides a scalable source of human manipulation demonstrations because it captures the operator’s viewpoint, visual attention, hand–object interactions, and action consequences. Large-scale datasets such as Ego4D and Ego-Exo4D have advanced the first-person activity understanding and fine-grained analysis of skilled behavior [4,6]. Building on these resources, recent human-to-robot approaches transfer human experience to robotic manipulation. R + X retrieves task-relevant clips from long, unlabeled videos for in-context robot execution [7], while EgoMimic combines egocentric human demonstrations with teleoperated robot data to reduce visual and kinematic embodiment gaps [17]. These methods demonstrate the value of egocentric data, but they mainly focus on local behaviors whose objects and goals remain observable over short temporal windows.

Long-horizon manipulation requires reasoning over multiple dependent stages, where early errors may invalidate later actions. Hierarchical learning addresses this difficulty by separating high-level task organization from low-level control. Learning Latent Plans from Play extracts latent plans from task-agnostic play data [10], and MimicPlay combines human-derived high-level plans with robot-specific visuomotor policies [11]. Skill-chaining methods further improve long-horizon feasibility by modeling compatibility between consecutive skills. Adversarial skill chaining regularizes skill transition states [18], whereas Generative Skill Chaining models skill preconditions, effects, and dependencies to generate feasible plans [19].

BioEgo-HSG brings hierarchical procedure representation and persistent procedural-state information into a common formulation. Untrimmed demonstrations are organized into phases, subtasks, and atomic skills, and the resulting representation is connected to a semantic robot execution interface.

R + X [7] and Neural Task Graphs [9] provide related-work context. The reported configurations include two internal controlled baselines: a Retrieval-based selector, which selects the next skill from similar past segments without persistent procedural memory, and a Sequence-only graph, which models temporal succession without explicit prerequisite or recovery edges. Their definitions specify the modeling alternatives studied in the feasibility protocol; neither baseline is a reproduction of the corresponding external system.

2.2. Memory-Augmented State Tracking and Hierarchical Skill-Graph Planning

Long-horizon embodied tasks are partially observable because objects may become occluded, leave the current view, or change state while remaining relevant to later decisions. History-based models such as Episodic Transformer and History-Aware Multimodal Transformer preserve past observations and actions [20,21], while ReMEmbR, DynaMem, and ConceptGraphs provide structured spatial or spatio-semantic memory [22,23,24]. However, these approaches rarely represent laboratory-specific procedural states, including object verification, the processing status, workspace cleanliness, task progress, and unresolved failures.

Graph-based methods support long-horizon planning by modeling skill dependencies and feasibility. Neural Task Graphs infer reusable task structures from demonstrations [9], SayCan combines semantic relevance with skill affordances [14], Inner Monologue uses execution feedback for replanning [25], and PDDLStream represents explicit preconditions and effects [26]. Nevertheless, these methods often assume that the symbolic state required for planning is already accurate and available.

BioEgo-HSG integrates procedural memory with hierarchical skill-graph planning. Its memory tracks object, zone, task-progress, and failure-recovery states, while the graph represents skill dependencies, alternatives, and recovery transitions. The planner filters candidate skills according to the current memory state and resumes execution from a valid graph node after failure.

2.3. Biomedical Laboratory Automation and Research Gap

Biomedical laboratory automation has improved throughput, reproducibility, and safety through integrated platforms combining liquid handling, imaging, cell culture, and measurement. Recent systems, including LabDroid-based platforms, autonomous optimization laboratories, and mobile laboratory robots, further demonstrate that general-purpose robots can operate human-oriented instruments and coordinate multi-stage experiments [1,27,28,29,30,31,32].

However, most existing systems remain protocol- and hardware-centric. Procedures must usually be manually converted into scripts or symbolic workflows, while object locations, device interfaces, and operation sequences are predefined. They provide limited support for learning complete procedures directly from untrimmed human demonstrations or for tracking persistent states across storage, inventory, workbench, and enclosed operating zones.

BioEgo-HSG formulates procedural learning from egocentric videos together with object, zone, task-progress, and failure-state memory. Its skill graph represents preconditions and recovery transitions, while its execution interface links semantic skill selection to local manipulation and postcondition feedback.

3. Problem Formulation

We formulate long-horizon biomedical laboratory automation as a partially observable hierarchical procedural-learning problem. The system infers procedural structure from untrimmed egocentric videos, maintains persistent task states, and selects executable skills under workflow and state constraints, while separating high-level reasoning from embodiment-specific control. Table 1 summarizes the main notation.

Table 1. Notation and definitions.

Notation	Definition	Type/Domain
$𝒱$ = {It|t = 1, …, T}	Untrimmed egocentric video	Sequence
ot	Structured observation at time t	Vector/set
st	Latent procedural state	State in S
bt(s)	Belief over the latent state	Distribution
Mt	Procedural memory at time t	Memory in M
Pk	Procedure phase	Phase
Sk,n	Functional subtask	Subtask
zk,n,l	Atomic skill	Skill
zj = ⟨aj, Ojarg, Zjsrc, Zjdst, cjpre, cjpost⟩	Atomic-skill specification	Tuple
G = ($𝒱$G, 𝓔G)	Hierarchical skill graph	Graph
vt	Current graph node	Node in G
A(Mt, G, vt)	Admissible skill set	Set of skills
zt+1*	Selected admissible skill; * marks the planner’s optimum	Skill
g	Workflow goal	Goal set
ut	High-level execution command	Command
πφ	Local manipulation policy	Policy
at,1:Htlow	Low-level robot actions	Action sequence

Note: uppercase Z denotes a functional zone, whereas lowercase z denotes a semantic atomic skill. Throughout the paper, the hierarchical skill graph is written as G = ($V_G$, $E_G$), where $E_G$ denotes the graph-edge set. A superscript asterisk (*) marks the solution selected by the stated optimization: a highest-scoring admissible skill for planning, or fitted model parameters for learning.

3.1. Long-Horizon Laboratory Workflow Under Partial Observability

The biomedical laboratory workspace contains four functional zones $\mathcal{Z}=\left\{Z^{\mathrm{storage}},Z^{\mathrm{inventory}},Z^{\mathrm{bench}},Z^{\mathrm{enclosed}}\right\}$. A long-horizon workflow may repeatedly transition among these zones, making the validity of a local action dependent on both current observations and earlier operations. The n-th untrimmed egocentric demonstration is represented as ${\mathcal{V}}^{\left(n\right)}=\left\{I_1^{\left(n\right)},I_2^{\left(n\right)},\dots ,I_{T_n}^{\left(n\right)}\right\}$, where $I_t^{\left(n\right)}$ is the RGB frame at time t. Each frame produces a structured observation $o_t$ containing visible entities, hand-object relations, object-state cues, and functional-zone evidence.

Because the complete procedure is only partially observable, the latent procedural state is defined as

$${\mathrm{s}}_t=\left({\mathrm{s}}_t^{\mathrm{obj}},{\mathrm{s}}_t^{\mathrm{zone}},{\mathrm{s}}_t^{\mathrm{prog}},{\mathrm{s}}_t^{\mathrm{fail}}\right)$$

(1)

where ${\mathrm{s}}_t^{\mathrm{obj}}$ denotes the states of task-relevant objects, ${\mathrm{s}}_t^{\mathrm{zone}}$ represents the status of functional zones, ${\mathrm{s}}_t^{\mathrm{prog}}$ describes procedural progress, and ${\mathrm{s}}_t^{\mathrm{fail}}$ records unresolved failures or abnormal conditions. Since $o_t$ reveals only part of ${\mathrm{s}}_t$, the system estimates the state belief from observation and skill histories:

$$b_t\left(\mathrm{s}\right)=p\left({\mathrm{s}}_t={\mathrm{s}|\mathrm{o}}_{1:t},{\mathrm{z}}_{1:j\left(t\right)-1}\right),$$

(2)

where $o_{1:t}$ denotes all observations available up to time $t$, $z_{1:j\left(t\right)-1}$ denotes the sequence of semantic skills completed before the current decision point, and $j\left(t\right)$ denotes the index of the skill to be selected at time $t$. Given a workflow goal $g$, BioEgo-HSG maintains a procedural memory $M_t$ based on current and historical evidence. As shown in Figure 1, the planner uses $M_t$, the hierarchical skill graph $G$, and $g$ to identify the set of admissible skills and select the next executable skill $z_{j\left(t\right)}^{*}$. Execution feedback is then used to update $M_t$ for subsequent planning.

Figure 1. Simplified problem formulation of BioEgo-HSG under partial observability. The current egocentric observation provides incomplete evidence about the latent procedural state. The procedural memory integrates current evidence with historical observations and executed skills, while the graph-constrained planner uses the updated memory, workflow goal, and hierarchical skill graph to select the next executable semantic skill. The superscript asterisk (*) marks the admissible skill selected by the planner in Equation (8).

3.2. Hierarchical Procedure Representation

A long-horizon laboratory procedure T is represented at three nested levels: procedure phases, functional subtasks, and atomic skills $T=\left\{P_1,P_2,\dots ,P_k\right\}$, $P_k=\left\{S_{k,1},S_{k,2},\dots ,S_{k,N_k}\right\}$, $S_{k,n}=\left\{z_{k,n,1},z_{k,n,2},\dots ,z_{k,n,L_{k,n}}\right\}$. Here, $P_k$, $S_{k,n}$, and $z_{k,n,l}$ denote a procedure phase, functional subtask, and reusable atomic skill, respectively. Their temporal intervals satisfy

$$\mathit{I}\left({\mathrm{z}}_{\mathrm{k},\mathrm{n},\mathrm{l}}\right)\subseteq \mathit{I}\left({\mathrm{S}}_{\mathrm{k},\mathrm{n}}\right)\subseteq \mathit{I}\left({\mathrm{P}}_{\mathrm{k}}\right),$$

(3)

ensuring that each skill is assigned to a temporally consistent subtask and phase.

Each atomic skill is represented as $z_j=⟨a_j,O_j^{\arg },Z_j^{\mathrm{src}},Z_j^{\mathrm{dst}},c_j^{\mathrm{pre}},c_j^{\mathrm{post}}⟩$, where $a_j$ is the semantic skill category, $O_j^{\arg }$ contains the object arguments, $Z_j^{\mathrm{src}}$ and $Z_j^{\mathrm{dst}}$ denote the source and target zones, and $c_j^{\mathrm{pre}}$ and $c_j^{\mathrm{post}}$ are the precondition and expected postcondition of the skill.

The learned procedure is further represented as a hierarchical skill graph $G=\left(V_G,E_G\right)$, where nodes represent phases, subtasks, skills, objects, zones, and latent states, while edges encode temporal, prerequisite, object-dependency, zone-transition, alternative, and recovery relations. This graph supports valid ordering variations and recovery paths rather than a single fixed sequence.

3.3. Persistent Procedural State

Because the latent state st is only partially observable, BioEgo-HSG maintains an explicit procedural memory $M_t=\left\{M_t^{\mathrm{obj}},M_t^{\mathrm{zone}},M_t^{\mathrm{prog}},M_t^{\mathrm{fail}}\right\}$. Here, $M_t^{\mathrm{obj}}$ stores the persistent object identity, location, condition, and processing status; $M_t^{\mathrm{zone}}$ records zone preparation, occupancy, and cleanliness; $M_t^{\mathrm{prog}}$ maintains the current phase, completed skills, pending requirements, and valid graph node; and $M_t^{\mathrm{fail}}$ stores failures, recovery attempts, outcomes, and unresolved risks.

Memory is updated from the previous state, current observation, and latest skill:

$$M_t=U\left(M_{t-1},o_t,z_{j\left(t\right)-1}\right), \mathrm{subject}\mathrm{to}\mathsf{\Phi }\left(M_{t-1},z_{j\left(t\right)-1},M_t\right)=1,$$

(4)

where U(⋅) is the memory-update function and Φ(⋅) enforces physical and procedural consistency. The memory preserves reliable states when evidence is missing and revises them when new high-confidence observations become available.

Operationally, a stored state is retained when the corresponding entity is not visible or when current evidence is below the confidence threshold. A state is overwritten only when the new observation is sufficiently confident and compatible with the transition allowed by the executed skill. If high-confidence evidence conflicts with the predicted transition, the system does not commit to task progress; it records the inconsistency in the failure memory and requests re-observation or a recovery skill. Thus, planned effects are never treated as completed effects without postcondition verification.

3.4. Learning, Planning, and Execution Objectives

The framework includes offline procedural learning and online skill planning. Given the training set $D={\left\{\left(V^{\left(n\right)},Y^{\left(n\right)}\right)\right\}}_{n=1}^N$, where $V^{\left(n\right)}$ is an untrimmed egocentric demonstration and $Y^{\left(n\right)}$ contains hierarchical, state, graph, and recovery annotations, the model predicts the procedure, memory sequence, and skill graph:

$$\left({\widehat{T}}^{\left(n\right)},{\widehat{M}}_{1:T_n}^{\left(n\right)},{\widehat{G}}^{\left(n\right)}\right)=F_{\theta }\left(V^{\left(n\right)}\right).$$

(5)

The offline objective is to minimize errors in hierarchical temporal parsing, state estimation, next-skill prediction, graph-relation inference, and constraint satisfaction:

$${\theta }^{*}=\arg \underset{\theta }{\min }{\displaystyle {\displaystyle \sum }_{n=1}^N}\left[L_{\mathrm{hier}}^{\left(n\right)}+{\lambda }_m{L}_{\mathrm{state}}^{\left(n\right)}+{\lambda }_n{L}_{\mathrm{next}}^{\left(n\right)}+{\lambda }_g{L}_{\mathrm{graph}}^{\left(n\right)}+{\lambda }_c{L}_{\mathrm{constraint}}^{\left(n\right)}\right].$$

(6)

During execution, the planner filters graph successors according to current memory:

$$A\left(M_t,G,v_t\right)=\left\{z\in \mathrm{Succ}\left(v_t;G\right)\left|M_t\right|=c^{\mathrm{pre}}\left(z\right)\right\},$$

(7)

where $\mathrm{Succ}\left(v_t;G\right)$ denotes the graph successors of $v_t$, and $\left|M_t\right|=c^{\mathrm{pre}}\left(z\right)$ means that the preconditions of skill z are satisfied by the current procedural memory.

The next skill is selected by balancing the procedural likelihood, goal relevance, state conflict, uncertainty, and recovery cost:

$$z_{t+1}^{*}=\arg \underset{z\in A\left(M_t,G,v_t\right)}{\max }\left[\begin{array}{c}\log p_{\theta }\left(z|M_t,o_t,G,g\right)+{\lambda }_q{R}_{\mathrm{goal}}\left(z,g\right)\\ -{\lambda }_c{C}_{\mathrm{conflict}}\left(z,M_t\right)-{\lambda }_u{U}_{\mathrm{exec}}\left(z\right)-{\lambda }_r{C}_{\mathrm{recovery}}\left(z,M_t\right)\end{array}\right].$$

(8)

The first term measures the likelihood of the candidate skill under the learned procedural model. $R_{\mathrm{goal}}$ measures its relevance to the current goal, $C_{\mathrm{conflict}}$ penalizes inconsistency with the persistent state, $U_{\mathrm{exec}}$ represents execution uncertainty, and $C_{\mathrm{recovery}}$ estimates the cost of returning to a valid procedural state if the skill fails.

Finally, the selected semantic skill is converted into a high-level execution command $u_t=⟨a_t,O_t^{\arg },Z_t^{target},c_t^{\mathrm{pre}},c_t^{\mathrm{post}}⟩$, which is passed to a local manipulation policy πϕ. The local policy maps the command and the robot’s current sensory state to embodiment-specific motor actions $a_{t,1:H_t}^{low}={\pi }_{\varnothing }\left(u_t,x_t^{robot}\right)$, where Ht denotes the execution horizon of the current skill. Execution feedback updates the memory and graph state. A workflow succeeds only when the goal predicates and final object and zone states are valid, with no unresolved failure.

4. BioEgo-HSG Methodology

BioEgo-HSG implements the formulation in Section 3 through a unified perception-memory-planning–execution architecture. Given an untrimmed egocentric video, structured observations are encoded as Laboratory Interaction-Centric Tokens. A multi-scale temporal parser then identifies procedure phases, functional subtasks, and atomic skills. Episodic procedural memory preserves object, zone, task-progress, and failure-recovery states under partial observability, while the inferred elements form a hierarchical skill graph.

During execution, the graph-constrained planner uses the current memory, graph node, observation, and workflow goal to select an admissible skill. The skill is converted into a high-level command and executed by a local manipulation policy. Execution feedback updates the memory and graph state and activates recovery transitions when necessary. The complete architecture is shown in Figure 2.

Figure 2. Overall architecture of BioEgo-HSG. An untrimmed egocentric video is converted into interaction-centric tokens, hierarchically parsed into phases, subtasks, and atomic skills, and organized as a skill graph. Episodic memory supports graph-constrained skill selection, local policy execution, state updating, and recovery planning under partial observability. Solid black arrows indicate inference and execution, dashed arrows indicate training supervision, and solid gray arrows indicate execution feedback. The superscript asterisk (*) marks the admissible skill selected by the planner in Equation (8).

4.1. Interaction-Centric Representation and Hierarchical Temporal Parsing

BioEgo-HSG converts each structured observation $o_t$ into Laboratory Interaction-Centric Tokens. For the i-th visible entity at time $t$,

$${\mathit{l}}_{i,t}=\mathrm{LN}\left({\mathit{W}}_l\left[{\mathit{e}}_{i,t}\Vert {\mathit{\beta }}_{i,t}\Vert {\mathit{\rho }}_{i,t}^L\Vert {\mathit{\rho }}_{i,t}^R\Vert {\mathit{q}}_{i,t}\Vert {\mathit{e}}_t^Z\Vert {\mathit{r}}_t\Vert {\kappa }_{i,t}\right]+{\mathit{p}}_t\right),$$

(9)

where ${\mathit{e}}_{i,t}$, ${\mathit{\beta }}_{i,t}$, ${\mathit{\rho }}_{i,t}^L$, ${\mathit{\rho }}_{i,t}^R$, and ${\mathit{q}}_{i,t}$ represent the appearance, geometry, hand–object relations, and object-state evidence, respectively. The remaining terms encode the functional zone, task progress, observation confidence, and temporal position.

For each hand $h\in \left\{L,R\right\}$, the relative hand-object geometry is

$${\mathit{\rho }}_{i,t}^h=\left[{\displaystyle \frac{x_{i,t}-x_t^h}{W}},{\displaystyle \frac{y_{i,t}-y_t^h}{H}},\log {\displaystyle \frac{w_{i,t}}{w_t^h}},\log {\displaystyle \frac{h_{i,t}}{h_t^h}},\mathrm{IoU}\left(B_{i,t},B_t^h\right),d\left(B_{i,t},B_t^h\right)\right],$$

(10)

where $B_{i,t}$ and $B_t^h$ denote the entity and hand bounding boxes, $W$ and $H$ are the frame width and height, and $d\left(\cdot ,\cdot \right)$ is the normalized distance between two regions. Entity tokens are aggregated into a frame-level interaction representation ${\overline{\mathcal{l}}}_t=AttnPool\left({\left\{{\mathcal{l}}_{i,t}\right\}}_{i=1}^{N_t}\right)$, where $N_t$ is the number of visible entities. The resulting representation ${\overline{\mathcal{l}}}_t$ summarizes the interaction state of the current frame while preserving the contributions of individual task-relevant entities.

The interaction token separates geometric measurements from semantic evidence. Bounding-box geometry and hand–object offsets are normalized by the frame size. Object-state, functional-zone, and task-progress information is represented by categorical probability vectors, together with an observation-confidence value. Appearance features, the normalized geometry, interaction relations, state cues, and confidence are concatenated and projected into a shared token space. A temporal-position embedding and layer normalization complete the representation.

To capture both local interactions and the long-range workflow context, fine- and coarse-scale temporal encoders are applied:

$$\left\{\begin{array}{c}{\mathit{h}}_t^f=E_f\left({\overline{\mathcal{l}}}_{t-w_f:t+w_f}\right) \\ {\mathit{h}}_t^c={\mathrm{Up}}_{\tau }{\left[E_c\left({\mathrm{Pool}}_{\tau }\left({\overline{\mathcal{l}}}_{1:T_n}\right)\right)\right]}_t\end{array}\right.,$$

(11)

where $w_f$ and $\tau$ denote the local temporal radius and pooling factor. Their outputs are fused as

$${\mathit{h}}_t=\mathrm{LN}\left({\overline{\mathcal{l}}}_t+W_h\left[h_t^f\Vert h_t^c\Vert \left(h_t^f\odot h_t^c\right)\right]\right),$$

(12)

where $\odot$ denotes element-wise multiplication. The fused feature ${\mathit{h}}_t$ therefore combines local interaction evidence with the long-range procedural context needed to distinguish visually similar operations performed at different workflow stages.

The temporal parser jointly predicts phase, subtask, and atomic-skill boundaries and labels:

$$\left\{\begin{array}{c}{\widehat{\mathit{b}}}_t=\sigma \left({\mathit{W}}_b{\mathit{h}}_t\right) \\ {\widehat{\mathit{p}}}_t^P=softmax\left({\mathit{W}}_P{\mathit{h}}_t\right) \\ {\widehat{\mathit{p}}}_t^S=softmax\left({\mathit{W}}_S\left[{\mathit{h}}_t\Vert {\mathit{E}}_P{\widehat{\mathit{p}}}_t^P\right]\right) \\ {\widehat{\mathit{p}}}_t^z=softmax\left(W_z\left[{\mathit{h}}_t\Vert {\mathit{E}}_P{\widehat{\mathit{p}}}_t^P\Vert {\mathit{E}}_S{\widehat{\mathit{p}}}_t^S\right]\right)\end{array}\right.,$$

(13)

The lower-level predictions are conditioned on the higher-level distributions instead of being produced by three independent classifiers. This design allows the interpretation of an atomic skill to depend on its functional subtask and procedure phase. To further suppress hierarchically incompatible predictions, two binary compatibility matrices, ${\mathit{C}}^{PS}$ and ${\mathit{C}}^{Sz}$, encode the valid phase–subtask and subtask–skill associations observed in the training set. The refined distributions are obtained through soft masking:

$$\left\{\begin{array}{c}{\tilde{p}}_t^S=\mathrm{Norm}\left[{\widehat{\mathit{p}}}_t^S\odot {\left({\mathit{C}}^{PS}\right)}^T{\widehat{\mathit{p}}}_t^P\right]\\ {\tilde{p}}_t^z=\mathrm{Norm}\left[{\widehat{\mathit{p}}}_t^z\odot {\left({\mathit{C}}^{Sz}\right)}^T{\tilde{\mathit{p}}}_t^S\right]\end{array}\right.,$$

(14)

where $\mathrm{Norm}\left[·\right]$ normalizes a non-negative vector to the unit sum. Soft masking preserves uncertainty over multiple plausible higher-level contexts while reducing the probability assigned to procedurally invalid combinations.

The hierarchical procedure is recovered through constrained temporal decoding:

$${\widehat{\mathcal{T}}}^{\left(n\right)}=\arg \underset{\mathcal{T}\in {\Omega }_{\mathrm{nest}}}{\max }{\displaystyle {\displaystyle \sum }_{t=1}^{T_n}}\left[\log {\widehat{p}}_t^P\left(P_t\right)+\log {\tilde{p}}_t^S\left(S_t\right)+\log {\tilde{p}}_t^z\left(z_t\right)+{\lambda }_b\log p\left({\widehat{b}}_t\mid \mathcal{T}\right)\right].$$

(15)

where ${\Omega }_{\mathrm{nest}}$ enforces the temporal nesting constraint in Equation (3). Skill arguments, source and target zones, and preliminary preconditions and postconditions are then inferred from the token attention, zone evidence, and segment-level state changes.

Finally, parsing confidence is computed as

$${\omega }_t=1-{\displaystyle \frac{1}{4}}\left[{\displaystyle \sum }_{r\in \left\{P,S,z\right\}}{\displaystyle \frac{\mathcal{H}\left({\tilde{p}}_t^r\right)}{\log C_r}}+{\displaystyle \frac{1}{3}}{\displaystyle \sum }_{r\in \left\{P,S,z\right\}}{\displaystyle \frac{{\mathcal{H}}_B\left({\widehat{b}}_t^r\right)}{\log 2}}\right],$$

(16)

where ${\tilde{p}}_t^P={\widehat{p}}_t^P,\mathcal{H}\left(\cdot \right)$ denotes categorical entropy, and ${\mathcal{H}}_B\left(\cdot \right)$ denotes binary entropy. A larger ${\omega }_t$ indicates that both the hierarchical labels and temporal boundaries are reliable. The decoded procedure and confidence values are passed to the procedural memory for state-consistent updating.

4.2. Episodic Procedural Memory and State-Consistent Updating

To preserve procedural information beyond the current view, BioEgo-HSG maintains $M_t=\left\{M_t^{\mathrm{obj}},M_t^{\mathrm{zone}},M_t^{\mathrm{prog}},M_t^{\mathrm{fail}}\right\}$, which stores object states, zone conditions, task progress, and failure-recovery history.

For each memory component $r\in \left\{\mathrm{obj},\mathrm{zone},\mathrm{prog},\mathrm{fail}\right\}$, current evidence ${\mathit{x}}_t^M$ produces a candidate state and retention gate:

$$\left\{\begin{array}{c}{\tilde{M}}_t^r=F_r\left(M_{t-1}^r,{\mathit{x}}_t^M\right) \\ {\gamma }_t^r=\sigma \left(W_{\gamma }^r\left[M_{t-1}^r\Vert {\mathit{x}}_t^M\right]+b_{\gamma }^r\right) \\ {\overline{M}}_t^r={\overline{\gamma }}_t^r\odot M_{t-1}^r+\left(1-{\overline{\gamma }}_t^r\right)\odot {\tilde{M}}_t^r\end{array}\right.,$$

(17)

where ${\mathit{x}}_t^M$ denotes the current memory evidence and ${\overline{\gamma }}_t^r$ is modulated by ${\omega }_t$ and entity visibility.

To prevent physically or procedurally invalid updates, BioEgo-HSG applies skill-conditioned state-transition constraints. For a discrete state variable $d$, the candidate distribution is refined using the transition matrix ${\mathit{T}}_d\left(z_{j\left(t\right)}\right)$:

$${\mathit{p}}_{i,t}^d=Norm\left[{\overline{\mathit{p}}}_{i,t}^d\odot {\mathit{T}}_d{\left(z_{j\left(t\right)}\right)}^T{\mathit{p}}_{i,t-1}^d\right].$$

(18)

Task progress is committed only when an atomic-skill boundary is detected and its expected postcondition is satisfied:

$${\delta }_t^{\mathrm{done}}=1\left({\widehat{b}}_t^z>{\tau }_b\right)1\left({\omega }_t>{\tau }_{\omega }\right)1\left(c_{j\left(t\right)}^{\mathrm{post}}\left(M_t\right)=1\right).$$

(19)

where ${\tau }_b$ and ${\tau }_{\omega }$ are boundary and confidence thresholds, respectively. During robot execution, successful policy feedback is also required.

At the inference time, the update follows three cases. First, missing or low-confidence evidence retains the previous memory value. Second, high-confidence evidence that agrees with the skill-conditioned transition matrix replaces the previous value. Third, high-confidence but transition-inconsistent evidence blocks progress and creates a failure-memory entry. When multiple observations conflict, transition-consistent evidence with higher confidence is preferred; otherwise, the state remains unresolved until re-observation or recovery.

If ${\delta }_t^{\mathrm{done}}=1$, the skill history and graph node are advanced; otherwise, the graph state is retained and the inconsistency is recorded in $M_t^{\mathrm{fail}}$.

4.3. Hierarchical Skill-Graph Learning and Recoverable Skill Planning

The hierarchical procedure and persistent states are organized into $G=\left(V_G,E_G\right)$, whose nodes represent phases, subtasks, skills, objects, zones, and states. Its edges encode temporal, prerequisite, object-dependency, zone-transition, alternative, and recovery relations. For candidate nodes $v_i$ and $v_j$, relation $r$ is inferred as

$$p_{ij}^r=\sigma \left(F_r\left[h_i\Vert h_j\Vert \mathsf{\Delta }{t}_{ij}\Vert \mathsf{\Delta }{M}_{ij}\right]\right), \left(v_i,v_j,r\right)\in E_{G}if{p}_{ij}^r>{\tau }_r,$$

(20)

where $\mathsf{\Delta }{t}_{ij}$ and $\mathsf{\Delta }{M}_{ij}$ denote temporal separation and the procedural-state change, and ${\tau }_r$ is the threshold for relation $r$. Graphs from multiple demonstrations are merged to form main, alternative, and recovery paths. Figure 3 illustrates how the learned hierarchical skill graph is coupled with the episodic procedural memory to filter executable skills and activate local recovery transitions. Here, upright Δ denotes a difference, whereas the lowercase δ in Equation (19) is a completion indicator.

Figure 3. Memory-aware hierarchical skill-graph planning. (a) The graph represents procedural elements and their temporal, prerequisite, transition, alternative, and recovery relations. (b) Episodic memory filters infeasible skills and supports recovery from a valid graph node after failure.

During execution, the planner filters graph-connected skills using their preconditions $\mathcal{A}\left(M_t,G,v_t\right)=\left\{z:\left(v_t,z\right)\in E_G,c_z^{\mathrm{pre}}\left(M_t\right)=1\right\}$. The next skill is selected by

$$\begin{array}{cc}\hfill {\mathcal{J}}_t\left(z\right)& =\log p_{\theta }\left(z\mid M_t,o_t,g,G\right)+{\lambda }_g{R}_{\mathrm{goal}}\left(z,g\right)\hfill \\ & -{\lambda }_s{C}_{\mathrm{state}}\left(z,M_t\right)-{\lambda }_{u}U\left(z\right)-{\lambda }_r{C}_{\mathrm{rec}}\left(z,M_t,G\right).\hfill \end{array}$$

(21)

Here, $R_{\mathrm{goal}}$ measures the relevance of skill $z$ to the workflow goal $g$, $C_{\mathrm{state}}$ penalizes conflicts with the current procedural memory, $U\left(z\right)$ denotes execution uncertainty, and $C_{\mathrm{rec}}$ estimates the expected recovery cost. This formulation balances task progress with state consistency and execution reliability. The selected skill is converted into the high-level command $u_t=⟨z_{t+1}^{*},{\mathcal{O}}_{t+1}^{\arg },Z_{t+1}^{\mathrm{src}},Z_{t+1}^{\mathrm{dst}},c_{t+1}^{\mathrm{pre}},c_{t+1}^{\mathrm{post}}⟩$, $a_{t,1:H_t}^{\mathrm{low}}\sim {\pi }_{\varphi }\left(u_t,o_t^{\mathrm{robot}}\right)$, where ${\pi }_{\varphi }$ is the local manipulation policy and $o_t^{\mathrm{robot}}$ denotes the current robot observation. Successful execution advances the graph state. Otherwise, the failure is recorded in $M_t^{\mathrm{fail}}$, and a feasible recovery edge returns execution to a valid graph state.

The low-level policy returns an execution-feedback tuple containing the success flag, uncertainty estimate, and observed postcondition. Memory is updated from verified outcomes rather than from the commanded action alone. Consequently, a failed grasp leaves the object location unchanged, and an inaccurate placement does not mark the destination zone as correctly occupied. A postcondition mismatch retains the current graph node, writes the failure type into M_fail, and activates a feasible recovery edge before normal execution can resume.

4.4. Joint Learning Objective and Implementation

BioEgo-HSG jointly optimizes temporal parsing, memory estimation, graph induction, next-skill prediction, and procedural consistency: the corresponding loss term is denoted as $L_{\mathrm{hierarchy}}$ throughout.

$$\begin{array}{c}\mathcal{L}={\lambda }_b{\mathcal{L}}_{\mathrm{boundary}}+{\lambda }_h{\mathcal{L}}_{\mathrm{hierarchy}}+{\lambda }_m{\mathcal{L}}_{\mathrm{memory}}\hfill \\ +{\lambda }_g{\mathcal{L}}_{\mathrm{graph}}+{\lambda }_n{\mathcal{L}}_{\mathrm{next}}+{\lambda }_c{\mathcal{L}}_{\mathrm{constraint}}+{\lambda }_a{\mathcal{L}}_{\mathrm{action}},\hfill \end{array}$$

(22)

where ${\mathcal{L}}_{\mathrm{boundary}}$ supervises phase, subtask, and skill boundaries; ${\mathcal{L}}_{\mathrm{hierarchy}}$ measures the corresponding label-prediction errors; ${\mathcal{L}}_{\mathrm{memory}}$ supervises object, zone, task-progress, and failure states; and ${\mathcal{L}}_{\mathrm{graph}}$ trains the relation classifier in Equation (20). The next-skill loss ${\mathcal{L}}_{\mathrm{next}}$ supervises the selection in Equation (21), while ${\mathcal{L}}_{\mathrm{constraint}}$ penalizes invalid state transitions and skills whose preconditions are not satisfied. The optional ${\mathcal{L}}_{\mathrm{action}}$ is used only when robot-specific action demonstrations are available.

The trainable parameters are obtained by

$${\Theta }^{*}=\arg \underset{\Theta }{\min }{\mathbb{E}}_{{\mathcal{V}}^{\left(n\right)}}\left[\mathcal{L}\right],$$

(23)

where $\Theta$ denotes the parameters of the visual projection, temporal encoders, memory-update networks, graph-relation predictor, and next-skill planner. The local manipulation policy ${\pi }_{\varphi }$ may be trained separately and connected to BioEgo-HSG through the high-level command $u_t$.

The learning procedure comprises three stages: temporal parsing with boundary and hierarchical labels, memory/graph learning with state and relation annotations, and joint fine-tuning. Hyperparameters are selected using the validation split. Graph construction is performed offline; memory updating, admissible-skill filtering, planning, and recovery operate online. The local manipulation policy is connected through the high-level semantic command interface and may be trained separately. Because the original run-level training configuration and logs are not available as independently traceable artifacts, newly reconstructed epoch counts, optimization schedules, architectural dimensions, runtime values, and hardware details are not presented as records of the reported experiments. Algorithm 1 summarizes the training and online planning procedure.

Algorithm 1: End-to-End Training and Online Planning of BioEgo-HSG.

Input: Untrimmed egocentric demonstrations  $\mathcal{D}$, workflow goals  $\mathit{g}$, robot observations  ${\mathit{o}}_{\mathit{t}}$

Output: Hierarchical skill graph  $\mathit{G}$, procedural memory model  $\mathit{M}$, executable skill sequence

Offline Training Stage:

For each egocentric video demonstration $V\in \mathcal{D}$:

Extract structured observations and construct Laboratory Interaction-Centric Tokens: $E_t=\mathsf{\Psi }\left(V_t\right)$ including visual appearance, object states, hand-object relations, zone information, and task-progress features.

Apply multi-scale temporal parsing to obtain hierarchical procedure structures: $V\to \left\{P_k,S_{k,n},z_{k,n,l}\right\}$ where $P_k$, $S_{k,n}$, and $z_{k,n,l}$ represent procedure phases, subtasks, and atomic skills.

Learn procedural states and update the episodic memory representation: $M_t=U\left(M_{t-1},o_t,z_t\right)$

Infer temporal, dependency, alternative, and recovery relations to construct the hierarchical skill graph: $G=\left({\mathcal{V}}_G,{\mathcal{E}}_G\right)$

Optimize the temporal parser, memory module, graph relation predictor, and skill planner jointly.

Online Planning and Execution Stage:

Given current observation $o_t$, memory state $M_t$, and graph state $G_t$, identify admissible skills according to graph constraints.

Select the next executable skill: $z_{t+1}=\arg \underset{z\in A\left(M_t,G_t\right)}{\max }S\left(z\right)$

Execute the selected semantic skill using the local manipulation policy.

Collect execution feedback, update memory, and activate recovery transitions if postconditions are violated.

Repeat the online observation, skill-selection, execution, and feedback cycle until the workflow goal is achieved.

5. Experimental Setup

The preliminary feasibility protocol links the offline egocentric-video procedure understanding to online robot skill chaining. It covers hierarchical temporal parsing, procedural-state tracking, skill-graph construction, next-skill prediction, invalid-skill detection, recovery planning, and physical execution. The setup specifies the multi-zone laboratory demonstrations and their annotations, the reported robot tasks, and the internal baseline and ablation configurations.

5.1. Experimental Environment and Dataset Construction

The dataset was collected in a multi-zone laboratory containing $\mathcal{Z}=\left\{Z^{\mathrm{storage}},Z^{\mathrm{inventory}},Z^{\mathrm{bench}},Z^{\mathrm{enclosed}}\right\}$, corresponding to storage, inventory, general-bench, and enclosed-workspace operations. Eight operators recorded untrimmed egocentric demonstrations, ${\mathcal{V}}^{\left(n\right)}={\left\{I_t^{\left(n\right)}\right\}}_{t=1}^{T_n}$, covering navigation, object search, verification, manipulation, zone transitions, and workspace restoration. Here, $I_t^{\left(n\right)}$ is the RGB frame at time $t$, and $T_n$ is the length of the demonstration.

The benchmark includes six representative workflows involving material retrieval, consumable preparation, container handling, weighing and recording, simulated transfer, and final storage or cleaning. Each operator completed three repetitions of every workflow, producing $N_{\mathrm{normal}}=8\times 6\times 3=144$ normal episodes. An additional 36 perturbation-recovery episodes contain errors such as incorrect object selection, skipped verification, improper placement, incomplete operations, or unresolved cleaning requirements.

The final dataset contains 180 episodes, each lasting approximately 10 min, for a total of about 30 h of video. Sensitive laboratory layouts, proprietary procedures, and sample information are excluded. Table 2 summarizes the dataset statistics.

Table 2. Dataset and workflow statistics.

Component	Scale	Description
Functional zones	4	Storage, inventory, general bench, enclosed workspace
Operators	8	Human demonstrators wearing egocentric recording devices
Workflow categories	6	Representative long-horizon biomedical laboratory procedures
Normal episodes	144	8 × 6 × 3 repetitions
Perturbation-recovery episodes	36	Episodes with procedural deviations and recovery operations
Total episodes	180	Normal and perturbation-recovery demonstrations
Average duration	~10 min	Complete multi-zone workflow per episode
Total egocentric video	~30 h	Long-horizon video corpus for offline learning
Primary purpose	-	Temporal parsing, memory tracking, graph learning, and recovery planning

The egocentric video collection involved adult operators performing routine laboratory procedures. Before recording, the participants were informed of the purpose of the study, the recording procedure, and the intended use of the collected data, and participant consent was obtained. Participant privacy and the appropriate use of the recorded materials were considered throughout data collection, processing, and manuscript preparation. No formal Institutional Review Board or ethics committee review was sought for this non-clinical study.

5.2. Annotation Protocol and Evaluation Splits

Each episode is annotated with hierarchical temporal labels, procedural states, graph relations, and recovery information ${\mathcal{Y}}^{\left(n\right)}=\left\{{\mathcal{Y}}_{\mathrm{temp}}^{\left(n\right)},{\mathcal{Y}}_{\mathrm{state}}^{\left(n\right)},{\mathcal{Y}}_{\mathrm{graph}}^{\left(n\right)},{\mathcal{Y}}_{\mathrm{rec}}^{\left(n\right)}\right\}$, where ${\mathcal{Y}}_{\mathrm{temp}}^{\left(n\right)}$ contains temporal boundaries and labels, ${\mathcal{Y}}_{\mathrm{state}}^{\left(n\right)}$ contains object, zone, progress, and failure states, ${\mathcal{Y}}_{\mathrm{graph}}^{\left(n\right)}$ contains skill-graph relations, and ${\mathcal{Y}}_{\mathrm{rec}}^{\left(n\right)}$ contains recovery annotations for perturbation episodes.

Temporal annotations include phase, subtask, and atomic-skill boundaries and labels, together with skill arguments, zones, preconditions, and postconditions. State annotations cover object, zone, task-progress, and failure-recovery variables. Graph annotations include temporal, prerequisite, object-dependency, zone-transition, alternative, and recovery edges. Recovery episodes additionally record failure points, recovery actions, and resumed graph nodes. A shared guideline, partial double annotation, and adjudication are used to improve annotation consistency.

The reported participant-disjoint partition separates operators across training, validation, and testing:

$$\mathcal{O}={\mathcal{O}}_{\mathrm{train}}\cup {\mathcal{O}}_{\mathrm{val}}\cup {\mathcal{O}}_{\mathrm{test}},{\mathcal{O}}_{\mathrm{train}}\cap {\mathcal{O}}_{\mathrm{val}}\cap {\mathcal{O}}_{\mathrm{test}}=\varnothing ,$$

(24)

with a 5/1/2 split corresponding to five training operators, one validation operator, and two test operators.

Under the normal-demonstration setting, the split produces 90 training episodes, 18 validation episodes, and 36 test episodes from the 144 normal episodes.

The participant-disjoint partition assigns five operators to training, one to validation, and two to testing. This yields 90/18/36 normal episodes and 21/6/9 perturbation-recovery episodes in the training/validation/test sets, respectively. Recovery episodes follow the same operator assignments as the normal demonstrations and are not independently shuffled across partitions. No operator appears in more than one split. The episode counts in Table 3 therefore describe the outcome of participant-level separation, not a separate recovery-episode allocation rule.

Table 3. Annotation categories and evaluation splits.

Component	Annotation/Split Item	Description
Temporal structure	$P_k,S_{k,n},z_{k,n,l}$	Phase, subtask, and atomic-skill boundaries and labels
Atomic-skill tuple	$a_j,{\mathcal{O}}_j^{\arg },Z_j^{\mathrm{src}},Z_j^{\mathrm{dst}},c_j^{\mathrm{pre}},c_j^{\mathrm{post}}$	Skill category, object arguments, zones, preconditions, and postconditions
Object state	$M_t^{\mathrm{obj}}$	Identity, visibility, location, verification, container condition, processing, and storage status
Zone state	$M_t^{\mathrm{zone}}$	Zone activity, preparation, occupancy, cleanliness, and restoration requirement
Task progress	$M_t^{\mathrm{prog}}$	Current phase, subtask, completed skills, pending requirements, and graph node $v_t$
Failure recovery	$M_t^{\mathrm{fail}}$	Failure type, violated postcondition, recovery attempt, outcome, and unresolved risk
Graph relations	$E_G$	Temporal, prerequisite, object-dependency, zone-transition, alternative, and recovery edges
Operator split	5/1/2	Five training operators, one validation operator, and two test operators
Normal episodes	90/18/36	Training/validation/test split under normal demonstrations
Recovery episodes	21/6/9	Recovery episodes follow the same operator assignments as the normal demonstrations and are not independently shuffled across partitions.

Validation data are used for hyperparameter selection, and test data are reserved for the final evaluation. Table 3 summarizes the annotation categories and evaluation split statistics. The 21/6/9 values denote episode counts after participant-level separation, not a separate recovery-episode allocation rule.

5.3. Robot Skill-Chaining Setup and Baselines

Robot experiments evaluate whether the learned procedural knowledge can guide physical execution through the high-level command $u_t$. The local policy executes each selected skill and returns its success status, uncertainty, and observed postcondition. Successful execution advances the skill graph; otherwise, the failure is recorded and a recovery transition is activated.

Each high-level command specifies the semantic action, object arguments, source and destination zones, preconditions, and expected postcondition. A local execution library carries out grasping, transport, placement, opening, closing, cleaning, and storage operations. Execution feedback includes a success flag and the observed postcondition; the skill graph advances only after the required postcondition is confirmed. The same local primitive library was used throughout the reported robot skill-chaining experiments so that the high-level planning configurations interfaced with a common low-level execution stack.

The reported robot setup includes the retrieve-and-place, dish/container arrangement, and state-aware placement, with 20 trials per task and 60 trials per configuration. These tasks involve cross-zone transfer, repeated manipulation, and skill selection conditioned on procedural state. The reported high-level configurations use a common local execution library.

Table 4 defines the internal baseline and ablation configurations. The flat temporal Transformer and skill-only model specify alternatives to the phase–subtask–skill hierarchy. The Retrieval-based selector and Sequence-only graph specify local-similarity selection and temporal-succession planning. Current-frame-only and fixed-window-history configurations specify alternatives to persistent memory. The ablations remove episodic memory, state-transition constraints, or recovery edges. Fixed action sequencing, memoryless replanning, and BioEgo-HSG are connected to the common local manipulation library for the reported robot tasks.

Table 4. Baselines and ablation settings.

Category	Method	Description
Temporal parsing	Flat temporal Transformer	Predicts labels without phase-subtask-skill hierarchy
Temporal parsing	Skill-only model	Predicts only atomic skills $z_{k,n,l}$
Next-skill prediction	Retrieval-based selector	Selects the next skill from similar past segments
Graph planning	Sequence-only graph	Keeps temporal edges but removes prerequisite and recovery edges
Memory	Current-frame only	Uses only $o_t$ for state and skill prediction
Memory	Fixed-window history	Uses a short history window instead of $M_t$
Ablation	w/o episodic memory	Removes all four memory components
Ablation	w/o state-transition constraints	Removes validity checks in memory updating
Ablation	w/o recovery edges	Removes recovery transitions from $G$
Robot execution	Fixed action sequence	Executes a predefined skill order
Robot execution	Memoryless replanning	Replans without persistent procedural memory
Robot execution	Full BioEgo-HSG	Uses $M_t$, $G$, admissible-skill filtering, and recovery planning

The Retrieval-based selector and Sequence-only graph represent internal modeling alternatives for next-skill selection and graph planning, respectively. Their definitions are given in Table 4.

5.4. Evaluation Metrics and Reporting Protocol

BioEgo-HSG is evaluated at four levels. For hierarchical temporal parsing, we report the boundary F1, F1@10/25/50, phase/subtask/skill accuracy, and segmental edit score. For procedural-state and graph evaluation, we report the state-tracking Macro-F1 over $M_t^{\mathrm{obj}}$, $M_t^{\mathrm{zone}}$, $M_t^{\mathrm{prog}}$, and $M_t^{\mathrm{fail}}$, together with the edge precision, recall, and F1 for the learned skill graph $G$. Next-skill planning is evaluated using Top-1/Top-5 accuracy and the invalid-skill detection rate.

For recovery and robot execution, we report the recovery success (valid-state resumption) rate, additional recovery steps, end-to-end completion rate, unrecovered failure rate, and average execution time. A robot trial is considered successful only if the final goal predicates are satisfied, all required postconditions hold, and no unresolved failure remains in $M_t^{\mathrm{fail}}$.

Long-horizon stability is measured using the skill-chain survival curve:

$$S\left(k\right)={\displaystyle \frac{1}{N}}{{\displaystyle \sum }}_{n=1}^{N}1\left(\mathrm{skills}1,\dots ,k\mathrm{remain}\mathrm{valid}\mathrm{in}\mathrm{trial}n\right),$$

(25)

and the normalized survival area:

$$AU{C}_H={\displaystyle \frac{1}{H}}{{\displaystyle \sum }}_{k=1}^{H}S\left(k\right),$$

(26)

where $H$ is the maximum evaluated skill-chain length. A larger $AU{C}_H$ indicates lower long-horizon error accumulation.

The reporting protocol uses the participant-disjoint partition in Section 5.2, selects hyperparameters based on the validation set, and reserves the test split for the reported evaluation. Recovery success and robot completion are expressed as proportions over episodes or physical trials, separately from variation across training runs.

Table 5 summarizes the evaluation metrics for offline and online experiments.

Table 5. Evaluation metrics for offline and online experiments.

Evaluation Target	Metrics
Temporal parsing	Boundary F1, F1@10/25/50, label accuracy, edit score
State tracking	Macro-F1, per-state F1
Skill-graph induction	Edge precision, recall, F1
Next-skill planning	Top-1/Top-5 accuracy, invalid-skill detection
Recovery planning	Recovery success, additional steps
Robot skill chaining	Completion rate, unrecovered failure rate, execution time
Long-horizon stability	Survival curve $S\left(k\right)$, $AU{C}_H$

Evidentiary status and reporting scope. Because complete raw run records and independently reproducible logs are unavailable for part of the previously reported offline summaries, those summaries are retained only as descriptive records of the reported evaluation. They are not independently validated estimates of repeated-run performance or statistical superiority. The seed-level numerical lists and associated significance claims remain withdrawn; no significance or reproducibility claim is made from the retained summaries. Physical robot trials are reported as feasibility observations within the tested platform and task setting. This reporting scope applies throughout Section 6.

The configurations in the reported summaries are the internal baselines and ablated variants defined in Table 4.

6. Preliminary Feasibility Observations

The results below should be interpreted under the reporting scope defined in Section 5.4.

6.1. Descriptive Evaluation and Ablation Summaries

Table 6 organizes the reported temporal, next-skill, state, graph, recovery, and survival summaries for the internal baseline and ablation configurations. The columns identify the measurement categories associated with each configuration.

Table 6. Descriptive summary of reported metrics for internal baselines and ablation configurations.

Method	Phase F1@50 (%)	Skill Acc. (%)	Next Top-1 (%)	State Macro-F1 (%)	Graph Edge F1 (%)	Recovery Success (%)	AUCH
Flat temporal Transformer	61.8 ± 2.4	66.4 ± 2.1	61.7 ± 2.5	60.8 ± 2.7	N/A	N/A	N/A
Retrieval-based selector	N/A	N/A	66.8 ± 2.2	N/A	N/A	N/A	N/A
Sequence-only graph	N/A	69.4 ± 2.0	67.9 ± 2.1	68.6 ± 2.5	56.3 ± 2.8	33.3	0.54 ± 0.04
w/o episodic memory	69.2 ± 1.9	74.0 ± 1.6	68.2 ± 2.1	67.5 ± 2.6	64.1 ± 2.4	55.6	0.60 ± 0.04
w/o state-transition constraints	70.1 ± 1.8	75.1 ± 1.7	70.2 ± 1.9	73.2 ± 2.1	65.5 ± 2.2	55.6	0.64 ± 0.03
w/o recovery edges	70.5 ± 1.7	75.8 ± 1.5	72.2 ± 1.7	77.1 ± 1.8	64.8 ± 2.3	44.4	0.61 ± 0.04
BioEgo-HSG	70.9 ± 1.7	76.8 ± 1.4	73.6 ± 1.6	78.4 ± 1.5	68.9 ± 2.0	66.7	0.77 ± 0.03

Note: N/A indicates that the corresponding metric is not applicable to, or was not evaluated for, that baseline under the shared evaluation protocol; it does not denote a zero score. Gray shading is used for table layout only; it does not encode results or statistical significance.

6.2. Hierarchical Procedure Understanding and Persistent State Tracking

Table 7 lists the reported hierarchical temporal-parsing and state-tracking summaries. Its temporal columns describe the boundaries, segment overlap, edit score, and label accuracy; the next-skill columns describe prediction of the following skill. These are distinct prediction targets.

Table 7. Descriptive summary of hierarchical procedure parsing, next-skill prediction, and persistent state tracking.

Method	Boundary F1 (%)	F1@10 (%)	F1@25 (%)	F1@50 (%)	Edit (%)	Phase Acc. (%)	Subtask Acc. (%)	Skill Acc. (%)	Next Top-1/Top-5 (%)	State Macro-F1 (%)
Flat temporal Transformer	68.5 ± 2.6	78.9 ± 2.3	70.5 ± 2.5	61.8 ± 2.4	65.3 ± 2.8	71.2 ± 2.4	68.0 ± 2.5	66.4 ± 2.1	61.7/82.4	60.8 ± 2.7
Skill-only model	70.1 ± 2.4	79.6 ± 2.2	72.8 ± 2.3	64.5 ± 2.2	67.8 ± 2.6	N/A	N/A	69.7 ± 2.0	64.8/84.1	N/A
Retrieval-based selector	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A	66.8/87.6	N/A
Current-frame only	66.2 ± 2.7	75.4 ± 2.8	67.9 ± 2.6	58.7 ± 2.5	62.1 ± 2.9	69.5 ± 2.6	65.8 ± 2.7	64.9 ± 2.5	60.3/81.7	63.5 ± 2.9
Fixed-window history	70.8 ± 2.2	80.7 ± 2.0	73.4 ± 2.1	65.9 ± 2.0	68.6 ± 2.3	75.3 ± 2.0	71.6 ± 2.2	70.5 ± 2.0	66.9/85.8	70.2 ± 2.4
BioEgo-HSG	76.4 ± 1.8	85.1 ± 1.6	78.6 ± 1.7	70.9 ± 1.7	76.2 ± 1.9	84.5 ± 1.5	80.2 ± 1.6	76.8 ± 1.4	73.6/91.2	78.4 ± 1.5

Note: N/A indicates that a metric is not applicable to, or was not evaluated for, the stated baseline. For the Retrieval-based selector, temporal skill-classification accuracy is not applicable: this baseline is evaluated only for next-skill prediction. Gray shading is used for table layout only; it does not encode results or statistical significance.

Figure 4 presents a representative workflow with reference labels, temporal predictions, and a memory-state trajectory. The example shows the phase–subtask–skill organization alongside identity verification, container condition, processing status, cleanliness, and return-to-storage states. It illustrates how the proposed representation carries procedural-state information across changes in visibility.

Figure 4. Representative hierarchical temporal parsing and persistent state-tracking example. The reference hierarchy, flat temporal prediction, BioEgo-HSG prediction, and memory-state trajectory are shown for one multi-zone laboratory workflow. The displayed predictions and memory trajectory illustrate how the framework organizes hierarchical labels and carries procedural-state information across changes in visibility.

Potential temporal ambiguities arise at gradual transitions between skills and when an action must be assigned consistently across phase, subtask, and skill levels. State tracking must also distinguish directly visible object and zone information from progress and failure states that depend on delayed effects and postcondition feedback. Missed detections, ambiguous observations, and incorrect boundary decisions remain possible error sources.

6.3. Skill-Graph Planning, Error Recovery, and Long-Horizon Stability

Table 8 organizes the reported graph-relation, invalid-skill, recovery, and survival summaries. Prerequisite and recovery relations connect skill selection to latent procedural states and corrective transitions.

Table 8. Descriptive summary of skill-graph, invalid-skill, recovery, and skill-chain survival metrics.

Method	Edge Precision (%)	Edge Recall (%)	Edge F1 (%)	Prerequisite F1 (%)	Recovery-edge F1 (%)	Invalid-skill Detection (%)	Recovery Success (%)	Additional Steps	AUCH
Sequence-only graph	60.2 ± 2.9	52.8 ± 3.1	56.3 ± 2.8	N/A	N/A	54.7 ± 3.2	33.3	3.8 ± 1.1	0.54 ± 0.04
Memoryless replanning	62.9 ± 2.8	58.0 ± 2.9	60.4 ± 2.6	52.1 ± 3.1	38.6 ± 3.8	61.5 ± 3.0	44.4	3.4 ± 1.0	0.57 ± 0.04
w/o state-transition constraints	68.3 ± 2.4	62.8 ± 2.5	65.5 ± 2.2	64.1 ± 2.5	55.2 ± 3.5	71.6 ± 2.7	55.6	3.0 ± 0.9	0.64 ± 0.03
w/o recovery edges	67.1 ± 2.5	62.6 ± 2.6	64.8 ± 2.3	65.5 ± 2.4	N/A	75.8 ± 2.5	44.4	3.1 ± 0.9	0.61 ± 0.04
BioEgo-HSG	71.8 ± 2.1	66.3 ± 2.2	68.9 ± 2.0	69.7 ± 2.1	61.4 ± 3.0	82.6 ± 2.2	66.7	2.4 ± 0.7	0.77 ± 0.03

Note: N/A indicates that the corresponding metric is not applicable to, or was not evaluated for, that baseline under the shared evaluation protocol; it does not denote a zero score. Gray shading is used for table layout only; it does not encode results or statistical significance.

Figure 5 visualizes a learned hierarchical skill graph for a representative laboratory workflow. The graph contains normal temporal edges, prerequisite edges, object-dependency edges, zone-transition edges, and recovery edges. This structure allows the planner to distinguish normal progress from corrective recovery transitions when a precondition or postcondition is violated.

Figure 5. Representative hierarchical skill graph with prerequisite and recovery transitions. The graph organizes phases, subtasks, atomic skills, objects, zones, and states. Solid arrows indicate temporal succession, dashed arrows indicate prerequisite relations, thick arrows indicate zone transitions, and dotted arrows indicate recovery transitions.

Invalid-skill checking compares candidate-skill preconditions with procedural memory and graph constraints. The modeled violations include processing an unverified container, storing an unclosed item, placing an object in an incompatible zone, and starting a new procedure while cleaning remains pending.

The reported recovery examples illustrate corrective transitions after a state conflict or postcondition mismatch. The represented branches include returning to search and verification after selecting a wrong container, inserting closing or rechecking before storage, and restoring workspace cleanliness before continuing.

Figure 6 presents the retained descriptive skill-chain survival summary. The execution loop checks skill preconditions, postconditions, and memory consistency as the chain advances.

Figure 6. Descriptive skill-chain survival summary. S(k) denotes the proportion of episodes that remain procedurally correct through the first k chained skills.

A recovery relation may admit several corrective skill sequences for the same failure. The graph represents these alternatives together with the procedural state required to resume the normal path.

6.4. Robot Skill-Chaining Demonstrations

The robot demonstrations connect the high-level procedural representation to physical execution through a semantic skill interface. The reported task configurations are retrieve-and-place, dish/container arrangement, and state-aware placement, with 20 trials per task and a common local manipulation library.

Table 9 records the reported completion counts for the three task configurations. For BioEgo-HSG, the listed counts are 17/20, 16/20, and 14/20, totaling 47/60. The listed totals for fixed action sequencing and memoryless replanning are 39/60 and 42/60. State-aware placement involves procedural variables such as whether a container has been verified, processed, closed, or designated for return.

Table 9. Descriptive record of reported robot skill-chaining trials across three tasks.

Method	Retrieve-and-Place	Dish/Container Arrangement	State-Aware Placement	Total Success	Avg. Time (s)	Unrecovered Failure (%)
Fixed action sequence	15/20	14/20	10/20	39/60	214.6 ± 34.2	35
Memoryless replanning	16/20	15/20	11/20	42/60	232.5 ± 39.8	30
BioEgo-HSG	17/20	16/20	14/20	47/60	241.8 ± 41.5	21.7

Note: Gray shading identifies the column headings. Bold in the body identifies the BioEgo-HSG row for reference only and does not indicate statistical significance.

The retrieve-and-place configuration involves object selection and cross-zone transport. The dish/container arrangement configuration includes repeated manipulation and spatial organization. These task descriptions specify the procedural contexts interfaced with the local skill library.

Table 9 records execution-time and unrecovered-failure summaries alongside completion counts. The BioEgo-HSG execution loop includes memory updating, postcondition verification, admissible-skill filtering, and recovery-branch selection.

Table 10 organizes the reported failure descriptions into high-level planning, state estimation, and local manipulation categories. These categories distinguish skill selection, observation interpretation, and physical execution within the modular interface. Grasp instability, alignment error, and placement imprecision are examples of local manipulation failures.

Table 10. Descriptive failure-type records for the reported BioEgo-HSG robot trials.

Failure Type	Number of Failed Trials	Percentage Among Failed Trials (%)	Typical Cause
High-level planning error	3	23.1	Incorrect recovery edge or wrong next-skill selection
State-estimation error	4	30.8	Incorrect object condition or zone-state update
Local manipulation failure	6	46.1	Grasp instability, alignment error, or placement imprecision
Total	13	100	—

Note: Gray shading identifies the column headings. Bold identifies the headings and the total row; it does not indicate statistical significance.

Figure 7 shows representative robot skill-chaining examples. In the normal execution case, BioEgo-HSG selects a sequence of semantic skills and updates procedural memory after each local manipulation. In the recovery case, the system detects a postcondition mismatch, writes the failure into memory, activates a recovery transition, and resumes execution from the latest valid graph node.

Figure 7. Representative robot skill-chaining and recovery examples in the reported laboratory setting. The upper row shows semantic skill execution and procedural-memory updates. The lower row shows a postcondition mismatch, failure-memory update, corrective skill selection, and resumption from a valid graph state.

The robot demonstrations show that the proposed procedural-memory and skill-graph mechanisms can be instantiated within the reported laboratory setup and interfaced with a physical manipulation system. These demonstrations are interpreted as feasibility observations within the tested setting rather than as evidence of general or quantitatively validated superiority over alternative planning approaches.

7. Discussion and Conclusions

BioEgo-HSG provides a methodological formulation for connecting egocentric procedural knowledge to long-horizon laboratory skill execution. The framework combines an interaction-centric hierarchical representation, episodic procedural memory, skill-graph constraints, and a semantic interface to a local robot execution library. The reported summaries and demonstrations describe how these mechanisms are instantiated within the studied setting.

The design commits task progress from postcondition feedback, retains procedural information when entities leave the current view, and represents corrective transitions as graph relations. These mechanisms define the proposed workflow and its interface contract between high-level procedural reasoning and local manipulation.

The study is limited by incomplete experiment provenance, a single participant-disjoint partition, limited recovery cases, one laboratory setting, and one robot embodiment. Further evaluation requires archived original run records, repeated runs, more operators and perturbations, and comparisons across workflows, sites, and robot platforms.