LLM-Based Control for Simulated Physical Reasoning: Modular Evaluation in the NeurIPS Embodied Agent Interface Challenge

Abstract

Benchmark-driven evaluation helps distinguish between planning quality and interface reliability when large language models are utilized for embodied reasoning in simulation. Our submission to the Embodied Agent Interface Challenge (EAI) is evaluated across four stages of the pipeline. These being goal interpretation, subgoal decomposition, action sequencing, and transition modeling. The tasks run in the BEHAVIOR and VirtualHome simulators, which use constrained action vocabularies, fixed-object inventories and symbolic state representations within a standard evaluation protocol. Our system accesses the OpenAI API using GPT-4.1 for BEHAVIOR, GPT-4.1-mini for VirtualHome, and GPT-5-mini in later exploratory experiments across both environments. The schemas for each task determine how the outputs are structured, and outputs are regenerated when they do not follow the specification. On the final public leaderboard, our system ranked eighteenth overall with a score of 57.92, achieving 68.88 on BEHAVIOR and 46.96 on VirtualHome. In this paper, we describe our approach and discuss what these observations suggest about the strengths and limitations of current language models when used for embodied reasoning.

1. Introduction

The NeurIPS Embodied Agent Interface Challenge evaluates how a system handles a sequence of reasoning steps for embodied tasks. It starts with a written goal and ends with predicted symbolic state changes [1]. The challenge separates this process into four tasks: goal interpretation, subgoal decomposition, action sequencing, and transition modeling. Each task is evaluated on its own, which makes it easier to see where errors occur. Because the inputs and outputs are symbolic, the benchmark allows reasoning behavior to be studied without dealing with perception or low-level control.

The tasks are based on two existing simulation environments, BEHAVIOR [2] and VirtualHome [3]. Both environments define a fixed set of objects, actions, and state transition rules. The evaluation uses predefined input and output formats, which keeps the comparison between systems consistent. Since the same reasoning stages are evaluated in both environments, the benchmark makes it possible to compare behavior across settings with different levels of detail. In this work, we implement the four-module pipeline required by the interface and follow the official schemas at each stage. Each module produces an explicit output that is saved separately. This design makes it possible to trace failures back to a specific step in the pipeline. The same implementation is used for both environments, with configuration files specifying the objects, actions, and state attributes.

Using large language models in this setting introduces constraints that are not always visible in free-form planning tasks. A model may produce a reasonable-looking plan but still fail because an object name is incorrect or the output does not match the expected format. In these cases, the failure is caused by a structural issue rather than by the reasoning itself. Looking only at final task success does not clearly show this difference. Evaluating each reasoning step separately helps address this issue. It becomes easier to distinguish between reasoning errors and interface formatting errors when intermediate outputs are available. This is important for language models, because outputs must follow strict schemas in order to be executable. Examining intermediate results provides a clearer picture of how these systems behave.

We present an interface-faithful implementation, meaning that we keep the benchmark interface unchanged: we use the official prompts, schemas, evaluator, and environment vocabularies for all four modules, and we do not fine-tune the model or add environment-specific rewriting rules. We describe the structural validation and regeneration policy used to keep outputs parsable and complete. Using an LLM as a control component raises a practical reliability problem that is easy to miss in text only evaluations. A plan can be sensible and still fail in execution because a required JSON field is missing, an object label does not match the environment inventory or a state update violates a transition rule.

This paper’s novelty is a reliability-focused analysis of LLM based control under a schema constrained embodied interface, using the Embodied Agent Interface benchmark to make these failures visible and measurable. We contribute (i) an interface-faithful implementation of the four-stage EAI pipeline, including structural validation and regeneration used to guarantee parsable artifacts; (ii) a modular diagnosis that separates structural and interface failures from groundedness failures using the benchmark’s task and execution metrics and planner and state metrics; (iii) a controlled cross-environment comparison under identical pipeline logic to isolate how closed-world vocabularies and schema constraints redistribute failure modes; and (iv) engineering implications that map benchmark signals to runtime reliability mechanisms integrating LLM-driven planning components into system-level autonomy frameworks.

2. Problem Statement and Scope

This paper studies reliability at the boundary between probabilistic generation and deterministic interfaces. The Embodied Agent Interface defines a pipeline of four modules. The four modules cover goal interpretation, subgoal decomposition, action sequencing, and transition modeling. The problem is to map the module input to the required structured output while staying within the simulator inventory and action set.

We keep the system interface faithful. We run the official four-stage workflow using the official prompt templates. We do not fine-tune a model. We do not add environment-specific rewriting rules. The only automated handling is regeneration when the output fails to parse or is missing required fields.

Our analysis uses the official evaluation outputs for the final public leaderboard submission [4]. We focus on two recurring failure families. The first is structural failures such as malformed JSON, missing keys and invalid nesting. The second is groundedness failures such as unsupported actions, references to missing objects, or effects that do not match the symbolic rules of the environment.

3. Related Work

Early studies of embodied reasoning emerged from research on symbolic planning, instruction-following, and hierarchical task networks. These systems showed that breaking a task into several smaller steps helps make the overall process more transparent. Research on hierarchical task networks has stressed the role of plans with well-defined preconditions and effects [5]. Shakey the Robot showed an early attempt to link symbolic planning with actions carried out in the real world [6].

Work on systems such as SHRDLU explored how natural language instructions could be interpreted and turned into actions in controlled settings [7]. Early learning-based systems such as ALVINN explored how perception and action could be connected in autonomous driving [8]. Later work on grounded language understanding explored how robots could interpret language-based commands for navigation and manipulation [9].

Simulation environments such as VirtualHome introduced program-like task descriptions in a three-dimensional household setting, allowing activities to be studied with a fixed set of objects and actions [3]. The BEHAVIOR and BEHAVIOR-1K benchmarks expanded this approach by introducing more realistic scenes, a wider range of tasks, and detailed symbolic state information [2,10].

The growth of large language models has renewed interest in using natural language for planning and high-level control in robotics. Surveys discuss symbolic, neural and hybrid approaches that ground language in perception and action [11,12]. Other surveys focus on multi-step instruction following, activity representations, and planning [13,14,15].

Reflexion introduced a form of self-correction in which an agent reviews its previous attempts and adjusts its next move [16]. Other studies examined agents that call tools or external services to gather information during a task [17,18]. Agentic system architecture work studies multi-step decision loops where a model conditions on its own intermediate outputs. In robotics settings, these loops make the interface explicit because the model must choose actions from a fixed vocabulary and produce structured intermediate state that other components can consume in later. Recent work has also explored combining temporal localization with causal reasoning in domain-specific multimodal tasks [19]. Under the EAI interface, this often appears as brittle goal interpretation, invalid action sequencing, or inconsistent state transitions.

4. Background

Modular benchmarks for embodied reasoning report the outputs of individual stages rather than only an overall success label. This structure makes it possible to identify where a pipeline breaks. Failures may arise in goal interpretation, subgoal generation, action selection, or in the way state changes are represented, and these failures need not look the same when viewed only through an end-to-end metric.

The Embodied Agent Interface specifies four modules and a schema for the output of each one [1]. From a natural language instruction and an environment description, the system is expected to produce a structured goal representation, a list of subgoals, a sequence of symbolic actions, and corresponding state updates. These outputs are passed from one module to the next in a fixed order. Figure 1 summarizes this interface and the information passed between modules.

Benchmark Abstractions and Assumptions

EAI is evaluated using symbolic simulators, which ensures that runs are deterministic while imposing a fixed set of modeling assumptions on the task. Both BEHAVIOR and VirtualHome provide a closed-world object inventory and a constrained action vocabulary [2,3]. State updates in the benchmark are encoded symbolically through explicit predicates and attribute assignments. These choices avoid perception noise and low-level control uncertainty, which helps isolate high-level reasoning. Under this setup, interface-level correctness becomes a gating condition, since outputs that violate the schema or reference undefined symbols are rejected even when the intended plan is otherwise coherent.

One example is a refrigerator cleaning task. It requires selecting the right objects, ordering actions such as opening the door and moving items, and tracking changes such as whether the door is open and whether a surface is clean. Within EAI, each of these operations must be encoded using the prescribed schemas. This makes it possible to attribute failures to a specific module rather than treating the pipeline as an opaque system.

5. Methods

We implemented the four-stage pipeline defined by the Embodied Agent Interface. For each task instance, the pipeline runs goal interpretation, subgoal decomposition, action sequencing, and transition modeling in sequence. Each stage takes the task input and the official prompt template for that module, calls a language model through the OpenAI API, and writes the result to disk for evaluation.

5.1. Workflow and Artifacts

The pipeline is organized to mirror the benchmark interface. Downstream modules take earlier outputs as inputs and each module emits an artifact in the format required by the official evaluator. This design keeps error attribution simple—when a failure occurs, it is possible to isolate whether it originated in a particular module or in the interface between modules. Figure 2 summarizes the workflow.

5.2. Schema Compliance Checks and Regeneration

Several modules in the pipeline require outputs in strict JSON format with predefined keys and nesting. Outputs are subjected to basic structural validation before being accepted. If an output does not parse as JSON or is missing required fields, the same prompt is issued again with an additional instruction that requests correction of structure only. This step addresses interface-level failures that would otherwise prevent evaluation despite a plausible underlying plan.

Because regeneration affects which outputs are ultimately evaluated, it is treated as part of the system implementation rather than as a property of a single model sample. Regeneration is applied only to enforce structural validity. No semantic adjustments are made, such as modifying object references or filtering actions against the allowed vocabulary.

5.3. Model Selection

During development we used three OpenAI API models: GPT-4.1-mini, GPT-4.1, and GPT-5-mini [20]. The models were accessed through the OpenAI API using those exact model identifiers. The final public leaderboard submission evaluated in this paper uses GPT-4.1 for BEHAVIOR and GPT-4.1-mini for VirtualHome. We also ran exploratory tests with GPT-5 mini to confirm prompt compatibility and output formatting, but we do not report a systematic model comparison.

6. Dataset

The experiments use the official dataset released for the Embodied Agent Interface Challenge and hosted on Hugging Face [1]. The dataset provides the complete set of task instances, environment configurations and reference specifications required to run the benchmark under the standardized interface.

The benchmark dataset is organized around two simulated environments, VirtualHome and BEHAVIOR. The VirtualHome portion contains 338 tasks spanning 26 activity categories, while the BEHAVIOR portion includes 100 tasks that emphasize complex, multi-step physical goals. Each task begins with a natural language goal description and is associated with environment-specific configuration files that enumerate the available objects, valid actions, and symbolic state attributes.

Each task includes formal annotations for goals and transitions that support interpretable and reproducible evaluation. The annotations comprise Linear Temporal Logic (LTL) goal specifications, symbolic trajectories and transition models expressed in a Planning Domain Definition Language (PDDL) style [21]. PDDL is a symbolic planning language that represents actions in terms of preconditions and effects, enabling formal reasoning about how sequences of actions transform a world state.

The BEHAVIOR tasks are derived from household manipulation scenarios and rely on a rich symbolic state representation. State attributes represent relations between objects, such as containment and support. They also encode object properties, including whether an object is open, closed, or held. This detailed state space increases the difficulty of transition modeling, as multiple attributes may need to be updated consistently across long action sequences.

The VirtualHome tasks are based on program like activity descriptions executed by an embodied agent in a simulated household. Exact matching of action signatures and object labels is required for correct execution and state tracking in VirtualHome. Action sequences are often shorter than in BEHAVIOR. The environment also differs in its object naming scheme and action vocabulary and applies stricter syntactic constraints on action parameters.

7. Evaluation Setup

The system is evaluated using the official evaluation protocol provided by the Embodied Agent Interface Challenge. All scores reported in this paper are produced by the challenge’s evaluation scripts without modification. The evaluation treats each stage of the pipeline separately, which allows performance to be measured at different points in the reasoning process.

The evaluator runs at the level of a single task instance. In practice, it reads the output produced by each module and checks two things. First, it checks the structure. This is the basic requirement that the output matches the expected schema (for example, valid JSON when JSON is required, and the required fields are present). Second, it checks content against the simulator constraints. This includes things like whether referenced objects exist in the environment inventory, whether the selected actions are part of the allowed action vocabulary, and whether the output is consistent with the symbolic state representation used by that environment. The same overall process is applied to all four stages.

Goal interpretation is scored based on whether the returned representation matches the goal structure expected by the interface. The output needs to follow the fields and structure defined by the task. Subgoal decomposition is scored in a similar way, but it focuses on whether the predicted subgoals use allowed subgoal types and whether the ordering is consistent with the task specification. In both cases, the evaluator compares structured symbolic fields instead of relying on text similarity.

The evaluator reports two scores for the action sequencing stage. The first one is task success, which focuses on whether the proposed action list achieves the goal under the simulator rules. The second one is execution success, which applies a stricter check where every step must be executable and a single unsupported action or invalid object reference causes failure. These metrics are reported in the Section 8.

The transition modeling is evaluated by comparing the predicted symbolic state updates to the reference state transitions defined by the simulator. In addition to an overall state prediction score, the evaluator reports a planner-level metric that measures how well the predicted transitions support task completion. Transition modeling scores depend on explicit symbolic state changes rather than narrative descriptions and missing or incorrect updates can lower the score even when the action sequence is largely correct. This stage requires consistent state tracking across multiple steps. The evaluator checks predicted updates against the expected effects of each action using the environment’s state attributes and relations.

All evaluation runs are performed separately for BEHAVIOR and VirtualHome. Scores are reported per environment and per module, as well as aggregated across modules. This makes it possible to compare performance not only between systems, but also between environments with different levels of detail and constraint. The final leaderboard ranking is computed using the official aggregation rules defined by the challenge.

In addition to schema validity, the evaluation uses module-specific symbolic metrics. For goal interpretation, the model maps natural language goals to Linear Temporal Logic specifications [22]. Scores are computed with precision, recall and F1 [1] and the evaluator reports these metrics for the state, relation and action goal types. This helps separate false positives from missing goal components.

For subgoal decomposition, a planner refines the predicted subgoal sequence into an executable trajectory. The evaluator then checks feasibility and goal satisfaction. For transition modeling, outputs are compared using logic–form matching over Planning Domain Definition Language (PDDL)-style preconditions and effects. The evaluator also reports a planner success metric for transition models. Overall performance is computed using the official aggregation formula across modules.

8. Results

This section presents the evaluation results for our submission to the public leaderboard of the 2025 NeurIPS Embodied Agent Interface Challenge during the evaluation phase, under the team name UNCC [4]. Results are presented separately for BEHAVIOR and VirtualHome, along with the aggregated overall score.

8.1. Overall Leaderboard Performance

The submission ranked 18th out of 50 participating teams on the public leaderboard with an overall score of 57.92. The rank and scores are taken from the public leaderboard listing used for the final evaluation [4]. BEHAVIOR averaged 68.88 and VirtualHome averaged 46.96.

Table 1. Overall leaderboard scores by environment.

Environment	Average Score	Overall Rank (Out of 50)
BEHAVIOR	68.88	18th
VirtualHome	46.96

reports the overall leaderboard scores by environment.

8.2. Module-Level Results

In VirtualHome, goal interpretation scored 22.10. In VirtualHome, transition modeling scored 40.60 for state prediction and 29.50 for the planner-level metric. Subgoal decomposition reached 61.80 at the task level and 79.50 at the execution level. Action sequencing produced task and execution success scores of 68.90 and 79.60.

Table 2. Module-level evaluation results reported by the official evaluator.

Stage	Metric	BEHAVIOR	VirtualHome
Goal Interpretation	Score	78.70	22.10
Subgoal Decomposition	Task level	49.00	61.80
Subgoal Decomposition	Execution level	54.00	79.50
Action Sequencing	Task success	75.00	68.90
Action Sequencing	Execution success	83.00	79.60
Transition Modeling	State prediction	58.60	40.60
Transition Modeling	Planner score	87.00	29.50

summarizes the module-level results reported by the official evaluator for both environments. In BEHAVIOR, goal interpretation scored 78.70. Action sequencing scored 75.00 at the task level and 83.00 at the execution level. Subgoal decomposition performed lower, with scores of 49.00 for task success and 54.00 for execution success. For transition modeling, the state prediction score was 58.60, and the planner-level metric reached 87.00.

The best publicly reported official task-level scores in the EAI challenge were 99.6/97.0/98.0/99.5 on BEHAVIOR and 65.4/78.7/82.6/99.9 on VirtualHome for goal interpretation, subgoal decomposition, action sequencing, and transition modeling, respectively. These scores were obtained using a different, task-specialized system based on fine-tuned Qwen3 models rather than an API-based setup like ours [23].

8.3. Derived Indicators of Interface Reliability

For action sequencing and subgoal decomposition, the evaluator reports two success rates, which are the task level and execution level. The task-level score measures how closely the predicted plan matches the reference symbolic plan. The execution-level score measures whether the plan can be executed under the simulator’s rules. The gap between these two scores indicates how much additional error is introduced by executability constraints beyond plan structure.

Table 3. Differences between execution-level and task-level scores (Exec−Task) as a coarse indicator of additional failures introduced at execution time. Transition modeling is shown as Planner−State.

Metric Gap	BEHAVIOR	VirtualHome
Subgoal Decomposition (Exec−Task)	5.00	17.70
Action Sequencing (Exec−Task)	8.00	10.70
Transition Modeling (Planner−State)	28.40	$-11.10$

reports the differences between execution-level and task-level scores and, for transition modeling, between planner and state scores.

Table 3 highlights two patterns. The first one is the execution gap being larger in VirtualHome for subgoal decomposition, suggesting that decompositions that look acceptable at the task level can still fail under execution constraints. The second one is that the transition modeling shows the largest separation in BEHAVIOR between the state prediction score and the planner metric, which is consistent with the evaluator rewarding plans that enable downstream planning even when some symbolic effects are missed. In VirtualHome, the planner metric is lower than the state prediction score, indicating that transition errors are more likely to break planner-level success in that environment.

8.4. Cross-Environment Comparison

The reported results indicate clear differences between the two environments across several stages of the pipeline. BEHAVIOR shows higher scores in goal interpretation and transition modeling, while VirtualHome exhibits higher execution level scores in action sequencing and subgoal decomposition. Since the same pipeline implementation and execution logic are used for both environments, these differences reflect properties of the environments and their evaluation constraints rather than environment specific processing in the system.

9. Error Analysis

EAI makes failures visible because each stage is evaluated separately and because the evaluator enforces schema constraints. In our development runs and in preparing the final submission artifacts, two broad categories of problems appeared repeatedly. The first category is structural validity: the output does not match the required schema and therefore cannot be scored. The second category is groundedness: the output is well formed but violates environment constraints, such as referencing an object that is not present in the inventory or selecting an action that is not in the allowed vocabulary.

9.1. Structural Validity Failures

Several modules require outputs in strict JSON format with required keys and nesting. Models sometimes produced outputs that were readable but not parseable or outputs that omitted required fields. These failures are particularly damaging because they can zero a module score and prevent downstream modules from being evaluated. We observed these issues more often in stages with deeper nesting and longer outputs, especially subgoal decomposition and transition modeling.

9.2. Groundedness Failures

Even when outputs were structurally valid, groundedness errors were common. Two types dominated. First, the model proposed actions that were not part of the constrained action vocabulary, or attached objects to actions in ways that are not allowed. Second, the model referred to objects that were not present in the task’s object inventory, often by using plausible but incorrect names. Because later stages consume earlier outputs without correction, these errors can propagate; an incorrect object name in goal interpretation or decomposition can persist through action sequencing and transition modeling.

9.3. Propagation Across Modules

A central reason to evaluate the pipeline in modular form is that errors propagate in predictable ways. Goal interpretation errors change the target that subsequent modules attempt to satisfy. Decomposition errors lead to action sequences that are internally consistent but misaligned with the intended goal. In our pipeline, which avoids post-hoc repair, a small mismatch early in the pipeline often persisted, which is consistent with the benchmark design and with the gap patterns in Table 3.

9.4. Transition Modeling Drift

Transition modeling is the most brittle stage because it requires predicting a sequence of symbolic state updates that remain consistent with both the action history and the simulator rules. We often observed drift on longer tasks, where later state updates became incomplete, contradictory, or detached from the earlier state. This tendency is reflected in the lower VirtualHome planner-level transition modeling score reported in Table 2.

9.5. Implications of Regeneration

Regeneration reduces the number of failures caused purely by formatting, but it also changes what is being measured. Instead of scoring the first model sample, the evaluator scores the first structurally valid sample produced under the regeneration policy. In our submission, regeneration is limited to structural repair; it does not correct groundedness errors. As a result, regeneration mainly improves interface reliability, while leaving most reasoning and grounding errors visible in the final scores.

10. Discussion

The Embodied Agent Interface is valuable because it imposes an explicit contract on intermediate reasoning artifacts. The model is not evaluated on free-form descriptions, but on whether it can produce artifacts that satisfy schemas and remain consistent with a closed set of objects, actions, and state predicates. Our results suggest that this contract is easy to violate, and that violations are not uniform across modules or environments.

10.1. Interpreting the Score Patterns

Two comparisons are especially informative. First, performance differs substantially between BEHAVIOR and VirtualHome, with the largest gaps in goal interpretation and transition modeling (Table 2). This pattern is consistent with VirtualHome tasks providing less contextual grounding, which increases ambiguity and raises the likelihood of vocabulary and reference errors. Second, task-level and execution-level metrics diverge (Table 3). When execution-level success falls relative to task-level success, the plan may be structurally plausible but fails under environment constraints.

10.2. Reliability Mechanisms Suggested by the Interface

From an engineering perspective, the interface suggests a layered reliability stack. Schema validation is the first gate—parse failures and missing keys should not reach downstream modules. Vocabulary checks are a natural second gate—action names and object references can be validated against the inventory provided by the task before acceptance. Transition modeling benefits from symbolic sanity checks, such as consistency of relations and mutually exclusive attributes. In our submission we applied structural checks and regeneration, but we did not apply systematic vocabulary filtering or symbolic invariants, which leaves groundedness errors visible in the results.

10.3. Conceptual Comparison to Heuristic or Hybrid Approaches

Several improvements are compatible with the EAI setting even without changing the language model. Constrained generation can enforce JSON validity by restricting decoding to a grammar. Post-generation repair can map near-miss tokens to the closest valid action or object name when the correction is unambiguous, followed by regeneration when it is not. A more structural option is to combine the model with a symbolic planner that enforces action preconditions and effects, using the LLM primarily for goal interpretation and decomposition. These approaches trade simplicity for stronger guarantees and would likely reduce the failure modes described in Section 9.

10.4. Open Research Questions

Main open research questions include how to measure reliability under partial observability when the object inventory is incomplete, observations are noisy and the agent must decide what to do without full information. Another question is how to evaluate uncertainty handling in a way that rewards safe behavior, calibrated confidence, and information seeking actions instead of relying on guesses. A third question is how to ensure grounding under schema constraints, so object references are linked to what was actually observed and remain consistent over time.

Transition modeling is difficult when the world model is approximate and the environment can tolerate possible small discrepancies. Score state updates in a way that ignores small numeric drift but catches real semantic mistakes, enforce basic rules like object conservation and track how errors build up over long sequences. Another practical question is how to test whether approximate transitions are still good enough for planning, state tracking and recovery via additional sensing and replanning.

10.5. Practical Significance

Modular evaluation mirrors how many embodied systems are engineered. Consider a simulated household assistant tasked with cleaning a refrigerator. Goal interpretation determines what counts as completion. Subgoal decomposition determines whether the plan includes necessary preparatory steps, such as obtaining a sponge or opening the door. Action sequencing must respect the action vocabulary and object bindings. Transition modeling determines whether the system’s internal state remains consistent with the action history. When a failure occurs, the module boundaries support targeted debugging and a JSON formatting issue requires different fixes than an incorrect object reference or an inconsistent state update.

11. Conclusions

We described an interface faithful pipeline for the Embodied Agent Interface Challenge and summarized its official evaluation results on BEHAVIOR and VirtualHome. The benchmark’s modular scoring makes it possible to separate failures in goal interpretation, decomposition, action selection, and transition modeling, rather than attributing all errors to a single end-to-end outcome.

The key result is that reliability is often limited by interface compliance and grounding, not by the ability to produce plausible text. Schema validity gates reduce avoidable failures, but groundedness errors and state tracking drift remain common, especially in lower context settings. These observations support the use of structured interfaces as both an evaluation tool and a practical contract for integrating language models into embodied control stacks.