Large Language Models for Structured Task Decomposition in Reinforcement Learning Problems with Sparse Rewards

Abstract

Reinforcement learning (RL) agents face significant challenges in sparse-reward environments, as insufficient exploration of the state space can result in inefficient training or incomplete policy learning. To address this challenge, this work proposes a teacher–student framework for RL that leverages the inherent knowledge of large language models (LLMs) to decompose complex tasks into manageable subgoals. The capabilities of LLMs to comprehend problem structure and objectives, based on textual descriptions, can be harnessed to generate subgoals, similar to the guidance a human supervisor would provide. For this purpose, we introduce the following three subgoal types: positional, representation-based, and language-based. Moreover, we propose an LLM surrogate model to reduce computational overhead and demonstrate that the supervisor can be decoupled once the policy has been learned, further lowering computational costs. Under this framework, we evaluate the performance of three open-source LLMs (namely, Llama, DeepSeek, and Qwen). Furthermore, we assess our teacher–student framework on the MiniGrid benchmark—a collection of procedurally generated environments that demand generalization to previously unseen tasks. Experimental results indicate that our teacher–student framework facilitates more efficient learning and encourages enhanced exploration in complex tasks, resulting in faster training convergence and outperforming recent teacher–student methods designed for sparse-reward environments.

1. Introduction

Imagine trying to master a complex new skill without any guidance or feedback until the moment the effort is fully completed. No indication of progress, no suggestions for improvement; only endless trial and error. This is the challenge of reinforcement learning (RL) agents in sparse-reward environments. The lack of intermediate guidance makes learning inefficient and exploration highly resource-intensive [1,2]. In such settings, traditional RL approaches relying on random exploration often fail to scale, especially when tasks involve multiple stages or dependencies.

To address these challenges, researchers have explored structured learning paradigms, such as curriculum learning (CL) [3], imitation learning [4], hierarchical reinforcement learning (HRL) [5], and teacher–student frameworks [6]. While CL provides a progressive learning path and HRL decomposes tasks into subtasks, both approaches face limitations in adaptability and dynamic feedback generation. The teacher–student paradigm, in contrast, introduces a guiding entity capable of tailoring subgoals to the agent’s current capabilities.

Recent advances in large language models (LLMs) offer a promising avenue for implementing such dynamic teachers [7]. LLMs possess the ability to interpret textual descriptions, develop task structures, and generate context-aware subgoals. This capability aligns with findings in natural language processing, where LLMs improve inferential reasoning when guided by rationales [8]. Similarly, LLMs can enhance exploration and generalization in RL by providing structured, human-like guidance.

Motivated by these capabilities, this study proposes a novel teacher–student RL framework that leverages LLMs as expert teachers, guiding agents in sparse-reward environments by decomposing tasks into different types of subgoals. While prior work has explored the potential of integrating LLMs into RL, existing approaches often fall short because they rely heavily on repeated LLM queries, which are computationally expensive and introduce significant latency. Moreover, prior work often focuses on generating subgoals that broaden the agent’s knowledge or facilitate exploration, rather than explicitly subdividing a single task into ordered steps that drive the agent towards successful completion (Figure 1). To address these limitations, we build upon our preliminary findings presented in [9] to introduce a scalable surrogate model that approximates the LLM’s behavior while avoiding overfitting to oracle-like supervision. In addition, we study how subgoals can be gradually decoupled, allowing the agent to benefit from them during training but ultimately operating at deployment without any reliance on explicit subgoals or LLM queries.

We evaluate our framework using three open-source LLMs (Llama, DeepSeek, and Qwen) across procedurally generated environments from the MiniGrid benchmark [10]. Our results demonstrate that LLM-guided subgoal generation accelerates learning, improves exploration, and enhances generalization to unseen tasks. The results show that LLM decomposition facilitates faster convergence, enabling our method to solve the evaluated environments in a fraction of the training steps required by other approaches [11,12], achieving over 90% improvement.

In summary, our contributions are as follows:

• We perform a detailed evaluation of different LLMs, identifying the most effective models for the teacher role in our proposed framework.
• We identify the most effective subgoal encoding that enables agents to perceive and act on them.
• We develop a scalable method for LLM integration, reducing computational costs and ensuring practical applicability in real-world settings.

The rest of the manuscript is structured as follows. In Section 2, we examine the related literature, providing an overview of existing approaches in the field. Section 3 outlines the proposed framework and methodology, including details of our approach. Section 4 introduces the experimental setup and Section 5 presents the obtained results, followed by a discussion of our findings. Section 6 concludes the paper and outlines future research directions.

2. Related Work

In this section, we review existing research and methodologies relevant to our study. We begin by exploring CL (Section 2.1), emphasizing the benefits of a well-structured learning process. Next, we discuss HRL (Section 2.2), which focuses on decomposing complex RL problems into more manageable subtasks. Finally, we analyze different strategies for integrating LLMs into the RL training process (Section 2.3).

2.1. Curriculum Learning

Organizing learning tasks in a structured sequence has emerged as an effective strategy in training agents. Originally conceived as “training from easy to hard”, CL has attracted attention due to its potential to expedite learning processes [11,13]. However, its efficacy is not universally consistent across implementations, where the curriculum—i.e., the structured sequence of tasks, environments, or goals presented to the agent in increasing order of difficulty to facilitate its learning—must be well-defined in order to be effective [14,15,16], suggesting that its application requires careful considerations.

Implementing CL in RL, therefore, requires deliberate design choices, often relying on teachers to guide the learning process. Randomized task sequences offered initial improvements over unguided training, but dynamically chosen tasks by a teacher helped maintain previously acquired skills [14,17]. Beyond these, to make curricula more adaptive, a buffer of high-potential learning tasks can be used, or tasks can be monitored for changes in success rates, ensuring training is concentrated where learning is most active [18,19].

Introducing subgoals allowed complex tasks to be decomposed into manageable objectives, while the teacher adaptively adjusted subgoal difficulty over time, effectively creating a curriculum tailored to the agent’s knowledge [20]. Adversarial frameworks complemented this process by evaluating the effectiveness of these subgoals without relying on environment rewards [21,22]. Subsequent work extended this idea by expressing subgoals in natural language, which not only provided richer guidance but also made the learning process interpretable to human observers [12].

2.2. Hierarchical Reinforcement Learning

HRL leverages structured abstractions to decompose complex tasks into manageable subgoals, improving generalization and sample efficiency [5,23]. Among these abstractions, language has proven to be especially powerful: high-level policies can use linguistic instructions to guide low-level controllers, enabling both flexible skill reuse across diverse tasks and improved interpretability [24]. Policies for these subgoals are typically trained from demonstrations, feedback, or language, allowing high-level planners to orchestrate them with structured commands [25,26]. Building on this paradigm, recent work integrates LLMs into HRL, where a high-level RL policy leverages LLMs to generate intermediate subgoals for the low-level agent, further enhancing the adaptability and scalability of its learning process [27,28].

2.3. LLM-Enhanced RL

Leveraging the power of LLMs has become an emerging strategy in the field of RL, exploiting the capabilities of natural language processing for RL frameworks. Models like the proprietary GPT series [29,30,31] or open-source alternatives such as Llama [32], Qwen [33], and DeepSeek [34] are nowadays being harnessed to guide RL agents.

Recent surveys in the field [7,35] have provided substantial insights into the diverse capabilities of LLMs, highlighting various studies that integrate them into RL frameworks. One prominent approach involves preprocessing RL environments into textual representations, including goals, histories, and observations, allowing LLMs to predict actions either in a zero-shot manner [36] or through fine-tuning on these textual environments [37]. Beyond acting as decision-makers, LLMs can play a central role in shaping reward functions, leveraging their ability to reason. By interpreting user-provided task descriptions, they can generate reward signals that reflect desired behaviors [38,39]. They can also elicit preferences over candidate behaviors to iteratively refine reward functions, guiding agents toward the acquisition of complex skills [40,41]. In addition, LLMs can serve as powerful information processors, offering new ways to simplify tasks. They can decompose tasks into smaller steps, suggest meaningful goals, and update plans based on outcomes, enabling agents to act effectively in diverse environments [42,43,44]. LLMs can also evaluate potential actions, combining natural language descriptions with value estimates to select the most promising behaviors [45], or leverage multimodal inputs and memory to generate and execute comprehensive plans, achieving state-of-the-art performance [46].

2.4. Contribution

Our approach leverages a pretrained teacher to generate subgoals, avoiding the need for the teacher to learn subgoal generation from scratch [12,21,27]. In contrast to methods that focus on diverse, non-sequential skills [43], we concentrate on sequential, task-specific subgoals that guide the agent through a structured learning sequence. Beyond simply assessing whether the LLM can generate subgoals [44], we examine their impact on RL training, including how the integration of these subgoals into the agent’s observations influences learning performance.

Table 1. Comparison of prior subgoal generation methods and our approach. Columns indicate (1) whether subgoals are sequential and task-specific; (2) whether an LLM is used as a teacher; (3) whether the method trains an RL agent; and (4) the main contribution.

Method	(1)	(2)	(3)	(4)
AMIGO [21]	✗	✗	✓	Trains an adversarial teacher to generate subgoals that guide the agent to explore the state space efficiently.
LAMIGO [12]	✗	✗	✓	Extends AMIGO by representing subgoals as natural language instructions, enabling text-based guidance.
LLMxHRL [27]	✓	✗	✓	Leverages an LLM to provide commonsense priors, which guide a hierarchical agent in generating and interpreting subgoals.
ELLM [43]	✗	✓	✓	Uses LLM to propose non-sequential, diverse skills, without task-specific sequencing.
CALM [44]	✓	✓	✗	LLM to decompose tasks into subgoals, evaluating only the accuracy of proposed subgoals without assessing RL performance.
Ours	✓	✓	✓	Generates sequential, task-specific subgoals with surrogate models from LLMs, guiding agents while reducing computational cost.

summarizes prior subgoal generation methods and highlights key differences in sequentiality, task-specificity, LLM usage, and RL integration. As shown in this table, our work differs from previous approaches in that we employ a LLM to train a surrogate model that acts as a teacher to decompose episodic, time-sensitive tasks, thereby enabling more efficient agent training and guiding learning through structured, sequential subgoals.

3. Proposed Framework and Methodology

Our framework integrates an LLM as an omniscient teacher that supervises an RL agent through sequences of subgoals. At the start of each episode, the LLM, with full access to the initial environment state $s_0\in \mathcal{S}$ and the global objective $g^{obj}$, generates a fixed sequence of N subgoals $g_0,g_1,\dots ,g_{N-1}$. These subgoals are designed to be unique ($g_{i+1}\ne g_i$) and progression-dependent, ensuring the agent advances toward the global goal $g^{obj}$. During interaction, the RL agent only observes partial states $o_t$, and at each time step it receives the triplet $(o_t,r_t,g_i)$, where $r_t$ is the environment reward and $g_i$ is the current subgoal provided by the teacher. Once a subgoal is achieved, such as reaching or interacting with the specified target, the agent immediately receives the next subgoal $g_{i+1}$. The agent thus learns a student policy ${\pi }_{student}$ that maps $(o_t,g_i)$ to actions, while the LLM provides the teacher policy ${\pi }_{teacher}$ by generating subgoals. The teacher remains static throughout training, serving as a source of prior knowledge, while the student adapts by learning how to effectively use the subgoals to achieve the global objective.

In this section, we first outline the methodology for generating subgoals (Section 3.1) and defining their representations (Section 3.2). Then, we describe our approach to prompt engineering, optimizing LLM performance for effective subgoal decomposition. Finally, we introduce a surrogate model that approximates LLM-generated subgoals, enabling efficient training without the need for repeated LLM queries (Section 3.3).

3.1. Subgoal Generation

Subgoal generation takes place at the beginning of each episode, where the LLM, with the complete environment state $s_0$, creates a sequence of subgoals $\{g_0,g_1,\dots ,g_N\}$, which progressively guide the agent toward the final goal of the RL task. These subgoals reduce the reward horizon by decomposing the problem into intermediate steps, thereby facilitating and shaping the learning process. In doing so, our approach constitutes an intermediate setup between sparse-reward cases (where feedback is limited to task completion) and dense-reward cases (where feedback is provided at a high frequency, e.g., at every step).

To incentivize both subgoal achievement and task completion, we augment the extrinsic reward $r_t^e$ with an auxiliary subgoal reward $r_t^g$. The total reward at each time step is defined as $r_t=r_t^e+\alpha ·r_t^g$, where $\alpha \in {\mathbb{R}}^{+}$ balances the emphasis on subgoal completion versus overall task success. The subgoal reward $r^g$ is structured to encourage efficient progress. It is computed as $r^g=1-{\displaystyle \frac{t^g}{t_{\max }}}$, where $t^g$ is the time step at which subgoal g is achieved, and $t_{\max }$, which depends on the environment, is the maximum number of steps allowed in an episode and must be chosen carefully as it can strongly impact learning. Using this formulation, we penalize excessive time spent on individual subgoals while rewarding timely progression.

3.2. Subgoal Representation

The form in which subgoals are presented plays a crucial role in how effectively the agent can use them. In our framework, we consider the following three distinct encoding strategies to represent teacher-provided subgoals:

• Position-based subgoals are defined in terms of coordinates provided by the environment (e.g., $\{x,y\}$ in 2D or $\{x,y,z\}$ in 3D). In discrete domains, these subgoals represent a point in space, whereas in continuous domains they denote a target region. While intuitive, this approach assumes the agent can infer directional relationships and may suffer from misalignment when the LLM inaccurately predicts object positions.
• Representation-based subgoals are defined in terms of identifiable components or features of the environment’s state, which the agent perceives through its observations $o_t$. In object-based domains, this could correspond to the representation of a specific object, while in image-based domains, it could correspond to an element detected through perception modules. This approach can be limited in novel or unseen scenarios since subgoals are tightly linked to predefined state representations.
• Language-based subgoals are derived from LLM-generated text, which is especially effective when the task cannot be directly expressed as an element of the environment. The text is then converted into embeddings using a pretrained language encoder, allowing the subgoals to be represented in a continuous vector space. To reduce the complexity of handling diverse natural language instructions, we first construct a fixed pool of environment-relevant subgoals. This pool is generated by enumerating all objects and their attributes and combining them with a set of predefined actions. Each resulting subgoal phrase is then converted into a vector embedding. When a new instruction is received, it is similarly converted into an embedding and matched to the closest canonical subgoal in the pool using cosine similarity.

Visual examples of these subgoal representations and the language subgoal pool for the RL environments considered in this work can be found in Appendix D.

3.3. Modeling LLM–Surrogate Subgoal Generation

While LLMs excel at decomposing tasks into subgoals, their direct integration into RL training poses scalability challenges. Many RL problems require training across hundreds of thousands of episodes, making real-time subgoal generation with LLMs computationally and economically impractical. For instance, modern environments can run at 1000–3000 frames per second, making a 1000-step task complete at roughly 1–3 Hz [47]. In contrast, end-to-end LLM queries might run at 0.2–0.5 queries per second [48]. This gap means that using an LLM inside the training loop would slow data collection by one to two orders of magnitude. Moreover, continuously running LLMs is often impractical due to resource constraints, making RL training both costly and unstable.

Beyond scalability and high resource demands, another key challenge arises when agents are trained with perfect subgoals but must be deployed under imperfect guidance. Training under flawless supervision, whether from a hypothetical perfect LLM or manually specified subgoals, creates a distribution mismatch: at deployment, agents encounter noisy or incomplete subgoals, rather than the ideal guidance seen during training. This can lead to overfitting, where agents rely on unrealistic guidance and fail to generalize properly when deployed. Moreover, generating perfect supervision through LLM prompting for every training step is costly and infeasible at the scale required for many RL tasks.

To address this issue, we introduce an offline surrogate framework (Algorithm 1) that approximates LLM-guided subgoal generation while avoiding repeated LLM queries during training. The surrogate is designed to mimic the variability of real LLM outputs, allowing agents to train under deployment-like conditions and improving robustness when oracle supervision is no longer available. Results in Appendix B confirm the effectiveness of this strategy for the RL environment later described in the experiments of this manuscript.

To take this idea to practice, we sample M levels from the environment’s level distribution and collect a set of subgoal sequences by prompting the LLM. Using an oracle, we label the optimal subgoal sequences as ground truth—the only supervision needed—and if no oracle is available, a human could provide the annotations. Finally, we statistically characterize the deviations of the LLM proposals from the oracle decompositions to construct a surrogate model, focusing on two error types:

• Positional errors: For position-based subgoals, we measure the distance between the LLMs’ proposed coordinates $G^m$ and the oracle’s optimal coordinates $O^m$.
• Semantic errors: For representation-based and language-based subgoals, we evaluate mismatches in object type, state, or sequence.

Algorithm 1 Surrogate LLM framework

1: M: The number of levels considered in the evaluation. 2: ${\mathcal{G}}^m$: The set of subgoal proposals at iteration m. 3: ${\mathsf{\Omega }}^m$: The set of optimal subgoals at iteration m. 4: $E=\{e_0,e_1,\dots ,e_N\}$: The set of all errors accumulated over all levels. 5: $ϵ$: The accuracy for the LLM, computed based on E. 6: $\widehat{S}$: The surrogate model, created based on the accuracies of the subgoals.   7: // Accuracy evaluation 8: for$m=1$ to M do 9:       Prompt the LLM for subgoal proposals ${\mathcal{G}}^m\leftarrow \mathrm{LLM}(L^m,s_0)$ 10:     Label the optimal subgoals using the oracle ${\mathsf{\Omega }}^m\leftarrow \{{\omega }_0^m,{\omega }_1^m,\dots ,{\omega }_N^m\}$ 11:     for $i=0$ to N do 12:         $(g_i^m,{\omega }_i^m)\leftarrow ({\mathcal{G}}^m\left[i\right],{\mathsf{\Omega }}^m\left[i\right])$ 13:         $e_i^m=\left\{\begin{array}{cc}\mathrm{Manhattan}\mathrm{distance}(g_i^m,{\omega }_i^m),\hfill & \mathrm{if}\mathrm{position}-\mathrm{based},\hfill \\ {\mathbb{1}}_{\mathrm{match}}(g_i^m,{\omega }_i^m),\hfill & \mathrm{if}\mathrm{semantic}-\mathrm{based}.\hfill \end{array}\right.$ 14:         Append $e_i^m$ to E 15:     end for 16: end for 17: $ϵ\leftarrow \mathrm{ComputeMean}\left(E\right)$   18: // Surrogate model application 19: for each episode do 20:     Sample level from environment 21:     for each ${\omega }_i\in \mathsf{\Omega }$ do 22:         $e_i^{\prime }\sim \mathcal{U}(0,1)$ 23:         $g_i^m=\left\{\begin{array}{cc}\widehat{S}({\omega }_i^m,\mathrm{modify}),\hfill & e_i^{\prime }<ϵ,\hfill \\ \widehat{S}({\omega }_i^m,\mathrm{retain}),\hfill & e_i^{\prime }\ge ϵ.\hfill \end{array}\right.$ 24:     end for 25:     Train RL agent with subgoals from $\widehat{S}$ 26: end for

Once trained, this surrogate can generate imperfect but realistic subgoals across the full distribution of levels, not just the M sampled ones. In doing so, it removes the scalability bottleneck of repeated LLM queries while preserving the benefits of LLM-guided task decomposition.

4. Experimental Setup

To evaluate the effectiveness of our proposed method, we conduct experiments across several challenging environments from the MiniGrid framework [10], a widely used benchmark for RL research. MiniGrid, part of the Farama Foundation ecosystem, provides lightweight, grid-based environments that simulate navigation, object interaction, and task completion tasks. MiniGrid environments are procedurally generated, meaning that each episode presents a unique level layout while maintaining consistent task structures (e.g., retrieving keys, unlocking doors, and navigating mazes). This procedural generation ensures that agents must learn to generalize across diverse instances rather than memorizing specific layouts, making MiniGrid an ideal testbed for evaluating the robustness and adaptability of our approach. We focus on six environments of varying complexity:

• MultiRoomN2S4 and MultiRoomN4S5: Navigation tasks requiring the agent to traverse multiple rooms.
• KeyCorridorS3R3 and KeyCorridorS6R3: Tasks involving key retrieval and door unlocking in increasingly complex layouts.
• ObstructedMaze1Dlh and ObstructedMaze1Dlhb: Challenging maze navigation tasks with obstructed paths and objects.

We first investigate how different LLMs affect subgoal generation and, consequently, agent training. To this end, we evaluate three open-source models (Llama3-70b [32], Qwen1.5-72b-chat [33], and DeepSeek-coder-33b [34]), using zero-shot chain-of-thought reasoning [30,49,50] to generate position-, representation- and language-based subgoals (for details on prompt design, see Appendix A). For language-based subgoals, embeddings are generated using all-MiniLM-L6-v2 [51]. We also examine the effect of reward balance on agent performance, as it can influence the agent’s ability to prioritize certain tasks over others. Finally, we conduct an ablation study to evaluate the impact of each subgoal component, specifically the subgoal representation in the agent’s observation and the associated reward, on overall performance.

4.1. Algorithm and Model Architecture

We train agents using the PPO [52] implementation by [53] with an actor–critic architecture tailored to the grid-based observation space. The model begins with three convolutional layers, each comprising 32 filters, a $3\times 3$ kernel, a stride of $2\times 2$, and padding of 1, followed by ELU activations to extract spatial features. The resulting 32-dimensional representation is mapped through a shared, fully connected layer to a 256-dimensional latent vector. From this shared representation, two separate fully connected networks parameterize the policy (actor) and the value function (critic).

4.2. Hyperparameter Configuration

Table 2. Hyperparameter values used for PPO in our experiments.

Hyperparameter	Description	Value
$\alpha$	Subgoal magnitude	0.2
$\gamma$	Discount factor	0.99
${\lambda }_{\mathrm{GAE}}$	Generalized Advantage Estimation (GAE) factor	0.95
$\mathrm{PPO}\mathrm{rollout}\mathrm{length}$	Number of steps to collect in each rollout per worker	256
$\mathrm{PPO}\mathrm{epochs}$	Number of epochs for each PPO update	4
$\mathrm{Adam}\mathrm{learning}\mathrm{rate}$	Learning rate for the Adam optimizer	0.0001
$\mathrm{Adam}ϵ$	Epsilon parameter for numerical stability	1 × ${10}^{-8}$
$\mathrm{PPO}\max \mathrm{gradient}\mathrm{norm}$	Maximum gradient norm for clipping	0.5
$\mathrm{PPO}\mathrm{clip}\mathrm{value}$	Clip value for PPO policy updates	0.2
$\mathrm{Value}\mathrm{loss}\mathrm{coefficient}$	Coefficient for the value function loss	0.5
$\mathrm{Entropy}\mathrm{coefficient}$	Coefficient for the entropy term in the loss function	0.0005

summarizes the hyperparameters used in our PPO implementation. These values are based on extensive tuning for MiniGrid tasks [53], ensuring optimal performance across diverse environments.

4.3. Metrics

We evaluate our model’s performance using distinct metrics for LLM-based subgoal generation and policy execution. On one hand, to assess the accuracy of subgoal prediction, we consider the following four metrics:

• Accuracy: The ratio of correct subgoal predictions to total predicted subgoals:

  $$\mathrm{Accuracy}={\displaystyle \frac{\#\mathrm{correct}\mathrm{subgoal}\mathrm{predictions}}{\#\mathrm{predicted}\mathrm{subgoals}}}$$

  (1)
• Correct SGs/Total: The number of correct subgoals identified by the LLM compared to the total number of subgoals.
• Correct EPs/Total: The number of complete episodes where all subgoals were correctly predicted.
• Manhattan distance: Average distance between predicted and actual subgoal positions:

  $$\mathrm{Manhattan}\mathrm{distance}={\displaystyle \frac{1}{M}}\sum _{i=1}^M\left|x_i^{\mathrm{pred}}-x_i^{\mathrm{true}}\right|+\left|y_i^{\mathrm{pred}}-y_i^{\mathrm{true}}\right|$$

  (2)

  with standard deviation:

  $${\sigma }_{\mathrm{Manhattan}}=\sqrt{{\displaystyle \frac{1}{M}}\sum _{i=1}^M{\left(\left|x_i^{\mathrm{pred}}-x_i^{\mathrm{true}}\right|+\left|y_i^{\mathrm{pred}}-y_i^{\mathrm{true}}\right|-\mathrm{mean}\right)}^2}$$

  (3)

A position-based subgoal is correct if the predicted position matches the object’s true location; otherwise, error is measured by Manhattan distance. Representation-based subgoals are correct if the object (and color, if relevant) is accurately described. Metric computation, including data cleaning and validation, is detailed in Appendix C.

For position-based subgoals, training errors are simulated by sampling alternative grid points, including infeasible ones, to reflect LLM inaccuracies (Figure 2—right). For representation- and language-based subgoals, training errors are approximated by selecting incorrect objects to mimic typical LLM mistakes (Figure 2—left).

With respect to the policy trained via RL with LLM subgoal proposals, we measure the expected return over the following episode:

$$G_t=\mathbb{E}\left[\sum _{k=0}^{\infty }{\gamma }^k{r}_{t+k}\right]=\mathbb{E}\left[\sum _{k=0}^{\infty }{\gamma }^k(r_{t+k}^e+\alpha ·r_{t+k}^g)\right].$$

(4)

5. Results

In this section, we present our results based on the experimentation introduced in Section 4. We begin by analyzing the performance of the surrogate LLM (Section 5.1), then evaluate RL training outcomes under this framework (Section 5.2), and conclude with an ablation study that isolates the contributions of individual subgoal components (Section 5.3).

5.1. Surrogate LLM Evaluation

We begin our experimental analysis by conducting a comprehensive evaluation by querying the aforementioned three LLMs across $M=1000$ distinct levels in each of the six environments under consideration. These levels are sampled uniformly at random from the full environment-level distribution. Prior work [4,47] has demonstrated that $M=1000$ levels are sufficient to capture the variability of the environment.

As summarized in

Table 3. Performance comparison of Llama, DeepSeek, and Qwen based on position (white background) and semantic (gray background) accuracy across six environments, in terms of accuracy; the number of correct subgoals (SGs) out of the total SGs; the number of correct episodes (EPs) out of the total EPs; and the average Manhattan distance with standard deviation. The model with the highest average performance in each environment is highlighted in bold.

Environment	LLM	Accuracy	Correct SGs /Total	Correct EPs /Total	Manhattan Distance
MultiRoomN2S4 (Position)	Llama	0.9210	1842/2000	859/1000	0.26 ± 1.01
	DeepSeek	0.3310	662/2000	125/1000	2.91 ± 2.54
	Qwen	0.7135	1427/2000	581/1000	1.12 ± 2.05
MultiRoomN2S4 (Semantic)	Llama	0.9295	1859/2000	871/1000	-
	DeepSeek	0.2570	514/2000	21/1000	-
	Qwen	0.5585	1117/2000	286/1000	-
MultiRoomN4S5 (Position)	Llama	0.5397	2159/4000	179/1000	3.69 ± 4.65
	DeepSeek	0.1200	480/4000	3/1000	7.92 ± 4.64
	Qwen	0.3600	1440/4000	56/1000	5.23 ± 4.94
MultiRoomN4S5 (Semantic)	Llama	0.5480	2192/4000	166/1000	-
	DeepSeek	0.1625	650/4000	7/1000	-
	Qwen	0.3517	1407/4000	38/1000	-
KeyCorridorS3R3 (Position)	Llama	0.9743	2923/3000	962/1000	0.18 ± 1.20
	DeepSeek	0.3337	1001/3000	18/1000	2.57 ± 2.54
	Qwen	0.9810	2943/3000	958/1000	0.07 ± 0.63
KeyCorridorS3R3 (Semantic)	Llama	0.9847	2954/3000	963/1000	-
	DeepSeek	0.9063	2719/3000	770/1000	-
	Qwen	0.9957	2987/3000	993/1000	-
KeyCorridorS6R3 (Position)	Llama	0.9137	2741/3000	802/1000	1.47 ± 5.02
	DeepSeek	0.3650	1095/3000	44/1000	7.24 ± 7.47
	Qwen	0.8810	2643/3000	732/1000	1.34 ± 4.25
KeyCorridorS6R3 (Semantic)	Llama	0.9323	2797/3000	806/1000	-
	DeepSeek	0.7617	2285/3000	536/1000	-
	Qwen	0.9587	2876/3000	895/1000	-
ObstructedMaze1Dlh (Position)	Llama	0.9870	2961/3000	984/1000	0.08 ± 0.82
	DeepSeek	0.4300	1290/3000	155/1000	2.98 ± 3.41
	Qwen	0.6687	2006/3000	486/1000	2.18 ± 3.53
ObstructedMaze1Dlh (Semantic)	Llama	0.9877	2963/3000	986/1000	-
	DeepSeek	0.7390	2217/3000	452/1000	-
	Qwen	0.7417	2225/3000	561/1000	-
ObstructedMaze1Dlhb (Position)	Llama	0.4485	1794/4000	0/1000	4.33 ± 4.21
	DeepSeek	0.2288	915/4000	2/1000	4.49 ± 3.39
	Qwen	0.4050	1620/4000	1/1000	3.85 ± 3.71
ObstructedMaze1Dlhb (Semantic)	Llama	0.4858	1943/4000	0/1000	-
	DeepSeek	0.5182	2073/4000	14/1000	-
	Qwen	0.4820	1928/4000	1/1000	-

, Llama consistently outperforms the other models, achieving the highest accuracy in five out of six position-based tasks and three out of six semantic-based tasks. On average, Llama’s accuracy in position-based tasks exceeds Qwen’s by approximately 10% and DeepSeek’s by roughly 50%. For semantic-based tasks, Llama maintains a similar advantage, outperforming Qwen by about 10% and DeepSeek by around 25%. Across all models, performance tends to be higher on semantic tasks than on position-based tasks, with DeepSeek showing the largest discrepancy, losing nearly 25% accuracy on position tasks. Additionally, accuracy generally decreases in larger, more complex environments, but Llama’s performance declines more gradually, indicating greater robustness and scalability.

5.2. Policy Training Evaluation

Building on the surrogate LLM, we next evaluate the performance of agents trained using these subgoal proposals across the six MiniGrid environments.

As shown in Figure 3, Llama consistently outperforms the other models across most environments. Agents trained with Llama-generated subgoals reach task completion significantly faster, achieving reductions in required training steps of up to $\stackrel{\sim }{4}$5% relative to DeepSeek in position-based subgoals. On the other hand, in representation-based and language-based subgoals, Llama increases the sample efficiency by 20%. Moreover, in most of the considered tasks, Llama also surpasses Qwen by 10–30% sample efficiency, depending on the environment and subgoal type. The gains are even more pronounced compared to LAMIGO and AMIGO: Llama required 90–99% fewer training steps to reach equivalent performance. This extreme difference occurs because in several environments, AMIGO and LAMIGO do not achieve task completion even after 1,000,000,000 steps, which highlights the improved learning when following the subgoal proposed by the LLMs. These results align with the findings in Table 3; higher subgoal proposal accuracy translates into more sample-efficient agent training.

Although DeepSeek generally exhibits the lowest subgoal accuracy across environments, as shown in Table 3, an intriguing exception arises in KeyCorridorS6R3 for representation- and language-based subgoals. Based on the data, DeepSeek does not always select the optimal subgoal, but in KeyCorridorS6R3, the environment’s larger size means that incorrect subgoals often lie closer to the agent’s current position. We hypothesize that this behavior can actually benefit learning: by offering intermediate waypoints that are easier for the agent to reach, DeepSeek effectively facilitates early-stage subgoal-following behavior. Furthermore, this mechanism is moderated by the fact that DeepSeek still proposes optimal subgoals with a non-negligible probability (approximately 75% for this environment and subgoal types, see Table 3), creating a balance between easier-to-reach subgoals and task-aligned subgoals. This combination may explain why, despite overall lower subgoal accuracy, DeepSeek achieves the highest task performance in these specific cases. Experimental results supporting this hypothesis are detailed in Appendix E.

To assess whether the observed differences in accuracy across LLMs reported in Table 3 are statistically significant, we conduct pairwise comparisons using a Wilcoxon signed-rank test. This non-parametric test evaluates whether two related samples differ significantly in their distributions, making it suitable for comparing algorithmic performance across multiple environments. Given the small number of environments considered (six), we report exact p-values rather than relying on asymptotic approximations. Additionally, we do not apply any correction for multiple comparisons, as the limited sample size would render such corrections overly conservative and potentially obscure meaningful differences.

The results of this test are summarized in

Table 4. Wilcoxon signed-rank test between the accuracy scores reported in Table 3 for every pair of LLMs.

LLM 1 AvgRank (pos/Semantic)	LLM 2 AvgRank (pos/Semantic)	Exact p-Value (Position)	Exact p-Value (Semantic)
Llama (1.33/1.83)	DeepSeek (2.66/2.00)	0.0312	0.0625
Llama (1.33/1.83)	Qwen (2.00/2.16)	0.0625	0.3125
DeepSeek (2.66/2.00)	Qwen (2.00/2.16)	0.0312	0.0938

for both position-based and semantic encoding strategies. Each row compares two algorithms, reporting their average ranks (lower values indicate better performance) and the exact p-value of the test. A p-value below a significance value of 0.05 suggests a statistically meaningful difference in performance. The average ranks provide a useful summary of overall performance across environments, while the p-values quantify the confidence in these differences.

As can be concluded from this table, Llama significantly outperforms DeepSeek in position-based subgoals ($p=0.0312$), and shows a trend toward outperforming Qwen ($p=0.0625$). Qwen significantly outperforms DeepSeek in position-based subgoals ($p=0.0312$). For semantic-based subgoals, none of the comparisons reach statistical significance, although Llama tends to rank better on average. These findings support the conclusion that Llama generally performs best, especially in position-based subgoal tasks, and that DeepSeek consistently underperforms relative to the other models.

5.2.1. LLMs’ Lack of Spatial Reasoning

A key insight from Table 3 results is that LLMs excel at tasks involving logical sequences (e.g., “pick up key → unlock door → retrieve ball”) but struggle with tasks requiring a deep understanding of spatial hierarchies and topological dependencies among subgoals. In the MultiRoom suite, the agent must traverse a series of interconnected rooms, where access to later rooms depends on resolving spatial relationships and progression constraints. According to our experiment (Table 3), LLMs seem to lack these capabilities, as they do not seem to infer geometric or topological relationships from static state descriptions. Nevertheless, as shown in Figure 3, agent training in environments requiring spatial reasoning appears to succeed despite these limitations. We hypothesize that training succeeds in these environments because their small size limits the impact of occasional subgoal errors. Moreover, subgoals almost always correspond to doors, the only interactive objects, so the task reduces to repeatedly opening doors to progress. This simplicity ensures that even imperfect subgoal proposals provide a sufficiently useful signal, allowing the agent to learn effective behavior despite the LLMs’ limited spatial reasoning.

5.2.2. Analysis of Subgoal Strategies

Table 3 compares position- and semantic-based subgoal strategies across Llama, DeepSeek, and Qwen. Overall, semantic subgoals provide better accuracies for DeepSeek (+4.2% to +57%) and moderate accuracy gains for Llama (+0.07% to +1.86%) and Qwen (+7% to +8%).

To better understand how differences in accuracy translate into learning, we analyze the training performance of position- and semantic-based (representation- and language-based) subgoals. As shown in Figure 3, the effectiveness of subgoal types depends on the reward horizon, defined as the distance from the agent’s starting position to the first subgoal and between subsequent subgoals. In smaller environments like MultiRoomN2S4 and N4S5, where targets lie within or near the agent’s $7\times 7$ observation window, semantic subgoals improve over position by 6–48%, enabling direct navigation using local information. By contrast, in long-horizon environments such as KeyCorridorS6R3, semantic subgoals achieve higher accuracy (Table 3) but are less effective during training due to limited observations. Position-based subgoals provide explicit stepwise guidance, improving navigation and sample efficiency—requiring 40% fewer samples than language-based and 7% fewer than representation-based subgoals, with Llama showing 84% and 27% gains, respectively. In the case of ObstructedMaze1Dlhb, representation-based subgoals outperform position- and language-based subgoals, while language performs worst (Figure 3). We hypothesize this is because overall accuracy in this environment is low (Table 3) and language-based subgoals, being higher-dimensional, make training more difficult for the agent.

The difference between the different subgoal strategies highlights a key trade-off: for local guidance, when subgoals are close or visible within the agent’s observation window, both representation- and language-based subgoals are highly effective. Representation-based subgoals allow the agent to immediately recognize and act upon the target object, accelerating task completion. Similarly, although more complex, language-based subgoals perform almost as well as representation-based ones and provide the added benefit of abstract, human-readable guidance. On the other hand, for global guidance, when subgoals are distant, position-based subgoals become more effective. By providing relative position information, these subgoals help the agent navigate toward far-off targets, even in situations where the agent cannot directly perceive the subgoal in its field of view.

Despite these advantages, all subgoal types face challenges under partial observability. While position-based subgoals mitigate long-horizon navigation, they risk leading agents into obstacles (e.g., walls) due to static instructions. Representation-based subgoals, on the other hand, fail to guide agents when targets are outside their perceptual field. In such cases, the agent must rely on its exploratory behavior and may reach the intended subgoal just by serendipity. Similarly, language-based subgoals share limitations with representation-based subgoals. While they offer an abstract form of guidance, they can still struggle when the agent cannot directly perceive the target object or when the subgoal requires more complex spatial reasoning beyond the provided language instructions.

5.2.3. Sensitivity Analysis of Subgoal Reward Balance

The balance between subgoal rewards and the overall task reward is critical for effective training. We evaluate this balance through the reward coefficient $\alpha$, which scales the subgoal reward $r_t^g$ relative to the extrinsic task reward $r_t^e$. Our experiments (Figure 4) reveal that a low $\alpha$ value (e.g., $\alpha =0.2$) strikes an optimal balance, ensuring that subgoals guide the agent without overshadowing the primary task objective.

The effect of $\alpha$ varies substantially across task suites. In the MultiRoom environments, the task is simple—traverse a few rooms and open doors—so even with low subgoal accuracy, a high $\alpha$ does not severely hinder training. Nevertheless, as shown in Figure 4, Llama tends to struggle with higher alpha values. On the other hand, in ObstructedMaze environments, where tasks involve more intricate dependencies, high $\alpha$ values amplify the impact of incorrect subgoals. This issue is evident in ObstructedMaze1Dlhb, where Llama achieves 44–48% accuracy and DeepSeek 22–52% (Table 3), making high $\alpha$ values impractical. By contrast, in the KeyCorridor suite, subgoal accuracy is much higher—above 90% for most models except DeepSeek (Table 3)—which allows the agent to benefit from larger $\alpha$ values, despite reduced sample efficiency in low alpha values (about 40% lower in KeyCorridorS6R3 with $\alpha =0.2$). Taken together, our experiments indicate that setting $\alpha =0.2$ provides the best balance across environments, offering sufficient guidance from subgoals without allowing subgoal errors to dominate learning, and ensuring consistent task performance even in more complex or error-prone settings.

5.3. Ablation Study: Impact of Subgoals and Rewards

To further understand the role of subgoals and rewards in agent training, we conduct an ablation study across all evaluated LLMs using representation-based subgoals (representation-based subgoals were chosen, as they demonstrated the most consistent performance across environments). We analyze two specific training conditions:

• No Reward: In this setup, agents receive representation-based subgoals in their observation, but no supplementary rewards for achieving them. Results show that agents fail to learn in environments with long reward horizons, namely, MultiRoomN4S5, KeyCorridorS3R3, KeyCorridorS6R3, and ObstructedMaze1Dlhb (Figure 5, first and third columns). This highlights the critical role of rewards in guiding exploration and reinforcing task completion, particularly in complex scenarios where subgoals alone are insufficient to bridge the reward gap. However, in environments with shorter reward horizons, such as MultiRoomN2S4 and ObstructedMaze1Dlh, agents successfully learn the task even without supplementary rewards, as the reduced complexity allows subgoals to provide sufficient guidance.
• No Subgoal: Agents receive supplementary rewards for subgoal completion but do not have access to subgoals in their observation space. Agents still perform well in this condition, achieving convergence comparable to setups where subgoals are explicitly provided (compare Figure 5, which shows training without subgoals, to Figure 3, which shows training with subgoals). For example, in ObstructedMaze1Dlh, access to subgoals leads to substantially faster convergence, with Qwen and Llama improving by roughly 20% and DeepSeek by about 12.5%. A similar trend is observed in KeyCorridorS6R3, where all models converge approximately 15% faster when subgoals are available. However, once the underlying policy has been fully acquired, subgoals in the observation space are no longer necessary for successful task completion, suggesting that incorporating subgoals into the agent’s observation space primarily serves to accelerate training.

These findings reveal that subgoals significantly reduce training time by providing intermediate milestones, particularly in environments with sparse rewards or long horizons. More importantly, supplementary rewards are essential for effective learning, especially in complex tasks where subgoals alone cannot bridge the reward gap. As a consequence, an important observation is that agents can be decoupled from subgoals once training is complete, reducing computational overhead and operational costs during deployment.

6. Conclusions and Future Work

This work has shown that LLMs can be effectively leveraged as subgoal generators to help RL agents learn in sparse environments, without requiring any fine-tuning of the language models. By providing structured guidance, LLMs help agents overcome reward sparsity and improve learning efficiency, particularly in long-horizon tasks.

We have conducted an extensive analysis of three different LLMs, evaluating their effectiveness in generating useful subgoals for RL agents. Our results demonstrate superior performance over recent teacher–student baselines within the MiniGrid benchmark. Additionally, we have explored three distinct subgoal-generation strategies, each tailored to different task requirements.

Furthermore, we have shown that our method can override the need for an LLM in deployment when the subgoal is not considered in the agent’s observation space. That is, we use the LLM to accelerate the training while maintaining scalability for real-world applications, where querying an LLM at inference time may not always be feasible. In addition, by not relying on perfect guidance, this approach improves the agent’s robustness and generalization, particularly when the quality of subgoals is degraded, which is critical in real-world scenarios such as robotic exploration in novel environments, content recommendation with shifting user preferences, or vehicular traffic management under varying inflow rate [54].

Future Work

We aim to extend our framework to a broader set of benchmarks. While our current results demonstrate the benefits of LLM-generated subgoals, testing on a wider variety of domains would provide deeper insights into their general applicability. Additionally, incorporating real-world tasks, like robotic assemblies or autonomous drone flights, could further validate the robustness of our approach.

Another promising direction is to make the proposed framework more adaptive to the agent’s knowledge progression. This can be approached in two ways. First, by dynamically adjusting the complexity of subgoals: breaking tasks into smaller, manageable steps and progressively increasing their difficulty as the agent improves. Second, by training agents to follow LLM-generated subgoals more closely, either by applying higher weighting factors (e.g., $\alpha \approx 1$) or by using dynamically adjusted weighting factors. These strategies could enhance the agent’s ability to generalize to entirely new environments. Consequently, the following key question arises: can an agent leverage LLM guidance to solve novel tasks in completely unfamiliar settings? Investigating this question could provide insights into how well LLM-driven subgoal generation supports cross-domain transfer and rapid adaptation in open-world settings.

Finally, our current framework includes an LLM surrogate component, which serves as an approximation of the reasoning capabilities of large models. While this surrogate approach is already effective, refining it could lead to even better subgoal quality and alignment with the agent’s capabilities.