Deep Reinforcement Learning in the Era of Foundation Models: A Survey

Abstract

Deep reinforcement learning (DRL) and large foundation models (FMs) have reshaped modern artificial intelligence (AI) by enabling systems that learn from interaction while leveraging broad generalization and multimodal reasoning capabilities. This survey examines the growing convergence of these paradigms and reviews how reinforcement learning from human feedback (RLHF), reinforcement learning from AI feedback (RLAIF), world-model pretraining, and preference-based optimization refine foundation model capabilities. We organize existing work into a taxonomy of model-centric, RL-centric, and hybrid DRL–FM integration pathways, and synthesize applications across language and multimodal agents, autonomous control, scientific discovery, and societal and ethical alignment. We also identify technical, behavioral, and governance challenges that hinder scalable and reliable DRL–FM integration, and outline emerging research directions that suggest how reinforcement-driven adaptation may shape the next generation of intelligent systems. This review provides researchers and practitioners with a structured overview of the current state and future trajectory of DRL in the era of foundation models.

1. Introduction

Deep reinforcement learning (DRL) has become a core part of modern artificial intelligence (AI), combining the representational power of deep learning with the sequential decision-making capabilities of reinforcement learning (RL) [1,2,3]. Over the past decade, DRL has enabled significant advances in game playing [4], robotics [5], autonomous navigation [6], and simulation-driven control [7]. In these systems, agents learn through interaction with their environment, optimizing long-horizon behavior under uncertainty using reward-guided objectives. Classic RL methods, including Q-learning and policy gradients, have historically struggled with high-dimensional state spaces. DRL addresses this limitation by incorporating deep neural networks into value functions, policies, and world models, enabling effective representation learning from raw sensory inputs. Milestones include Deep Q-Networks (DQN) [8], which achieved human-level Atari performance, and model-based agents such as Dreamer [9], which learn latent dynamics for long-horizon planning.

Meanwhile, foundation models (FMs) have reshaped generative AI, with large language models (LLMs) such as GPT and Claude demonstrating robust performance, emergent reasoning capabilities, and multimodal understanding [10,11]. Foundation models, although developed primarily for supervised and self-supervised learning, incorporate reinforcement mechanisms during their refinement. The most prominent example is reinforcement learning from human feedback (RLHF), which played a crucial role in aligning models such as instruction-tuned LLMs [12]. Reinforcement learning from AI feedback (RLAIF) extends this paradigm by using strong teacher models to generate preference judgments. Beyond alignment, recent work uses DRL to support tool use, interactive decision-making, and embodied action in FM-driven agents, illustrating a deepening connection between the two research areas.

Despite rapid progress, the literature at the intersection of DRL and foundation models remains fragmented: alignment-focused reviews tend to emphasize feedback pipelines and preference optimization, while DRL-focused reviews emphasize learning dynamics, stability, and generalization in control settings. As a result, it is often unclear how pretrained priors, feedback sources, and reinforcement signals jointly shape agent behavior when foundation models are placed inside closed-loop decision-making systems [13,14,15,16,17,18,19].

This fragmentation leaves gaps in the current literature. To the best of our knowledge, no prior survey provides a consolidated synthesis of how DRL and foundation models interact across alignment, tool-augmented behavior, multimodal reasoning, embodied control, and scientific discovery—domains where reinforcement-driven adaptation is becoming central to model reliability, safety, and real-world applications. In order to fill these gaps, this survey offers a comprehensive and integrated examination of DRL in the era of foundation models. This review is timely given the accelerating use of reinforcement mechanisms in FM training and the increasing deployment of FM-driven systems in real-world settings. The key contributions are as follows:

• A unified taxonomy of DRL–FM integration, distinguishing model-centric, RL-centric, and hybrid paradigms, and clarifying their implications for representation, optimisation, and interaction.
• A detailed synthesis of application domains including language and multimodal agents, autonomous and robotic control, scientific discovery, and societal and ethical alignment.
• A unified evaluation and benchmarking contribution that consolidates metrics, protocols, and stress-tests for comparing DRL–FM systems across tasks, safety, robustness, and efficiency dimensions.
• An in-depth analysis of the challenges that limit scalable and safe DRL–FM integration, alongside forward-looking research opportunities with emphasis on stability, interpretability, and trustworthy DRL-driven FM systems.

The rest of the paper is organized as follows: Section 2 summarizes related surveys. Section 3 provides some background on DRL and FM, and Section 4 presents a taxonomy of DRL–FM integration. Section 5 reviews technical foundations of DRL in FM contexts. Section 6 examines key applications. Section 7 discusses open challenges and future directions, which Section 8 concludes the study.

Review Methodology

This survey follows a structured and systematic review protocol to ensure comprehensive coverage and reproducibility. The Scopus, IEEE Xplore, SpringerLink, arXiv, and ACM Digital Library databases were queried between January 2020 and March 2025 using keywords such as “deep reinforcement learning”, “foundation models”, “RLHF”, “alignment”, and “autonomous agents”. An initial set of 612 studies was retrieved. After duplicate removal and title–abstract screening, 184 papers were shortlisted for full-text review. Exclusion criteria included works without explicit DRL components, alignment discussions unrelated to foundation models, or redundant workshop versions.

After full-text screening, 28 were excluded, leaving 156. The final corpus comprised 126 primary studies, 18 review papers, and 12 benchmark proposals. Each article was annotated by research focus (model-centric, RL-centric, or hybrid), methodological contribution (training, optimization, or evaluation), and application domain (language/multimodal, control, scientific, or societal). This coding schema enabled cross-sectional comparison across integration paradigms and performance criteria.

Bibliometric analysis was performed to capture publication trends, author contributions, and venue distribution. Citation data were normalized by year to control for recency bias. Qualitative synthesis focused on identifying conceptual convergence between reinforcement learning and foundation model alignment, while quantitative evidence was summarized using citation frequencies and methodological clusters. The resulting taxonomy and discussion sections are grounded in these systematically collected insights.

2. Related Reviews

The research landscape on reinforcement learning and foundation models has expanded rapidly, producing several surveys that address related but distinct themes. For example, Kaufmann et al. [13] conducted one of the most comprehensive reviews of RLHF, covering the full pipeline from preference data collection to reward modeling and policy optimization. Their work identified key limitations in human supervision scalability and emphasized the necessity of automated or hybrid feedback systems. In a related study, Wang et al. [14] presented a detailed taxonomy of large model alignment techniques, including RLHF and RLAIF. They discussed how divergence constraints, reward modeling strategies, and reference policies influence alignment stability. Zhou et al. [15] extended this discussion by focusing on alignment for large language model (LLM) agents, emphasizing preference-driven optimization, social alignment, and decision-making under uncertainty. These reviews clarify the mechanics of alignment but devote limited attention to the role of DRL as a theoretical and algorithmic foundation underlying feedback-based optimization.

Beyond alignment, Plaat et al. [16] surveyed the emerging paradigm of agentic large language models that reason, act, and interact across open environments. They categorized research into reasoning, tool-augmented behavior, and multi-agent coordination, highlighting progress toward autonomous systems capable of planning and execution. While their analysis captures the capabilities of modern foundation models, it pays little attention to reinforcement-driven adaptation, sample efficiency, and policy optimization, which are core aspects of DRL that enable these agentic behaviors to scale reliably.

From the DRL perspective, several foundational reviews have provided valuable insights but primarily focus on algorithmic aspects independent of foundation models. Landers and Doryab [19] surveyed verification methods for DRL, proposing taxonomies for robustness evaluation and formal safety guarantees. Their contribution complements alignment research by introducing verification standards but does not explore FM-driven policy learning or preference-based fine-tuning. Similarly, Moerland et al. [17] offered an exhaustive overview of model-based reinforcement learning, detailing how learned dynamics and uncertainty-aware planning bridge learning and control. While this work establishes the foundation for modern world-model agents, it predates the tight coupling between DRL and FMs that now enables reasoning across multimodal representations.

Offline and data-efficient learning have also been the subject of active review. Prudencio et al. [20] and Levine et al. [18] surveyed offline reinforcement learning, emphasizing conservative policy estimation, distributional robustness, and dataset quality—concepts increasingly relevant when optimizing foundation models using static preference datasets or synthetic feedback. In robotics, Tang et al. [21] summarized the application of DRL to real-world control, including sim-to-real transfer, hierarchical policies, and safety-aware exploration. Their findings demonstrate the practical viability of DRL systems and acknowledge the growing influence of pretrained representations; however, the review remains narrowly focused on robotics rather than general-purpose foundation model integration.

Complementary domains have examined interpretability and explainability. Puiutta and Veith [22] provided a systematic survey of explainable reinforcement learning (XRL), classifying interpretation methods and evaluation protocols for analyzing learned policies. Their work informs how transparency can be embedded within reinforcement-driven systems but does not address the challenges of aligning foundation models using reinforcement objectives.

Overall, these surveys (summarized in

Table 1. Summary of related reviews and surveys on DRL, foundation models, and alignment.

Reference	Year	Description
Kaufmann et al. [13]	2024	Comprehensive analysis of RLHF pipelines, covering preference data, reward modeling, and policy optimization, with emphasis on supervision scalability.
Wang et al. [14]	2024	Taxonomy of alignment techniques for large models, including RLHF, RLAIF, and DPO, focusing on optimization stability and divergence controls.
Zhou et al. [15]	2025	Review of alignment and decision-making in LLM agents, examining preference optimization, safety, and interactive behaviour.
Plaat et al. [16]	2025	Analysis of agentic LLMs with emphasis on reasoning, tool use, and multi-agent interaction; limited coverage of reinforcement-based adaptation.
Landers and Doryab [19]	2023	Overview of verification and safety evaluation methods for DRL, highlighting robustness concerns but without FM-specific integration.
Moerland et al. [17]	2023	Extensive review of model-based reinforcement learning, detailing learned dynamics and planning but not foundation-model coupling.
Levine et al. [18]	2020	Foundational survey of offline reinforcement learning and conservative policy learning, relevant to preference datasets used in FM alignment.
Prudencio et al. [20]	2023	Examination of offline RL methods, focusing on distribution shift, robustness, and data constraints.
Tang et al. [21]	2025	Survey of DRL in robotics, including sim-to-real transfer and safety-aware exploration; coverage of pretrained representations remains limited.
Puiutta and Veith [22]	2020	Systematic review of explainable reinforcement learning, addressing transparency but not DRL–FM integration.
This survey	2025	Unified synthesis of DRL–FM integration covering FM-centric DRL architectures, RL-centric alignment, and hybrid multimodal agents, coupled with a unified evaluation framework that specifies shared metric groups, protocol templates, and minimum reporting artifacts for cross-domain comparability and auditing.

) form a broad yet disjointed picture: alignment-focused works often overlook DRL’s algorithmic role, while DRL-focused surveys rarely engage with foundation model alignment or multimodal reasoning. Motivated by this disconnect, this survey focuses on the coupling mechanisms that link pretrained foundations with reinforcement-driven adaptation in practice: how representations, planners/policies, world models, and feedback signals are composed inside the reinforcement loop. This lens is then used to organize the remainder of the paper, connecting technical foundations to application patterns and the recurring challenges that limit scalable and trustworthy DRL–FM systems.

3. Preliminaries

3.1. Reinforcement Learning Fundamentals

Reinforcement learning formalizes sequential decision making as a Markov decision process (MDP) $\mathcal{M}=(𝒮,\mathcal{A},P,R,\gamma )$ with states $s\in 𝒮$, actions $a\in \mathcal{A}$, transition kernel $P\left(s^{\prime }\right|s,a)$, reward $R(s,a)$, and discount $\gamma \in [0,1)$. An agent seeks a policy $\pi \left(a\right|s)$ that maximizes the expected discounted return [23]

$$J\left(\pi \right)={\mathbb{E}}_{\pi }\left[\sum _{t=0}^{\infty }{\gamma }^t{r}_t\right].$$

(1)

The state and action value functions under $\pi$ are

$$V^{\pi }\left(s\right)={\mathbb{E}}_{\pi }\left[\sum _{t=0}^{\infty }{\gamma }^t{r}_t|s_0=s\right],Q^{\pi }(s,a)={\mathbb{E}}_{\pi }\left[\sum _{t=0}^{\infty }{\gamma }^t{r}_t|s_0=s,a_0=a\right],$$

(2)

satisfying the Bellman expectation equations, with optimal counterparts $V^{*}$ and $Q^{*}$ defined via maximization over actions. Policy optimization follows the policy gradient theorem, where a differentiable policy ${\pi }_{\theta }\left(a\right|s)$ is updated in the direction [24,25]

$${\nabla }_{\theta }J\left({\pi }_{\theta }\right)={\mathbb{E}}_{(s,a)\sim {\pi }_{\theta }}\left[{\nabla }_{\theta }log{\pi }_{\theta }\left(a\right|s)Q^{{\pi }_{\theta }}(s,a)\right],$$

(3)

often estimated with an advantage baseline $A^{\pi }(s,a)=Q^{\pi }(s,a)-V^{\pi }\left(s\right)$ to reduce variance. Practical trust-region ideas constrain policy drift between updates; for instance, proximal policy optimization (PPO) maximizes a clipped surrogate with ratio $r_{\theta }={\pi }_{\theta }\left(a\right|s)/{\pi }_{{\theta }_{\mathrm{old}}}\left(a\right|s)$ and an entropy bonus to encourage exploration [26,27].

3.2. Deep Reinforcement Learning

DRL extends classical reinforcement learning by incorporating deep neural networks as function approximators for policies, value functions, and environment models [1,28]. As shown in Figure 1, standard reinforcement learning consists of an agent interacting with an environment by receiving states, selecting actions, and optimizing expected rewards based on feedback. Deep neural networks (DNNs), on the other hand, provide powerful nonlinear mappings from inputs to outputs through multiple hidden layers, enabling the extraction of high-dimensional features and representations. DRL combines these paradigms by embedding a deep network within the agent to approximate optimal policies and value functions directly from raw, high-dimensional inputs such as pixels or sensor data, allowing end-to-end learning of perception and control. Recent work has also revisited the choice of function approximators used inside DRL agents, with potential implications for parameter efficiency and interpretability; we discuss this further in Section 4 [29,30].

Model-free DRL algorithms, including Asynchronous Advantage Actor–Critic (A3C), PPO, and Soft Actor–Critic (SAC), achieve a balance between stability and sample efficiency through on-policy and off-policy updates, entropy regularization, and parallelized learning [27,31,32]. Model-based variants learn predictive world models of environment dynamics and rewards, enabling planning and imagination-based rollouts before executing actions. MuZero exemplifies this paradigm by jointly learning latent dynamics, policy, and value functions, achieving superhuman performance in Atari, Go, chess, and shogi [33]. Similarly, world-model agents such as DreamerV3 leverage learned latent representations to simulate trajectories and optimize behavior through imagination, attaining state-of-the-art results across diverse continuous-control domains [9]. More recently, transformers have reframed control as sequence modeling—illustrated by the Decision Transformer—which predicts actions conditioned on prior states, actions, and desired returns, thereby eliminating the need for explicit value estimation while competitive with strong offline RL baselines on several benchmarks [34].

3.3. Foundation Models and Alignment

FMs are large pretrained architectures––language, vision–language, or multimodal––that can be adapted to diverse tasks through instruction tuning and alignment [35]. Reinforcement learning plays a central role in refining these models by optimizing their behavior according to evaluative feedback. In RLHF and related approaches, a pretrained model generates candidate outputs that are ranked by human annotators or teacher models. These preferences are used to train a reward model, which provides scalar feedback for fine-tuning the policy. Variants such as RLAIF and DPO modify this pipeline by incorporating synthetic evaluators or direct preference optimization objectives [13,14]. Alignment methods have also been extended to LLM-based agents, where reinforcement signals influence reasoning, tool use, and decision-making in interactive environments [15].

3.4. Evaluation Paradigms and Benchmarks

Assessing the capabilities of DRL and FMs requires benchmarks that capture not only performance and efficiency, but also reasoning, generalization, and alignment. Yet current evaluation paradigms remain largely siloed by domain, preventing direct comparison of agents that integrate perception, language, and control. The major benchmark categories relevant to DRL–FM systems are summarized below and tabulated in

Table 2. Summary of benchmark categories commonly used to evaluate DRL–FM systems, highlighting what each category measures and what it typically misses.

Category	Benchmarks	Primary Evaluation Focus	Typical Limitations for DRL–FM Evaluation
Classical DRL benchmarks	Atari 2600 [36]; MuJoCo [37]; ProcGen [38]	Policy learning performance; exploration; sample efficiency; robustness and generalization in controlled settings	Limited language and multimodal reasoning; weak coverage of alignment behavior and governance-oriented reporting
Embodied and multimodal testbeds	MineDojo [39]; Habitat [40]; BEHAVIOR [41]	Long-horizon tasks; grounding language in perception/action; embodied manipulation and navigation; compositional skills	Often task-centric; alignment assessment is indirect or inconsistent; scenario coverage and evaluation protocols vary widely
Foundation model benchmarks	BIG-Bench [42]; HELM [43]; GAIA [44]	Reasoning and language capabilities; calibration and bias; tool use; safety and social compliance	Limited closed-loop control and reinforcement feedback; weak representation of exploration and online adaptation

.

3.4.1. Classical Reinforcement Learning Benchmarks

Classical DRL evaluation is anchored in structured control environments designed to measure policy optimization, exploration, and generalization under controlled conditions. The Atari 2600 suite [36] remains a standard for testing visual perception, temporal credit assignment, and robustness to sparse or delayed rewards in discrete-action settings. In contrast, MuJoCo [37] targets continuous-control dynamics, emphasizing stable locomotion and manipulation, fine-grained motor coordination, and sample efficiency in physics-based simulation. To better probe generalization beyond fixed test sets, ProcGen [38] uses procedural generation to evaluate robustness and zero-shot generalization across unseen levels, highlighting overfitting risks that can be masked by static environments. While these benchmarks provide rigorous measures of learning performance and efficiency, they offer limited coverage of multimodal reasoning, language grounding, and alignment-oriented behaviors that become central when integrating DRL with foundation models.

3.4.2. Embodied and Multimodal Testbeds

Embodied and multimodal testbeds extend evaluation toward agents that must couple perception, language, and action over longer horizons. MineDojo [39] leverages the open-ended structure of Minecraft to test long-horizon exploration, tool use, and instruction following in environments that blend vision and language with complex action sequences. Simulation platforms such as Habitat [40] and BEHAVIOR [41] further emphasize realistic 3D interaction, where agents must interpret natural language instructions, navigate cluttered scenes, and manipulate semantically meaningful objects. Collectively, these environments probe multi-step reasoning, grounding, and coordination across modalities, while also exposing practical failure modes such as compounding errors and brittle perception-action coupling. However, evaluation is often tightly tied to task success and episode-level rewards, which limits systematic assessment of alignment-related properties across heterogeneous scenarios.

3.4.3. Foundation Model Benchmarks

Foundation model benchmarks primarily evaluate reasoning, language understanding, and alignment-related attributes, typically in settings that do not require closed-loop control. BIG-Bench [42] and HELM [43] span broad collections of linguistic and reasoning tasks and include measurement axes such as calibration, bias, and fairness, enabling comparative analysis of model behavior across diverse prompts and evaluation protocols. GAIA [44] extends this scope toward multi-step problem solving with tool use, including external API interaction and agentic task decomposition, thereby providing a closer proxy to real-world assistant behavior. Despite their usefulness for benchmarking reasoning quality and ethical performance, these suites only weakly represent reinforcement-driven adaptivity, exploration under uncertainty, and feedback loops that are central to DRL.

This separation complicates comparison of integrated DRL–FM agents across closed-loop performance, robustness, and alignment, motivating the unified evaluation framework introduced in Section 7.2.

4. A Taxonomy of DRL–Foundation Model Integration

This section proposes a taxonomy for how DRL and foundation models are coupled within the reinforcement loop. Rather than classifying systems by application domain, we distinguish integration patterns by where pretrained representations, planning, and feedback-driven adaptation enter the learning pipeline. This framing supports a consistent comparison of methods spanning language-conditioned control, multimodal robotics, and alignment-oriented fine-tuning.

4.1. Paradigms of Integration

The integration of DRL and FMs can be categorized into three primary paradigms based on how learning, perception, and reasoning are coupled within the reinforcement loop. These paradigms are: (i) FM-centric DRL architectures, where foundation models act as policy, planner, or world-model components; (ii) RL-centric foundation models, where DRL drives fine-tuning and alignment of large pretrained models; and (iii) hybrid or multimodal frameworks, where both paradigms are unified in interactive or embodied agents. Here, “FM-centric” denotes integration in which a pretrained foundation model is the primary policy/planner, in contrast with classical model-based RL, where the central object is an explicit environment dynamics model learned for planning. To reduce boundary ambiguity, we distinguish paradigms by (a) which module performs deployment-time action selection or planning, and (b) which parameters are primarily shaped by reward- or preference-driven objectives (i.e., whether the FM itself is the main object being optimized versus an attached controller). When the dominant “model” is a transition model trained primarily from scratch for planning, the setting aligns with classical model-based RL rather than FM-centric integration, even if pretrained encoders are used. The following subsections describe each paradigm in detail.

4.1.1. FM-Centric DRL Architectures

In this paradigm, the foundation model forms the central computational substrate for policy learning or planning, while reinforcement learning supplies task- and environment-specific adaptation around that pretrained backbone. In this paradigm, a pretrained foundation model is embedded directly into the reinforcement learning pipeline, often serving as the policy ${\pi }_{\theta }\left(a\right|s)$, planner, or environment model. Rather than learning policy parameters $\theta$ from scratch, the FM provides pretrained representations $f_{\varphi }\left(s\right)$ that encode semantic or multimodal information from the state space [45,46]. Because downstream control quality depends on the robustness of these learned representations under complex scenes and appearance variation, advances in robust feature learning for perception provide relevant grounding for FM-centric agent design [47]. The policy optimization objective then becomes

$$\underset{\theta }{max}J\left(\theta \right)={\mathbb{E}}_{{\pi }_{\theta }}\left[\sum _{t=0}^{\infty }{\gamma }^{t}r(s_t,a_t)\right],\mathrm{with}{a}_t\sim {\pi }_{\theta }\left(a_t\right|f_{\varphi }\left(s_t\right)).$$

(4)

Here, $f_{\varphi }$ represents the feature extractor or encoder inherited from the foundation model, and ${\pi }_{\theta }$ maps these embeddings to actions. This formulation allows the policy to benefit from the pretrained model’s generalization while still being fine-tuned for downstream control or decision tasks. A notable example is the Decision Transformer, which formulates RL as sequence modeling. The model predicts an action sequence $a_{1:T}$ conditioned on prior states $s_{1:T}$, rewards-to-go ${\widehat{R}}_t={\sum }_{k=t}^T{r}_k$, and actions, using an autoregressive transformer:

$$P\left(a_{1:T}\right|s_{1:T},{\widehat{R}}_{1:T})=\prod _{t=1}^{T}P\left(a_t\right|s_{\le t},a_{<t},{\widehat{R}}_{\le t};\theta ).$$

(5)

By leveraging pretrained transformer architectures, such methods transform the RL problem into a supervised learning task over trajectory datasets [34]. Other systems such as Gato [48], SayCan [49], PaLM-E [50], and RT-2 [51] further extend this concept by integrating visual, textual, and proprioceptive modalities into unified policies. These approaches treat the FM as a differentiable policy backbone that grounds language and perception into control signals, resulting in generalist agents capable of zero-shot task adaptation.

When foundation models are used as world-model components in this paradigm, they typically serve as pretrained generative priors or simulators that complement (or initialize) learned transitions, rather than mirroring the classical setting where the transition model is the primary object learned from scratch for planning. Accordingly, FM-centric approaches may incorporate pretrained models as differentiable world models $p_{\varphi }(s_{t+1},r_t|s_t,a_t)$, which enable simulation-based policy optimization. The learning objective in such cases is to minimize model prediction error while maximizing long-term returns under imagined trajectories:

$$\underset{\varphi }{min}\mathbb{E}\left[\parallel s_{t+1}-{\widehat{s}}_{t+1}{\parallel }^2\right],\underset{\theta }{max}{\mathbb{E}}_{p_{\varphi }}\left[\sum _{t=0}^{\infty }{\gamma }^t{r}_{\varphi }(s_t,a_t)\right].$$

(6)

This dual objective, used in systems like DreamerV3 [9], enables efficient planning and transfer learning by combining learned dynamics with the representational priors of foundation models.

4.1.2. RL-Centric Foundation Models

In RL-centric paradigms, reinforcement learning provides the optimization mechanism for fine-tuning and aligning foundation models. The model (policy) ${\pi }_{\theta }$ generates an output y given a prompt x, and a learned reward model $r_{\varphi }(x,y)$ provides feedback based on human or synthetic preferences. The objective function is expressed as

$$\underset{\theta }{max}{\mathbb{E}}_{y\sim {\pi }_{\theta }(·|x)}\left[r_{\varphi }(x,y)\right]-\beta \mathrm{KL}\left({\pi }_{\theta }(·|x)\parallel {\pi }_{\mathrm{ref}}(·|x)\right),$$

(7)

where ${\pi }_{\mathrm{ref}}$ is a reference model (often the pretrained base model) and $\beta$ regulates the degree of deviation. This general framework underlies methods such as RLHF, RLAIF, and DPO, which optimize large models for qualities like helpfulness, coherence, and safety [13,14]. Furthermore, in this paradigm, the foundation model itself is the primary object being updated by reinforcement objectives, rather than being embedded as a planner or controller within an external environment loop. While these methods originated in natural language processing, they extend naturally to multimodal systems, where feedback may include visual, auditory, or behavioral signals. In essence, DRL provides the mechanism through which pretrained models learn value alignment and behavioral consistency beyond pure likelihood optimization.

4.1.3. Hybrid and Multimodal Frameworks

Hybrid paradigms couple FM-centric decision-making with reinforcement-driven adaptation in interactive settings, combining pretrained reasoning or planning with environment-facing control and feedback. Hybrid paradigms unify model-centric and RL-centric elements by coupling pretrained representation learning with reinforcement-based adaptation and feedback. These frameworks operate across multiple modalities—text, vision, and action—enabling agents that can perceive, reason, and act in real-world settings. Systems such as Voyager [52] exemplify this approach: foundation models provide high-level reasoning and planning, while DRL components handle continuous interaction, feedback, and environment adaptation. The composite objective typically combines supervised pretraining loss ${\mathcal{L}}_{\sup }$ with reinforcement objectives ${\mathcal{L}}_{\mathrm{RL}}$:

$${\mathcal{L}}_{\mathrm{total}}={\mathcal{L}}_{\sup }+\lambda {\mathcal{L}}_{\mathrm{RL}},$$

(8)

where $\lambda$ controls the trade-off between imitation learning and reinforcement-driven improvement. This paradigm supports the development of embodied agents capable of learning from both instruction data and real-time feedback, paving the way for scalable autonomous systems that integrate perception, cognition, and control.

These paradigms separate whether foundation models drive deployment-time decision-making, whether reinforcement objectives primarily reshape the foundation model itself, or whether both mechanisms co-exist in an interactive loop with distinct planning and control roles. Collectively, these three paradigms illustrate complementary pathways for integrating DRL with foundation models. Model-centric designs emphasize representational transfer and sample efficiency, RL-centric paradigms focus on alignment and optimization, and hybrid frameworks seek unification through multimodal reasoning and closed-loop interaction. Together, they represent the foundational structure for understanding how reinforcement learning and large-scale pretraining coalesce in the next generation of intelligent agents.

4.2. Architectural Patterns

Architectural innovations underpin the practical realization of DRL–FM integration. These architectures determine how perception, memory, reasoning, and control interact across model-centric, RL-centric, and hybrid paradigms. The key trend is the unification of sequence modeling, representation learning, and world modeling within a single policy structure. In addition to these macro-architectures, recent work revisits the choice of function approximators used inside policies, critics, and dynamics models, which can materially affect interpretability and parameter efficiency.

4.2.1. Transformer-Based Architectures

Transformer-based architectures dominate modern DRL–FM integration due to their ability to model temporal dependencies and multimodal context. By interpreting reinforcement learning trajectories as sequences of tokens—comprising states, actions, and rewards—transformer policies can learn long-term dependencies across time steps [53,54,55]. A general autoregressive policy can be expressed as

$${\pi }_{\theta }\left(a_t\right|s_{\le t},a_{<t},r_{<t})=\mathrm{softmax}\left(W_o{h}_t\right),$$

(9)

where $h_t=f_{\theta }(s_{\le t},a_{<t},r_{<t})$ represents the hidden representation computed by a transformer encoder–decoder and $W_o$ projects the latent state to an action distribution. This sequence-based design enables direct adaptation of large transformer backbones, such as GPT-style or T5 architectures, for policy modeling and trajectory prediction.

4.2.2. Hierarchical Architectures

Hierarchical architectures extend this formulation by separating decision-making into high-level planning and low-level control. A hierarchical policy can be formalized as a two-level structure:

$$\pi \left(a_t\right|s_t)=\sum _{z_t}{\pi }_L\left(a_t\right|z_t,s_t){\pi }_H\left(z_t\right|s_t),$$

(10)

where ${\pi }_H$ denotes the high-level policy that selects abstract goals or latent skills $z_t$, and ${\pi }_L$ represents the low-level controller [50]. Foundation models contribute to this hierarchy by encoding task semantics or linguistic goals that condition ${\pi }_H$, while DRL learns ${\pi }_L$ through continuous interaction. This design underlies agents such as PaLM-E [50] and RT-2 [51], where multimodal encoders feed symbolic intentions into continuous controllers.

4.2.3. World-Model Architectures

World-model architectures form another critical pattern. These systems jointly learn a latent dynamics model $p_{\varphi }\left(s_{t+1}\right|s_t,a_t)$ and a policy that plans within the learned latent space. The combined objective typically includes both reconstruction and return-maximization terms [56,57]:

$${\mathcal{L}}_{\mathrm{world}}=\mathbb{E}\left[\parallel s_{t+1}-{\widehat{s}}_{t+1}{\parallel }^2\right]-\eta {\mathbb{E}}_{p_{\varphi }}\left[\sum _{t=0}^T{\gamma }^t{r}_{\varphi }(s_t,a_t)\right],$$

(11)

where $\eta$ controls the trade-off between predictive accuracy and reward optimization. DreamerV3 [9] and MuZero [33] are canonical examples, with recent variants incorporating pretrained vision–language encoders or large language models (LLMs) as priors for richer latent representations.

4.2.4. Kolmogorov–Arnold Networks

A recent architectural development is KANs, proposed as a potential replacement for MLP blocks in function approximation. KANs shift learnable nonlinearities from fixed activations at nodes to learnable univariate functions on edges, often parameterized using spline bases, while nodes primarily aggregate additive contributions. This design has been argued to support more interpretable component functions and, in some regimes, improved parameter efficiency relative to standard MLPs [29]. A canonical Kolmogorov–Arnold form can be written as

$$f\left(x\right)=\sum _{q=1}^{2n+1}{\Phi }_q\left(\sum _{p=1}^n{\phi }_{q,p}\left(x_p\right)\right),$$

(12)

where $x=(x_1,\dots ,x_n)\in {\mathbb{R}}^n$ denotes the input vector with n scalar components; ${\phi }_{q,p}(·)$ are learnable univariate component functions; and ${\Phi }_q(·)$ are learnable univariate outer functions that map the aggregated inner sum to a scalar contribution, and practical KAN implementations generalize this template to wider and deeper networks with spline-parameterized edge functions [29]. In DRL–FM systems, KANs are relevant wherever MLPs act as default approximators: (i) actor and critic heads in actor–critic methods, (ii) reward and dynamics heads in world-model agents, and (iii) auxiliary prediction modules used for representation learning or uncertainty modeling. This creates a DRL-specific mapping in which KANs can serve as policy approximators, value approximators, or components of learned environment models, while leaving higher-level FM backbones unchanged. From a practical standpoint, KAN performance can be sensitive to basis choices, grid resolution, regularization, and training dynamics, and recent guidance consolidates these implementation trade-offs and positioning for practitioners [30].

4.2.5. Retrieval-Augmented and Memory-Based Architectures

Finally, retrieval-augmented and memory-based architectures enhance sample efficiency and reasoning depth by coupling DRL policies with external knowledge [58,59]. In such systems, a retriever module $\mathcal{R}\left(q\right)$ fetches relevant context from a memory bank $\mathcal{M}$ given a query q, producing augmented state representations ${\tilde{s}}_t=[s_t;\mathcal{R}\left(s_t\right)]$. The policy then conditions on ${\tilde{s}}_t$ rather than raw observations, allowing foundation models to act as dynamic knowledge bases. This mechanism supports long-term credit assignment and continual learning—core requirements for scalable agentic intelligence [60,61].

4.3. Interaction and Feedback Loops

Interaction and feedback loops determine how DRL–FM systems adapt over time. Human-in-the-loop supervision provides preference labels, rankings, or evaluative signals that guide alignment and policy refinement [62,63]. These signals are typically converted into preference datasets that support reward modeling or direct optimization methods [13,14]. Furthermore, AI-in-the-loop feedback extends this paradigm by using teacher models to generate synthetic evaluations, improving scalability and consistency across tasks [64,65]. Self-improvement mechanisms further enable policies to critique their own outputs, adjust task difficulty, and iteratively generate new training samples [66,67]. These feedback structures support continual learning and autonomy in open-ended environments.

These feedback mechanisms—human-in-the-loop, AI-in-the-loop, and self-improving loops—form the backbone of adaptive intelligence in DRL–FM systems. They transform static pretrained models into interactive, evolving agents capable of learning from evaluation, reasoning over feedback, and refining their policies over time. These interaction patterns complement the architectural foundations discussed earlier, together defining the dynamic learning ecosystem that underlies scalable, aligned, and generalizable reinforcement-driven foundation models.

5. Training, Alignment, and Optimization Methods

Having outlined the major paradigms through which DRL and foundation models interact, the next step is to examine the underlying mechanisms that enable these systems to learn from preference signals, structured rewards, and interaction dynamics. The training and alignment methods discussed in this section operationalize the taxonomy presented earlier by detailing how policies are optimized, how reward signals are constructed, and how reinforcement objectives shape the behavior of foundation models in practice.

5.1. Reward Modeling and Preference Learning

Reward modeling forms the foundation of alignment in DRL and foundation model integration. It provides the evaluative signal that drives policy optimization, ensuring model behavior reflects human or task-specific values [68,69,70]. Classical reinforcement learning assumes access to an explicit scalar reward $r_t=R(s_t,a_t)$ that captures task success. In contrast, foundation models operate in high-dimensional, ambiguous domains—such as language, vision, or multimodal reasoning—where reward functions are not predefined but must instead be learned from preference data or human feedback.

Reward modeling uses the same preference-learning formulation introduced in Section 4.1.2, where a reward model is trained from pairwise preferences and subsequently used to guide policy optimization. In DRL–FM systems, this component becomes the primary mechanism for shaping high-level behaviors such as reasoning quality, factuality, and safety. The overall RLHF framework is illustrated in Figure 2. A pretrained language model generates candidate outputs that are evaluated by a learned reward or preference model trained from human feedback [62]. The trained policy model is then updated via PPO using the reward signal and a Kullback–Leibler (KL) regularization term, which constrains deviations from the base model to preserve linguistic fluency and prevent exploitation of the reward model [71,72]. The KL divergence term used in RLHF optimization is expressed as

$$D_{\mathrm{KL}}\left({\pi }_{\theta }\parallel {\pi }_{\mathrm{ref}}\right)={\mathbb{E}}_x\left[\sum _y{\pi }_{\theta }\left(y\right|x)log{\displaystyle \frac{{\pi }_{\theta }\left(y\right|x)}{{\pi }_{\mathrm{ref}}\left(y\right|x)}}\right],$$

(13)

where ${\pi }_{\theta }$ denotes the current policy and ${\pi }_{\mathrm{ref}}$ the frozen reference model. This regularization encourages the updated model to remain aligned with its pretrained distribution while integrating reinforcement signals effectively.

Modern systems often adopt multi-objective reward modeling to balance competing values such as helpfulness, harmlessness, and honesty. The composite reward is represented as

$$r_{\varphi }(x,y)=\sum _{i=1}^K{w}_i{r}_{{\varphi }_i}(x,y),\mathrm{where}\sum _{i=1}^K{w}_i=1,$$

(14)

with $r_{{\varphi }_i}$ representing individual reward components and $w_i$ their corresponding priorities. This flexible formulation enables dynamic trade-offs among multiple alignment criteria during training. Recent extensions include language-conditioned rewards, where evaluative criteria are expressed through natural language rather than explicit labels [73,74]. The reward is computed using an evaluator model $E_{\psi }$ queried with a meta-instruction:

$$r_{\psi }(x,y)=E_{\psi }(“\mathrm{Is}y\mathrm{an}\mathrm{appropriate}\mathrm{response}\mathrm{to}x?”),$$

(15)

thus converting open-ended qualitative judgments into quantitative reinforcement signals. Despite these advances, reward-model exploitation and bias amplification remain persistent challenges.

5.2. Policy Optimization in the FM Era

Policy optimization defines how learned rewards translate into improved model behavior. In the context of foundation models, reinforcement learning typically fine-tunes a pretrained policy ${\pi }_{\theta }$ using feedback from a reward model $r_{\varphi }$ [68,75]. The optimization objective follows the KL-regularized policy update described previously, where reinforcement signals from the reward model are balanced with deviation constraints from the reference policy. The KL regularization term penalizes excessive divergence from the pretrained distribution, maintaining linguistic fluency and stability during optimization.

This structure forms the basis of several modern policy optimization methods summarized in

Table 3. Comparison of policy optimization methods for foundation model alignment.

Method	Key Principle	Advantages	Limitations
Proximal Policy Optimization [27]	On-policy gradient ascent with clipped updates and KL penalty.	Strong update stability under clipping and KL control when properly tuned.	Low sample efficiency due to rollout dependence; high sensitivity to KL strength, clipping thresholds, reward scaling, and learning rates.
Direct Preference Optimization [76]	Closed-form objective optimizing preference log odds.	Rollout-free and typically more sample-efficient than PPO; simpler optimization pipeline.	Stability depends on preference consistency and evaluator quality; sensitive to objective scaling and dataset mixture.
Implicit Preference Optimization [76]	Implicitly optimizes pairwise preference likelihood via reparameterized gradient.	Rollout-free with strong efficiency; smoother optimization and improved convergence relative to DPO.	Depends on correct objective specification and scaling; still constrained by preference noise and limited standardized tuning practices.
Offline RL	Optimizes policy from static datasets using precomputed feedback.	Safe and reproducible; no online exploration required.	Limited adaptability; distributional bias possible.

. Among these, PPO remains the most widely adopted approach for alignment, performing gradient updates with clipped probability ratios to ensure stable improvement. More recent methods, such as Direct Preference Optimization (DPO) and Implicit Preference Optimization (IPO) [76], extend this foundation by reformulating preference-based optimization objectives. DPO removes explicit reward modeling by directly encoding preference differences, whereas IPO leverages implicit likelihood reparameterization for improved convergence properties.

Across alignment pipelines, PPO, DPO, and IPO differ systematically along three practical axes: stability, sample efficiency, and sensitivity to hyperparameters. PPO is typically stable when its clipping thresholds and KL regularization are well tuned, but its on-policy nature makes it sensitive to reward-model noise, learning-rate schedules, and KL coefficients, while also incurring high sample and compute costs due to repeated rollouts. In contrast, DPO avoids on-policy rollouts entirely and operates directly on preference pairs, yielding substantially higher sample efficiency and a simpler optimization pipeline, but its stability depends strongly on preference consistency, evaluator calibration, and dataset quality. IPO further modifies the preference-based objective to smooth optimization dynamics and reduce variance, often improving convergence behavior and reducing sensitivity relative to DPO, while still inheriting fundamental limitations tied to preference noise, objective scaling, and data coverage.

Offline or implicit reinforcement learning further adapts this paradigm by learning from static datasets rather than online interactions [18,77]. The objective for offline RL is formulated as

$${\mathcal{L}}_{\mathrm{offline}}\left(\theta \right)=-{\mathbb{E}}_{(x,y)\sim \mathcal{D}}\left[{\widehat{r}}_{\varphi }(x,y)\right]+\beta \mathrm{KL}\left({\pi }_{\theta }(·|x)\parallel {\pi }_{\mathrm{ref}}(·|x)\right),$$

(16)

where $\mathcal{D}$ denotes pre-collected human or synthetic preference data. This approach enhances safety and reproducibility by decoupling policy updates from real-time exploration, but limits adaptability under distribution shift. A further development involves hybrid optimization, where behavior cloning (BC), a supervised imitation-learning method, is often combined with reinforcement learning for stability [78]. The total objective is defined as

$${\mathcal{L}}_{\mathrm{total}}={\mathcal{L}}_{\mathrm{BC}}+\lambda {\mathcal{L}}_{\mathrm{RL}},$$

(17)

with $\lambda$ controlling the trade-off between imitation learning and reinforcement adaptation. In summary, these methods indicate that policy optimization in the FM era is governed less by novel reinforcement algorithms and more by managing trade-offs between stability, efficiency, preference quality, and optimization sensitivity at scale.

5.3. Safety, Robustness, and Constraints

Safety and robustness remain critical challenges in DRL–FM integration. Constrained reinforcement learning imposes explicit limits on risk or undesired behavior through cost signals and thresholded expectations [79,80]. In reward-model–driven alignment, a central failure mode is a feedback-loop dynamic: systematic bias in the reward model can be exploited by the policy during optimization, leading to progressive bias amplification and, in extreme cases, reward hacking under distribution shift [81]. These constraints help prevent policies from exploiting weaknesses in reward models or drifting toward unsafe actions.

Adversarial training and red-teaming evaluate robustness by exposing models to perturbed inputs, adversarial prompts, or challenging edge cases [82]. Beyond one-off stress tests, periodic audits and targeted red-team cycles can detect reward-hacking behaviors that only surface after repeated policy improvement, when the policy learns to optimize evaluator blind spots rather than the intended objective [81,83]. Mitigations that reduce bias amplification include reward-model ensembles, uncertainty-aware reward signals, and adversarially curated preference data, which make overoptimization harder and increase the chance that failure modes are surfaced before deployment.

Interpretability methods, including policy probing and rationale extraction, further support safety auditing by revealing the internal decision features that drive policy behavior. In practice, these tools are most effective when paired with constraint mechanisms that bound optimization pressure, so that evaluator imperfections do not propagate unchecked into policy behavior [84,85].

5.4. Scaling Laws and Data Efficiency

Scaling behavior in DRL–FM systems follows empirical regularities analogous to those observed in supervised and self-supervised learning [86,87]. The expected performance E can often be approximated as a power-law function of model size N, data volume D, and environment complexity $\xi$:

$$E=k{N}^{\alpha }{D}^{\beta }{\xi }^{-\delta },$$

(18)

where $\alpha$, $\beta$, and $\delta$ are scaling exponents and k is a task-dependent constant [86]. Empirical studies have shown that reward quality and sample efficiency improve sublinearly with scale until diminishing returns set in [88,89]. These relationships guide model design and resource allocation for large reinforcement-trained systems.

Data efficiency remains an active research frontier. Techniques such as synthetic preference generation, experience replay, and active sampling have been developed to improve learning from limited feedback. Synthetic data generation uses teacher models or simulators to produce preference pairs $(x,y^{+},y^{-})$, which augment scarce human annotations while preserving reward diversity. Experience replay buffers past trajectories for off-policy optimization, ensuring better sample reuse. Active learning strategies adaptively select queries for which model uncertainty is highest, thus maximizing information gain per feedback instance.

Lastly, scaling and efficiency insights reveal that the effectiveness of DRL–FM systems depends not only on architectural capacity but also on the quantity, diversity, and quality of feedback data. As models grow larger and environments more complex, balancing computational cost with reward fidelity and safety alignment becomes the central optimization challenge.

6. Applications of DRL in Foundation Model Contexts

Building on the preceding taxonomy and optimization principles, the following sections survey how DRL–FM integration manifests across real-world domains, illustrating concrete implementations of model-centric, RL-centric, and hybrid paradigms.

6.1. Language and Multimodal Agentic Systems

Language-first agents have become a central proving ground for reinforcement-based refinement of foundation models. RLHF remains the most widely deployed strategy for aligning generative behavior with user expectations. Ouyang et al. [12] showed that combining supervised instruction tuning with preference-optimized policy updates enables moderately sized models to outperform significantly larger pretrained systems on human preference metrics. Rafailov et al. [76] advanced this direction with direct preference optimization, a formulation that dispenses with explicit reward modeling while matching the alignment performance of PPO-based RLHF across summarization and stylistic control tasks. More recently, Singh et al. [90] demonstrated that explicit reinforcement-driven reasoning schedules can further improve the consistency of tool-use decisions and stepwise reasoning, indicating that alignment and agentic competence can be shaped jointly.

The transition from passive text generation to utility-driven action becomes particularly evident in tool-augmented agents. Yao et al. [91] showed that interleaving chain-of-thought traces with concrete environment actions improves stability and reduces hallucinations relative to prompting-only approaches. Schick et al. [92] further demonstrated that models can learn tool-use behaviors through self-supervised objectives that implicitly reward actions contributing to downstream success. Building on this direction, Jiang et al. [93] introduced VerlTool, a framework that integrates reinforcement learning with dynamic tool-selection mechanisms, addressing fragmentation in earlier tool-use pipelines and enabling more coherent multi-step interaction.

These ideas extend naturally to embodied and grounded settings, where language, perception, and control must operate jointly. Driess et al. [50] embedded continuous perceptual signals into PaLM-E, enabling transfer of knowledge from vision–language tasks to robotic manipulation. Zitkovich et al. [51] expanded this approach in RT-2 by co-training on web-scale visual corpora and robot data, achieving improved generalization to previously unseen objects and task compositions. Kim et al. [94] contributed OpenVLA, an open-source alternative demonstrating that parameter-efficient reinforcement-style adaptation can port policies across heterogeneous robotic embodiments. These developments show how pretrained representations serve as semantic priors, while reinforcement learning supplies the corrective feedback necessary for fine-grained action grounding.

The limitations of prompt-only control become even more evident in open-ended digital environments. Zhou et al. [95] introduced WebArena, a realistic multi-site benchmark where language-model agents frequently fail without structured feedback or explicit credit assignment. Wang et al. [52] illustrated this in Minecraft: an automatic curriculum, self-verification, and iterative skill learning enabled substantially faster competence acquisition than imitation-driven or prompting-only baselines. Zheng et al. [96] reached similar conclusions in real web environments, showing that reinforcement-guided exploration and retrieval yield more reliable long-horizon research behavior than static prompting approaches. These studies reflect a consistent pattern: in open-world settings with branching action spaces, reinforcement-style shaping is essential for compressing exploration and avoiding compounding errors.

As language models increasingly function as autonomous agents rather than passive generators, new benchmarks have emerged to evaluate their decision-making and instruction-following capabilities. Qi et al. [97] introduced AGENTIF, a suite of tasks designed to measure the robustness of instruction following under interactive and constraint-heavy conditions. Results across multiple foundation models show substantial gaps between static prompting performance and reinforcement-refined agent behavior, highlighting the need for explicit feedback loops when models operate in environments with persistent state and long-horizon dependencies.

These developments illustrate how deep reinforcement learning transforms foundation models into more reliable and action-capable systems. Pretrained representations provide broad generalization and semantic grounding, while reinforcement-driven updates—whether from human preferences, synthetic evaluators, or environment returns—supply the corrective signals required for stable reasoning, tool use, and multi-step interaction. This synergy underlies the growing shift from text-only generation toward fully agentic systems capable of operating across web environments, software APIs, robotics tasks, and other complex domains.

Table 4. Applications of DRL in language and multimodal agentic systems.

Reference	Year	Domain	Contribution Summary
Ouyang et al. [12]	2022	RLHF	Introduced instruction tuning + preference-optimized RL, enabling smaller models to outperform larger LMs on human preferences.
Rafailov et al. [76]	2023	Alignment	Proposed Direct Preference Optimization, matching PPO-based RLHF without explicit reward models.
Singh et al. [90]	2025	Agentic RL	Showed that reinforcement-shaped reasoning schedules improve tool-use reliability and agentic consistency.
Yao et al. [91]	2022	Tool Use	Combined chain-of-thought reasoning and environment actions for more stable utility-driven behavior.
Schick et al. [92]	2023	Tool Learning	Demonstrated self-supervised API usage through implicit reward modeling.
Jiang et al. [93]	2025	Tool RL	Introduced VerlTool, integrating RL with dynamic tool-selection for coherent multi-step interaction.
Driess et al. [50]	2023	Embodied VLA	Introduced PaLM-E, merging continuous perception with language reasoning for robotics.
Zitkovich et al. [51]	2023	Robotics	Released RT-2, combining web-scale VLM pretraining with robot action sequences for improved generalization.
Kim et al. [94]	2024	Open VLA	Presented OpenVLA, enabling parameter-efficient adaptation across embodiments.
Zhou et al. [95]	2023	Web Agents	Released WebArena, showing LMs fail without RL-style feedback in long-horizon web tasks.
Wang et al. [52]	2023	Open-Ended RL	Demonstrated curriculum-driven RL for Minecraft–based lifelong skill acquisition.
Zheng et al. [96]	2025	Research Agents	Showed RL-driven retrieval and exploration improves long-horizon reasoning in real-world web environments.
Qi et al. [97]	2025	Benchmarking	Introduced AGENTIF, evaluating instruction-following reliability in agentic scenarios.

summarizes the studies discussed in this section.

6.2. Autonomous Decision-Making and Control

Autonomous control systems provide another setting where pretrained representations and reinforcement optimization interact to yield substantial gains. Model-based DRL architectures remain fundamental to this space. Hafner et al. [9] introduced DreamerV3, a latent world-model agent that leverages imagination rollouts to optimize long-horizon policies from high-dimensional observations, achieving strong data efficiency across diverse benchmarks. Schrittwieser et al. [33] demonstrated with MuZero that effective planning does not require explicit knowledge of environment dynamics; its combination of latent dynamics learning and policy optimization continues to influence hybrid architectures that embed pretrained vision or language encoders into downstream control pipelines.

The integration of foundation models into robotic systems has accelerated progress in real-world manipulation and navigation. Brohan et al. [98] trained RT-1 on large-scale real-world trajectories, showing that transformer sequence models paired with reinforcement refinement produce robust manipulation behavior. RT-2 [51] extended this paradigm by combining web-scale visual pretraining with robot data, enabling generalization to unseen objects and novel task compositions. O’Neill et al. [99] advanced this direction further with the Open X-Embodiment collection, a multi-embodiment dataset and model suite demonstrating that shared representations across robots substantially enhance cross-task and cross-platform transfer, particularly when reinforced through interaction.

Recent work has also begun to examine broader multimodal representations for control. Rajasegaran et al. [100] proposed token-to-token video learning mechanisms that capture temporal structure in continuous streams, providing a richer perceptual backbone for visuomotor policies. Yan et al. [101] introduced RoboMM, a multimodal foundation model for robotic manipulation that integrates vision, language, and action signals. Their framework highlights how large pretrained encoders can supply semantic grounding while reinforcement-based fine-tuning tailors the policy to platform-specific constraints.

Simulation-to-reality transfer remains a persistent challenge, and reinforcement-guided grounding has emerged as a key method for bridging the gap. Ahn et al. [49] combined high-level language reasoning with value-based affordances in SayCan, constraining action plans to feasible behaviors and improving real-world manipulation success. Chebotar et al. [102] showed that transformer-based attention stabilizes Q-learning across multitask manipulation settings, enhancing data efficiency and robustness. Nasiriany et al. [103] contributed RoboCasa, a large-scale household simulation environment that supports both VLA-style pretraining and reinforcement-driven adaptation, enabling controlled experimentation with sim-to-real pipelines under realistic task complexity.

Driving and aerial robotics exhibit similar advantages from hybrid DRL–FM architectures. Xu et al. [104] introduced DriveGPT-4, a multimodal driving system that integrates vision–language reasoning with reinforcement-style policy updates, improving collision avoidance and route-following stability relative to imitation-only training. Shao et al. [105] developed LMDrive, demonstrating that reinforcement refinement helps address long-tail edge cases that challenge purely supervised systems. These systems underscore the value of pretrained semantic understanding combined with reinforcement-driven adaptation for handling dynamic, safety-critical environments.

Table 5. Applications of DRL in autonomous decision-making and control.

Reference	Year	Domain	Contribution Summary
Hafner et al. [9]	2023	World Models	DreamerV3 with imagination rollouts for long-horizon control.
Schrittwieser et al. [33]	2020	Planning RL	MuZero with learned latent dynamics and tree search.
Brohan et al. [98]	2022	Robotics	RT-1 transformer-based robot control from large-scale trajectories.
Zitkovich et al. [51]	2023	Robotics	RT-2 combining VLM pretraining with robot data.
O’Neill et al. [99]	2024	Embodiment	Open X-Embodiment dataset for multi-robot generalization.
Rajasegaran et al. [100]	2024	Video Learning	Token-to-token video representations for visuomotor control.
Yan et al. [101]	2024	VLM Robotics	RoboMM multimodal model for manipulation.
Ahn et al. [49]	2022	Language→Action	SayCan integrates language reasoning with value-based affordances.
Chebotar et al. [102]	2023	Q-Learning	Transformers to stabilize multi-task Q-learning.
Nasiriany et al. [103]	2024	Simulation	RoboCasa simulation environment for VLA+RL research.
Xu et al. [104]	2024	Driving	DriveGPT-4 integrates VLM reasoning with RL-style refinement.
Shao et al. [105]	2024	Driving	LMDrive addresses long-tail driving cases using RL.

provides a summary of the applications. Across these developments, a consistent pattern is evident: foundation models supply broad perceptual and semantic structure, while reinforcement learning provides the task-specific corrective signals required for stable, generalizable decision-making. This interaction has reshaped the landscape of embodied intelligence, enabling policies that adapt under uncertainty, generalize across embodiments, and maintain robustness in real-world, interactive settings.

6.3. Scientific Discovery and Complex Systems

Scientific discovery often unfolds in spaces that are vast, combinatorial, and sparsely supervised, making it a natural setting for uniting foundation-model priors with reinforcement-driven search. In molecular design, Bagal et al. [106] demonstrated that transformer decoders can generate syntactically valid and chemically novel compounds competitive with specialized neural baselines. Frey et al. [107] expanded this direction with ChemGPT, documenting scaling laws that enable billion-parameter chemical models to serve as powerful priors for targeted property optimization. Ye et al. [108] further showed that augmenting such priors with proximal policy optimization yields molecular candidates that better satisfy structural and pharmacological constraints than those obtained via supervised fine-tuning alone.

Materials science provides a parallel illustration of how pretrained structure can accelerate the search over large physical and chemical design spaces. Riebesell et al. [109] introduced Matbench Discovery to evaluate model performance under realistic, prospective screening conditions, finding that reinforcement-guided surrogate search significantly improves hit rates for stable materials. Tang et al. [110] extended these ideas with MatterChat, a multimodal foundation model that incorporates structural, textual, and physical information; its representations offer a flexible substrate for reinforcement-driven optimization of materials with complex cross-property trade-offs.

Pretrained scientific models are also increasingly used as reasoning engines that guide algorithmic or symbolic search. Fawzi et al. [111] demonstrated the feasibility of framing algorithm design as a reinforcement learning problem with AlphaTensor, which discovered matrix multiplication routines outperforming long-standing human-derived schemes. Mankowitz et al. [112] later extended this line of work by discovering faster sorting algorithms using deep reinforcement learning, indicating that algorithmic domains once dominated by theory-driven heuristics can benefit from policy-guided exploration. SciGLM [113], a scientific multimodal language model, complements these efforts by providing a versatile prior for tasks involving symbolic manipulation, hypothesis evaluation, and multi-step reasoning.

Climate and geoscience research present additional domains where foundation models and reinforcement learning intersect. Rasp et al. [114] established WeatherBench as a standardized benchmark for data-driven forecasting, enabling systematic comparison of physics-free and hybrid models. Pathak et al. [115] introduced a high-resolution global forecasting system based on adaptive Fourier operators that now serves as a surrogate for downstream reinforcement-guided scheduling and resource allocation. Reviews by Michailidis et al. [116] and de Burgh-Day et al. [117] highlight growing evidence that coupling learned simulators with reinforcement controllers improves decision quality in climate mitigation, energy optimization, and environmental policy planning.

In these scientific domains (summarized in

Table 6. Applications of DRL in scientific discovery and complex systems.

Reference	Year	Domain	Contribution Summary
Bagal et al. [106]	2021	Molecules	MolGPT produced chemically valid and novel molecules using transformer decoders.
Frey et al. [107]	2023	ChemGPT	Established scaling laws for chemical foundation models.
Ye et al. [108]	2024	Drug Design	Used PPO to steer molecular generation toward desired properties.
Riebesell et al. [109]	2023	Materials	Introduced Matbench Discovery for prospective evaluation of materials ML.
Tang et al. [110]	2025	Materials LLMs	Introduced MatterChat, a multimodal LM for materials science and cross-property reasoning.
Zhang et al. [113]	2024	Scientific LMs	SciGLM provided self-refined scientific reasoning and symbolic manipulation capabilities.
Fawzi et al. [111]	2022	Algorithms	AlphaTensor used deep RL to discover faster matrix-multiplication schemes.
Mankowitz et al. [112]	2023	Algorithms	Extended RL-based algorithm discovery to produce faster sorting routines.
Rasp et al. [114]	2020	Climate	Released WeatherBench for standardized climate-forecast benchmarking.
Pathak et al. [115]	2022	Climate	Introduced FourCastNet, a global high-resolution surrogate model.
Michailidis et al. [116]	2025	Climate RL	Reviewed RL-augmented climate and energy decision systems.
de Burgh-Day et al. [117]	2024	Geoscience RL	Surveyed RL-enhanced environmental modeling and simulation control.

), the interaction between broad, pretrained priors and reinforcement-based adaptation has transformed how researchers navigate vast hypothesis spaces. Foundation models provide structured, domain-aware representations, while reinforcement learning offers a principled mechanism for goal-directed search, enabling faster discovery, deeper reasoning, and more efficient use of experimental or computational resources.

6.4. Societal, Ethical, and Alignment Applications

As foundation models expand their presence in socially embedded settings, reinforcement-driven alignment has become a core method for shaping behavior toward human expectations. Chaudhari et al. [118] demonstrated that RLHF substantially increases perceived helpfulness and preference consistency, while Bai et al. [119] showed that constitutional principles combined with AI-generated critique can reduce harmful outputs without relying exclusively on costly human annotation. Heo et al. [120] further showed that non-expert feedback can provide surprisingly coherent preference signals during early-stage tuning, underscoring the flexibility of reinforcement-based alignment methods.

However, the growing evidence around reward hacking indicates that RLHF pipelines can behave unpredictably unless reward models are carefully constrained. Miao et al. [121] proposed an information-theoretic approach (InfoRM) that suppresses spurious reward predictors within the reward model, reducing overoptimization and improving alignment stability. Related work by Casper et al. [63] highlighted how misgeneralization, proxy gaming, and evaluator blind spots arise in practice, particularly when evaluators—human or model-based—lack adequate robustness. This concern is amplified by findings from Perez et al. [122], who showed that language models acting as adversarial red-teamers can reliably elicit unsafe behavior from reinforcement-tuned systems.

Scaling alignment beyond human feedback introduces an additional layer of complexity. Chen et al. [123] proposed uncertainty-aware reward-model pipelines that incorporate adversarially curated preference data, improving robustness in reinforcement optimization. Complementing these ideas, Fu et al. [124] examined reward shaping strategies designed to reduce reward hacking in RLHF, showing that bounded and smoothed reward functions produce more stable optimization dynamics. These findings indicate that alignment procedures must be treated as full-stack systems in which reward design, data curation, and optimization dynamics interact in nontrivial ways.

When reinforcement-trained systems operate within multi-agent environments, additional social risks emerge. Dafoe et al. [125] examined how strategic behaviors—such as defection, collusion, and exploitation—arise naturally in multi-agent reinforcement learning unless cooperative constraints are explicitly modeled. Constrained reinforcement learning provides one viable direction. Gao et al. [126] showed that harm-sensitive constraints can produce policies that remain within ethical bounds even under ambiguous reward signals, while Shperberg et al. [127] introduced a relaxed exploration framework that enforces strict safety constraints only at deployment time. Haydari et al. [128] demonstrated similar benefits in safety-critical control settings, further illustrating how constraint-based formulations complement preference-based alignment strategies. These approaches integrate naturally with broader evaluation frameworks such as HELM [43], which assess robustness, fairness, and safety alongside task-oriented performance.

Finally, reinforcement-based alignment raises governance and societal challenges that extend beyond purely technical interventions. Yao et al. [129] highlighted the risks of culturally narrow or demographically skewed preference datasets, which can cause RLHF pipelines to encode restrictive normative assumptions. Lindström et al. [130] examined practical limitations in human feedback collection, noting that scalable alignment requires transparent annotation protocols and diversified evaluators. Xu et al. [131] similarly argued that reinforcement-driven alignment demands reproducible governance mechanisms to ensure stability under distribution shift. Across these analyses, a consistent pattern emerges: reinforcement-based alignment is effective at shaping model behavior, but its success ultimately depends on robust oversight structures that integrate technical, organizational, and societal considerations. These applications are tabulated in

Table 7. Representative applications of DRL in societal, ethical, and alignment contexts.

Reference	Year	Theme	Contribution Summary
Chaudhari et al. [118]	2025	RLHF	Improved preference consistency and helpfulness.
Bai et al. [119]	2022	Constitutional AI	Rule-based critique + LM revision reduces harm.
Heo et al. [120]	2025	Human Feedback	Found stable early-stage alignment from non-experts.
Miao et al. [121]	2024	Reward Modeling	InfoRM reduces reward hacking via information constraints.
Casper et al. [63]	2023	Alignment Risks	Analyzed misgeneralization, proxy gaming, evaluator failures.
Perez et al. [122]	2022	Red-Teaming	LM adversaries reveal unsafe RLHF behaviors.
Chen et al. [123]	2025	RLAIF	Uncertainty-aware reward modeling + adversarial curation.
Fu et al. [124]	2025	Reward Shaping	Bounded/smoothed rewards improve RLHF stability.
Dafoe et al. [125]	2020	Social RL	Examined emergent collusion and defection in multi-agent RL.
Gao et al. [126]	2024	Safe RL	Harm-sensitive constraints for ethical policy learning.
Shperberg et al. [127]	2024	Constrained RL	Relaxed exploration with deployment-time safety guarantees.
Haydari et al. [128]	2024	Fair RL	Fairness-aware constrained RL for traffic control.
Liang et al. [43]	2022	Evaluation	HELM evaluation framework for safety, robustness, and bias.
Yao et al. [129]	2023	Cultural Bias	Showed demographic skew in preference data distorts RLHF.
Lindström et al. [130]	2024	Oversight	Identified feedback-collection bottlenecks and evaluator bias.
Xu et al. [131]	2025	Governance	Advocated reproducibility and governance for RLHF pipelines.

.

7. Challenges, Solutions, and Future Research Directions

As DRL becomes increasingly intertwined with foundation models, substantial technical, methodological, and governance-related challenges remain unresolved. The issues are not isolated; they interact across optimization, evaluation, human supervision, and societal deployment. This section consolidates the central open problems and outlines research pathways capable of advancing the integration of DRL and FMs toward more reliable, generalizable, and socially aligned systems.

7.1. Challenges and Open Problems

The integration of DRL with FMs introduces several unresolved challenges spanning optimization, supervision, interaction dynamics, and governance. These challenges reflect underlying tensions between the flexibility of reinforcement learning and the scale, opacity, and social impact of modern FMs.

7.1.1. Optimization Instability

A core difficulty lies in the instability of reinforcement-based optimization when applied to foundation-scale policies. Algorithms such as PPO and DPO remain sensitive to hyperparameter choices, reward shaping, and small errors in reward model predictions. Even minor inaccuracies can induce reward hacking or behavioral drift, especially in high-dimensional policy spaces where updates propagate unpredictably [119]. As model size increases, controlling these instabilities becomes more challenging, and current techniques lack mechanisms for ensuring consistent improvement during iterative alignment.

7.1.2. Credit Assignment in Language-Conditioned Environments

Another structural challenge concerns credit assignment when rewards depend on long-horizon linguistic or multimodal criteria. Many DRL–FM tasks require assessing latent constructs—factual accuracy, ethical compliance, or multistep tool use—but the scalar rewards typically available provide insufficient granularity for identifying which intermediate actions, tool decisions, or reasoning steps contributed to success [50,52]. In practice, reward models and automated evaluators often score an entire response or episode, producing delayed, noisy, and sometimes contradictory signals; this causes “credit diffusion” where many distinct trajectories receive similar scores, while small evaluator errors can dominate the learning signal. When the environment is language-conditioned, the action space expands to token- or instruction-level decisions, and the state can be partially implicit (e.g., hidden assumptions, private intermediate reasoning, or external tool side effects), making standard temporal-difference style assignment unreliable under sparse or delayed feedback.

Emerging approaches attempt to restore structure by moving from step-local rewards to trajectory-aware and decomposition-aware supervision. Trajectory-level reward modeling assigns value to coherent segments (plans, subgoals, tool-use phases) rather than isolated steps, while hierarchical policies and subgoal-conditioned critics restrict the search space so that rewards map to fewer, more interpretable decisions. Complementary methods add process-level feedback via verifier signals, intermediate checkpoints, or structured preference comparisons over partial solutions, aiming to densify supervision without introducing brittle hand-crafted shaping. However, these designs remain vulnerable to evaluator gaming and distribution shift: as policies improve, they generate out-of-distribution reasoning patterns that can break learned evaluators, leading to unstable updates or misaligned shortcuts.

A second line of work explores causal-influenced credit assignment, treating tool calls and reasoning interventions as potential causes and estimating which decisions materially change outcomes via counterfactual rollouts, causal graphs, or influence-style attributions [132]. In parallel, self-critique mechanisms embed an internal critic—often an FM-based verifier or revision loop—that tests, edits, or rejects intermediate steps before execution, converting sparse task rewards into richer self-generated supervision [119]. These mechanisms can improve learnability in long-horizon settings, but they introduce new failure modes, including critic–policy collusion, over-optimization to the critique format, and increased compute and governance burden for auditing critic reliability at scale.

7.1.3. Scalability and Reliability of Human Feedback

High-quality human preference data remain expensive, inconsistent, and difficult to scale. RLAIF [119] alleviates the annotation bottleneck but introduces new risks: synthetic feedback inherits the blind spots, cultural biases, and normative assumptions of the teacher models that generate it. As a result, feedback pipelines face a tension between scalability and representational fidelity. The lack of systematic tools for measuring these distortions complicates efforts to build durable and trustworthy alignment mechanisms.

7.1.4. Emergent Multi-Agent Dynamics

Large models increasingly interact with one another in multi-agent ecosystems—through communication, negotiation, shared tool use, or competitive tasks. These interactions can yield emergent behaviors such as collusion, deception, or manipulative strategies [125]. Standard single-agent RL algorithms are not designed to detect or constrain such dynamics, and existing multi-agent RL techniques lack proven stability guarantees at FM scale. Without robust mechanisms for shaping cooperative norms and monitoring emergent strategies, FM-driven ecosystems risk producing unintended and difficult-to-audit behaviors.

7.1.5. Reproducibility, Transparency, and Governance Limitations

The field continues to face persistent reproducibility and governance gaps. Variation in pretraining corpora, reward model architectures, evaluator instructions, and computational budgets can significantly alter experimental outcomes [43,133]. These discrepancies hinder the comparability of results and limit the ability of auditors to evaluate alignment claims. Transparent reporting, standardized evaluation, and reproducible pipelines remain underdeveloped, creating barriers to responsible deployment in socially sensitive environments.

7.2. Unified DRL–FM Evaluation Framework

A recurring limitation in DRL–FM research is that evaluation practice does not match system complexity: agents are trained with multimodal perception, tool use, preference feedback, and long-horizon interaction, yet are often reported with narrow single-domain metrics. This subsection proposes a unified evaluation framework intended to be usable across classical control, embodied environments, and language-first agentic settings. The goal is not to replace existing benchmarks, but to standardize how results are reported so that claims become comparable across environments, feedback sources, and deployment contexts.

7.2.1. Design Principles

• Cross-domain comparability: report a shared set of metric groups so results from control, web agents, and multimodal settings can be compared along common axes.
• Closed-loop and offline evaluation: evaluate both interactive behavior under environment dynamics and offline behavior under fixed datasets or preference logs, because many systems mix both regimes.
• Safety and robustness as first-class objectives: treat unsafe behavior, policy exploits, and adversarial fragility as primary outcomes rather than secondary notes.
• Governance and reproducibility artifacts: publish minimum reporting details (seeds, compute, data, evaluator prompts, and reward-model settings) required to reproduce headline results.

7.2.2. Metric Groups and Minimum Outputs

We recommend reporting metrics in seven groups. Each group includes example measures that can be instantiated across environment types.

• Capability and return: success rate, episodic return, task completion rate, constraint satisfaction rate, and trajectory-level goal completion.
• Efficiency: environment samples to threshold, wall-clock time to threshold, training tokens, compute proxy measures, regret curves, and evaluation-time cost.
• Generalization: performance under out-of-distribution splits, procedural generalization, tool or API shift, environment shift, and evaluator shift.
• Alignment and helpfulness: preference win-rate, refusal correctness, harmlessness rate, truthfulness proxies, and instruction-following reliability under constraints.
• Credit assignment quality: stability of step- or segment-level attributions under perturbations, consistency of identified causal steps across seeds, and sensitivity of outcomes to intermediate action edits [19,22].
• Robustness and security: adversarial prompts, observation perturbations, reward-model attacks, jailbreaking resistance, and traceability checks such as watermark or provenance hooks when applicable.
• Reproducibility: seed sensitivity, variance across runs, reporting of hyperparameters and reward-model details, and ablations that isolate the contribution of the FM and the reinforcement component.

7.2.3. Evaluation Protocol Template

To reduce ambiguity across papers, evaluation should explicitly specify:

• Interaction mode: online, offline, or mixed, and the exact point at which preference or reward signals enter the loop.
• Evaluator type: human, model-based, rule-based, or hybrid, including prompt templates and any safety policies used by evaluators.
• Aggregation: mean and dispersion across seeds, and the number of independent runs used for reported metrics.
• Failure analysis: a mandatory set of failure modes aligned with the benchmark blueprint in

  Table 8. Benchmark blueprint for the proposed unified DRL–FM evaluation: environment types, required metric groups, protocol expectations, and common failure modes.

  Environment Type	Required Metric Groups	Evaluation Protocol	Common Failure Modes
  Classical control (Atari, MuJoCo, ProcGen)	Capability and return; Efficiency; Generalization; Robustness; Reproducibility	Multiple seeds; fixed training budget; OOD or procedural splits; perturbation stress tests	Reward hacking; brittle exploration; overfitting to levels; instability across seeds
  Embodied multimodal tasks (Habitat, BEHAVIOR, MineDojo)	Capability and return; Generalization; Credit assignment quality; Robustness; Alignment	Instruction variants; tool or object shift; long-horizon evaluation; safety constraint checks	Compounding errors; goal drift; unsafe interactions; shortcut learning from spurious cues
  Language and tool agents (web, APIs, assistants)	Alignment; Capability and return; Generalization; Robustness and security; Reproducibility	Held-out tasks; tool-shift tests; adversarial prompt suite; evaluator shift comparisons	Hallucinated actions; tool misuse; jailbreak susceptibility; preference overfitting
  Scientific discovery and combinatorial search	Capability and return; Efficiency; Generalization; Robustness; Reproducibility	Prospective evaluation; constrained validity checks; compute reporting; sensitivity analysis	Mode collapse; proxy optimization; constraint violations; poor transfer across targets
  Multi-agent settings (cooperative or competitive)	Capability and return; Robustness; Alignment; Reproducibility	Cross-play evaluation; opponent variation; policy exploit tests; monitoring for emergent strategies	Collusion; deception; exploitability; non-stationarity instability

  .

7.2.4. Benchmark Blueprint

Table 8 provides a blueprint that links environment types to required metric groups, recommended protocols, and typical failure modes. It is intended as a minimum standard that authors can extend rather than a restrictive checklist.

7.2.5. Minimum Reporting Standard

To make results auditable and comparable, we recommend a minimum reporting standard that includes: (i) model and adapter details, including which components are frozen or fine-tuned; (ii) reward or preference pipeline details, including evaluator prompts and reward-model training data sources; (iii) training regime and compute proxies; (iv) number of seeds and variance measures; and (v) a short failure-mode report aligned with Table 8. This standard is designed to be lightweight enough for survey-derived comparisons while still enabling reproducibility and governance review.

7.3. Future Research Directions

Solving these problems requires progress across modeling, optimization, evaluation, and governance. Foundation-scale world models offer one avenue: systems such as DreamerV3 [9] illustrate how temporally coherent latent dynamics can support long-horizon credit assignment, reduce reliance on sparse scalar rewards, and stabilize optimization. Extending such techniques to FM architectures could provide shared state representations that integrate perception, reasoning, and action.

A second direction centers on self-reflective and self-improving agents. Reflection and critique mechanisms—such as iterative self-evaluation and revision pipelines [95,119]—may mitigate reward hacking, correct reasoning errors, and facilitate consistent long-horizon behavior. Embedding these loops within recurrent or memory-augmented architectures would provide structured pathways for internal error detection, enabling more robust credit assignment and reducing fragility in policy updates.

Further progress may arise from neurosymbolic reinforcement learning, which integrates logical constraints, causal structure, and symbolic rules with neural policy optimization. Such approaches offer pathways toward interpretability and verifiability, addressing governance and reproducibility concerns while improving robustness in high-stakes settings. Complementary work in federated and resource-efficient DRL aims to reduce computational cost and improve privacy-preserving deployment, which becomes increasingly important as FMs proliferate across distributed infrastructures.

Sustainability and energy efficiency represent additional priorities. Energy-aware training and evaluation—consistent with principles from Green AI [134,135]—should become integral to reinforcement learning objectives, with carbon- and compute-aware reward shaping enabling more responsible scaling trajectories.

Finally, the development of sovereign, culturally adaptive, and legally grounded alignment practices is essential for global deployment. Region-specific reward functions, inclusive preference datasets, and auditable oversight structures [136] are critical for mitigating bias propagation and ensuring that reinforcement-trained models operate within diverse normative contexts. These directions suggest a research landscape in which DRL functions not only as an optimization tool but as part of a broader governance framework shaping the evolution of foundation models. Meanwhile,

Table 9. Summary of key challenges and corresponding future research directions for DRL–FM integration.

Challenge	Description	Future Research Direction
Optimization Instability	Sensitivity to reward misspecification and brittle policy updates in large FM spaces.	Foundation-scale world models; structured trajectory-level RL; robust reward modeling.
Credit Assignment for Language-Conditioned Rewards	Difficulty linking high-level linguistic rewards to intermediate reasoning steps.	Self-reflective agents; hierarchical RL; memory-augmented reasoning architectures.
Scalability of Human and Synthetic Feedback	High cost of human annotation; bias propagation in RLAIF pipelines.	Hybrid human–AI feedback; adversarially curated preference datasets; evaluator diversification.
Emergent Multi-Agent Strategic Behavior	Risk of collusion, deception, and adversarial dynamics between interacting FMs.	Multi-agent safety mechanisms; game-theoretic alignment protocols; cooperative constraint learning.
Reproducibility, Governance, and Accountability	Lack of standardized reporting, inconsistent datasets, and opaque reward models.	Transparent governance frameworks; standardized evaluation; sovereign and culturally aware alignment.
Sustainability and Energy Constraints	Rising computational cost and environmental impact of DRL–FM training.	Energy-aware objectives; carbon-efficient RL; federated and decentralized DRL.

summarizes the challenges presented in this section and the corresponding future research directions.

8. Conclusions

The growing interplay between DRL and FMs marks a significant shift in how artificial agents perceive, reason, and act across complex environments. This survey examined how DRL complements the broad generalization, semantic priors, and multimodal capabilities of FMs, enabling systems that extend beyond passive prediction to become adaptive, interactive, and goal-directed. Across language agents, embodied control, scientific discovery, and societal alignment, the combined use of pretrained representations and reinforcement-based optimization has produced substantial gains in reliability, task efficiency, and long-horizon competence.

Furthermore, DRL–FM integration faces substantial technical and governance-related challenges. Optimization instability, limited credit assignment for language-conditioned tasks, and the difficulty of scaling high-quality human feedback continue to constrain system reliability. Addressing these shortcomings will require not only algorithmic advances but also transparent experimental reporting, inclusive evaluation practices, and stronger institutional mechanisms for oversight.

A practical takeaway of this survey is the unified DRL–FM evaluation framework introduced in Section 7, which operationalizes cross-domain comparability through shared metric groups, protocol templates, and a benchmark blueprint that links environment types to expected evaluation outputs and failure modes. The framework supports more consistent comparison across model-centric, RL-centric, and hybrid systems, and enables stronger auditing of safety, robustness, and governance claims.

Looking ahead, progress will likely come from coupling foundation-scale world models and structured long-horizon reinforcement learning with more reliable feedback and evaluation pipelines, while treating safety, reproducibility, and compute constraints as design objectives rather than afterthoughts. Culturally grounded governance and reward design further suggest that alignment is context-dependent, so DRL–FM systems must be evaluated and audited under the norms and deployment conditions they will actually face.