Towards Self-Evolving Agents: A Dual-Process Framework for Continual Context Refinement

Abstract

Large Language Models (LLMs) have become essential for interactive AI systems, yet they remain fundamentally static after deployment: they cannot update their parameters from interaction feedback and often repeat the same mistakes across long interaction streams. We propose Dual-Process Agent (DPA), a framework for continual context refinement that enables learning without modifying a frozen model backbone. Inspired by dual-process theory from cognitive science, DPA decomposes each interaction episode into two complementary processes: a fast System 1 that retrieves compact, relevant context from an explicit long-term memory and generates responses, and a slow System 2 that reflects on outcomes and writes curated updates back into memory. To prevent memory degradation over extended interactions, DPA maintains bulletized memory entries with utility statistics and employs a conservative curator gate that filters generic, redundant, or conflicting insertions. Experiments on six diverse benchmarks demonstrate that DPA consistently outperforms vanilla prompting and competitive baselines on both GPT-5.1 and Llama-3.1-8B backbones, achieving the best overall performance across multiple reasoning and knowledge-intensive tasks.

1. Introduction

Large Language Model (LLM)-based autonomous agents have progressed rapidly in recent years, demonstrating promising capabilities in task planning, knowledge-intensive reasoning, and multi-turn interaction [1]. However, relying solely on parametric knowledge encoded in model weights still entails substantial limitations, particularly a heightened risk of hallucinations when operating in open and dynamically evolving information environments [2]. To mitigate these issues, Retrieval-Augmented Generation (RAG) [3] has been proposed, which augments the generation process with external knowledge sources and enables the model to dynamically acquire evidence that is highly relevant to the current task.

A typical RAG system consists of two collaborative components: a vector-based retrieval module that identifies content most relevant to the input from large-scale knowledge corpora, and a generation module that conditions on both the retrieved evidence and the original query to produce more accurate, coherent, and factually grounded responses. By integrating external knowledge into the inference process, RAG reduces uncertainty arising from the model’s reliance on purely parametric memory and achieves greater reliability and robustness in knowledge-intensive tasks.

Although RAG expands an agent’s access to knowledge within a single inference step, its overall mechanism remains essentially reactive and oriented toward immediate response, lacking the ability to persist and update information produced during interaction. Conventional RAG pipelines provide little support for revisiting generated outputs through systematic review, evaluation, or strategy adaptation, preventing the agent from internalizing experience and from building a memory structure that evolves with its environment. In addition, RAG typically relies on a static external knowledge base where the system can only passively consume pre-existing information, with no principled mechanism for updating, organizing, or restructuring the underlying knowledge. This limits its capacity to absorb new information and maintain consistency as the world changes. Consequently, while retrieval enriches the model’s external knowledge sources, the overall paradigm still operates in a fast, one-shot reasoning mode, falling short of the deeper reflective processes and self-evolution capabilities required by autonomous agents in long-horizon settings.

Building on these observations, we argue that to build truly self-evolving agents that can continuously learn and improve autonomously, we need to introduce a complementary process that can reflect on, integrate, and evolve the interaction process. Inspired by Kahneman’s dual-process theory [4], we view fast reasoning and deep reflection as two independent yet collaborative processes in the agent’s memory system: the former focuses on immediate decision-making and efficient generation, while the latter is responsible for abstracting experience, evaluating behavior, and updating internal representations and long-term knowledge based on that evaluation.

To this end, we propose a Dual-Process Agent (DPA) framework for continual context refinement (as shown in Figure 1b). Our design draws on both cognitive dual-process theory and established AI cognitive architectures. In cognitive science, System 1 denotes fast, automatic, associative processing, while System 2 denotes slow, deliberate, rule-based reasoning [5,6]. Similarly, in DPA, System 1 corresponds to feed-forward inference, where the agent retrieves relevant context via pattern matching and generates a response in a single pass, while System 2 corresponds to deliberate meta-cognition, where the agent revisits the interaction trace, evaluates outcomes, and decides whether to update the long-term memory store. This setup reflects dual-process views in which System 1 produces a default response that System 2 can refine through reflective monitoring. This decomposition also connects to AI cognitive architectures such as SOAR [7] and ACT-R [8], where production systems distinguish between automatic pattern-matched actions and deliberate goal-directed reasoning.

In contrast to conventional linear RAG pipelines that rely on static knowledge bases and lack any post-interaction update mechanism, our framework augments the fast reactive loop with an evolution-and-reflection phase that systematically compresses, abstracts, and integrates the reasoning trajectory, retrieved evidence, and final outputs after each interaction, thereby driving the dynamic evolution of long-term memory. Concretely, a user query is first passed to a retrieval module, which selects the most relevant memory snippets from a continually updated long-term memory store and feeds them, together with the current query, into the LLM to produce the final response. The full interaction trace is then distilled in the background into reusable memory units and written back into long-term memory, allowing the memory store to continually adapt and grow with each episode. This closed-loop mechanism enables the agent to accumulate experience over long-horizon episodes and maintain an evolving long-term memory structure without updating backbone parameters.

To validate the effectiveness of DPA, we conduct extensive experiments on six benchmarks spanning factual QA, multi-hop reasoning, instruction following, and mathematical reasoning (TruthfulQA, StrategyQA, IFEval, HotpotQA, FreshQA, and GSM-IC) using both GPT-5.1 and Llama-3.1-8B as backbones. Experimental results demonstrate that DPA achieves the strongest average performance across all benchmarks, with an average accuracy of 77.3% on GPT-5.1 and 67.0% on Llama-3.1-8B. We summarize our main contributions as follows:

• We propose a cognitively-inspired dual-process autonomous agent framework that explicitly separates fast response and deep reflection into two complementary processes, enabling the agent to combine immediate reasoning with continuous self-improvement.
• We design an evolvable long-term memory system that automatically distills, integrates, and reorganizes interaction experiences, enabling continual context refinement and self-improvement over extended interaction streams.
• We instantiate DPA as a complete end-to-end pipeline with retrieval, reflection, curation, and memory maintenance modules, and conduct comprehensive experiments and ablation studies to validate its effectiveness.

2. Related Work

2.1. Retrieval-Augmented Generation

RAG augments language models with external non-parametric knowledge and has become one of the primary approaches for mitigating hallucinations and improving factuality and timeliness in generation [9,10,11]. Early RAG systems [3,12,13] typically perform a single retrieval step over a static vector index, concatenate a small set of similar passages with the input, and fuse them either at the encoder side or during decoding, achieving strong performance on open-domain QA and other knowledge-intensive tasks. To extend the applicability of RAG, RETRO [14] performs fine-grained neighborhood retrieval over a trillion-token-scale corpus for each generation chunk, and injects retrieved neighbors into an autoregressive model via chunked cross-attention, demonstrating that explicit external memory can substantially enhance model capability without increasing parameter count.

More recently, researchers have proposed finer-grained designs for RAG with a focus on adaptive and robust retrieval strategies. Self-RAG [15] introduces internal reflection tokens that allow the model to trigger retrieval on demand and assess the usefulness of retrieved passages. Adaptive-RAG [16] selects retrieval intensity based on estimated query complexity and model uncertainty, automatically choosing among different retrieval paths. To improve robustness to retrieval noise and retrieval failures, CRAG [17] employs a separate quality estimator to filter low-confidence retrieved documents and trigger fallback strategies when necessary. Despite the improvements these methods bring in single-step accuracy, robustness, and flexibility of retrieval strategies, they still treat retrieval as a one-off operation conditioned on the current input: retrieved documents and intermediate reasoning are discarded once the interaction ends, and the external knowledge store remains static during use, making it difficult to accumulate interaction experience and structurally evolve the context over time. In contrast, our work introduces an evolving memory store where retrieval quality directly influences memory curation through persistent utility tracking.

2.2. Memory-Augmented Language Agents

As LLM-based agents are deployed in increasingly complex, long-horizon scenarios, the hard limits of the context window and the absence of principled memory management have become major bottlenecks for further capability scaling [18,19]. Early work [20,21,22] shows that augmenting language models with explicit external memory modules can improve long-term behavioral coherence, context maintenance, and personalized interaction. Subsequent research has explored memory-structure design and multi-turn dialogue modeling in a more systematic manner. LD-Agent [23] decouples event perception, persona extraction, and response generation, and combines a two-tier storage scheme of short-term and long-term memories with dynamic persona modeling to enhance cross-session contextual coherence and personalization. THEANINE [24] introduces a timeline-based memory structured by temporal and causal relations, organizing past events into a retrievable memory timeline to support state tracking and long-range dependency modeling in lifelong dialogue. RMM [25] incorporates both prospective and retrospective reflection into long-term personalized dialogue memory management, writing multi-granularity summaries into a personalized memory store and dynamically adapting retrieval strategies based on actually cited evidence, thereby forming a reflection-driven memory update loop.

Recent work has further shifted the focus toward the structured organization of long-term memory and its interaction-driven evolution. H-MEM [26] organizes memories into a multi-level hierarchical structure and employs index-based routing to balance representational capacity and access efficiency for long-horizon reasoning, whereas A-MEM [27] treats the memory system itself as an agentic module that automatically creates links and triggers updates to existing entries when new memories are written, yielding a memory graph that can be continuously reorganized based on experience. Taken together, these approaches move memory-augmented agents beyond simple external caches toward dynamically evolving memory systems. Along this line, our work further introduces a dual-process perspective at the levels of interaction and memory, enabling continual context refinement over explicit long-term memory.

2.3. Self-Evolving Agents

Self-evolution has become an important design paradigm for LLM-based agents, emphasizing that agents should autonomously improve their capabilities through continual interaction with the environment [28]. One major line of work introduces self-evaluation and self-correction loops at inference time: the agent diagnoses and revises candidate outputs via natural language reflection or tool-assisted verification, thereby improving factuality and robustness without updating underlying model parameters [29,30,31,32]. Such methods typically first produce an initial answer, then analyze the reasoning process and intermediate conclusions based on self-reflection, confidence or consistency checks, and feedback from external tools, and finally generate revised outputs, yielding a multi-round self-correcting inference procedure at test time. In parallel, another line of work [33,34,35] treats agent-generated interaction trajectories as training signals, leveraging policy-level reflection and search, experience distillation, or test-time finetuning to gradually improve overall decision-making ability over multiple tasks and extended deployment.

Context serves as a critical medium for self-evolving agents: through context engineering, an agent can continuously optimize its behavior while keeping model parameters frozen. SAGE [36] combines iterative feedback, reflective reasoning, and memory optimization based on the Ebbinghaus forgetting curve to adaptively adjust strategies and information retention in multi-task, long-horizon settings, thereby strengthening the agent’s sustained decision-making ability. The symbolic learning framework of Ou et al. [37] treats an agent as a symbolic network composed of prompts, tools, and their orchestration, and employs a symbolic optimizer that updates these “symbolic weights” during deployment based on data feedback, enabling data-driven self-evolution at the workflow and policy levels. ACE [38] restructures context as an evolvable playbook and, via modular procedures for generation, reflection, and curation, continually optimizes context configurations in both offline and online settings, achieving significant performance gains while relying primarily on natural execution feedback. In contrast to these approaches, we focus on explicit long-term memory and, within a dual-process framework, realize context self-evolution for long-horizon tasks through insertion, refinement, and pruning of interaction trajectories.

3. Methods

DPA augments a frozen backbone language model with an explicit, non-parametric memory that evolves online. As illustrated in Figure 2, DPA runs a fast reactive loop that answers each instance using a compact retrieved context, and a slower reflective loop that audits completed episodes and commits conservative, auditable delta writes back to memory. Instead of updating model parameters, DPA assigns credit to retrieved memory bullets (helpful vs. harmful) and uses these signals to refine future context selection while keeping $\theta$ fixed.

3.1. Overview

We consider a stream of instances $(q_1,\dots ,q_T)$ drawn from an unknown distribution ${\mathcal{D}}_{\mathrm{test}}$, where each $q_t$ may be a question, instruction, or task specification. At step t, the agent produces an output $a_t$ and may receive a feedback signal $y_t$ (e.g., a reference answer, verifier outcome, or environment reward). Since the backbone parameters $\theta$ are fixed, DPA adapts by evolving an external memory state ${\mathcal{M}}_t$ during deployment. Concretely, drawing on dual-process cognition [4], DPA couples a fast reactive procedure (System 1) that retrieves a compact context and generates $a_t$ under a strict budget, with a slower reflective procedure (System 2) that revisits completed episodes, assigns credit to the retrieved memory items, and applies conservative edits to ${\mathcal{M}}_t$. By decoupling execution from memory evolution, DPA remains responsive at inference time while avoiding unstable, large-scale memory rewrites over long streams.

The long-term memory store ${\mathcal{M}}_t$ consists of short natural-language entries (“bullets”) that encode reusable strategies, warnings, and compact corrections distilled from experience. We represent each entry $m_i\in {\mathcal{M}}_t$ as

$$m_i=({\mathrm{text}}_i,{\mathrm{id}}_i,s_i,h_i^{+},h_i^{-},r_i),$$

(1)

where ${\mathrm{text}}_i$ is the bullet content, ${\mathrm{id}}_i$ is a stable identifier, $s_i\in [0,1]$ is a curator-assigned quality score, $(h_i^{+},h_i^{-})$ are helpful/harmful counters used for credit assignment, and $r_i$ is a recency indicator updated whenever the entry is retrieved. Importantly, $s_i$ and $(h_i^{+},h_i^{-})$ play complementary roles. The quality score $s_i$ is assigned once at write time by the curator gate and remains fixed. By contrast, $(h_i^{+},h_i^{-})$ are runtime counters that accumulate as the agent uses an entry and observes its effects on outcomes. During retrieval (System 1), we select context based on net utility $\mathrm{usage}\left(m_i\right)=h_i^{+}-h_i^{-}$ and recency $r_i$ (Equation (4)), since empirical utility is more reliable than the initial write-time estimate. During memory maintenance, these signals guide pruning of consistently harmful entries and, under a fixed memory budget, determine which entries are retained when capacity is exceeded.

Algorithm 1 summarizes the end-to-end inference & evolution loop.

Algorithm 1 Dual-Process Agent (DPA) Inference & Evolution

Require: Stream ${\left\{q_t\right\}}_{t=1}^T$, frozen backbone $f_{\theta }$, memory ${\mathcal{M}}_1$, context size k, gate threshold

$\tau$, budget B

1:  for $t=1$ to T do

2:     // Process I: Fast Reaction

3:     ${\mathcal{R}}_t\leftarrow \mathrm{Retrieve}(q_t,{\mathcal{M}}_t)$

4:     ${\tilde{\mathcal{R}}}_t\leftarrow \mathrm{Rerank}\left({\mathcal{R}}_t\right)$                       // by relevance

5:     ${\mathcal{C}}_t\leftarrow {arg\; topk}_{m\in {\tilde{\mathcal{R}}}_t}(\mathrm{usage}\left(m\right),r_m)$            // usage-aware selection

6:     Refresh recency $r_m$ for each selected entry $m\in {\mathcal{C}}_t$

7:     $a_t\leftarrow f_{\theta }(q_t,{\mathcal{C}}_t)$                       // Generate response

8:     // Process II: Reflection

9:     if $\mathrm{Trigger}(q_t,a_t,y_t)$ then

10:         $({\mathcal{T}}_t,{\mathcal{I}}_t)\leftarrow R(q_t,a_t,{\mathcal{C}}_t,y_t)$

11:         for all $m\in {\mathcal{C}}_t$ do

12:             $h_m^{+}\leftarrow h_m^{+}+\mathbb{I}[{\mathcal{T}}_t\left(m\right)=\mathtt{helpful}]$

13:             $h_m^{-}\leftarrow h_m^{-}+\mathbb{I}[{\mathcal{T}}_t\left(m\right)=\mathtt{harmful}]$

14:         end for

15:         $\Delta {\mathcal{M}}_t\leftarrow G({\mathcal{I}}_t,q_t,a_t,{\mathcal{M}}_t;\tau )$

16:         ${\mathcal{M}}_{t+1}\leftarrow \mathrm{Maintain}({\mathcal{M}}_t\cup \Delta {\mathcal{M}}_t;B)$

17:     else

18:         ${\mathcal{M}}_{t+1}\leftarrow {\mathcal{M}}_t$

19:     end if

20:  end for

3.2. System 1: Retrieval-Augmented Answer Generation

Given $q_t$ and ${\mathcal{M}}_t$, System 1 constructs a compact context bank ${\mathcal{C}}_t$ (size k) and generates the response. We first retrieve and rerank a candidate set via a two-stage pipeline:

$${\tilde{\mathcal{R}}}_t=\mathrm{Rerank}\left(\mathrm{Retrieve}(q_t,{\mathcal{M}}_t)\right),$$

(2)

where $\mathrm{Retrieve}(·)$ performs dense retrieval using a bi-encoder to obtain an initial candidate pool, and $\mathrm{Rerank}(·)$ applies a cross-encoder reranker to refine the ranking by query-document relevance.

To incorporate long-horizon utility into context selection, we define a usage statistic based on accumulated credit:

$$\mathrm{usage}\left(m_i\right)=h_i^{+}-h_i^{-}.$$

(3)

After relevance-based reranking, we apply a lightweight usage-aware selection and choose the top-k entries according to the lexicographic key $(\mathrm{usage}\left(m\right),r_m)$:

$${\mathcal{C}}_t=\underset{m\in {\tilde{\mathcal{R}}}_t}{arg\; topk}(\mathrm{usage}\left(m\right),r_m).$$

(4)

We update $r_m$ whenever m is selected into ${\mathcal{C}}_t$ (Algorithm 1). The backbone model then produces

$$a_t=f_{\theta }(q_t,{\mathcal{C}}_t).$$

(5)

Optionally, the generator emits cited bullet identifiers to make the subsequent audit step attributable.

3.3. System 2: Reflection, Credit Assignment, and Conservative Commits

System 2 runs when an episode terminates (e.g., dialogue/task completion) and when a feedback signal $y_t$ becomes available. It has two responsibilities: (i) assign credit to the retrieved context items, and (ii) commit a small set of curated memory edits. We denote the reflector by $R(·)$ and the curator gate by $G(·)$.

Given the episode tuple $(q_t,a_t,{\mathcal{C}}_t,y_t)$, the reflector outputs bullet-level tags and a small set of candidate insights:

$$({\mathcal{T}}_t,{\mathcal{I}}_t)=R(q_t,a_t,{\mathcal{C}}_t,y_t),$$

(6)

where ${\mathcal{T}}_t$ assigns each retrieved bullet $m\in {\mathcal{C}}_t$ a tag in $\{\mathtt{helpful},\mathtt{harmful},\mathtt{neutral}\}$ and ${\mathcal{I}}_t$ contains distilled reusable corrections, strategies, or warnings. The reflector also emits structured diagnostics for analysis; only ${\mathcal{I}}_t$ is forwarded to the curator gate as candidate memory, while ${\mathcal{T}}_t$ updates the utility counters (see Appendix B for the full prompt template and JSON output schema). We then update the helpful/harmful counters for each $m\in {\mathcal{C}}_t$ using the tags:

$$h_m^{+}\leftarrow h_m^{+}+\mathbb{I}[{\mathcal{T}}_t\left(m\right)=\mathtt{helpful}],h_m^{-}\leftarrow h_m^{-}+\mathbb{I}[{\mathcal{T}}_t\left(m\right)=\mathtt{harmful}],$$

(7)

which yields a persistent utility signal that influences future context selection via Equation (4).

Candidate insights may be noisy, redundant with existing bullets, or inconsistent with the current memory. To stabilize long-horizon behavior, we apply a conservative curator gate $G(·)$ that evaluates each candidate insight $I_j\in {\mathcal{I}}_t$ and assigns it a quality score $c_j\in [0,1]$. Only insights with $c_j\ge \tau$ are committed to memory, where $\tau$ is a conservativeness threshold. The gate also detects near-duplicates (merged into canonical entries) and conflicts (resolved via rewriting). Formally, the resulting localized increment is

$$\Delta {\mathcal{M}}_t=G({\mathcal{I}}_t,q_t,a_t,{\mathcal{M}}_t;\tau ),$$

(8)

which contains only high-quality, non-redundant insights that pass the threshold.

Finally, memory is updated by

$${\mathcal{M}}_{t+1}=\mathrm{Maintain}\left({\mathcal{M}}_t\cup \Delta {\mathcal{M}}_t;B\right),$$

(9)

where B is a fixed budget (max bullets). $\mathrm{Maintain}(·)$ performs lightweight stabilization operations, including (i) normalization-based deduplication, (ii) pruning of persistently harmful entries (when $h_m^{-}-h_m^{+}$ exceeds a threshold), and (iii) budget-based truncation that prioritizes high-quality and high-utility bullets. All updates are non-parametric and do not require modifying $\theta$. Prompt schemas and output constraints for each component are provided in Appendix B.

4. Experiments

We evaluate DPA on six benchmarks spanning factual QA, multi-hop reasoning, instruction following, and mathematical reasoning. Our experiments address three questions: (i) whether DPA improves over static baselines and existing adaptive methods, (ii) which System 2 components contribute most to performance, and (iii) whether memory evolution remains stable over long streams.

4.1. Experimental Setup

Datasets. We select six benchmarks that collectively test different aspects of memory-augmented reasoning: knowledge accuracy, multi-step inference, instruction compliance, temporal sensitivity, and robustness to distraction.

TruthfulQA [39] evaluates whether models avoid generating false but plausible-sounding answers. We use the binary-choice format under the January 2025 knowledge cutoff. Distinguishing truth from plausible misconceptions requires nuanced world knowledge, so we evaluate this benchmark on GPT-5.1.

StrategyQA [40] requires implicit multi-step reasoning to answer yes/no questions (e.g., “Did Aristotle use a laptop?”). Memory entries capturing successful decomposition strategies can transfer across questions with similar implicit structure. We evaluate this dataset using GPT-5.1.

IFEval [41] tests instruction-following with verifiable constraints such as word count limits, formatting requirements, and content restrictions. We report prompt-level accuracy (all constraints satisfied). We evaluate on both GPT-5.1 and Llama-3.1-8B for this dataset.

HotpotQA [42] is a multi-hop QA benchmark requiring reasoning over multiple supporting documents. We use the validation split and report token-level F1 on the final answer. Unlike the original open-domain setting where models can retrieve from Wikipedia or provided supporting documents, we adopt a closed-book setting where the model receives only the question and must answer based on its parametric knowledge and accumulated memory (i.e., we perform no external document/corpus retrieval). This setting is more challenging and is intended to test whether DPA can distill reusable multi-hop reasoning knowledge into memory to improve downstream performance, rather than to evaluate external document retrieval capability. We evaluate on GPT-5.1.

FreshQA [43] contains time-sensitive factual questions whose answers may change over time (e.g., “Who is the current CEO of Twitter?”). We follow the FreshEval protocol with a deterministic LLM judge (temperature 0). We evaluate this benchmark using Llama-3.1-8B, and use GPT-5.1 as the judge model. This benchmark tests whether DPA can learn meta-strategies for handling temporal uncertainty.

GSM-IC [44] augments grade-school math problems with irrelevant context designed to distract the model. We report Exact Match (EM) on the final numerical answer. The underlying math is straightforward, so we evaluate on Llama-3.1-8B to test whether memory can help smaller models learn to ignore distracting information.

Baselines. We compare against five systems: Vanilla (direct prompting), Abstention Prompting [45,46,47] (instructing the model to reply “Unknown” when uncertain), Self-Refine [30] (iterative refinement without external memory), SwiftSage [48] (dual-process fast/slow reasoning), and Dynamic Cheatsheet [49] (memory-augmented hint accumulation; we use the Cumulative Memory configuration from the original paper). These baselines span prompting-only, self-correction, and memory-augmented approaches, allowing us to isolate the contribution of DPA’s specific design choices.

Implementation. We evaluate on GPT-5.1 via API and Llama-3.1-8B locally. For Llama-3.1-8B, we utilize vLLM for efficient inference. For memory-enabled methods, we fix a deterministic stream order (seed 42) to ensure reproducibility. During each episode, the model receives only the current question $q_t$ (plus retrieved context for memory-based methods); no ground truth is available during answer generation. Feedback is revealed only after the episode and is then used for evaluation and, where applicable, memory updates. Memory-augmented methods receive the same post-episode feedback signal under each task, while non-memory baselines do not use cross-episode updates by design.

Table 1. Hyperparameter settings for DPA.

Hyperparameter	Value
System 1: Retrieval & Generation
Context bank size k	5
Initial retrieval size	50
Embedder model	all-MiniLM-L6-v2
Reranker model	BGE-reranker-large
Reranker batch size	16
QA temperature	0.0 a
Max output tokens (QA)	512 b
System 2: Reflection & Curation
Insights per episode	2 c
Curator gate threshold $\tau$	0.6 d
Max sentences per bullet	3
Reflector temperature	0.0
Max output tokens (reflector)	384
Memory Management
Memory budget B	1000
Pruning threshold ($h_m^{-}-h_m^{+}$)	2

^a GSM-IC and IFEval use temperature 0.2. ^b StrategyQA and TruthfulQA use 32 tokens; IFEval uses 768 tokens. ^c FreshQA uses 1 insight per episode. ^d Aligned with the curator scoring rubric (Appendix B): scores $\le 0.5$ indicate vague or non-actionable insights (rejected), while $\ge 0.6$ indicates salvageable content worthy of rewriting or acceptance. This threshold represents the natural boundary between clearly low-quality and potentially useful entries in the rubric.

summarizes key hyperparameters; we use consistent settings across benchmarks with minor task-specific adjustments noted in the footnotes.

4.2. Main Results

Table 2. Comparison with baselines on GPT-5.1. Best results are bolded; second-best are underlined. Overall is the macro-average across datasets.

Method	TruthfulQA	StrategyQA	IFEval	HotpotQA	Overall
	Acc.	Acc.	Acc.	F1	Avg.
vanilla	89.4	80.6	89.9	9.3	67.3
Abstention Prompting	89.9	79.5	92.9	10.1	68.1
Self-Refine	83.5	71.6	84.5	24.4	66.0
SwiftSage	83.5	77.6	93.1	11.6	66.5
Dynamic Cheatsheet	87.5	76.1	54.5	58.5	69.2
DPA (Ours)	92.3	79.2	90.6	47.1	77.3

presents results on GPT-5.1 across four benchmarks. DPA achieves the highest overall performance (77.3% macro-average). It attains the best accuracy on TruthfulQA (92.3%) and the second-best F1 on HotpotQA (47.1%), indicating strong performance on both factual accuracy and multi-hop reasoning. While SwiftSage achieves slightly higher accuracy on IFEval (93.1% vs. 90.6%), DPA remains the strongest method on average across the evaluated GPT-5.1 settings.

Table 3. Comparison with baselines on Llama-3.1-8B. Best results are bolded; second-best are underlined. Overall is the macro-average across datasets.

Method	FreshQA	GSM-IC	IFEval	Overall
	Acc.	EM	Acc.	Avg.
vanilla	21.5	91.7	69.3	60.8
Abstention Prompting	22.3	91.1	73.2	62.2
Self-Refine	21.7	82.9	64.5	56.4
SwiftSage	19.5	76.0	56.0	50.5
Dynamic Cheatsheet	15.0	53.2	34.2	34.1
DPA (Ours)	34.7	91.8	74.7	67.0

reports results on Llama-3.1-8B. DPA achieves the strongest overall performance across the three evaluated benchmarks: FreshQA (34.7%), GSM-IC (91.8%), and IFEval (74.7%). The improvement over vanilla is particularly notable on FreshQA (+13.2%) and IFEval (+5.4%), suggesting that accumulated memory entries capture reusable strategies for time-sensitive QA and constraint satisfaction.

4.3. Learning Dynamics

A key design goal of DPA is stable memory evolution over extended streams. Figure 3 shows cumulative accuracy (left) and memory size (right) on IFEval as functions of episode index. Both backbones exhibit stable learning dynamics, and memory growth remains controlled, indicating that the curator gate filters redundant or low-quality insights.

The memory store grows sub-linearly, indicating that the curator gate successfully filters redundant or low-quality insights. This controlled growth validates the conservative commit strategy (Section 3.3), which prioritizes high-confidence updates over aggressive expansion.

4.4. System 2 Component Ablations

To isolate the contribution of each System 2 component, we disable one component at a time while keeping all other settings fixed. Specifically, we evaluate three ablation variants: w/o Refl. disables the reflector, so memory entries are committed directly without structured outcome analysis or credit assignment; w/o Gate disables the curator quality gate, allowing all candidate insights to pass through without filtering; and w/o Prune disables the pruning mechanism, so persistently harmful entries (high $h^{-}-h^{+}$) are never removed.

Table 4. System 2 component ablations (GPT-5.1). Best results are bolded; second-best are underlined.

Variant	TruthfulQA	StrategyQA	IFEval	HotpotQA	Overall
	Acc.	Acc.	Acc.	F1	Avg.
Vanilla	89.4	80.6	89.9	9.3	67.3
w/o Refl.	90.9	81.8	89.2	36.8	74.7
w/o Gate	91.4	80.6	88.7	40.6	75.3
w/o Prune	92.8	79.0	88.0	40.0	75.0
Full	92.3	79.2	90.6	47.1	77.3

and

Table 5. System 2 component ablations (Llama-3.1-8B). Best results are bolded; second-best are underlined.

Variant	FreshQA	GSM-IC	IFEval	Overall
	Acc.	EM	Acc.	Avg.
Vanilla	21.5	91.7	69.3	60.8
w/o Refl.	29.5	90.4	74.1	64.7
w/o Gate	29.0	89.7	73.4	64.0
w/o Prune	29.0	89.2	73.2	63.8
Full	34.7	91.8	74.7	67.0

present results for GPT-5.1 and Llama-3.1-8B.

Disabling the reflector (w/o Refl.) causes notable degradation on most benchmarks. On Llama, FreshQA accuracy drops from 34.7% to 29.5% ($-5.2$%), confirming that explicit outcome analysis is essential for generating useful memory updates. Interestingly, on GPT-5.1, the w/o Refl. variant remains competitive on StrategyQA (81.8%), suggesting that stronger backbones may partially compensate for the lack of structured reflection.

Removing the curator gate (w/o Gate) degrades performance, particularly on Llama where the overall average drops from 67.0% to 64.0%. Without quality filtering, memory accumulates noisy or conflicting entries that mislead subsequent retrieval—underscoring the importance of conservative writes for long-horizon stability.

Disabling pruning (w/o Prune) yields the lowest overall performance on Llama (63.8%), with consistent degradation across benchmarks. The pruning mechanism removes persistently harmful entries based on credit assignment (Equation (7)); its absence allows detrimental memories to persist and negatively influence context selection.

4.5. Qualitative Analysis

Figure 4 illustrates how accumulated memory entries influence model behavior through three representative cases where DPA succeeds but the vanilla baseline fails.

In Case 1 (FreshQA), the question contains a false premise—the 2022 Winter Olympics were held in Beijing, not Russia. The vanilla baseline catastrophically fails, entering a repetition loop. DPA retrieves meta-strategies about verifying premises and produces a stable, correct response.

In Case 2 (TruthfulQA), the question asks which U.S. state produces the most peaches. Although Georgia is nicknamed “The Peach State”, California actually leads in production. Vanilla selects the culturally prevalent but incorrect answer (Georgia), while DPA retrieves evaluative heuristics that encourage skepticism toward surface-level associations.

In Case 3 (IFEval), the instruction forbids using “can” and “ride” in a story about learning to bike. Vanilla violates the constraint; DPA successfully avoids both words by using alternatives such as “glided” and “pedaling”.

These cases illustrate three types of transferable knowledge: meta-strategies for premise verification, skepticism toward common misconceptions, and procedural awareness for constraint satisfaction. Crucially, none of these insights encode task-specific facts; they capture reasoning patterns that generalize across benchmarks.

4.6. Cost Analysis

Figure 5 presents the token cost breakdown (GPT-5.1 DPA runs; same four benchmarks as Table 2). Reflection and curation account for 13.5% (TruthfulQA), 15.6% (IFEval), 38.4% (StrategyQA), and 62.3% (HotpotQA) of total tokens, reflecting both task difficulty and how often System 2 is triggered. This overhead can be reduced further by triggering System 2 selectively (e.g., only on incorrect predictions or low-confidence outputs).

5. Discussion

Our experiments reveal a trade-off in memory-augmented inference with frozen backbones: persistent context can mitigate recurring failure modes, but only when it remains query-aligned, compact, and robust to noisy writes. DPA improves performance by storing a small set of reusable strategies and retrieving them selectively. In contrast, methods that inject a growing, query-agnostic state can introduce interference, with effects that are more pronounced for smaller backbones.

5.1. On the Interaction Between Memory Design and Model Scale

We observe notable performance differences between memory-augmented approaches when evaluated on models of varying scale. Specifically, on Llama-3.1-8B (Table 3), we find that while DPA maintains consistent improvements over the vanilla baseline, Dynamic Cheatsheet (DC) [49] exhibits performance below the baseline across all three benchmarks, with average accuracy dropping from 60.8% to 34.1%. This divergence is particularly interesting given that both methods maintain persistent state across episodes, suggesting that architectural choices in memory management may interact differently with backbone model scale.

We identify three potential factors that may contribute to this phenomenon, which offer insights into the design space of memory-augmented systems for resource-constrained settings:

(1) Trade-offs in context injection strategies. DC employs a design where the entire evolving cheatsheet is provided to every episode, independent of query content. While this ensures comprehensive coverage, it can result in contexts containing information spanning multiple domains. In our experiments, cheatsheets often grow to thousands of characters. For smaller models with more limited reasoning capabilities, distinguishing task-relevant information from a large heterogeneous context may present additional challenges, potentially leading to outputs that deviate from expected formats or violate task-specific constraints.

(2) Sensitivity to output format conventions. DC uses a structured response template together with a dedicated extractor. In our experiments on Llama, strict compliance with the template is not always reliable. On IFEval, the extractor occasionally fails to identify a valid final answer. This is especially important for instruction-following tasks, where small formatting deviations can materially change evaluation results. Extra headings, meta-commentary, or minor structural changes may be sufficient to break extraction. In these cases, the resulting errors tend to reflect surface-form constraints, including punctuation, casing, and length, rather than semantic content alone.

(3) Considerations for memory update policies. DC updates its cheatsheet after each episode without explicit quality filtering or credit assignment mechanisms. While this ensures comprehensive coverage of experiences, it may allow less informative or task-misaligned entries to persist. Over extended interaction streams, such entries could be repeatedly reintroduced, potentially creating feedback loops between generation patterns and memory state.

These observations suggest an important design consideration: as model scale decreases, the benefits of selective retrieval and quality-controlled memory updates may become more pronounced. More broadly, unfiltered cumulative memory appears substantially more fragile than selective retrieval with conservative curation in resource-constrained settings, particularly on tasks with strict output constraints. We emphasize that these findings reflect specific experimental conditions and implementation choices, and different design decisions or hyperparameter configurations might yield different outcomes.

5.2. When Is Memory Helpful—and When Can It Hurt?

The ablations on GPT-5.1 indicate that memory is not uniformly beneficial; its value depends on whether System 2 produces high-signal, reusable updates and whether the system prevents low-quality writes from accumulating (Table 4).

Memory helps when a small set of strategies generalizes. On some tasks, a handful of persistent rules can correct recurring blind spots. For example, on IFEval the full system admits very few memory entries over hundreds of episodes, yet achieves higher prompt-level accuracy than variants without reflection or without quality control. This supports the view that memory quality can dominate memory quantity: compact, reusable strategies can improve constraint satisfaction without large stores.

Memory hurts when updates are noisy or miscalibrated. Two failure modes are visible in the ablations. First, removing the curator gate causes the system to commit nearly all proposed updates, increasing memory size and reducing performance on three of the four GPT-5.1 benchmarks; on HotpotQA, F1 drops from 47.1 to 40.6. Second, disabling pruning degrades long-horizon performance (HotpotQA: 47.1 → 40.0), suggesting that some memories are net harmful and should be down-weighted or removed when they repeatedly correlate with errors.

System 2 is not always worth triggering. Removing the reflector improves StrategyQA (81.8 vs. 79.2), even though it hurts the other benchmarks. When the backbone is already strong and the feedback signal is coarse, reflection may generate low-signal heuristics that compete with the base model’s decision rule. Binary correctness on short answers provides limited information for meaningful reflection. In such regimes, a conservative policy may be preferable: writing less or triggering System 2 only under stronger evidence of uncertainty. In the current implementation, System 2 is triggered after every episode for simplicity and reproducibility. However, several adaptive triggering strategies could reduce the 13–62% System 2 token overhead reported in our cost analysis (Figure 5) while preserving the benefits where they matter most: (i) confidence-based triggering, where System 2 activates only when the backbone’s output probability or self-reported confidence falls below a threshold; (ii) consistency-based triggering, where multiple responses are sampled and reflection is triggered only when they disagree, signaling genuine uncertainty; and (iii) constraint-based triggering, where a lightweight verifier checks whether the output satisfies known structural constraints before committing to full reflection. Designing and evaluating such adaptive triggers is an important direction for future work.

5.3. Limitations and Open Problems

Order dependence and calibration. Memory is learned sequentially and is therefore sensitive to stream order and the gate threshold. The ablations suggest that the optimal write policy can be task-dependent: for some datasets, aggressive reflection can inject noise (StrategyQA), while for others pruning and gating are essential (HotpotQA). Future work should quantify variance across multiple random seeds and explore adaptive gate calibration.

Scalability and scope. Our memory is a lightweight strategy bank, not a knowledge base: it stores procedural heuristics and error-avoidance rules, and may not help when improvements require new external facts or long-form, compositional skills. Moreover, on high-variance tasks the store can grow to hundreds of entries, raising questions about retrieval saturation, compression, and long-term stability.

Cross-task transfer and robustness. Our experiments maintain an independent memory store per benchmark, and we do not study cross-task transfer. Although some meta-level heuristics may generalize, others are likely task-specific and can induce negative transfer when retrieved out of context, especially under distribution shift. Systematically evaluating cross-task memory sharing and developing safeguards against such negative transfer remain important directions for future work.

Open-domain applicability. Our benchmarks involve well-defined, single-turn tasks with clear feedback signals. Deploying DPA in open-domain settings such as real-time dialogue or open-ended reasoning would introduce additional challenges, including sparse and noisy feedback, unbounded interaction streams, and stronger cross-domain retrieval interference. Addressing these issues would likely require more aggressive memory compression, better session-level memory management, and topic-aware organization of long-term memory. Despite these challenges, DPA’s core mechanisms (selective retrieval, conservative curation, credit-based pruning) are domain-agnostic by design and should transfer to open-domain settings with appropriate adaptations.

Security and privacy considerations. Memory-augmented LLM agents introduce security and privacy concerns that are distinct from stateless models. An adversary may influence the input stream or feedback signal to inject persistent harmful memories, creating an attack surface related to backdoor, prompt-injection, and data-poisoning risks [50,51]. In addition, stored reasoning artifacts may inadvertently expose sensitive information from prior interactions. Mitigating these risks requires stronger memory access control, storage-layer filtering, and privacy-preserving memory management.

6. Conclusions

We presented the Dual-Process Agent (DPA), a framework for continual context refinement that keeps the backbone LLM frozen while evolving an explicit, editable long-term memory. DPA couples a fast System 1 that answers each instance with retrieval-augmented context and a conservative System 2 that audits outcomes, assigns credit to retrieved memories, and commits curated, localized edits to memory.

Across six benchmarks, DPA improves performance over static prompting and competitive baselines by accumulating reusable, deployment-time memories that refine future context selection. These findings also suggest several concrete directions for future work.

Adaptive gate calibration. The current curator gate uses a fixed threshold $\tau$, but the optimal write policy is task-dependent. Learning the gate threshold from memory state or task difficulty could make the system more aggressive when reflection is consistently useful and more conservative when it risks injecting noise.

Memory compression and cross-task transfer. Our experiments maintain independent memory stores per benchmark. Compressing task-specific entries into higher-level meta-strategies could support knowledge sharing across tasks while reducing redundancy and negative transfer.

Selective System 2 triggering. Triggering reflection after every episode incurs unnecessary overhead when the model is already confident and correct. Confidence-based or consistency-based triggers could reduce computational cost while preserving the benefits of reflection where they matter most.

Integration with parameter-efficient fine-tuning. DPA currently keeps the backbone frozen. Combining memory evolution with lightweight parameter updates such as LoRA [52] could allow persistent high-confidence patterns to be internalized into model weights while preserving the flexibility of external memory.

Together, these directions aim to make continual context refinement reliably beneficial across a wider range of backbones, tasks, and deployment regimes.