Graph-Structured Persistent Memory for Efficient LLM-Based Computer Use Agents

Abstract

Large language model (LLM)-driven computer use agents (CUAs) automate graphical user interface (GUI) tasks but often re-solve previously encountered subtasks, increasing token use and latency. We address this limitation with a directed graph-based persistent memory in which nodes represent observable GUI states and edges encode executable action sequences. We formalize the memory-augmented agent as $\mathcal{S}=⟨A,\mathrm{\Sigma },\mathcal{G},\delta ,\pi ,\mathrm{\Phi }⟩$, define task reachability and memory-coverage conditions inspired by functional stability theory, and derive token-cost efficiency bounds. In control-theoretic terms, the Manager–Worker architecture can be interpreted as a closed-loop system where memory provides experience-based feedback; this interpretation is used as an analogy rather than a full model-reference adaptive control proof. Experiments on OSWorld show that the proposed agent cuts both the LLM token consumption and execution time by about 50% versus a memoryless baseline while preserving comparable success rates (≈36.9% on 15-step and ≈46.9% on 50-step tasks). The demonstrated contribution is therefore operational efficiency through reusable graph memory, not a claim of improved task success or classical Lyapunov stability.

1. Introduction

In the era of increasingly autonomous information systems, the efficiency, recoverability, and controllability of software agents that interact with complex, dynamic environments constitute fundamental challenges at the intersection of control theory, decision-making science, and artificial intelligence [1,2,3]. Large language model (LLM)-driven computer use agents (CUAs) represent a compelling instantiation of this challenge: they must reliably interpret graphical user interface (GUI) states, plan multi-step action sequences, and execute them in real desktop and web environments [4]. Despite significant progress, contemporary CUAs suffer from a critical operational-reuse deficit—they lack persistent procedural memory and consequently “reinvent the wheel” on recurring subtasks, repeatedly reasoning through previously solved subproblems [4,5]. This redundancy leads to excessive token consumption, increased latency, and weaker operational consistency over extended operation.

The concept of functional stability—broadly understood as the ability of an information system to perform its assigned tasks over a necessary period, at least partially, despite structural or environmental perturbations—has been extensively studied in the context of network architectures and distributed systems [1,2,6]. A classical criterion for functional stability requires that the vertex connectivity ${\lambda }_G\left(G\right)$ and edge connectivity $\chi \left(G\right)$ of the system’s structural graph G both satisfy ${\lambda }_G\left(G\right)\ge 2\wedge \chi \left(G\right)\ge 2$, ensuring a minimal reserve against single-point failures [1,7]. Although originally formulated for communication networks, the underlying principle—that structural redundancy in a system’s knowledge graph can support recoverability under explicitly stated assumptions—motivates, but does not by itself prove, analogous design criteria for the memory architectures of autonomous agents.

From a control-theoretic perspective, contemporary CUA architectures such as Agent S2 [5] employ a hierarchical Manager–Worker decomposition that parallels classical cascade control [8]. The Manager interprets user instructions and decomposes tasks (the controller), while the Worker executes low-level GUI actions (the plant/actuator). However, without persistent memory, this control loop is entirely open: each task execution is independent, with no feedback from prior experience. The introduction of a memory graph closes this loop in an engineering sense, creating an experience-based feedback mechanism that can reduce repeated planning. We use model-reference adaptive control terminology [9,10] only as an interpretive analogy: the present architecture does not specify a continuous adaptive law or Lyapunov proof for a closed-loop plant.

The agent’s operational decision at each step—whether to retrieve a memorized action trajectory or invoke fresh LLM planning—constitutes a formal decision-making problem under uncertainty. This binary choice balances exploitation (reusing known solutions) against exploration (generating novel plans), a trade-off well studied in reinforcement learning and multi-criteria decision theory [11,12]. Our graph memory architecture provides a structured decision support system that reduces both the uncertainty and the computational cost of this decision by maintaining an indexed repository of verified action sequences.

Several prior works have explored memory mechanisms for GUI agents, though with notable limitations. MobileGPT/MemoDroid [13] augments a mobile automation agent with hierarchical memory, achieving approximately 70% reduction in LLM usage, but operates primarily as an action replay log without structured state representation. AppAgentX [14] compresses action trajectories into reusable macros, yet organizes knowledge as chronological chains rather than a navigable state graph. Agent S2 [5] advances perception and planning through a generalist–specialist composition and Mixture-of-Grounding but retains no long-term memory between tasks. Recent surveys on LLM agent memory [4,15] and specialized memory systems such as MemGPT [16] and A-Mem [17] highlight the general importance of persistent, structured memory but do not address the specific requirements of GUI state–action representation. Compared with these approaches, GBMA treats memory as an explicit state–action graph: retrieval is graph traversal over observed GUI states, tool generation abstracts repeated paths into parameterized descriptors, and maintenance operations prune or merge stale graph elements. This positioning separates the proposed architecture from general episodic-memory stores, which retrieve textual experience, and from tool-using agents, which invoke external functions but typically do not retain a persistent graph of prior GUI transitions. Related work by the present authors on retrieval-augmented generation systems in a domain-specific analytical setting further underscores the value of structured retrieval and reusable knowledge organization [18]. Recent distributed-control work on reinforcement-learning consensus in Takagi–Sugeno fuzzy multi-agent systems [19] also motivates the broader view that learning-based agents require explicit coordination and stability-oriented design, although our setting concerns GUI automation rather than continuous multi-agent consensus. This broader perspective motivated the graph-based architecture presented here.

The contribution of this paper is threefold. First, we propose a directed graph-based persistent memory architecture for CUAs in which nodes correspond to observable GUI states, edges encode executable action sequences, and parameterized task descriptors enable hierarchical reuse at both low-level (individual actions) and high-level (multi-step procedures) granularities. Second, we provide a formal mathematical framework that models the memory-augmented agent as an algebraic tuple, defines task reachability and coverage conditions inspired by functional stability theory, and derives efficiency bounds on token consumption. Third, we present experimental validation on the OSWorld benchmark [20], demonstrating that the memory-augmented agent achieves approximately 50% reductions in both token cost and execution time while maintaining comparable task success rates. Thus, the empirical contribution is primarily efficiency and operational memory reuse rather than higher task success or a classical stability guarantee.

The remainder of this paper is organized as follows. Section 2 presents the formal mathematical model and system design, including the tuple-based formalization, reachability and coverage definitions, and the cost model. Section 3 reports the experimental evaluation. Section 4 interprets the results through reachability, efficiency, and control-analogy frameworks. Section 5 summarizes the contributions and outlines future work.

2. Materials and Methods

2.1. Formal Model of the Memory-Augmented Agent

We define the memory-augmented computer use agent as a formal system:

$$\mathcal{S}=\langle A,\mathrm{\Sigma },\mathcal{G},\delta ,\pi ,\mathrm{\Phi }\rangle ,$$

(1)

where:

• A is the finite set of available actions, encompassing both primitive GUI operations (mouse clicks, keyboard inputs, scrolling) and composite tool invocations (parameterized macros such as SearchDrive(query));
• $\mathrm{\Sigma }$ is the set of observable GUI states, where each state $\sigma \in \mathrm{\Sigma }$ is characterized by a visual screenshot, OCR-extracted text, and accessibility-tree metadata;
• $\mathcal{G}=(N,E,D)$ is the memory graph—a directed graph with node set N, edge set $E\subseteq N\times N$, and task descriptor set D;
• $\delta :N\times A\to N$ is the deterministic state transition function that governs action replay from memory;
• $\pi :\mathrm{\Sigma }\times \mathcal{G}\to A\cup \{\perp \}$ is the decision policy, which returns either a memorized action $a\in A$ or the null symbol ⊥ to trigger fresh LLM planning;
• $\mathrm{\Phi }:\mathrm{\Sigma }\to N\cup \{⌀\}$ is the state recognition function that maps a current observation to its corresponding memory graph node, or to ⌀ if no matching node exists.

2.2. Memory Graph Structure

The memory graph $\mathcal{G}=(N,E,D)$ is realized as a persistent directed graph stored in a graph database. Formally, each node $n\in N$ is a tuple:

$$n=(x,{\mathcal{T}}_n,A_n),$$

(2)

where x denotes the visual representation of the screen (a perceptual hash of the screenshot combined with extracted text features), ${\mathcal{T}}_n\subseteq D$ is the set of task descriptors executable from this state, and $A_n\subseteq A$ is the set of actions available at this node.

Each directed edge $e\in E$ takes the form

$$e=(n_i,n_j,a_{ij},c_{ij}),$$

(3)

where $n_i,n_j\in N$ are the source and target nodes, $a_{ij}\in A$ is the action (or action sequence) that transitions the system from state $n_i$ to state $n_j$, and $c_{ij}\in {\mathbb{R}}_{\ge 0}$ is the associated execution cost (measured in tokens or time). In practice, $a_{ij}$ may be a Python pyautogui script encoding a sequence of mouse clicks and keyboard inputs.

A task descriptor $d\in D$ is a parameterized functional definition:

$$d=(\mathrm{name},\mathrm{params},n_{\mathrm{start}},n_{\mathrm{end}},P_d),$$

(4)

where $\mathrm{name}$ is a human-readable identifier (e.g., ExportReport), $\mathrm{params}$ is a parameter schema, $n_{\mathrm{start}}$ and $n_{\mathrm{end}}$ are the initial and terminal nodes, and $P_d=(e_1,e_2,\dots ,e_k)$ is the ordered path of edges constituting the procedure. This hierarchical structure allows complex multi-step workflows to be stored as single callable procedures.

The graph is initialized with a single root node corresponding to the desktop home screen. As the agent completes tasks, new nodes and edges are added, and the Tool Generation module abstracts frequently traversed paths into reusable task descriptors, causing $\mathcal{G}$ to grow and become increasingly dense over time (Figure 1).

2.3. Reachability and Coverage Conditions for the Memory Graph

Drawing on the theory of functional stability for information systems [1,2,7], we define reachability and coverage conditions for the memory-augmented agent system. These conditions are weaker than Lyapunov or input–output stability: they characterize whether reusable task paths exist in the memory graph, not whether a continuous dynamical system is stable under perturbations.

Definition 1

(Task Reachability of the Memory Graph). The memory graph $\mathcal{G}=(N,E,D)$ is said to be task-reachable with respect to a task set $\mathcal{T}$ if, for every task $\tau \in \mathcal{T}$, there exists at least one path $P_{\tau }=(n_0,n_1,\dots ,n_k)$ in $\mathcal{G}$ from a reachable initial node $n_0$ to a goal node $n_k$ such that executing the edge actions along $P_{\tau }$ accomplishes τ.

This definition is closer to a reachability condition than to classical control-theoretic stability. It parallels the graph-connectivity intuition behind the classical requirement ${\lambda }_G\left(G\right)\ge 2$ for information systems [7], but it is adapted to the directed, task-oriented nature of the memory graph and should not be read as a Lyapunov stability claim. We further define a quantitative coverage measure.

Definition 2

(Memory Coverage Ratio). For a task set $\mathcal{T}$ and memory graph $\mathcal{G}$, the memory coverage ratio is

$$\rho (\mathcal{G},\mathcal{T})={\displaystyle \frac{\left|\right\{\tau \in \mathcal{T}:\exists P_{\tau }\mathrm{in}\mathcal{G}\left\}\right|}{\left|\mathcal{T}\right|}}.$$

(5)

The system is $\alpha$-covered if $\rho (\mathcal{G},\mathcal{T})\ge \alpha$ for a prescribed threshold $\alpha \in$ [0, 1]. The threshold α is therefore a deployment parameter: safety-critical or highly repetitive workflows should use a higher target coverage level, while open-ended exploratory tasks may tolerate lower coverage because fallback planning remains available.

Definition 3

(Stabilization of Memory Utilization). Let $T=(t_1,t_2,\dots )$ be a sequence of tasks presented to the agent. Define the memory utilization ratio after k tasks as

$${\mu }_k={\displaystyle \frac{{\sum }_{i=1}^k{s}_i^{\mathrm{mem}}}{{\sum }_{i=1}^k{s}_i^{\mathrm{total}}}},$$

(6)

where $s_i^{\mathrm{mem}}$ is the number of steps in task $t_i$ served from memory and $s_i^{\mathrm{total}}$ is the total number of steps. The memory system exhibits stabilizing memory utilization if $\left\{{\mu }_k\right\}$ is eventually non-decreasing and bounded above by some ${\mu }^{*}<1$:

$${\mu }_k\le {\mu }_{k+1}\le {\mu }^{*}\forall k\ge k_0,$$

(7)

 for some warm-up period $k_0\in \mathbb{N}$.

This criterion expresses procedural reuse over a recurring task stream. It is analogous to the settling of an adaptive system [9,10], but it does not establish parameter convergence or closed-loop Lyapunov stability; the upper bound ${\mu }^{*}<1$ reflects the irreducible novelty in any realistic task distribution.

2.4. Perturbation Sensitivity and Graceful Degradation

A key design goal of the memory-augmented system is graceful degradation under perturbation, analogous to resilience considerations in functionally stable information systems [6,21], but this goal is treated as an engineering design criterion rather than as a formal robustness guarantee. Let ${\sigma }^{\prime }$ denote a perturbed observation (e.g., a modified UI layout) such that $\mathrm{\Phi }\left({\sigma }^{\prime }\right)=⌀$ (the state recognition function fails to match any known node). In this case, the decision policy defaults to

$$\pi ({\sigma }^{\prime },\mathcal{G})=\perp ,$$

(8)

triggering the standard LLM planning procedure. Under an ideal fallback assumption, the agent is intended to retain the baseline planning path: the memory graph provides an efficiency opportunity rather than replacing baseline planning. Formally, if $R_{\mathrm{mem}}\left(\tau \right)$ and $R_{\mathrm{base}}\left(\tau \right)$ denote the success indicators for a task $\tau$ under the memory-augmented and baseline agents, respectively, then

$$\mathbb{E}\left[R_{\mathrm{mem}}\left(\tau \right)\right]\ge \mathbb{E}\left[R_{\mathrm{base}}\left(\tau \right)\right]\forall \tau \in \mathcal{T},$$

(9)

Inequality (9) is therefore a conditional design target under policy-class containment, not an unconditional empirical guarantee: every execution path available to the baseline agent remains available to the memory-augmented agent through $\pi ({\sigma }^{\prime },\mathcal{G})=\perp$, while the memory agent additionally has access to verified, potentially more efficient trajectories. In practice, this bound can fail if state recognition returns an incorrect node, if a stored action sequence becomes stale, or if retrieval overhead changes the execution context; these failure modes are discussed in Section 4.4.

2.5. Token Cost Model

We define the token cost function for a task $\tau$ as the additive cost of its executed steps:

$$C\left(\tau \right)={\displaystyle \sum _{i=1}^{\left|\tau \right|}}c\left(s_i\right),$$

(10)

where $\left|\tau \right|$ is the number of steps in task $\tau$ and $c\left(s_i\right)$ is the token cost of step $s_i$. The additive form is chosen because commercial LLM APIs charge approximately linearly in input and output tokens per invocation, and because GUI actions decompose naturally into sequential Manager–Worker steps. Non-token overheads, such as database lookup and GUI latency, are measured separately in Section 3.4. For the baseline agent, each step requires full LLM reasoning:

$$C_{\mathrm{base}}\left(\tau \right)={\displaystyle \sum _{i=1}^{\left|\tau \right|}}{c}_{\mathrm{LLM}}\left(s_i\right),$$

(11)

where $c_{\mathrm{LLM}}\left(s_i\right)$ includes prompt construction, chain-of-thought reasoning, and action generation.

For the memory-augmented agent, a step served from memory incurs only a retrieval cost $c_{\mathrm{ret}}\ll c_{\mathrm{LLM}}$:

$$C_{\mathrm{mem}}\left(\tau \right)={\displaystyle \sum _{i=1}^{\left|\tau \right|}}\left[⊮[\pi ({\sigma }_i,\mathcal{G})\ne \perp ]·c_{\mathrm{ret}}+⊮[\pi ({\sigma }_i,\mathcal{G})=\perp ]·c_{\mathrm{LLM}}\left(s_i\right)\right],$$

(12)

where $⊮[·]$ is the indicator function. Defining the fraction of memory-served steps as ${\mu }_{\tau }={\displaystyle \frac{\left|\right\{{\sigma }_i:\pi ({\sigma }_i,\mathcal{G})\ne \perp \left\}\right|}{\left|\tau \right|}}$, we obtain

$${\displaystyle \frac{C_{\mathrm{mem}}\left(\tau \right)}{C_{\mathrm{base}}\left(\tau \right)}}\le 1-{\mu }_{\tau }\left(1-{\displaystyle \frac{c_{\mathrm{ret}}}{{\overline{c}}_{\mathrm{LLM}}}}\right),$$

(13)

where ${\overline{c}}_{\mathrm{LLM}}$ is the average per-step LLM cost. Inequality (13) is obtained by partitioning the task steps into memory-served and LLM-served subsets, substituting ${\mu }_{\tau }\left|\tau \right|$ and $(1-{\mu }_{\tau })\left|\tau \right|$ for their cardinalities, and upper-bounding individual LLM costs by the average per-step baseline cost. Since the effective retrieval-to-LLM cost ratio $c_{\mathrm{ret}}/{\overline{c}}_{\mathrm{LLM}}\approx 0.02$ in our implementation, a memory utilization of ${\mu }_{\tau }=0.5$ yields a cost ratio of approximately $0.51$, consistent with the observed ∼50% reduction.

2.6. Hierarchical Control Architecture

The agent’s architecture is interpreted as a hierarchical feedback control system (Figure 2). The Manager module functions as the high-level controller: it receives the user’s task instruction, interprets the goal, decomposes it into subtasks, and issues commands to the Worker. The Worker module functions as the plant/actuator: it perceives the current GUI state through screenshot capture and OCR, then executes low-level actions (clicks, keystrokes, scrolling).

The Memory Graph serves as the feedback controller: before each action, the Manager queries the memory graph to determine whether a known trajectory exists for the current state and objective. The operational cycle thus forms a closed loop:

$$\mathrm{Observe}{\sigma }_t\stackrel{\mathrm{\Phi }}{\to }{n}_t\stackrel{\pi ({\sigma }_t,\mathcal{G})}{\to }{a}_t\stackrel{\mathrm{Execute}}{\to }{\sigma }_{t+1}\stackrel{\mathrm{Update}}{\to }{\mathcal{G}}^{\prime },$$

(14)

where ${\sigma }_t$ is the observation at time t, $n_t=\mathrm{\Phi }\left({\sigma }_t\right)$ is the recognized node, $a_t=\pi ({\sigma }_t,\mathcal{G})$ is the selected action, ${\sigma }_{t+1}$ is the resulting observation, and ${\mathcal{G}}^{\prime }=\mathrm{Update}(\mathcal{G},{\sigma }_t,a_t,{\sigma }_{t+1})$ is the updated memory graph.

This architecture resembles model-reference adaptive control (MRAC) [9] at the level of system organization: the memory graph plays a reference-like role by storing desired state–action trajectories, while the LLM-based planner generates new trajectories when memory is insufficient. The analogy is limited. We do not define a continuous plant model, an adaptive law, or a Lyapunov function for the closed-loop system; instead, the graph-update rule $\mathrm{Update}(\mathcal{G},{\sigma }_t,a_t,{\sigma }_{t+1})$ is the discrete learning mechanism that enriches the reference-like memory over time and reduces the need for LLM-based intervention.

2.7. Decision Policy: Exploitation vs. Exploration

The decision policy $\pi :\mathrm{\Sigma }\times \mathcal{G}\to A\cup \{\perp \}$ implements a structured decision-making process that balances exploitation of known solutions against exploration via LLM reasoning [11,12]. The policy operates as follows:

• Task Recognition: The Manager interprets the user instruction and queries the memory graph for matching task descriptors. The search combines textual similarity (instruction vs. stored task names/descriptions) with state matching (current UI context vs. stored node states).
• Memory-Driven Execution (Exploitation): If a matching task descriptor $d\in D$ is found, the Manager retrieves the associated path $P_d$ and the Worker executes the stored action sequence with minimal LLM involvement—only for success verification and minor adaptations.
• LLM-Driven Planning (Exploration): If $\pi ({\sigma }_t,\mathcal{G})=\perp$, the standard planning procedure is invoked: the Manager decomposes the task and the Worker uses LLM reasoning at each step. Upon successful completion, the new trajectory is integrated into $\mathcal{G}$.
• Memory Update: After task completion (via either path), the graph is updated: new nodes and edges for newly visited screens and actions, updated usage statistics for memory-served paths, and the Tool Generation module abstracts reusable subsequences into new task descriptors.

This decision framework can be viewed as an instance of knowledge-based decision support [22], where the memory graph constitutes a structured knowledge base that reduces the complexity of the decision space from the full LLM planning problem to a graph traversal problem.

2.8. State Recognition and Hashing

The state recognition function $\mathrm{\Phi }:\mathrm{\Sigma }\to N\cup \{⌀\}$ employs a composite matching strategy:

$$\mathrm{\Phi }\left(\sigma \right)=arg\underset{n\in N}{min}{d}_{\mathrm{hash}}(h\left(\sigma \right),h\left(n\right))+w_{\mathrm{text}}·d_{\mathrm{text}}(\mathrm{OCR}\left(\sigma \right),\mathrm{OCR}\left(n\right)),$$

(15)

where $h(·)$ denotes a perceptual hash function applied to the screenshot, $d_{\mathrm{hash}}$ is the Hamming distance between hashes, $\mathrm{OCR}(·)$ extracts text content, $d_{\mathrm{text}}$ is a text similarity metric, and $w_{\mathrm{text}}>0$ is a weighting parameter. We use $w_{\mathrm{text}}$ rather than $\lambda$ to avoid ambiguity with graph vertex connectivity ${\lambda }_G\left(G\right)$. If ${min}_n[·]>\theta$ for a threshold $\theta$, then $\mathrm{\Phi }\left(\sigma \right)=⌀$, indicating a novel state. The implementation utilizes the Neo4j graph database with vector indexing for efficient semantic search, analogous to techniques used in AppAgentX [14].

2.9. Memory Graph Evolution and Maintenance

The memory graph evolves continuously through the following mechanisms:

• Node addition: When $\mathrm{\Phi }\left(\sigma \right)=⌀$, a new node $n_{\mathrm{new}}=(x_{\sigma },⌀,A_{\sigma })$ is created and added to N.
• Edge addition: After successfully transitioning from state ${\sigma }_i$ to ${\sigma }_j$ via action a, the edge $(n_i,n_j,a,c)$ is added to E, where $n_i=\mathrm{\Phi }\left({\sigma }_i\right)$ and $n_j=\mathrm{\Phi }\left({\sigma }_j\right)$.
• Tool generation: The Tool Generation module analyzes completed trajectories to identify reusable subsequences. Frequently traversed paths are abstracted into parameterized task descriptors $d\in D$ and attached to the appropriate source node.
• Pruning: Periodically, maintenance operations merge duplicate nodes, generalize parameters, and remove low-utility edges to prevent graph bloat and preserve efficient retrieval.

Over the lifetime of the agent, this evolution tends to increase $\left|N\right|$ and $\left|E\right|$ up to pruning and can increase the coverage ratio $\rho (\mathcal{G},\mathcal{T})$, moving the system toward greater task coverage and reuse. Parameter selection is therefore practical rather than universal: the matching threshold $\theta$ should be calibrated on validation screens to balance false retrievals against missed retrievals, $w_{\mathrm{text}}$ should be increased when OCR text is stable and decreased for visually driven applications, and pruning thresholds should preserve high-success, recently used paths while removing stale or duplicate edges.

3. Results

3.1. Experimental Setup

We evaluate the proposed memory architecture on the OSWorld benchmark [20], a comprehensive suite of over 300 computer use tasks executed in real desktop and web application environments. OSWorld tasks span file management, form completion, web browsing, and multi-step workflows, making it representative of realistic GUI automation scenarios. Each task has a defined initial state (e.g., open applications and existing files) and a target state, and is classified by execution length (15-step or 50-step).

The baseline agent is S2-Base: the Agent S2 framework [5] using the Manager–Worker architecture with Claude 4.5 Sonnet as the underlying LLM but retaining no memory between tasks. Each task is solved from scratch. We compare S2-Base against S2-Mem, our memory-augmented variant that adds the graph memory and tool generation modules described in Section 2. Both agents use identical perception components (screenshot capture and OCR) and have access to the same set of low-level actions. The only difference is S2-Mem’s ability to store, retrieve, and reuse prior experience.

We evaluate S2-Mem in two configurations:

• S2-Mem Cold: Memory is initialized with only a small set of basic tools (e.g., login sequences for common applications), simulating a “cold start” scenario.
• S2-Mem Warm: Memory is pre-populated with tools accumulated from prior task executions, simulating an agent that has been operational for some time.

3.2. Evaluation Metrics

We measure three primary metrics tied to the formal model:

1.

Token Consumption $C\left(\tau \right)$: The total number of LLM tokens (input + output) consumed during task execution as defined in Equation (10).

2.

Execution Time $T\left(\tau \right)$: Wall-clock time from instruction receipt to task completion, encompassing both LLM processing delays and GUI interaction latencies.

3.

Success Rate $R\left(\tau \right)\in \{0,1\}$: Binary indicator of task completion—1 if the final state matches the target specification, 0 otherwise.

Additionally, we track the number of LLM invocations per task and the average number of steps, which serve as auxiliary measures of computational efficiency.

3.3. Task Success Rates

Table 1. Comparison of task success rates (%) on OSWorld benchmark. Results are reported for 15-step and 50-step task subsets.

Method	15-Step Tasks	50-Step Tasks
Agent S2 w/Claude-4.5-Sonnet (S2-Base)	36.9	46.5
S2-Mem w/Claude-4.5-Sonnet Cold	36.9	46.7
S2-Mem w/Claude-4.5-Sonnet Warm	36.9	46.9

presents the success rates across the evaluated 15-step and 50-step task subsets for each agent configuration. The near-identical success rates should be interpreted as evidence that graph memory preserves baseline task quality while reducing cost, not as evidence of a material improvement in task success.

The results are consistent with the ideal-fallback degradation bound in Inequality (9): the memory-augmented agent achieves success rates comparable to the baseline in all configurations. The warm-start configuration shows only a marginal improvement of 0.4 percentage points on 50-step tasks. We therefore interpret the main empirical gain as efficiency rather than improved task quality or robustness.

3.4. Token Cost and Execution Time

Figure 3 presents the average per-task token cost and execution time across the evaluated configurations.

The baseline agent (S2-Base) incurs an average cost of $0.21 per task with an average execution time of 61 s for 15-step tasks. The cold-start memory agent (S2-Mem Cold) shows a moderate increase to $0.25 in cost and 73 s in time, reflecting the overhead of memory population during initial encounters. However, the warm-start agent (S2-Mem Warm) achieves $0.10 per task and 34 s, representing reductions of 52% and 44%, respectively.

In terms of the formal cost model (Equation (13)), the observed cost ratio $C_{\mathrm{mem}}/C_{\mathrm{base}}\approx 0.48$ for the warm-start configuration implies an effective memory utilization rate of

$$\widehat{\mu }\approx {\displaystyle \frac{1-0.48}{1-0.02}}\approx 0.53,$$

(16)

indicating that approximately 53% of execution steps were served from memory, consistent with independent counts of LLM invocations (22 per task for S2-Mem vs. 45 for S2-Base on average).

3.5. LLM Invocation Analysis

The reduction in LLM calls provides further evidence of the memory graph’s efficiency effect. For the warm-start configuration, the agent required only 15 LLM calls for a task that demanded 38 calls under the baseline—a 60% reduction driven by memory-based replay of the login and navigation sequences. The overall average across all tasks dropped from 45 invocations (S2-Base) to 22 invocations (S2-Mem Warm), with individual tasks showing reductions up to 69% for highly repetitive workflows.

3.6. Component-Level Interpretation

The evaluation above measures the full S2-Mem system rather than a complete factorial ablation of all modules. The observed reductions in token use, latency, and LLM invocations are consistent with the intended division of labor among the components: the graph memory stores reusable state–action paths, tool generation abstracts frequently repeated paths into callable procedures, and the decision policy determines when retrieval is preferable to fresh planning. The strongest measured effect appears in repetitive workflows, where login, navigation, and document-manipulation steps can be replayed from memory with only lightweight verification.

Because the available experimental record does not contain repeated runs for each ablated configuration, we do not report statistical significance tests or component-wise causal estimates. We therefore interpret the present results conservatively: they show that the complete memory-augmented system can preserve baseline success rates while reducing cost and latency, but they do not by themselves quantify the isolated contribution of graph memory, tool generation, and state recognition. A full ablation study with repeated runs, confidence intervals, and per-task category breakdowns is left as future work.

3.7. Qualitative Behavior

The qualitative behavior of the system matches this conservative interpretation. In recurring workflows, the memory graph reduces repeated reasoning by replaying verified trajectories and invoking the LLM mainly for task-specific verification or adaptation. In novel workflows, or when the GUI state cannot be matched confidently, the policy falls back to ordinary LLM planning. The main failure mode is therefore not loss of baseline planning capability but missed or incorrect memory retrieval caused by state-recognition errors. This failure mode motivates the mitigation strategies discussed in Section 4.4.

4. Discussion

4.1. Interpretation Through the Reachability Framework

The experimental results support the reachability and coverage framing developed in Section 2.3. The memory-augmented agent exhibits three operational properties inspired by functional-stability theory [1,23]:

• Monotonic coverage growth: As the agent completes more tasks, the memory graph accumulates nodes and edges, increasing the coverage ratio $\rho (\mathcal{G},\mathcal{T})$. The transition from cold-start to warm-start performance (Table 1, Figure 3) illustrates this growth: cost decreases from $0.25 to $0.10 as memory populates.
• Stabilizing efficiency: The memory utilization ratio ${\mu }_k$ (Definition 3) increases over the evaluation period. Initial tasks require extensive LLM planning ($\mu \approx 0$), while later tasks with overlapping subtasks benefit from stored trajectories ($\mu \approx 0.53$ on average).
• Graceful degradation as a design target: When the agent encounters novel states not represented in the memory graph ($\mathrm{\Phi }\left(\sigma \right)=⌀$), it falls back to LLM-based planning. This mechanism is intended to avoid degradation relative to the baseline under the ideal fallback assumption, while practical failure modes are treated explicitly as limitations.

These properties mirror, at the level of design intuition, the resilience criteria established for communication network topologies [6,7]; they do not constitute a classical stability proof for the GUI agent as a dynamical system.

4.2. Control-Theoretic Interpretation

The closed-loop architecture (Equation (14)) demonstrates characteristics of adaptive control [8,9]. The initial cold-start phase corresponds to the transient response of an adaptive controller before the reference model has been adequately identified. During this phase, performance is comparable to the open-loop (baseline) system. As the memory graph fills, the system enters the steady-state regime, where the reference model (memory graph) provides accurate state–action mappings and the adaptive mechanism (LLM planner) is invoked only for novel situations.

The behavior of the memory utilization ratio ${\mu }_k$ (Definition 3) is analogous only at a high level to parameter convergence in MRAC systems [10]: both include an initial transient followed by a more stable operating regime. Unlike MRAC, however, the present system lacks an explicit adaptive law and Lyapunov analysis, so we treat this comparison as explanatory rather than as a theorem.

An important aspect of the control architecture is the separation of concerns between the Manager (high-level controller) and Worker (low-level actuator). This mirrors the well-known cascade control structure [8], where the outer loop operates at a slower time scale (task decomposition) and the inner loop at a faster time scale (individual GUI actions). The memory graph enhances the outer loop’s decision quality by reducing the search space from the exponential space of possible LLM plans to the polynomial space of graph traversals.

4.3. Decision-Making Perspective

From the perspective of decision-making theory [12,22], the memory graph functions as a knowledge-based decision support system (DSS) that transforms the agent’s operational decisions. Without memory, every step requires solving a full planning problem—essentially, the agent faces a Markov decision process (MDP) where each state requires policy computation from scratch. With memory, the agent’s decision reduces to a binary classification: “Is this state known?” followed by either efficient lookup or standard planning. This reduction in decision complexity directly manifests as the observed 50% reduction in computational cost.

The balance between exploitation (memory retrieval) and exploration (LLM planning) in the policy $\pi$ can be analyzed through the multi-armed bandit framework [11]. The memory graph effectively eliminates the exploration cost for previously solved subproblems, allowing the agent to allocate its computational budget toward genuinely novel challenges. This is especially useful when the task stream contains a mixture of recurring and previously unseen subtasks.

4.4. Limitations and Perturbation Sensitivity

Several limitations merit discussion in terms of recoverability, robustness boundaries, and the absence of formal stability guarantees:

State recognition fragility. The state recognition function $\mathrm{\Phi }$ (Equation (15)) relies on perceptual hashing and text matching, which are sensitive to visual perturbations. Even minor interface changes (theme changes, resolution scaling, or application updates) can cause $\mathrm{\Phi }\left(\sigma \right)=⌀$, preventing memory utilization. This constitutes a practical limitation of the recognition layer: small observation changes can disable memory retrieval even when the underlying task remains the same. Worse, an incorrect non-null match can replay a stale action sequence. Concrete mitigations include threshold calibration on held-out UI states, confidence-aware fallback when the best match is close to $\theta$, structural parsing of GUI elements, periodic revalidation of high-use paths, and invalidation of edges after repeated replay failure. More robust methods such as structural analysis through models like OmniParser [14] could improve $\mathrm{\Phi }$’s accuracy.

Scalability. As $\left|N\right|$ and $\left|E\right|$ grow to hundreds of nodes and thousands of edges, graph traversal and search operations may become more costly. We partially address this through vector search indexing, task-descriptor filtering, Neo4j’s graph query optimization, and the pruning/merging of low-utility or duplicate nodes. In large deployments, memory maintenance must also handle stale edges, application-version drift, privacy constraints for stored screenshots/OCR text, and compaction of rarely used trajectories. A comprehensive empirical analysis of asymptotic scaling remains necessary.

Generalization boundary. The memory graph is bound to specific application environments. A login sequence learned for one website cannot be directly transferred to another without abstract concept matching. Multi-level semantic memory organization [15,16,17]—where abstract concepts (“authenticate” and “export document”) exist at higher levels with environment-specific implementations below—represents a promising direction.

Cold-start overhead. The cold-start configuration shows slightly higher cost than the baseline (Figure 3), reflecting the overhead of memory population. This transient cost is the “price of learning” inherent in any adaptive system [9] and is amortized over subsequent task executions.

4.5. Quality of Generated Tools

The Tool Generation module successfully abstracts many generalized instruments. Tools such as Login(app, user), OpenFile(filename), and SendEmail(recipient) proved versatile across application contexts. In some cases, the generator produced overly specific tools (e.g., ExportPDFReport, bound to a specific format sequence); such cases highlight the need for action parameterization and composition of similar tools [3]. Nevertheless, the availability of reusable modules enabled the agent to effectively solve complex multi-step tasks through composition. For instance, a 50-step task (“Take a dataset from a spreadsheet, build a diagram, and insert it into a presentation”) was solved by composing three previously learned tools with minimal new planning, resulting in approximately 60% fewer LLM tokens compared to the baseline agent’s full planning approach.

5. Conclusions

This paper presented a graph-based persistent memory architecture for LLM-driven computer use agents and analyzed it through reachability, hierarchical control, and decision-making perspectives. The core scientific contribution is the directed graph memory system—where nodes represent GUI states and edges encode executable action sequences—integrated into the Agent S2 framework and evaluated on the OSWorld benchmark.

The key findings are as follows. First, the memory-augmented agent achieves an approximately 50% reduction in token consumption and a substantial reduction in execution time relative to the memoryless baseline, while maintaining comparable task success rates. Second, the formal analysis shows that, under an ideal fallback assumption, the memory graph can degrade to baseline planning when encountering novel states; this is an operational recoverability property rather than a universal performance guarantee. Third, the growth of the memory coverage ratio $\rho (\mathcal{G},\mathcal{T})$ and utilization ratio ${\mu }_k$ indicates increasing procedural reuse over time.

The contributions extend beyond the immediate application domain. By formalizing the memory-augmented agent as a tuple $\mathcal{S}=\langle A,\mathrm{\Sigma },\mathcal{G},\delta ,\pi ,\mathrm{\Phi }\rangle$ and defining quantitative reachability, coverage, and utilization criteria, we provide a mathematical framework applicable to the broader class of experience-driven autonomous systems. The analogy between memory graph connectivity and classical functional stability conditions [1,7] suggests that tools from graph theory and network reliability analysis can inform the design of agent memory systems, provided that their assumptions and limits are stated explicitly.

Several shortcomings remain. The current experiments primarily establish efficiency gains; they do not yet provide full repeated-run variance, statistical significance tests, per-task breakdowns, or direct quantitative comparison with all memory-based baselines. The theoretical analysis is intentionally moderate: it gives graph reachability and cost-efficiency arguments but not a Lyapunov proof, MRAC adaptive law, or closed-loop convergence theorem. Finally, state recognition remains sensitive to UI changes, and large-scale memory graphs require further maintenance and privacy analysis. Future work will therefore pursue the formal analysis of memory-utilization dynamics under specified task distribution assumptions, ablation and statistical studies on larger OSWorld subsets, multi-agent shared memory, verification of graph-maintenance operations, and more robust state recognition methods based on structural GUI analysis.