Abstract
Large language model agents can automate data science workflows, but cloud-centric deployment exposes sensitive context and edge-only deployment limits analytical capability. We present FinDS-Agent, a cloud–edge framework that keeps raw records and program execution at the trusted edge while providing a policy-screened, sanitized context to support cloud planning. FinDS-Agent integrates a Three-Stage Cascaded Privacy Gate (TCPG), a Multi-Dimensional Joint Router (MJR), contract-guided ToolGraph planning, edge-side verification, and bounded repair. On 222 DataSciBench tasks over three runs, FinDS-Agent achieved a 69.93% completion rate and 57.06% success rate, improving over Edge-Only by 9.50 and 5.71 percentage points while invoking the cloud for 32.27% of eligible task-runs. On FinDS-Privacy-Bench, TCPG increased sensitive-field recall from 58.20% to 98.10%; no payload-leakage event was observed in the full set (0/200; Wilson 95% CI: 0–1.8845%) or blind split (0/100; 0–3.6993%) under the specified audit and threat model. External evaluation gave pass rates of 33.2%, 53.1%, and 62.3% for Edge-Only, FinDS-Agent, and Cloud-Only on DS-1000. These empirical results support selective cloud planning while delimiting statistical, privacy, and transfer claims.
1. Introduction
Over the past decades, financial institutions have made substantial progress in enterprise data governance, particularly at the headquarters level. The centralization of core business data, construction of data warehouses, and standardization of data-management processes have established a relatively mature foundation for business intelligence and regulatory reporting [1]. Meanwhile, analytical requirements are evolving from descriptive reporting toward more complex data science workflows. Business users increasingly need to perform data cleaning and preprocessing, exploratory and statistical analysis, visualization, predictive modeling, data mining and pattern discovery, as well as model interpretation and report generation [2].
Traditionally, these workflows have relied heavily on professional data analysts and manually developed scripts. Such processes are often time-consuming, difficult to reuse, and highly dependent on the expertise of individual analysts. Recent advances in large language models (LLMs) and autonomous agents offer a potential means of reducing this dependence [3]. By integrating natural-language understanding, task planning, tool invocation, code generation, and execution feedback, LLM-based agents can transform business requirements into executable analytical workflows [4]. Data science benchmarks such as DS-1000 [5], DSBench [6], MLE-bench [7], and DataSciBench [8] have further demonstrated the potential of LLM agents in realistic programming, data analysis, and machine-learning tasks.
However, deploying data science agents at financial branches presents three interrelated challenges. First, constrained local infrastructure and limited analytical expertise restrict the ability of lightweight edge models to generate reliable code and complete complex workflows involving multi-stage preprocessing, modeling, visualization, and report generation. Second, although cloud-hosted LLMs generally provide stronger reasoning and code-generation capabilities [9], transmitting user requests, data samples, intermediate results, or raw financial records outside the institutional security domain introduces substantial privacy and regulatory risks. Finally, the probabilistic nature of LLM generation may produce hallucinated APIs, unsafe operations, incomplete artifacts, or silently incorrect analytical logic, even when the generated code executes successfully [2]. Practical financial data agents must therefore jointly address limited edge capability, unreliable model-generated programs, and sensitive-data exposure. These challenges create a fundamental deployment dilemma, as illustrated in Figure 1.
Figure 1.
Edge-only versus cloud-only deployment for financial analytics.
Existing LLM-based data agents remain limited in privacy-sensitive enterprise settings. Cloud-centric approaches risk exposing sensitive context, whereas edge-only agents may lack sufficient capability for complex analytical tasks. Existing cloud–edge methods often make offloading decisions without jointly considering task intent, analytical complexity, data sensitivity, and policy constraints. Moreover, privacy control, program verification, execution recovery, and artifact validation are typically addressed separately rather than within a unified workflow. Current evaluations also provide limited support for jointly assessing analytical utility and payload-level privacy leakage [10]. These limitations reveal a research gap in selectively exploiting cloud planning while retaining raw data and program execution at the edge. FinDS-Agent addresses this gap through privacy-aware routing, constrained cloud collaboration, edge-local execution, and integrated privacy evaluation.
To address these limitations, we propose FinDS-Agent, a privacy-aware cloud–edge collaborative framework for enterprise data science that decouples cloud-assisted planning from edge-side execution. Under the evaluated policy and threat model, raw records and data-dependent execution remain in the trusted edge environment, while only a screened and sanitized context is eligible for cloud planning. The framework combines three system-level mechanisms: TCPG performs hierarchical privacy assessment; MJR considers task intent, analytical complexity, and sensitivity when selecting edge or cloud-assisted planning; and ToolGraph constrains planning through registered skills, dependencies, execution contracts, and bounded repair. The contribution is their policy-governed coordination with edge-local execution and observable-payload auditing, rather than a claim that privacy screening, routing, tool graphs, or repair are individually novel primitives.
This paper makes three primary contributions toward trustworthy cloud–edge agent systems for enterprise data science:
- We formulate and implement a policy-governed cloud–edge workflow that separates cloud-assisted planning from edge-side data custody, verification, and execution. Its system-level contribution is the coordination of TCPG, MJR, ToolGraph, Code-Down communication, and bounded recovery under explicit privacy and artifact contracts.
- We make the routing and privacy boundaries auditable: the MJR parameters, calibration offset, pilot composition, probability estimates, fallback rule, and exploratory sensitivity profiles are disclosed, while payload leakage is defined as a benchmark event with explicit denominators rather than as a formal security guarantee.
- We extend the empirical evidence beyond the original internal configurations through same-model framework comparisons, DS-1000 and KramaBench evaluation, compound-fault injection, and a separate five-run nine-arm ablation. Statistical intervals and paired task-cluster analyses preserve the task, run, benchmark, and version boundaries of each experiment.
The remainder of this paper is organized as follows. Section 2 reviews related work, and Section 3 defines the system, task, and threat models. Section 4 presents FinDS-Agent and its principal mechanisms, while Section 5 describes the experimental evaluation and results and Section 6 describes the failure analysis. Finally, Section 7 summarizes the findings, limitations, and future directions.
2. Related Work
FinDS-Agent is related to three research directions: data science agents and cloud–edge routing, constrained agent execution, and privacy-preserving agents and leakage evaluation. This section reviews representative studies in these areas and summarizes the remaining challenges in privacy-sensitive financial analytics.
2.1. Data Science Agents and Cloud–Edge Routing
LLM-based agents extend language models with reasoning, tool invocation, code execution, and iterative feedback [1]. Representative approaches include ReAct [2], which interleaves reasoning and task-specific actions; Toolformer [3], which studies self-supervised tool use; and Reflexion, which uses linguistic feedback from failed trajectories to improve subsequent attempts [4]. Data science agents further require schema understanding, tabular-data manipulation, statistical analysis, predictive modeling, and structured artifact generation [5].
Existing benchmarks evaluate these capabilities from complementary perspectives. DS-1000 focuses on execution-grounded data science programming [6]; DSBench evaluates realistic analysis and modeling workflows [7]; MLE-bench targets machine-learning engineering tasks [8]; and DataSciBench assesses executable analytical outcomes with multiple requirements [9]. Although these benchmarks reveal continuing limitations in complex workflows, they generally assume centralized access to data, models, tools, and execution resources [10].
Cloud–edge collaboration seeks to combine lightweight local models with more capable cloud LLMs through request assignment, model partitioning, early exit, and token- or subtask-level routing [11]. RouteLLM selects between strong and weak models using preference data [12]; hybrid SLM–LLM inference combines local drafts with selective cloud verification [13]; and other routing approaches optimize quality, latency, cost, or dependency-aware resource allocation [14]. However, these approaches rarely consider task semantics, data sensitivity, and execution locality together, which limits their applicability to regulated financial analytics [15].
2.2. Constrained and Verifiable Agent Execution
Execution feedback and structured interfaces can improve the reliability of tool-using agents. Reflexion [4] uses failed trajectories to support iterative refinement, while SWE-agent [16] demonstrates the importance of agent–computer interfaces and runtime observations in software-engineering tasks. Nevertheless, executable code may still contain invalid APIs, unsafe operations, incomplete outputs, or silent semantic errors [17].
Skill-based orchestration provides additional structural guidance [18]. GraSP represents retrieved skills as typed dependency graphs with explicit preconditions, effects, verification, and localized repair [19]. Meanwhile, the Model Context Protocol (MCP) [20] and Agent-to-Agent (A2A) [21] protocol standardize interactions among models, tools, context providers, and peer agents. These developments improve interoperability and skill composition, but they do not inherently enforce domain-specific artifact contracts, privacy-consistent program behavior [22], or edge-local execution. Reliable financial data agents therefore require execution constraints that extend beyond successful code generation [23].
2.3. Privacy-Preserving Agents and Leakage Evaluation
Agentic systems may disclose sensitive information through tool arguments, intermediate messages, shared memory [24], execution traces, cloud payloads, and final outputs. AGENTDAM evaluates data-minimization compliance in autonomous web agents [25]; PrivacyLens-Live studies privacy risks in dynamic MCP and A2A workflows [26]; and AgentLeak shows that output-only evaluation may fail to detect leakage through internal coordination channels [27]. These studies demonstrate that privacy evaluation must cover the complete agent trajectory rather than only the final response [28].
Local sanitization provides an architectural means of limiting cloud exposure [29]. For example, MemPrivacy identifies sensitive spans locally, replaces them with typed placeholders, and retains reversible mappings at the edge [30]. However, most existing studies focus on conversational text, web interactions, memory, or general multi-agent communication. Structured financial data introduce additional risks associated with identifiers, financial attributes, protected characteristics, semantically obfuscated columns, and indirect disclosure through row-level reporting or model interpretation [31]. Existing privacy benchmarks provide limited coverage of these risks in executable cloud–edge data science workflows [32].
2.4. Positioning of This Work
The literature reveals three connected gaps: cloud–edge routers rarely incorporate task semantics and data sensitivity jointly; tool-using agents lack domain-specific execution and artifact controls; and privacy evaluations seldom address structured financial data throughout an executable agent workflow. This work investigates these challenges within a unified cloud–edge data-agent setting, with particular emphasis on planner selection, verifiable local execution, and empirical privacy evaluation.
The underlying ideas have precedents: model routing selects among heterogeneous inference resources, privacy filters constrain disclosures, tool graphs structure actions, and verifier-guided repair uses execution feedback. FinDS-Agent’s distinction is therefore architectural and empirical: TCPG decisions constrain the admissible cloud-visible context before MJR routing; ToolGraph contracts and bounded repair operate with edge-local execution; and the evaluation links analytical outcomes to payload-level audit events. Appendix A (Table A1) states these boundaries component by component.
3. Preliminaries
This section defines the system setting, data science task, privacy and threat assumptions, and the optimization problem considered in this work. The concrete mechanisms used to satisfy these requirements are presented in Section 4.
3.1. System Setting
We consider a cloud–edge system for privacy-sensitive enterprise data science, represented as
where denotes an authorized enterprise user or business application, denotes the trusted edge environment, denotes an external cloud planning service, is the edge-resident dataset, and is the applicable enterprise privacy policy. The user submits a natural-language request and identifies an authorized data source. Authorization to access a data source does not imply that every submitted request is policy compliant; each request remains subject to privacy and execution-policy checks.
In a practical deployment, may correspond to an on-premises server, a branch-level workstation, a private computing cluster, or an institutional data platform operating within the enterprise trust boundary. It stores the original dataset, enforces local access and outbound-communication policies, hosts the edge planner , verifies generated programs, executes admitted code, validates the resulting artifacts, and returns them to the authorized user. The edge may access local files, databases, or institutional data services for which the user has been granted permission.
The cloud component may be implemented as a commercial LLM API, an externally hosted foundation model, or a remote model service outside the enterprise trust boundary. It provides a cloud planner for planning and code generation but has no direct interface to , the edge file system, or the local execution environment. Cloud–edge interaction is restricted to an edge-to-cloud task representation and a cloud-to-edge planning response containing a structured plan and candidate program. The admissibility condition for outbound messages is defined in Section 3.3, and the concrete cloud payload is specified in Section 4.4.3.
Accordingly, planner selection may change the source of planning and code generation but does not change data custody or the location of data-dependent execution. All admitted programs, whether generated by or , are verified and executed within . The adversarial behavior of the cloud and the remaining trust assumptions are specified separately in Section 3.3.
3.2. Task Definition
A data science task is represented as
where is a natural-language request, is the edge-resident dataset introduced in Section 3.1, is the set of required analytical artifacts, and denotes the applicable privacy, safety, and execution constraints. Here, the calligraphic symbol denotes task-level constraints and is distinct from the cloud participant .
Given a candidate program , let
denote its execution result in the trusted edge environment. Task success is defined as
where is the indicator function and indicates whether the execution result satisfies the output specification of artifact . Therefore, a task is considered successful only when the candidate program executes correctly and produces all mandatory artifacts in their required forms.
3.3. Privacy and Threat Model
Let denote the set of sensitive columns in , including personally identifiable information, financial or business-sensitive attributes, protected attributes, and quasi-identifiers. Let denote the corresponding set of raw sensitive values. For a task , let denote the set of messages transmitted from the edge to the cloud. An admissible cloud interaction must satisfy
where denotes the unmasked literal values contained in message . Reversible masking information and other locally retained privacy metadata must not be included in .
Equation (4) specifies an operational non-disclosure requirement rather than a formally verified privacy guarantee. Its practical satisfaction depends on the accuracy of sensitive-information detection, the construction of cloud-visible messages, and the adopted leakage-audit procedure. The cloud component is modeled as honest-but-curious: it follows the prescribed interaction protocol and returns the requested planning results but may inspect or retain any information transmitted to it. Communication between and is assumed to be protected by transport-layer encryption. We trust the edge environment , including its local storage, privacy-policy enforcement, program verifier, and execution sandbox. The cloud is assumed not to deliberately violate the interaction protocol or generate programs specifically intended to exfiltrate data. Physical compromise of the edge environment, cryptographic attacks, malicious-cloud code exfiltration, reconstruction from released aggregates, membership inference, side-channel attacks, and risks arising from the secondary use of cloud logs are outside the evaluated threat model.
An observed payload-leakage event occurs when a request enters an externally exposed processing path and the serialized cloud-visible payload retains at least one gold sensitive field or an unmasked gold sensitive value under the adopted benchmark audit. This event differs from sensitive-field recall, the benchmark action decision, sensitive content in a final local output, and a formal privacy property. Accordingly, a zero event count means only that no payload-leakage event was observed under the specified benchmark, audit procedure, and threat model. The study provides neither differential-privacy nor cryptographic or non-interference guarantees, and it does not estimate risks from reconstruction, membership inference, side channels, malicious cloud-generated exfiltration code, or secondary use of provider logs.
3.4. Problem Formulation
For each task , the system must determine whether the task should be rejected or planned using an edge or cloud resource. We denote this decision by
where is the system-level planning decision. denotes policy-based rejection, whereas and denote edge-local and cloud-assisted planning, respectively. These decisions determine the source of planning rather than the execution location because all admitted programs execute within .
Given a collection of tasks, the system-level objective is
where denotes analytical utility, denotes cloud-resource consumption, denotes residual privacy risk, and denotes end-to-end latency. The nonnegative coefficients , , , and represent deployment-specific preferences.
The optimization is subject to
The first constraint requires a task that violates its applicable constraints to be assigned the route. The second permits cloud-assisted planning only when every outbound message contains no raw sensitive value under the adopted detection model. The third requires every admitted candidate program, whether generated by the edge or cloud planner, to be executed within the trusted edge environment . These constraints respectively formalize policy compliance, outbound non-disclosure, and edge-local execution. Section 4 specifies how FinDS-Agent operationalizes these requirements.
4. Methods
This section presents FinDS-Agent, a privacy-aware cloud–edge framework that selectively uses cloud planning while retaining raw data and data-dependent execution within the trusted edge environment. The method consists of privacy-aware task preparation, multidimensional planner selection, contract-guided planning, and edge-side verification and execution.
4.1. System Overview
Within the system setting defined in Section 3.1, FinDS-Agent operationalizes the interaction among the authorized user , the trusted edge environment , and the cloud planning service . As illustrated in Figure 2, FinDS-Agent comprises three edge-resident stages and an optional cloud planner. Privacy-aware task preparation constructs a structured local context and determines what information, if any, may leave the edge. Adaptive orchestration then selects an edge or cloud-assisted planner and grounds code generation with task-specific skills and constraints. Finally, secure execution verifies generated programs, executes admissible code in a local sandbox, and applies bounded recovery when failures are detected.
Figure 2.
Cloud–edge architecture of FinDS-Agent.
The central idea is to decouple planning capability from data custody and program execution. The cloud is treated as a selectively invoked planning resource rather than a default execution environment. Privacy constraints first determine whether cloud interaction is admissible; planner selection is subsequently based on task intent, analytical complexity, and residual sensitivity. Thus, cloud assistance can improve planning capability without transferring the original dataset or relocating data-dependent execution.
At runtime, the user request and authorized data sources are converted into a structured local context. TCPG either blocks the task or constructs a privacy report and sanitized context. MJR then selects the edge or cloud-assisted planner, and ToolGraph provides the selected planner with relevant skills, artifact requirements, and execution constraints. The generated program is verified and executed locally. Detectable failures may trigger bounded repair, deterministic fallback, or policy-permitted cloud replanning. The process terminates when the required artifacts have been validated and returned to the authorized user. Algorithm 1 summarizes this end-to-end procedure.
| Algorithm 1 FinDS-Agent Cloud–Edge Execution Pipeline |
| Require: Task = (q, D, , ); privacy policy Πpriv; complete ToolGraph registry ; MJR parameter set ΘMJR containing the fixed score coefficients and routing thresholds; repair budget K Ensure: Validated artifact set or a safe terminal response Section 4.2: Context Building and Three-Stage Cascaded Privacy Gate
Section 4.3: Multi-Dimensional Joint Router
Planner invocation
Section 4.4: Agent Orchestration Engine Edge-side verification, execution, and validation
Deterministic fallback
Conditional cloud escalation
|
4.2. Context Building and Three-Stage Cascaded Privacy Gate
FinDS-Agent first transforms the task defined in Equation (2) into a structured representation suitable for privacy assessment, routing, and constrained code generation. Privacy decisions are made before planner selection so that estimated cloud capability cannot override mandatory protection requirements.
4.2.1. Context Building
The edge resolves the authorized data sources into the local dataset and extracts schema, structural, and lightweight statistical information. Raw rows may be inspected locally for profiling and privacy detection, but complete records are not incorporated into cloud-visible messages. The resulting task context is
where is the user request, contains the locally extracted schema and structural metadata, and contains lightweight profile information, including dataset dimensions, missing-value ratios, column cardinalities, and selected descriptive statistics. The artifact set and constraint set are inferred from the request and may be supplemented by the corresponding execution or benchmark adapter.
The context remains entirely within the trusted edge domain. It provides a common representation from which TCPG derives privacy controls, MJR derives routing features, and ToolGraph derives artifact and execution constraints. The privacy policy introduced in Section 3.1 instantiates the privacy-related requirements in , including sensitive-field classifications, prohibited disclosure patterns, aggregate-only conditions, and outbound-message admission rules.
4.2.2. Three-Stage Cascaded Privacy Gate
The Three-Stage Cascaded Privacy Gate (TCPG) combines policy-guided detection, local semantic review, and query-intent arbitration. The cascade follows a recall-oriented design: deterministic rules identify high-confidence sensitive fields, while uncertain candidates are retained for local semantic analysis rather than being prematurely discarded, as shown in Figure 3.
Figure 3.
Three-stage privacy assessment in TCPG.
For candidate field , let denote its rule-based sensitivity confidence and let be the semantic-review threshold. The rule and ambiguous sets are
The confidence measures the reliability of the rule-based sensitive-field classification rather than the severity of the field. A high-confidence field may therefore be assigned any policy-defined sensitivity level.
The rule scan incorporates financial-data lexicons, column-name patterns, value-pattern regular expressions, and query indicators associated with row-level or indirect disclosure. Fields in are subsequently reviewed by an on-premises 7B model using column-level metadata and a limited number of truncated values. Complete rows are not provided to the model. If denotes the fields classified as sensitive through semantic review, the final detected-field set is
For each ambiguous field, the local reviewer returns a structured finding containing the field identifier, sensitivity category, sensitivity level, classification rationale, and a binary sensitive-field decision. Only fields classified as sensitive are included in . The semantic stage resolves domain-specific abbreviations and contextual meanings that cannot be reliably captured by surface-form rules. Its output does not authorize cloud transmission; instead, the findings are passed to query-intent arbitration.
Query-intent arbitration jointly evaluates and . Requests involving prohibited row-level disclosure, indirect disclosure, or high-risk explanation over sensitive attributes are blocked. Permitted aggregate requests over high-sensitivity data are assigned an aggregate-only restriction. The decision is expressed as
where denotes the policy-based query-intent arbitration function, indicates whether the request must be rejected, and indicates whether only aggregate processing and output are permitted.
If , no cloud-visible context is constructed and . Otherwise, TCPG produces
where is the redacted request, is the sanitized schema, and contains a limited number of masked samples. Reversible masking information and complete raw rows are excluded.
TCPG further derives a task-level sensitivity label . Its complete output is
The privacy report defines the admissible information boundary for subsequent processing. TCPG therefore determines whether and in what form a task may interact with the cloud, whereas MJR determines whether cloud assistance is analytically justified.
4.3. Multi-Dimensional Joint Router
The Multi-Dimensional Joint Router (MJR) operationalizes the planner-selection problem in Equation (5). At the system level, the resulting route is one of Blocked, Edge, or CloudAssist. The Blocked route is propagated directly from TCPG, whereas MJR selects between Edge and CloudAssist for a non-blocked task. These routes determine the source of planning rather than the execution location because all approved programs execute at the edge, as shown in Figure 4.
Figure 4.
Privacy-constrained planner selection in MJR. Privacy rules are applied before the expected benefit of cloud assistance is evaluated.
Let denote the fixed router parameters, including the calibration offset, intent-dependent prior weights, privacy-risk coefficients, cloud-use penalties, and routing thresholds and .
MJR represents task as
where denotes task intent, denotes analytical complexity, and denotes TCPG-derived sensitivity.
Task intent characterizes the requested operation, including profiling, aggregation, visualization, predictive modeling, advanced mining, and row-level retrieval. Analytical complexity is represented by . For natural-language tasks, profiling, aggregation or visualization, predictive modeling, and advanced mining are generally mapped to increasingly demanding levels. For code-oriented tasks, structural features of the request provide additional refinement.
Task intent and analytical complexity play distinct but complementary roles. Task intent identifies the requested analytical operation and determines the operation-specific routing prior, whereas analytical complexity estimates the planning and code-generation capability required to complete the task. These two dimensions are operationally coupled for natural-language tasks but are not treated as interchangeable variables. Data sensitivity imposes an orthogonal privacy constraint that determines whether cloud interaction is admissible and, if permitted, contributes to the residual privacy-risk penalty. Dataset scale and cloud-use cost are incorporated as routing penalties rather than additional task dimensions.
MJR applies privacy constraints lexicographically: a blocked task is rejected without computing a cloud-benefit score, and a high-sensitivity aggregate-only task at complexity or remains at the edge. For other admissible tasks, the estimated cloud gain is
where and are capability estimates for the corresponding intent–complexity bucket, is a fixed calibration offset, and restricts the result to .
The intent-dependent prior is
This quantity is a fixed prior weight rather than a calibrated probability.
Residual privacy risk is approximated by
where indicates row-level disclosure intent. Blocked tasks are excluded because they have already been handled by the privacy constraint.
Let and denote the numbers of rows and columns in . The cloud-use penalty is
This term is a routing penalty rather than an observed monetary cost or runtime measurement.
The final routing score is
The score approximates the utility, cloud-resource, and privacy-risk terms in Equation (6). Latency is measured empirically rather than predicted by the current router. All utility-score coefficients are fixed routing hyperparameters rather than calibrated probabilities or a jointly learned policy. Privacy prohibitions are lexicographic hard rules and cannot be overridden by utility optimization. The current three-run main experiment used the explicitly disclosed calibration offset ; the equation and Table 1 therefore retain that value.
Table 1.
Distribution of DataSciBench evaluation entries. One task may contribute to multiple categories.
The development pilot contained 240 tasks in a balanced design: five intents, four analytical-complexity levels, three privacy levels, and four tasks per cell. Edge and cloud-sanitized arms were each repeated three times, yielding 1440 execution records. DataSciBench evaluation tasks were not used to select the reported coefficients. A post-hoc automated comparison of the 240 prompts and data trees against 4180 benchmark prompts flagged no overlap; because this audit was not preregistered and cannot rule out all semantic paraphrases, we describe the set as a pilot rather than an independently confirmed calibration benchmark.
For an intent–complexity bucket, let and be its successes and trials. The smoothed posterior mean is , where is a fallback prior. The implementation searches, in order, (i) other intents at the same complexity, (ii) other complexities for the same intent, and (iii) all other buckets for the same arm; the first pool with at least 20 records is used, with the global pool serving as the final fallback. When , the operational probability is the parent prior itself; otherwise, it is the smoothed posterior mean. and are computed separately from the two arms. Their bucket difference is the cloud gain before the current main-version offset is applied.
A later, versioned zero-bias router was evaluated exploratorily with three frozen profiles (privacy-priority, default, and utility-priority), each on 222 tasks over three runs. The mean CR was 77.53%, 73.14%, and 76.30%; the maximum spread was 4.39 percentage points, while initial cloud-selection rates ranged from 91.59% to 98.47%. The profile names encode policy preferences rather than calibrated risk levels. These results show observable quality–cloud-use variation and do not establish parameter invariance. Because that router removed the historical bias, its statistics are not attached to Table 2 and are reported separately in Appendix A (Table A3).
Table 2.
Performance under different inference configurations. Results for locally evaluated configurations are mean sample SD over three runs of 222 tasks; CR/SR 95% intervals are reported in the text. The GPT-4o-2024-05-13 result is taken from the official DataSciBench evaluation and is reported only as an external capability reference.
For low-complexity mining tasks, MJR additionally defines
where . This exception applies primarily to code-oriented tasks whose complexity is refined independently of the default intent mapping.
Combining the TCPG blocking decision with the MJR planner selection, the system-level initial route is
where is the default routing margin. Privacy hard rules are evaluated before the score-based conditions. Consequently, cloud assistance is selected only when the combined cloud-capability gain and intent-dependent offload prior exceed the privacy-risk and cloud-use penalties by the positive margin . Neither nor the associated thresholds should be interpreted as calibrated success probabilities.
Capability estimates are retrieved from the most specific available intent–complexity bucket. When observations are insufficient, the router uses predefined priors. MJR is therefore a hybrid rule-based and partially frequency-calibrated policy. Each decision records its input dimensions, score components, selected route, and rationale for subsequent auditing.
The execution record stores both the initial route and the final route . The two values differ when an initially selected cloud route falls back to edge planning or when an edge-planned task subsequently escalates to the cloud. This distinction prevents failure-driven escalation from being interpreted as an additional routing dimension.
4.4. Agent Orchestration Engine
Once the route has been selected, the Agent Orchestration Engine generates and evaluates a task-specific program. FinDS-Agent uses Mini-Agent [29] as the underlying interaction substrate, while ToolGraph introduces domain-specific planning structure and machine-readable constraints. Mini-Agent provides the underlying agent loop, tool interface, contextual memory, and model-invocation mechanism but does not itself provide the privacy, routing, or execution controls required by FinDS-Agent. TCPG, MJR, ToolGraph grounding, PlanVerifier, Code-Down, and failure recovery are introduced around this substrate to enforce the privacy and execution requirements defined in Section 3.
4.4.1. ToolGraph-Guided Planning
ToolGraph represents analytical skills and their dependencies as a task-conditioned graph. Its nodes encode data science operations, privacy contracts, and execution adapters, while typed edges encode prerequisite and output relationships. Let denote the complete ToolGraph registry, which contains registered analytical skills, privacy contracts, artifact specifications, execution constraints, and repair policies. For task , ToolGraph selects and expands a task-conditioned subgraph from , yielding the grounding context . For task , ToolGraph produces
where and are the selected skill nodes and dependency edges, is the machine-readable realization of , , and are derived from , and contains the applicable repair policies.
Candidate skills are scored using the request, task intent, analytical complexity, task type, sensitivity information, and the active planner-route hint. The highest-scoring analytical skill is selected as the primary node. If no candidate receives a positive score, a deterministic general-purpose analytical skill is used as the grounding fallback. Because the active route may affect planner-specific grounding, is reconstructed when a cloud-selected task falls back to edge planning or when an edge-planned task escalates to cloud-assisted replanning.
Graph construction proceeds by selecting the analytical skill most relevant to the task representation, recursively expanding its dependencies, and aggregating artifact, privacy, execution, and repair constraints. A privacy-contract node is included in every selected path, and the path is bounded to five nodes to limit prompt length and planning complexity. The resulting context is provided to either planner through a common interface.
ToolGraph constrains rather than predetermines program generation. The planner may produce task-specific code instead of concatenating fixed templates, but its output is subsequently checked against the graph-derived contracts. This separation preserves generative flexibility while making expected artifacts and constraints explicit.
ToolGraph grounding narrows the planning space but does not establish program safety, artifact correctness, or analytical optimality. These properties are assessed separately through PlanVerifier, sandbox execution, and post-execution artifact validation.
4.4.2. Verification and Artifact Validation
A generated program may be syntactically valid while remaining unsafe or incomplete. PlanVerifier checks Python syntax and abstract syntax tree structure, prohibited imports and external-access operations, consistency with aggregate-only and other privacy restrictions, references to expected artifacts, and registered MUST_CALL and MUST_WRITE constraints. The privacy-consistency check acts as a defense-in-depth control after TCPG rather than replacing the earlier privacy decision.
Let denote the expected artifacts referenced by candidate program . Contract coverage is
Post-execution artifact validation uses only the user-visible task specification and the artifacts generated in the local run directory; it does not inspect hidden benchmark answers.
The coverage score is a lightweight completeness signal rather than evidence of correctness. Execution admission depends on all blocking checks, and artifact validity is assessed again after sandbox execution. Programs with non-repairable violations, or programs that exhaust the repair budget, are denied access to the local dataset.
4.4.3. Code-Down Protocol and Failure Recovery
Cloud assistance may be selected initially by MJR or invoked after a detectable edge-side failure. In both cases, communication between and follows the interface defined in Section 3.1, and the cloud receives only a policy-approved representation of the task.
Before transmission, the complete serialized message must satisfy
where denotes the reversible masking map retained at the edge. The message is additionally checked against an outbound allowlist and scanned for recognizable sensitive values. These controls operationalize Equation (4) under the adopted detection model; they do not constitute a formal non-inference guarantee.
The Code-Down workflow is
Consistent with Section 3.1, returns a structured planning response, and denotes the candidate-program component of that response. The operator performs bounded verification and repair. The cloud has no execution path to ; every returned program is verified and executed at the edge.
Failure-Driven Recovery
Recovery is applied sequentially and is guided by observable failure evidence. A repairable static violation produces structured blocking reasons, which are combined with the applicable policies in and returned to the active planner for one constraint-guided revision. The revised program must pass PlanVerifier again before entering the sandbox. If an admitted program fails during execution, its exception type, execution trace, and standard error are returned to the edge planner under a bounded retry budget. These signals provide localized evidence for repairing syntax, dependency, API-usage, execution, and artifact-generation failures.
If bounded repair does not produce an admissible and executable program, a deterministic fallback may construct a contract-oriented program for supported task categories. The fallback is also verified and executed locally. Persistent execution failure or post-execution detection of missing or malformed artifacts may trigger cloud-assisted replanning only when TCPG permits cloud interaction. Every cloud-generated or repaired program subsequently re-enters the same verification, sandbox-execution, and artifact-validation pipeline.
Recovery is triggered only by observable signals, including static-policy violations, syntax errors, prohibited operations, runtime exceptions, and missing or malformed artifacts. It therefore improves robustness against detectable failures but cannot guarantee the correctness of plausible yet semantically incorrect analytical results.
5. Experiments
We organize the evaluation around three research questions: (RQ1) What trade-off does FinDS-Agent achieve among analytical utility, cloud-resource consumption, and end-to-end latency relative to edge-only and cloud-planner-only configurations? (RQ2) How effectively does FinDS-Agent restrict observable sensitive-data disclosure during cloud-assisted planning under the adopted privacy-benchmark audit? (RQ3) How do the principal components of FinDS-Agent contribute to its analytical utility, privacy enforcement, cloud utilization, and execution reliability?
5.1. Experimental Design
5.1.1. Benchmarks
DataSciBench
DataSciBench is used to evaluate general data science capability. It covers data processing, statistical analysis, visualization, machine learning, data mining, and report generation. Each task requires the agent to interpret an analytical request, generate and execute code, and produce task-specific artifacts over tabular data.
The experimental split contains unique task prompts associated with 519 Intention–Function–Code and Task–Function–Code evaluation entries. Because one prompt may contain multiple requirements, the 519 entries are not treated as independent tasks as shown in Table 1. All 222 prompts are included in the denominators of Completion Rate and Success Rate. A timed-out, rejected, or failed task receives zero credit for unsatisfied criteria.
DataSciBench was selected because its executable and artifact-oriented evaluation directly measures the planning, code-generation, verification, and recovery capabilities targeted by FinDS-Agent. DataSciBench remains the principal capability benchmark; Section 5.2 and Appendix A therefore report separate DS-1000, KramaBench, and bounded DSBench transfer evidence without averaging their official metrics.
FinDS-Privacy-Bench
FinDS-Privacy-Bench evaluates privacy protection in Chinese financial data science. It contains 200 manually specified requests over four synthetic datasets representing personal banking, human-resource management, loan applications, and securities trading. Synthetic data avoid the use of real customer records while preserving representative schemas and privacy-sensitive task patterns.
Each request is annotated with sensitive columns, data-classification labels, an expected privacy action, and a privacy-complexity level . These levels range from direct row-level disclosure to indirect semantic-disclosure risks and are distinct from the – analytical-complexity levels used by MJR. The benchmark contains 48 , 68 , 54 , and 30 tasks. Among the 200 requests, 195 involve sensitive fields, and 20 constitute an obfuscated-column subset requiring semantic recognition.
Two annotators with experience in financial data governance independently labeled the benchmark, and disagreements were resolved by consensus. The gold actions are block, sanitize, and allow, corresponding respectively to rejection, protected or aggregate-only processing, and policy-compliant processing without additional suppression.
Only 5 of the 200 requests have the gold action allow. This count is insufficient to estimate benign-query false blocking, unnecessary intervention, or over-sanitization with useful stability. We define false blocking as predicting block for a gold-allow request; unnecessary intervention as predicting block or sanitize for a gold-allow request; and over-sanitization as removing information or functionality that policy did not require to be removed from an otherwise admissible request. Consequently, the 97.5% action accuracy is dominated by class imbalance and is not evidence of a low false-blocking rate. A separate balanced set of 100 requests (40 allow, 30 sanitize, and 30 block) was used only as a development diagnostic; it was not an independently adjudicated confirmatory test, and no performance number from it is reported as a main result.
A predefined 100-task subset was withheld from TCPG prompt refinement and used as a blind split. Because the complete 200-task results include this subset, the full-set results are reported descriptively rather than as an independent test set.
5.1.2. Compared Configurations
The DataSciBench evaluation includes four locally evaluated configurations and one externally reported capability reference:
- Edge-Only (w/o FinDS-Agent): The local 7B model directly generates code without MJR, ToolGraph, Code-Down, or failure-driven escalation;
- Edge-Only: FinDS-Agent is used with cloud assistance disabled, so planning, verification, repair, and execution remain at the edge;
- FinDS-Agent (Cloud–Edge): The complete system, in which TCPG enforces privacy constraints and MJR selects edge or cloud-assisted planning;
- Cloud-Planner-Only: MJR is bypassed and the cloud planner is used for every non-blocked task. Privacy screening, sanitized context, edge-side verification, and local execution are retained.
- GPT-4o-2024-05-13 (Official Reference): The result of GPT-4o-2024-05-13 is taken from the official DataSciBench evaluation on the complete benchmark. It provides an external capability reference but was not reproduced under our prompting, execution, evaluator, or hardware environment. It is therefore excluded from controlled performance, cloud-efficiency, and latency comparisons.
The privacy evaluation compares Rule-Only, Full TCPG, and Cloud-Raw. Rule-Only excludes local semantic review; Full TCPG uses all three privacy stages; and Cloud-Raw intentionally bypasses privacy screening and sanitization. Cloud-Raw is used only as an unsafe privacy baseline and is not a permitted FinDS-Agent deployment mode.
5.1.3. Evaluation Metrics
Analytical utility is measured using the Completion Rate (CR) and Success Rate (SR). For task , let and denote the numbers of satisfied and applicable evaluation criteria. Then
CR measures criterion-level completion and permits partial credit, whereas SR requires all applicable criteria and artifacts to be satisfied.
Cloud Utilization Rate (CUR) measures the proportion of route-eligible task-runs that invoke the cloud planner at least once:
where , is the set of non-blocked route-eligible tasks in run , and is the number of cloud-planner invocations. Initial cloud selection and failure-driven escalation are both counted, but each task-run contributes at most once.
Average Cloud-Token Consumption (CTC) is
where task-runs without cloud invocation contribute zero tokens. CTC is reported in thousands of tokens per task-run.
Average end-to-end latency is
The measurement includes context construction, privacy assessment, routing, planning, verification, repair, execution, and artifact validation.
For FinDS-Privacy-Bench, sensitive-field detection is evaluated using column-level precision, recall, and F1. Binary action accuracy evaluates whether the system correctly distinguishes tasks requiring protection (block or sanitize) from tasks that may be allowed. Privacy-Benchmark Sensitive Leak Rate is defined as
where is the number of benchmark tasks, counts sensitive tasks incorrectly allowed without protection, and counts externally processed tasks whose sanitized payload retains at least one gold sensitive field.
Column-level recall and measure different properties. Recall includes sensitive fields from tasks that are subsequently blocked, whereas records only disclosure along an externally exposed processing path. Thus, a missed field in a blocked task reduces recall but does not constitute payload leakage. A measured of zero indicates no leakage observed under the benchmark and audit rules, not a formal zero-leakage guarantee.
5.1.4. Implementation Details
Experiments were conducted on an edge node with an NVIDIA RTX 3090 GPU (24 GB), an Intel Xeon CPU, and 128 GB RAM under Ubuntu 22.04 LTS. The implementation used Python 3.11.8, PyTorch 2.6.0, CUDA 12.4, pandas 2.2.3, NumPy 2.1.1, and scikit-learn 1.6.
Qwen2.5-7B-Instruct served as the edge planner, and the official DeepSeek-V4-flash API served as the cloud planner. Unless otherwise stated, the MJR routing margin was , the mining threshold was , and the repair budget was . MiMo-v2.5 was used as the evaluator with temperature 1, caching disabled, and semantic shim disabled.
All comparable configurations use identical task inputs, artifact requirements, execution environments, and evaluation functions. All locally evaluated configurations are evaluated in three repeated runs over the same 222 DataSciBench tasks, yielding 666 task-runs per configuration.
5.1.5. Statistical Analysis
Run-to-run variability is summarized by the sample standard deviation, and run-level 95% intervals use Student’s distribution. Table 2 uses only run identifiers 0–2 from the same archived three-run batch as its reported means. For paired method contrasts, the unique task is the inferential unit: repeats are averaged within task, task-cluster paired bootstrap intervals are computed by resampling tasks, paired Wilcoxon tests assess signed differences, and Holm adjustment is applied within each CR or SR hypothesis family. Repeated executions are not treated as additional independent tasks or privacy scenarios. Zero-event proportions report the event count, denominator, and two-sided Wilson 95% interval. Scores from different benchmarks are kept separate rather than averaged.
5.2. Main Results
5.2.1. Analytical Utility, Cloud Efficiency, and Latency
To answer RQ1, we compare FinDS-Agent with the edge-only and cloud-planner-only configurations in terms of analytical utility, cloud utilization, cloud-token consumption, and end-to-end latency. Table 2 reports results for the four locally evaluated configurations together with the externally reported GPT-4o-2024-05-13 reference.
Introducing the FinDS-Agent workflow without cloud assistance increases CR from 55.20% to 60.43%, while SR changes from 51.05% to 51.35%. This result suggests that constrained planning and verification primarily improve criterion-level completion under the capability limit of the local model.
FinDS-Agent with cloud–edge collaboration achieves a CR of 69.93% and an SR of 57.06%, improving over Edge-Only by 9.50 and 5.71 percentage points, respectively. Only 32.27% of route-eligible task-runs invoke the cloud planner, with an average consumption of 2.82 K cloud tokens per task-run. Selective cloud assistance therefore improves utility while most eligible task-runs remain edge planned.
Cloud-Planner-Only achieves the highest CR and SR, at 80.48% and 65.32%, but invokes the cloud for every route-eligible task-run and consumes 5.81 K cloud tokens per task-run. Relative to this configuration, FinDS-Agent retains 86.9% of CR and 87.4% of SR while reducing CUR by 67.73 percentage points and CTC by 51.4%.
The official DataSciBench evaluation reports a CR of 68.44% and an SR of 66.31% for GPT-4o-2024-05-13 on the complete benchmark. These values are reported as an external capability reference. Because the model was not rerun under our prompting, execution, evaluator, or hardware settings, the result is not used for controlled performance, cloud-efficiency, or latency comparisons.
FinDS-Agent has an average latency of 20.19 s, compared with 19.17 s for Edge-Only and 16.31 s for Cloud-Planner-Only. The additional latency is associated with edge-side privacy assessment, routing, grounding, verification, and recovery. The results therefore characterize FinDS-Agent as an intermediate operating point between fully local planning and unconditional cloud planning, rather than as the highest-utility or lowest-latency configuration.
The corresponding three-run 95% intervals for CR/SR are 51.36–59.05%/45.76–56.34% for Edge-Only without FinDS-Agent, 55.70–65.16%/48.39–54.31% for Edge-Only, 66.44–73.41%/54.24–59.87% for FinDS-Agent, and 78.82–82.13%/64.20–66.43% for Cloud-Planner-Only. These wide intervals reflect the three-run uncertainty and are not replaced by statistics from the separate ten- or five-run batches.
Under a separate same-model comparison, FinDS-Agent, Data Interpreter/MetaGPT, Open Interpreter, and a Mini-Agent v1 execution-substrate baseline processed the same 222 DataSciBench tasks with Qwen2.5-7B-Instruct, a common 900-s outer task budget, and three runs per framework (2664 task-runs). Mean CR was 66.02%, 60.02%, 19.27%, and 48.02%, respectively. The paired FinDS-Agent-minus-Data-Interpreter difference was 6.00 percentage points with a task-cluster bootstrap 95% interval of to 12.10 percentage points, so we do not claim statistical superiority. Internal interaction loops, model-call counts, token use, and a disclosed two-way deadline-recovery phase for unfinished Mini-Agent/Open-Interpreter cells differed; the comparison is therefore observational under the disclosed configurations, and latency, cost, token, completion, or failure differences cannot be attributed solely to architecture. Appendix A (Table A2) reports the full summary.
External evaluation tested transfer beyond DataSciBench. On DS-1000 (1000 tasks per arm), pass rates were 33.2% for Edge-Only, 53.1% for FinDS-Agent, and 62.3% for Cloud-Only. On KramaBench (104 tasks and three seeds per arm), mean official scores were 0.1971 for Edge-Only, 0.5093 for the frozen FinDS-Agent route, and 0.5368 for Cloud-Only Sanitized. A pilot-trained learned router and a random router matched at a 40.38% initial cloud budget scored 0.4503 and 0.4829; their paired difference was (95% bootstrap CI: to 0.0095), providing no same-budget advantage for the learned candidate. The pilot-trained router is exploratory because its training set was used for candidate selection. DSBench contained 27 cases and 306 questions, but its benchmark-informed routing arm is retained only as development evidence and not as sealed confirmation. Official scores across these benchmarks are not averaged. Appendix A (Table A4) provides the external summary.
5.2.2. Privacy Protection Under the Adopted Audit Protocol
To answer RQ2, we evaluate the privacy-protection behavior of FinDS-Agent on FinDS-Privacy-Bench. The evaluation compares the complete privacy pipeline with rule-only screening and the unsafe Cloud-Raw baseline, covering sensitive-field detection, privacy-action decisions, and observable leakage through externally exposed payloads. Table 3 reports the privacy results on FinDS-Privacy-Bench. The local semantic reviewer receives only column metadata and limited de-identified samples from the first 50 rows, rather than complete records.
Table 3.
Privacy evaluation on FinDS-Privacy-Bench. Recall includes fields from blocked tasks, whereas the last column reports observed payload-leakage events and two-sided Wilson 95% intervals on externally exposed benchmark paths.
On the full benchmark, TCPG increases sensitive-field recall from 58.20% to 98.10% (+39.90 percentage points) and changes precision from 28.60% to 25.60% ( percentage points). The observed payload-leakage count decreases from the Rule-Only rate of 30.00% to 0/200 for Full TCPG; on the blind split it is 0/100. The two-sided Wilson 95% intervals are 0–1.8845% and 0–3.6993%, respectively. Thus, no payload-leakage event was observed under the specified benchmark, audit procedure, and threat model; the intervals and uncovered threats preclude a claim of zero true leakage probability.
The recall gain and precision loss reflect a conservative, recall-oriented policy. Because only five full-set requests are labeled allow, the benchmark cannot establish reliable false-blocking, unnecessary-intervention, or over-sanitization rates. Cloud-Raw exposes sensitive content in 195/200 requests and is an intentionally unsafe baseline, not a deployable FinDS-Agent mode.
The utility-side privacy-gate intervention was evaluated separately on DataSciBench: the configuration without privacy-gate enforcement obtained 69.99% CR and 57.06% SR, whereas full FinDS-Agent obtained 69.93% and 57.06%, a CR change of percentage points with unchanged SR. Because the privacy and utility changes come from different benchmarks, they do not form a same-request Pareto curve and do not identify an optimal privacy–utility operating point.
The audit covers gold sensitive fields and unmasked gold values in serialized cloud-visible payloads. It does not establish differential privacy, cryptographic confidentiality, non-interference, or safety against reconstruction, membership inference, side channels, malicious exfiltration code, short-value ambiguity, or secondary use of cloud logs. A six-category adversarial privacy set was used only for development diagnostics and is not reported as confirmatory attack performance.
5.2.3. Component Analysis
To answer RQ3, we conduct controlled component interventions to quantify the contributions of privacy-gate enforcement, ToolGraph grounding, and failure-driven recovery. Table 4 reports their effects on analytical utility, cloud utilization, token consumption, and end-to-end latency. Edge-Only and Cloud-Planner-Only are treated as deployment-boundary configurations rather than component ablations.
Table 4.
Component analysis of FinDS-Agent. CR and SR are mean sample SD over the original three-run batch. The separate five-run expanded ablation is reported in Appendix A.
Removing privacy-gate enforcement produces negligible change on DataSciBench: CR changes by 0.06 percentage points and SR remains unchanged. This result indicates that privacy enforcement introduces little utility loss on the capability benchmark; it does not measure the privacy contribution of TCPG, which is evaluated separately on FinDS-Privacy-Bench.
Removing ToolGraph reduces CR by 7.55 percentage points and SR by 2.11 percentage points. The larger reduction in CR suggests that explicit artifact and dependency constraints primarily improve criterion-level completion. Because ToolGraph removal also changes cloud utilization and failure behavior, these values represent its system-level contribution rather than an isolated prompt effect.
Removing failure-driven repair produces the largest degradation: CR decreases by 11.75 percentage points and SR by 7.66 percentage points. CUR simultaneously decreases from 32.27% to 7.80% because repair-triggered cloud escalation is disabled. The observed difference therefore represents the joint contribution and resource cost of bounded repair, fallback, and conditional escalation.
Overall, the component analysis associates ToolGraph with more complete artifact production and failure recovery with robustness against detectable verification, execution, and artifact failures. The privacy contribution of TCPG is supported by the dedicated privacy benchmark rather than the DataSciBench utility ablation.
For the three matching runs, the 95% intervals are 66.44–73.41% CR and 54.24–59.87% SR for the full system, 60.65–64.11% and 53.84–56.07% without ToolGraph, and 53.97–62.40% and 45.47–53.33% without repair. A clean matching run-level archive was not recoverable for the original 69.99% privacy-gate intervention, so no interval is imputed for that row.
The separate R15 analysis used nine arms, 222 paired tasks, and five runs (9990 task-runs). Averaging repeats within task, repair increased CR by 9.29 percentage points (bootstrap 95% CI: 6.04–12.68; Holm-adjusted ) and SR by 6.85 points (3.15–10.63; adjusted ). ToolGraph increased CR by 5.21 points (1.27–9.09; adjusted ), whereas its SR effect was 0.90 points (–4.82; adjusted ). The CR interaction estimate was 4.33 points with a bootstrap interval crossing zero (–9.55), despite an adjusted Wilcoxon ; we therefore treat the interaction as weak and inference-method dependent. The SR interaction was 1.08 points (–6.40; adjusted ). Raw and sanitized cloud-only arms are not interchangeable, and five payload events were observed in the sanitized cloud-only arm. Table A5 and Table A6 report the distinct five-run results and factorial effects.
6. Failure Analysis
To better understand the remaining limitations of FinDS-Agent, we manually analyzed the unsuccessful tasks and categorized them according to their primary root causes. As summarized in Table 5, functional correctness errors constitute the largest proportion of failures (52.1%), followed by result-quality deficiencies (33.2%). Together, these two categories account for more than 85% of all failure cases, whereas environment-related issues, artifact generation failures, dependency problems, and schema inconsistencies represent only a small fraction of the remaining failures.
Table 5.
Distribution of failure categories observed in the experimental runs.
The analysis reveals that the primary challenge for LLM-driven data science agents is no longer execution feasibility but task-level semantic correctness. Most unsuccessful cases are not caused by runtime exceptions; instead, the generated programs can be executed but fail to satisfy the expected analytical objectives, such as incomplete analytical workflows, incorrect statistical procedures, or outputs that do not meet the evaluation requirements. This observation indicates that successful code execution does not necessarily imply successful task completion.
Furthermore, the relatively small proportion of environment- and dependency-related failures suggests that the proposed execution framework, including ToolGraph-guided tool orchestration and the verification–repair pipeline, effectively mitigates common runtime issues. The remaining limitations are mainly associated with semantic reasoning and analytical correctness, which remain challenging for current LLM-based autonomous agents.
Overall, these findings suggest that future improvements should focus on task-level verification, schema-aware semantic validation, and result-quality assessment, rather than relying solely on iterative execution repair. Such mechanisms are expected to further improve the reliability and robustness of cloud–edge collaborative data science agents.
Multi-Cause Failure Handling
The runtime processes compound failures in a bounded order: policy violations terminate or sanitize before routing; planning/tool-format errors trigger constrained replanning; code-execution and verifier/artifact errors trigger one local repair; and eligible unresolved failures may invoke a bounded sanitized cloud fallback. Once the repair/fallback budget is exhausted, the system terminates safely and retains the diagnostic record rather than treating an incomplete artifact as success.
We evaluated 1260 fault cases plus 60 controls spanning six single faults and 15 fault pairs. Of 1871 activated fault instances, micro-averaged identification F1 was 85.08%. Automatic recovery completed 805/1222 cases with an activated fault (65.88%); 198 remaining terminal cases satisfied the safe-termination contract (198/198). Under fault injection, official CR and SR were 43.43% and 28.65%, and mean CR decreased by 7.06 percentage points relative to matched controls. Masking and non-additive interactions were common, so these outcomes do not imply independent detection or recovery of every configured cause. The result demonstrates bounded recovery and safe termination under the injected conditions, not failure immunity.
7. Conclusions
This paper presented FinDS-Agent as a policy-governed cloud–edge data-science workflow in which screened cloud planning is separated from edge-side data custody, verification, and execution. The contribution is the coordinated use of TCPG, MJR, ToolGraph contracts, and bounded recovery under explicit empirical boundaries.
On the original three-run DataSciBench batch, FinDS-Agent achieved 69.93% CR and 57.06% SR while using the cloud in 32.27% of eligible task-runs. Same-model framework results were observational rather than architecture-only causal evidence. Transfer results improved over Edge-Only on DS-1000 and KramaBench but remained below cloud-only variants. The five-run factorial analysis supported CR effects for repair and ToolGraph, while the ToolGraph SR effect and both interaction interpretations were limited or inconsistent.
On FinDS-Privacy-Bench, TCPG increased sensitive-field recall to 98.10%, but precision decreased to 25.60%. No payload-leakage event was observed in the full set (0/200; Wilson 95% CI: 0–1.8845%) or blind split (0/100; 0–3.6993%) under the specified audit and threat model. This is an empirical zero-event observation, not a formal privacy guarantee. The five allowed requests cannot establish low false blocking, and the balanced privacy/attack sets remain development diagnostics.
Limitations include fixed hand-selected routing parameters; the version difference between the main router and the exploratory zero-bias sensitivity batch; a post-hoc pilot-overlap audit; only three main-result runs; benchmark, model, and evaluator dependence; non-identical framework interaction budgets; and untested reconstruction, membership-inference, side-channel, malicious-code, and provider-log threats. Future work should preregister calibration and overlap tests, evaluate larger independently adjudicated benign/privacy sets, and compare learned and rule-based routing under identical data and cloud budgets.
Author Contributions
Conceptualization, X.D. and C.W.; Methodology, X.D., R.D. and C.W.; Software, C.W. and S.H.; Experiments, C.W., S.H. and F.Z.; Testing, C.W., S.H., and F.Z.; Writing—original draft preparation, X.D.; Writing—review and editing, R.D., S.H., F.Z., S.F. and Z.L.; Supervision, S.H., F.Z. and Z.L. All authors have read and agreed to the published version of the manuscript.
Funding
This work was supported by grants from the National Natural Science Foundation of China (72595845, 72595840).
Data Availability Statement
DataSciBench is publicly available from its original repository. The FinDS-Privacy-Bench datasets, annotations, evaluation scripts, and anonymized experimental results are available from the corresponding author upon reasonable request. The source code and configuration files will be made publicly available upon publication.
Conflicts of Interest
Author Cheng Wang was employed by GienTech Technology Co., Ltd. The remaining authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Appendix A. Supplementary Evidence
Table A1.
Component positioning and claim boundary.
Table A2.
Same-model DataSciBench framework comparison (222 tasks × 3 runs per framework).
Table A3.
Exploratory sensitivity of the separate zero-bias MJR version (222 tasks × 3 runs per profile).
Table A4.
External benchmark and matched-routing summary.
Table A5.
Separate five-run nine-arm ablation. CR interval is a run-level Student interval; cloud-call rate and payload events are descriptive.
Table A6.
Task-cluster factorial effects from the five-run ablation (222 paired tasks).
References
- Luo, Y.; Li, G.; Fan, J.; Tang, N. Data Agents: Levels, State of the Art, and Open Problems. In Proceedings of the 2026 International Conference on Management of Data (SIGMOD-Companion), Bengaluru, India, 31 May–5 June 2026; pp. 571–579. [Google Scholar] [CrossRef] [Scilit]
- Yao, S.; Zhao, J.; Yu, D.; Du, N.; Shafran, I.; Narasimhan, K.; Cao, Y. ReAct: Synergizing Reasoning and Acting in Language Models. In Proceedings of the 11th International Conference on Learning Representations (ICLR), Kigali, Rwanda, 1–5 May 2023. [Google Scholar]
- Schick, T.; Dwivedi-Yu, J.; Dessì, R.; Raileanu, R.; Lomeli, M.; Hambro, E.; Zettlemoyer, L.; Cancedda, N.; Scialom, T. Toolformer: Language Models Can Teach Themselves to Use Tools. Adv. Neural Inf. Process. Syst. 2023, 36, 68539–68551. [Google Scholar] [CrossRef] [Scilit]
- Shinn, N.; Cassano, F.; Gopinath, A.; Narasimhan, K.; Yao, S. Reflexion: Language Agents with Verbal Reinforcement Learning. Adv. Neural Inf. Process. Syst. 2023, 36, 8634–8652. [Google Scholar] [CrossRef] [Scilit]
- Biswas, S.; Wardat, M.; Rajan, H. The Art and Practice of Data Science Pipelines: A Comprehensive Study of Data Science Pipelines in Theory, in-the-Small, and in-the-Large. In Proceedings of the 44th International Conference on Software Engineering (ICSE), Pittsburgh, PA, USA, 21–29 May 2022; pp. 2091–2103. [Google Scholar] [CrossRef] [Scilit]
- Lai, Y.; Li, C.; Wang, Y.; Zhang, T.; Zhong, R.; Zettlemoyer, L.; Yih, W.-T.; Fried, D.; Wang, S.; Yu, T. DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation. Proc. Mach. Learn. Res. 2023, 202, 18319–18345. [Google Scholar]
- Jing, L.; Huang, Z.; Wang, X.; Yao, W.; Yu, W.; Ma, K.; Zhang, H.; Du, X.; Yu, D. DSBench: How Far Are Data Science Agents from Becoming Data Science Experts? In Proceedings of the 13th International Conference on Learning Representations (ICLR), Singapore, 24–28 April 2025. [Google Scholar]
- Chan, J.S.; Chowdhury, N.; Jaffe, O.; Aung, J.; Sherburn, D.; Mays, E.; Starace, G.; Liu, K.; Maksin, L.; Patwardhan, T.; et al. MLE-Bench: Evaluating Machine Learning Agents on Machine Learning Engineering. In Proceedings of the 13th International Conference on Learning Representations (ICLR), Singapore, 24–28 April 2025. [Google Scholar]
- Zhang, D.; Zhoubian, S.; Cai, M.; Li, F.; Yang, L.; Wang, W.; Dong, T.; Hu, Z.; Tang, J.; Yue, Y. DataSciBench: An LLM Agent Benchmark for Data Science. In Proceedings of the Association for Computational Linguistics: ACL 2026, San Diego, CA, USA, 2–7 July 2026; pp. 3685–3728. [Google Scholar] [CrossRef] [Scilit]
- Chen, Z.; Li, S.; Smiley, C.; Ma, Z.; Shah, S.; Wang, W.Y. ConvFinQA: Exploring the Chain of Numerical Reasoning in Conversational Finance Question Answering. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP), Abu Dhabi, United Arab Emirates, 7–11 December 2022; pp. 6279–6292. [Google Scholar] [CrossRef] [Scilit]
- Li, S.; Wang, H.; Xu, W.; Zhang, R.; Guo, S.; Yuan, J.; Zhong, X.; Zhang, T.; Li, R. Collaborative Inference and Learning between Edge SLMs and Cloud LLMs: A Survey of Algorithms, Execution, and Open Challenges. arXiv 2025, arXiv:2507.16731. [Google Scholar] [CrossRef] [Scilit]
- Ong, I.; Almahairi, A.; Wu, V.; Chiang, W.-L.; Wu, T.; Gonzalez, J.E.; Kadous, M.W.; Stoica, I. RouteLLM: Learning to Route LLMs with Preference Data. In Proceedings of the 13th International Conference on Learning Representations (ICLR), Singapore, 24–28 April 2025. [Google Scholar]
- Hao, Z.; Jiang, H.; Jiang, S.; Ren, J.; Cao, T. Hybrid SLM and LLM for Edge–Cloud Collaborative Inference. In Proceedings of the Workshop on Edge and Mobile Foundation Models (EdgeFM), Tokyo, Japan, 3–7 June 2024; pp. 36–41. [Google Scholar] [CrossRef] [Scilit]
- Dong, J.; Li, J.; Zheng, T.; Lin, W. HybridFlow: Resource-Adaptive Subtask Routing for Efficient Edge–Cloud LLM Inference. In Proceedings of the 43rd International Conference on Machine Learning (ICML), Seoul, Republic of Korea, 6–11 July 2026. [Google Scholar]
- Hollmann, N.; Müller, S.; Hutter, F. Large Language Models for Automated Data Science: Introducing CAAFE for Context-Aware Automated Feature Engineering. Adv. Neural Inf. Process. Syst. 2023, 36, 44753–44775. [Google Scholar] [CrossRef] [Scilit]
- Yang, J.; Jimenez, C.E.; Wettig, A.; Lieret, K.; Yao, S.; Narasimhan, K.; Press, O. SWE-Agent: Agent–Computer Interfaces Enable Automated Software Engineering. In Proceedings of the 38th Annual Conference on Neural Information Processing Systems (NeurIPS 2024), Vancouver, BC, Canada, 10–15 December 2024. [Google Scholar]
- Fan, Z.; Gao, X.; Mirchev, M.; Roychoudhury, A.; Tan, S.H. Automated Repair of Programs from Large Language Models. In Proceedings of the 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE), Melbourne, Australia, 14–20 May 2023; pp. 1469–1481. [Google Scholar]
- Ghareeb, A.E.; Chang, B.; Mitchener, L.; Yiu, A.; Szostkiewicz, C.J.; Laurent, J.M.; Wright, S.M.; Razzak, M.T.; White, A.D.; Hinks, M.M.; et al. A Multi-Agent System for Automating Scientific Discovery. Nature 2026, 655, 497–505. [Google Scholar] [CrossRef] [Scilit] [PubMed]
- Xia, T.; Hu, L.; Sun, Y.; Xu, M.; Xu, L.; Wang, S.; Xu, W.; Jiang, J. GraSP: Graph-Structured Skill Compositions for LLM Agents. arXiv 2026, arXiv:2604.17870. [Google Scholar] [CrossRef] [Scilit]
- Anthropic. Model Context Protocol Specification. Available online: https://modelcontextprotocol.io/ (accessed on 19 July 2026).
- Google. Agent2Agent Protocol (A2A). Available online: https://github.com/google/A2A (accessed on 19 July 2026).
- Gao, L.; Madaan, A.; Zhou, S.; Alon, U.; Liu, P.; Yang, Y.; Callan, J.; Neubig, G. PAL: Program-Aided Language Models. In Proceedings of the 40th International Conference on Machine Learning (ICML), Honolulu, HI, USA, 23–29 July 2023; pp. 10764–10799. [Google Scholar]
- Madaan, A.; Tandon, N.; Gupta, P.; Hallinan, S.; Gao, L.; Wiegreffe, S.; Alon, U.; Dziri, N.; Prabhumoye, S.; Yang, Y.; et al. Self-Refine: Iterative Refinement with Self-Feedback. Adv. Neural Inf. Process. Syst. 2023, 36, 46534–46594. [Google Scholar] [CrossRef] [Scilit]
- Evans, J.; Bratton, B. Agentic AI and the Next Intelligence Explosion. Science 2026, 391, eaeg1895. [Google Scholar] [CrossRef] [Scilit] [PubMed]
- Zharmagambetov, A.; Guo, C.; Evtimov, I.; Pavlova, M.; Salakhutdinov, R.; Chaudhuri, K. AGENTDAM: Privacy Leakage Evaluation for Autonomous Web Agents. In Proceedings of the 39th Conference on Neural Information Processing Systems, Datasets and Benchmarks Track (NeurIPS), San Diego, CA, USA, 2–7 December 2025. [Google Scholar]
- Wang, S.; Yu, F.; Liu, X.; Qin, X.; Zhang, J.; Lin, Q.; Zhang, D.; Rajmohan, S. Privacy in Action: Towards Realistic Privacy Mitigation and Evaluation for LLM-Powered Agents. In Proceedings of the Association for Computational Linguistics: EMNLP 2025, Suzhou, China, 4–9 November 2025; pp. 17055–17074. [Google Scholar] [CrossRef] [Scilit]
- El Yagoubi, F.; Badu-Marfo, G.; Al Mallah, R. AgentLeak: A Benchmark for Internal-Channel Privacy Leakage in Multi-Agent LLM Systems. IEEE Access 2026, 14, 94960–94978. [Google Scholar] [CrossRef] [Scilit]
- Papadopoulou, A.; Yu, Y.; Lison, P.; Øvrelid, L. Neural Text Sanitization with Explicit Measures of Privacy Risk. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (AACL-IJCNLP), Online, 20–23 November 2022; Volume 1: Long Papers, pp. 217–229. [Google Scholar] [CrossRef] [Scilit]
- Ferrag, M.A.; Lakas, A.; Tihanyi, N.; Debbah, M. Securing LLM Agents: From Prompt Sanitization to Autonomous Red Teaming and Beyond. Internet Things Cyber-Phys. Syst. 2025, 5, 185–209. [Google Scholar] [CrossRef] [Scilit]
- Chen, Y.; Zhao, J.; Tang, B.; Wang, H.; Zhang, Y.; Huang, F.; Xiong, F.; Li, Z. MemPrivacy: Privacy-Preserving Personalized Memory Management for Edge–Cloud Agents. arXiv 2026, arXiv:2605.09530. [Google Scholar] [CrossRef] [Scilit]
- Li, N.; Zhang, K.; Polley, K.; Ma, J. Security Considerations for Artificial Intelligence Agents. arXiv 2026, arXiv:2603.12230. [Google Scholar] [CrossRef] [Scilit]
- Pilán, I.; Lison, P.; Øvrelid, L.; Papadopoulou, A.; Sánchez, D.; Batet, M. The Text Anonymization Benchmark (TAB): A Dedicated Corpus and Evaluation Framework for Text Anonymization. Comput. Linguist. 2022, 48, 1053–1101. [Google Scholar] [CrossRef] [Scilit]
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2026 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license.



