Context-Aware Sequential Ranking for Dynamic Test Case Prioritization in Continuous Integration

Abstract

Regression testing in continuous integration (CI) helps preserve delivered software value by exposing regressions early, but limited feedback windows make test order consequential. This study evaluates a leakage-safe dynamic test case prioritization framework that reconstructs job–build–commit provenance, combines strictly prior-build history with change–test similarity, applies class-conditional conformal calibration, and evaluates an operator-level deep Q-network (DQN). The primary held-out evaluation contains 273 failure-bearing jobs and 45,774 job–test rows from four Java projects. Pretrained history–semantic fusion achieved an equal-project macro failing-test-entity APFD surrogate (FTE-APFD surrogate) of 0.8840, compared with 0.8689 for history-only and 0.7023 for pretrained semantic-only ranking; the fusion–history difference did not survive Holm correction. A train-only TF–IDF control reached 0.7502 for semantic-only ranking and 0.8871 when fused with history in a seed-averaged diagnostic, so the benchmark does not establish unique superiority of the pretrained representation. A protocol-aligned RETECS reimplementation achieved 0.8296 ± 0.0111 FTE-APFD surrogate across five seeds. Adaptive conformal calibration reduced the candidate fraction from 0.3171 to 0.2598 while retaining 0.9058 failing-test coverage. The candidate-informed gated DQN remained below strong deterministic fusion rankings at 10%, 25%, and 50% budgets; five-seed, short-budget, hyperparameter, pre-execution-budget, and project-exclusion sensitivities did not establish a consistent DQN advantage. An additional three-project operational analysis containing 444 passing test jobs showed high failing-row coverage but strongly project-dependent candidate burden. Overall, the evidence favors strong leakage-safe deterministic rankings with explicit uncertainty control, while the evaluated reinforcement-learning design remains a local negative result rather than a general conclusion about reinforcement learning.

1. Introduction

Continuous integration and continuous delivery shorten the intervals between code change, automated verification, and developer feedback. In agile development, however, the objective of regression testing is not to create business value directly. Its practical role is to protect already delivered functionality and to reveal changes that may degrade that value before they propagate further through the delivery process. Recent value-aware regression-testing work makes this distinction explicit by linking test decisions to the preservation of business value under CI constraints [1]. In this setting, earlier exposure of regression-related failures is useful because it can reduce feedback latency and the time for which a potentially value-degrading change remains unresolved.

As systems evolve, regression suites can become too expensive to execute completely within every feedback window. Regression testing therefore uses complementary strategies such as minimization, selection, and prioritization [2,3]. This study focuses on test case prioritization (TCP): it orders the available tests so that the most informative prefix is executed first. Prioritization is deliberately distinguished from permanent selection. A non-candidate test is never removed from the suite in the proposed framework; it can still execute after higher-priority tests or in a later part of the CI window. This choice is appropriate when the immediate question is which tests should run first under a constrained feedback budget, rather than which tests may be omitted entirely. To reflect particularly short agile/CI feedback windows, the evaluation includes an additional 5% budget sensitivity alongside the original 10%, 25%, and 50% settings.

Learning-based TCP usually relies on prior verdicts, failure frequency, execution duration, coverage-related variables, and other historical signals [3,4,5,6]. These features are valuable but mostly retrospective. They may not represent whether the meaning of the current source-code change is related to a particular test. The present study therefore evaluates change–test similarity as an additional contextual signal. The goal is not to assume that a pretrained representation must outperform simpler alternatives. The experiments explicitly compare the frozen code-language representation with a train-only TF–IDF lexical baseline under the same chronological protocol.

Dynamic TCP is treated here as a finite sequential ordering problem with uncertain failures, evolving code context, and limited execution resources. The framework combines exact provenance reconstruction, leakage-safe historical risk, change–test similarity, set-valued conformal calibration, and an operator-level sequential controller. These components are established techniques or adaptations of established techniques; the contribution is their integration, formalization under explicit admissible-information constraints, and controlled empirical evaluation. Accordingly, this study is positioned primarily as software-engineering and regression-testing research rather than as a claim of a new general-purpose machine-learning algorithm.

The framework reconstructs many-to-many job–build–commit provenance, represents versioned commit hunks and test artifacts, estimates project-specific failure risk from strictly prior-build history and semantic context, and produces class-conditional conformal failure-candidate sets. Four independently trained DQN policies then select among six deterministic ranking operators. A protocol-aligned RETECS reimplementation provides an external learning-based TCP baseline, while TF–IDF provides a lightweight lexical control. Additional analyses evaluate a 5% budget, five-seed DQN stability, limited learning-rate/architecture/training-duration sensitivity, a pre-execution duration budget and reward normalization, Buck- and OkHttp-excluded results, and an operational population that includes passing jobs where archival build provenance can be reconstructed.

The main contributions are summarized as follows:

• Exact archived many-to-many job–build–commit provenance is reconstructed, and dynamic TCP is formalized as a leakage-safe sequential ordering problem over failing-test exposure and execution cost.
• Versioned source-code changes are aligned with test artifacts using a frozen pretrained code-language encoder, and the resulting semantic signal is compared with both history-only risk and a project-specific train-only TF–IDF lexical control.
• Static and build-wise adaptive class-conditional conformal failure-candidate sets are evaluated as an uncertainty-communication and candidate-set-control mechanism rather than as an assumed ranking improvement or hard pruning rule.
• A clean $2\times 2$ DQN ablation separates candidate-state information from candidate-first gating, and a protocol-aligned RETECS reimplementation provides an external TCP comparison under the same chronological data split.
• Robustness is examined through very short budgets, five DQN seeds, limited hyperparameter/training-duration diagnostics, a pre-execution duration budget and reward normalization, alternative duration treatments, project exclusion, and a three-project passing-job operational sensitivity.

The contributions are therefore methodological and empirical. The study does not claim a new conformal-validity theorem, a new reinforcement-learning convergence result, or algorithmic novelty in logistic risk modeling, TF–IDF, split conformal prediction, adaptive conformal updating, RETECS, or DQN optimization. The formal contribution lies in the admissible-information restrictions, chronology, stopping rules, candidate-set construction, and fixed operator-level control problem; the engineering contribution lies in provenance reconstruction and versioned change–test alignment; and the empirical contribution is a controlled evaluation that also reports negative and non-significant findings. In particular, a non-significant fusion–history comparison is interpreted as insufficient evidence of uniform superiority, not as evidence of equivalence.

The remainder of this paper is organized as follows: Section 2 reviews regression-test prioritization in CI, value-aware and context-aware testing, semantic representations, reinforcement learning, and conformal uncertainty. Section 3 presents the formal problem. Section 4 describes provenance reconstruction, historical and semantic modeling, lexical and external baselines, conformal calibration, and sequential control. Section 5 defines the evaluation protocol and measures. Section 6 reports the findings, Section 7 discusses their practical interpretation, and Section 8 examines threats to validity. Section 9 concludes the paper.

2. Related Work

This section positions this study at the intersection of continuous-integration test case prioritization, learning-based ranking, semantic change–test analysis, reinforcement learning, and uncertainty-aware decision support. The discussion is deliberately limited to claims supported by the cited studies; numerical results from prior work are not compared directly with ours because the datasets, failure definitions, split protocols, and budget semantics differ between studies.

2.1. Regression Test Prioritization in Continuous Integration

Regression testing includes minimization, selection, and prioritization strategies for reducing the cost of re-validating evolving software [2]. In CI, the problem is intensified by frequent builds, changing test pools, and short feedback expectations. A systematic mapping dedicated to CI environments establishes TCP as a recurring response to these constraints [3]. Value-aware regression testing further argues that test decisions can be interpreted in relation to preserving delivered business value rather than treating fault discovery as business value in itself [1]. The present work adopts that narrower interpretation: early failing-test exposure is a technical feedback objective that can help protect value by identifying potential regressions sooner.

Pan et al. reviewed 29 machine-learning-based test selection and prioritization studies published from 2006 to 2020. Their synthesis identifies execution history, textual information, and coverage-related information as major feature families, while also documenting substantial variation in experimental design, metrics, and reproducibility practices [4]. Khatibsyarbini et al. reviewed 110 primary studies on machine-learning techniques for TCP and reported APFD as one of the commonly used outcome measures in the field [7]. Practical applicability remains a separate concern: Greca et al. examined the availability of tools, replication material, and evidence of use in practice, highlighting that many regression-testing proposals remain difficult to reproduce or transfer into operational workflows [8].

This work uses RTPTorrent, an open dataset introduced for regression test prioritization research [9]. Unlike studies that use the dataset only as a table of test outcomes, the implemented pipeline reconstructs archived job–build–commit provenance and derives source-level change and test artifacts. This reconstruction is a methodological contribution of the current work and should not be attributed to the original dataset paper.

2.2. Learning-Based and Context-Aware Prioritization

Learning-based TCP typically estimates a risk or ranking score from information available before the current test executes. Yaraghi et al. proposed a CI data model and a broad feature set, and they evaluated learning-based prioritization on 25 open-source systems selected to have meaningful regression-test duration and failed builds [5]. Zhao et al. compared 11 representative machine-learning-based CI prioritization techniques on 11 open-source subjects under a unified setup. They reported that performance changes across CI cycles and identified the amount of available training data as an important explanatory factor [6]. These findings motivate chronological evaluation and explicit control of the information available at each build.

Contextual information can also alter the balance between early failure exposure and execution cost. Roza et al. introduced contextual versions of multi-armed-bandit and random-forest prioritizers using readily collected test information. Across six systems and three budgets, the contextual variants improved time-reduction indicators, but they did not uniformly improve NAPFD or APFDc [10]. This result motivates the experimental design because it shows that adding context need not dominate a simpler model on every objective. Therefore, history-only, semantic-only, and fused risk models are evaluated separately, rather than assuming that additional features consistently improve every metric.

Deep learning has also been used for CI prioritization. DeepOrder applies deep learning to test ordering in continuous integration [11], while later work has explored recurrent architectures and automated model selection [12,13]. While these studies provide evidence that nonlinear sequence models are relevant to TCP, their reported performance cannot be directly transferred to the present setting due to differences in evaluation subjects and protocols.

2.3. Reinforcement Learning and Sequential Policies

Reinforcement learning (RL) provides a direct formulation for sequential test-selection decisions. RETECS is an early CI study of RL for automatic test case prioritization and selection [14]. Bagherzadeh et al. subsequently modeled TCP as sequential interaction between a CI environment and a ranking agent and investigated three alternative ranking models [15]. Chen et al. focused specifically on prioritizing new test cases in continuous-integration testing [16]. Qian et al. combined locally linear embedding, K-means clustering, and a dynamic priority factor with RL to improve the initial learning environment and ranking process across CI cycles [17].

These studies establish RL as a credible TCP formulation, but they do not imply that any particular RL design will outperform a strong deterministic ranking. The proposed DQN therefore uses a deliberately small action space: it selects among six deterministic ranking operators rather than directly choosing from a variable set of raw test identifiers. The evaluation also includes a protocol-aligned reimplementation of RETECS under the same project-wise chronological split. Because the original RETECS implementation, data schema, and training protocol differ from the present benchmark, this comparison is explicitly treated as a protocol-aligned reimplementation rather than an exact reproduction. The DQN result is interpreted only for its implemented state, action, reward, architecture, training population, and budgets.

2.4. Semantic and Language-Model-Based Signals

Textual and semantic information has a longer history in TCP than recent language models. TF–IDF is a classical term-weighting representation for automatic text retrieval [18]. Thomas et al. investigated static prioritization using topic models [19]. The review by Pan et al. identifies textual artifacts and code-similarity analysis as useful feature sources and describes studies that compare changed code with test artifacts using information-retrieval techniques [4]. More recently, Mahmoud et al. investigated LLM-enhanced commit messages as inputs to machine-learning-based TCP [20]. These studies motivate both the pretrained change–test representation and the TF–IDF lexical control. The lexical comparison is important because a gain from textual similarity should not automatically be attributed to a pretrained code-language encoder if a lightweight retrieval representation provides comparable evidence.

A substantial part of the LLM-for-testing literature concerns test generation rather than prioritization. For example, Schäfer et al. empirically evaluated LLMs for automated unit-test generation [21], and Alagarsamy et al. studied text-to-test-case generation with prompting and fine-tuning across multiple language-model families and open-source Java projects [22]. A tertiary study by Amalfitano et al. maps the wider use of artificial intelligence across software-testing activities [23]. These works motivate code-aware representation learning, but they are not direct TCP baselines. Accordingly, the method is described as using a frozen pretrained code-language embedding model, not as using a generative LLM to make ranking decisions.

2.5. Uncertainty Calibration and Research Positioning

Conformal prediction provides set-valued uncertainty quantification based on calibration data [24]. Adaptive conformal inference extends this line of work to settings with distribution shift [25]. In the TCP studies reviewed for this study, uncertainty is usually represented by model scores, stochastic policies, or contextual exploration. The specific combination of class-conditional conformal failure sets and build-wise prequential updating used in the implemented pipeline was not identified in the reviewed prior work. Static Mondrian split conformal and an ACI-inspired adaptive procedure are therefore evaluated separately. The formal coverage interpretation of the adaptive method is restricted to empirical prequential evaluation rather than extended to arbitrary-shift finite-sample validity.

The reviewed literature contains strong work on historical learning, contextual features, semantic representations, RL-based ranking, and uncertainty quantification, but these components are generally evaluated in isolation or in partial combinations. Within the reviewed literature, no single study was identified that jointly reconstructs exact archived job–build–commit provenance, aligns commit hunks with versioned test-source representations, fuses semantic and strictly prior-build historical risk, calibrates class-conditional failure sets, and evaluates an operator-level budget-conditioned DQN (deep Q-network) under a leakage-safe temporal protocol. The contribution of this study is therefore not an assumption that every added component must improve performance. Instead, the framework enables controlled component-wise evaluation of semantic fusion, conformal calibration, and sequential policy control under one reproducible protocol. This is an integration, formalization, and evaluation contribution; the underlying embedding model, conformal constructions, adaptive update principle, and DQN algorithm remain established methods from the cited literature.

Table 1. Conservative positioning of representative related work. A check mark indicates that the cited study explicitly addresses the dimension; a dash means that the dimension is not used as a defining element in the cited work. The table does not imply direct quantitative comparability across datasets (✓: addressed; –: not addressed).

Study	CIContext	History/Context	SemanticText/Code	RL/Sequential	Budget/Cost	ConformalUncertainty
DeepOrder [11]	✓	✓	–	–	–	–
RETECS [14]	✓	✓	–	✓	–	–
Bagherzadeh et al. [15]	✓	✓	–	✓	–	–
Roza et al. [10]	✓	✓	–	–	✓	–
Thomas et al. [19]	–	–	✓	–	–	–
Mahmoud et al. [20]	–	–	✓	–	–	–
This work	✓	✓	✓	✓	✓	✓

uses a deliberately strict coding rule: a dimension is checked only when it is explicit in the title, abstract, or inspected full text available to this study. An unchecked cell must not be read as proof that the study excludes the dimension; it indicates only that the dimension was not used as a defining characteristic in this conservative comparison.

3. Problem Formulation

This study is formulated at the level of continuous-integration (CI) builds, jobs, mapped source-code commits, and test executions. This distinction is essential because a build may contain concurrent jobs, a job may be associated with more than one commit, and the same commit may be associated with more than one job. The formulation therefore preserves the observed many-to-many provenance rather than replacing it with a single presumed “primary” commit. The prioritization target is early exposure of failing tests under an execution budget; RTPTorrent does not provide distinct fault identities, so the formulation does not interpret failing tests as separate software faults. Classical TCP and APFD provide the broader optimization context [2,26], but the definitions below are aligned with the information and outcomes available in the implemented pipeline.

3.1. CI Entities and Exact Provenance

Let $\mathcal{P}$ denote the set of projects. For each project $p\in \mathcal{P}$, let

$${\mathcal{B}}_p=\left(b_{p,1},b_{p,2},\dots ,b_{p,N_p}\right)$$

(1)

be its chronologically ordered sequence of real CI builds, where $b_{p,i}$ denotes the ith build and $N_p$ denotes the total number of builds for project p. Ties in the archived build number are resolved by build identifier. Each build $b\in {\mathcal{B}}_p$ contains a set of concurrent Travis jobs ${\mathcal{J}}_b$. For a job $j\in {\mathcal{J}}_b$, let ${\mathcal{T}}_j$ denote the set of tests executed by that job. The archived job–commit relation is represented by

$$\mathcal{M}\subseteq \mathcal{J}\times \mathcal{C},$$

(2)

where $\mathcal{J}={\bigcup }_{p,b}{\mathcal{J}}_b$ and $\mathcal{C}$ is the set of archived Git commits. The exact commit set of job j is

$${\mathcal{C}}_j=\{c\in \mathcal{C}:(j,c)\in \mathcal{M}\}.$$

(3)

No restriction $\left|{\mathcal{C}}_j\right|=1$ is imposed. Likewise, a commit may belong to multiple jobs. For each mapped commit c, ${\mathcal{H}}_c$ denotes the extracted set of changed-code hunks. The current test outcome and realized duration are

$$y_{j,t}\in \{0,1\},{\tau }_{j,t}>0,$$

(4)

where $y_{j,t}=1$ means that test $t\in {\mathcal{T}}_j$ failed in job j. These variables are retrospective outcomes: they are not available to the prioritization policy before the corresponding test executes.

Table 2. Principal notation used in the problem formulation.

Symbol	Meaning
${\mathcal{B}}_p$	Chronologically ordered real CI builds of project p.
${\mathcal{J}}_b$	Concurrent jobs belonging to build b.
${\mathcal{T}}_j$	Tests executed in job j.
$\mathcal{M}$, ${\mathcal{C}}_j$	Exact many-to-many job–commit relation and the commit set mapped to job j.
${\mathcal{H}}_c$	Changed-code hunks extracted for commit c.
$y_{j,t}$	Binary failing-test indicator for test t in job j.
${\tau }_{j,t}$, ${\widehat{\tau }}_{j,t}$	Realized test duration and pre-execution historical duration estimate.
${\mathbf{h}}_{j,t}$, ${\mathbf{s}}_{j,t}$	Strictly prior-build historical features and current change–test semantic features.
$p_{j,t}^H$, $p_{j,t}^S$, $p_{j,t}^F$	History-only, semantic-only, and fused failure-risk scores.
${\Gamma }_{j,t}$, $g_{j,t}$	Conformal prediction set and failure-candidate indicator.
$B_j\left(\beta \right)$	Nominal execution budget for job j at budget fraction $\beta$.
${\pi }_j$	Sequential test order produced for job j.

summarizes the main notation.

3.2. Leakage-Safe Historical and Semantic Risk

For a job j in build $b\left(j\right)$, let ${\mathcal{F}}_{b{\left(j\right)}^{-}}$ be the information generated only by builds that strictly precede $b\left(j\right)$ in the project-specific chronology. The historical feature vector must satisfy

$$h_{j,t} \mathrm{is} \mathrm{measurable} \mathrm{with} \mathrm{respect} \mathrm{to} {\mathcal{F}}_{b{\left(j\right)}^{-}}.$$

(5)

Consequently, jobs that run concurrently in the same build receive the same pre-build history and cannot update one another. After all jobs in a build have been observed, the test-level build outcome is aggregated before the history state is updated. In the implemented history engine, failure is aggregated by the maximum job-level failure indicator and duration by the median observed duration for the test within that build. Historical features include prior-build failure and error rates, recency and rolling failure indicators, and historical duration summaries. Current-job outcomes, current realized duration, original execution index, and current counts are excluded from model inputs.

Semantic information is constructed from the exact current commit set. Let $E(·)$ be the frozen code-language encoder followed by ${\mathsf{\ell }}_2$ normalization. For hunk $h\in {\mathcal{H}}_c$ and the versioned representation of test t at commit c, define

$${\mathbf{z}}_h={\displaystyle \frac{E\left(x_h\right)}{\parallel E\left(x_h\right){\parallel }_2}},{\mathbf{z}}_{c,t}={\displaystyle \frac{E\left(x_{c,t}\right)}{\parallel E\left(x_{c,t}\right){\parallel }_2}}.$$

(6)

When the test source cannot be resolved at the commit snapshot, $x_{c,t}$ is an identifier-only fallback rather than an invented source artifact. The hunk-to-test cosine scores for commit c are summarized by

$$\begin{array}{cc}\hfill s_{c,t}^{\max }& =\underset{h\in {\mathcal{H}}_c}{\max }{\mathbf{z}}_h^{\top }{\mathbf{z}}_{c,t},\hfill \end{array}$$

(7)

$$\begin{array}{cc}\hfill s_{c,t}^{\mathrm{mean}}& ={\displaystyle \frac{1}{|{\mathcal{H}}_c|}}\sum _{h\in {\mathcal{H}}_c}{\mathbf{z}}_h^{\top }{\mathbf{z}}_{c,t},\hfill \end{array}$$

(8)

$$\begin{array}{cc}\hfill s_{c,t}^{\mathrm{top}3}& ={\displaystyle \frac{1}{\min (3,|{\mathcal{H}}_c\left|\right)}}{\displaystyle \sum _{\mathsf{\ell }=1}^{\min (3,|{\mathcal{H}}_c\left|\right)}}{s}_{c,t}^{\left(\mathsf{\ell }\right)},\hfill \end{array}$$

(9)

where $s_{c,t}^{\left(\mathsf{\ell }\right)}$ is the ℓ-th largest hunk-level cosine similarity. For a job mapped to several commits, the semantic vector

$${\mathbf{s}}_{j,t}={\mathrm{Agg}}_{c\in {\mathcal{C}}_j}\left(s_{c,t}^{\max },s_{c,t}^{\mathrm{mean}},s_{c,t}^{\mathrm{top}3},\mathrm{coverage} \mathrm{and} \mathrm{count} \mathrm{indicators}\right)$$

(10)

uses both maxima and arithmetic means across the complete mapped commit set, together with semantic-availability and provenance-count variables. Thus, the many-to-many relation is retained in the features rather than collapsed by selecting one commit.

Three project-specific risk functions are defined:

$$\begin{array}{cc}\hfill p_{j,t}^H& =f_H\left({\mathbf{h}}_{j,t}\right),\hfill \end{array}$$

(11)

$$\begin{array}{cc}\hfill p_{j,t}^S& =f_S\left({\mathbf{s}}_{j,t}\right),\hfill \end{array}$$

(12)

$$\begin{array}{cc}\hfill p_{j,t}^F& =f_F\left([{\mathbf{h}}_{j,t};{\mathbf{s}}_{j,t}]\right),\hfill \end{array}$$

(13)

where $p_{j,t}^H,p_{j,t}^S,p_{j,t}^F\in [0,1]$ estimate the probability that t fails in j. The policy uses the mean score across the three fixed training seeds. For the fused model, the across-seed standard deviation

$$u_{j,t}^F={\mathrm{sd}}_{r\in \{42,45,2026\}}\left(p_{j,t}^{F,\left(r\right)}\right)$$

(14)

is retained only as a heuristic across-seed variability proxy. With three risk-model seeds, it is not interpreted as a calibrated epistemic-uncertainty estimate; the conformal prediction set provides the study’s separate set-valued calibration mechanism.

3.3. Failing-Test-Entity Ranking Targets

For a complete ordering ${\pi }_j=({\pi }_{j,1},\dots ,{\pi }_{j,n_j})$ of $n_j=\left|{\mathcal{T}}_j\right|$ tests, let $m_j={\sum }_{t\in {\mathcal{T}}_j}{y}_{j,t}$ be the number of failing tests and let ${\rho }_{j,1},\dots ,{\rho }_{j,m_j}$ be their one-based positions in ${\pi }_j$. For jobs with $m_j>0$, the failing-test-entity APFD surrogate used in the study is

$$\mathrm{FTE}-{\mathrm{APFD}}_j\left({\pi }_j\right)=1-{\displaystyle \frac{{\sum }_{q=1}^{m_j}{\rho }_{j,q}}{n_j{m}_j}}+{\displaystyle \frac{1}{2n_j}}.$$

(15)

Equation (15) treats failing test entities as detection targets. To avoid conflating this quantity with classical fault-level APFD, it is named the failing-test-entity APFD surrogate (FTE-APFD surrogate) throughout this study. If several failing tests arise from one latent defect, that defect can contribute more than once because RTPTorrent does not provide fault identities with which to collapse correlated detections.

The budgeted problem uses an exogenous nominal budget $B_j$. In the retrospective benchmark, the budget is defined as a fraction of the full realized job cost. This construction standardizes evaluation prefixes after the fact; it is not a deployable wall-clock limit known from future job cost, and the policy is not supplied with the realized full-job total as an input:

$$B_j\left(\beta \right)=\beta \sum _{t\in {\mathcal{T}}_j}{\tau }_{j,t},\beta \in \{0.10,0.25,0.50\}.$$

(16)

The confirmatory experiments use $\beta \in \{0.10,0.25,0.50\}$; an additional short-budget sensitivity evaluates $\beta =0.05$ without changing the original DQN training budget set.

Because Equation (16) uses the retrospectively observed full-job cost, a separate pre-execution sensitivity defines

$${\widehat{B}}_j\left(\beta \right)=\beta \sum _{t\in {\mathcal{T}}_j}{\widehat{\tau }}_{j,t},$$

(17)

where every ${\widehat{\tau }}_{j,t}$ is available before the current test executes and is derived only from prior builds. In that sensitivity, the DQN cost penalty is normalized by ${\widehat{B}}_j\left(\beta \right)$, and the same predicted budget defines the stopping threshold. Realized test duration is observed only after execution. This removes dependence on the future full-job realized-duration total from both the budget and cost-normalization denominator; the terminal recall component remains an offline training reward computed from logged labels and is not supplied to the pre-execution state.

The policy does not observe the current realized duration ${\tau }_{j,t}$ before selecting t; it may use only the historical estimate ${\widehat{\tau }}_{j,t}$. If tests are selected sequentially as ${\pi }_{j,1},{\pi }_{j,2},\dots$, the stopping time is

$$K_j\left(\beta \right)=\min \left(\left\{k:{\displaystyle \sum _{\mathsf{\ell }=1}^k}{\tau }_{j,{\pi }_{j,\mathsf{\ell }}}\ge B_j\left(\beta \right)\right\}\cup \left\{n_j\right\}\right).$$

(18)

The budget-crossing test is executed because its current duration becomes known only after execution. The realized selected cost, failure recall, and relative budget excess are therefore

$$\begin{array}{cc}\hfill C_j\left(\beta \right)& ={\displaystyle \sum _{\mathsf{\ell }=1}^{K_j\left(\beta \right)}}{\tau }_{j,{\pi }_{j,\mathsf{\ell }}},\hfill \end{array}$$

(19)

$$\begin{array}{cc}\hfill R_j\left(\beta \right)& ={\displaystyle \frac{{\sum }_{\mathsf{\ell }=1}^{K_j\left(\beta \right)}{y}_{j,{\pi }_{j,\mathsf{\ell }}}}{m_j}},\hfill \end{array}$$

(20)

$$\begin{array}{cc}\hfill O_j\left(\beta \right)& ={\displaystyle \frac{\max (0,C_j\left(\beta \right)-B_j\left(\beta \right))}{B_j\left(\beta \right)}}.\hfill \end{array}$$

(21)

Accordingly, nominal budget, realized cost fraction, and budget excess are distinct quantities.

3.4. Class-Conditional Conformal Failure Sets

The fused risk score is converted into a set-valued prediction by class-conditional conformal calibration [24,27]. For a failure probability p, the nonconformity scores for labels 0 and 1 are

$$A_0\left(p\right)=p,A_1\left(p\right)=1-p.$$

(22)

For class $k\in \{0,1\}$ with $n_k$ calibration scores sorted in ascending order, the finite-sample quantile is

$$q_{k,\alpha }=A_k^{\left(r_k\right)},r_k=\min \left(n_k,⌈(n_k+1)(1-\alpha )⌉\right).$$

(23)

The prediction set for test t in job j is

$${\Gamma }_{j,t}\left(\alpha \right)=\{0:p_{j,t}^F\le q_{0,\alpha }\}\cup \{1:1-p_{j,t}^F\le q_{1,\alpha }\}.$$

(24)

The conformal failure-candidate indicator is

$$g_{j,t}\left(\alpha \right)=\mathbf{1}\{1\in {\Gamma }_{j,t}\left(\alpha \right)\}.$$

(25)

Depending on membership, a prediction set may be nonfailure-only, failure-only, ambiguous, or empty. Candidate status is used as a soft prioritization layer; a non-candidate test is not permanently removed from the action space.

The adaptive variant is ACI-inspired [25]. All rows in a real build use the same pre-build class-specific quantiles and alpha states. After the full build has been predicted and observed, the class-specific empirical error $e_{k,b}$ updates the next-build alpha state as

$${\alpha }_{k,b+1}={\mathrm{clip}}_{[0.001,0.50]}\left({\alpha }_{k,b}+0.01\left({\alpha }^{\star }-e_{k,b}\right)\right),$$

(26)

where ${\alpha }^{\star }$ is the target error rate. If class k is absent from build b, its state is unchanged. Calibration scores from the completed build are appended only after all predictions in that build have been made. The primary configuration uses fused risk and ${\alpha }^{\star }=0.10$. The update coefficient 0.01 and clipping interval $[0.001,0.50]$ are fixed stabilization choices rather than theory-derived optima or parameters selected on the held-out test partition; they bound one-build adaptation and prevent degenerate alpha states. This adaptive procedure is evaluated prequentially; Equation (26) is not presented as an arbitrary-shift finite-sample coverage guarantee, and no claim is made that the selected update controls are optimal.

3.5. Budget-Conditioned Operator-Level Sequential Control Model

The final sequential problem is implemented as a finite episodic DQN control model, following the general DQN framework [28] and prior RL formulations of CI test prioritization [14,15]. An episode corresponds to one failure-bearing CI job and one budget fraction $\beta$. The full environment state includes the complete remaining test set and accumulated execution history, but the agent observes only the aggregate representation defined below. Consequently, the learned process is most precisely interpreted as an approximate state-aggregated MDP and may be partially observable from the agent’s perspective; Markov sufficiency of the 15- or 16-dimensional summary is not assumed.

Let ${\mathcal{R}}_{j,k}$ be the unexecuted tests before decision step k, $E_{j,k}$ the realized elapsed cost, and $F_{j,k}$ the number of failures already found. Let ${\mu }_{\mathcal{R}}\left(v\right)$ and ${\max }_{\mathcal{R}}\left(v\right)$ denote the mean and maximum of a test-level quantity v over ${\mathcal{R}}_{j,k}$, with value zero when the remaining set is empty. The common 15-dimensional state is

$$\begin{array}{cc}{\tilde{\mathbf{x}}}_{j,k}=& (\beta ,{\left(1-{\displaystyle \frac{E_{j,k}}{B_j\left(\beta \right)}}\right)}_{+},{\displaystyle \frac{|{\mathcal{R}}_{j,k}|}{n_j}},\min \left({\displaystyle \frac{F_{j,k}}{3}},1\right),\mathbf{1}\{F_{j,k}>0\},{\displaystyle \frac{k-1}{n_j}},\\ & {\mu }_{\mathcal{R}}\left(p^F\right),\underset{\mathcal{R}}{\max }\left(p^F\right),{\mu }_{\mathcal{R}}\left(p^H\right),\underset{\mathcal{R}}{\max }\left(p^H\right),{\mu }_{\mathcal{R}}\left(p^S\right),\underset{\mathcal{R}}{\max }\left(p^S\right),\\ & {\mu }_{\mathcal{R}}\left(u^F\right),{\displaystyle \frac{\log (1+|{\mathcal{R}}_{j,k}\left|\right)}{10}},{\displaystyle \frac{\log \left(1+\frac{{\sum }_{t\in {\mathcal{R}}_{j,k}}{\widehat{\tau }}_{j,t}}{B_j\left(\beta \right)}\right)}{10}}).\end{array}$$

(27)

The candidate-aware policies append

$$q_{j,k}={\displaystyle \frac{1}{|{\mathcal{R}}_{j,k}|}}\sum _{t\in {\mathcal{R}}_{j,k}}{g}_{j,t}$$

(28)

to Equation (27), producing a 16-dimensional state. Pure and gate-only policies use only ${\tilde{\mathbf{x}}}_{j,k}$. No gate indicator is included in any state. For an empty remaining set, all remaining-set summaries are defined as zero.

The action space is fixed across jobs:

$$\mathcal{A}=\left\{A_F,A_H,A_S,A_{F/\widehat{\tau }},A_{\min \widehat{\tau }},A_U\right\}.$$

(29)

The operator-level action space converts a job-dependent raw-test action set into six fixed outputs that can be shared across jobs of different sizes. This reduces output dimensionality and preserves interpretable deterministic fallbacks, but it also restricts the agent to choosing among predefined ranking rules rather than learning arbitrary test-level actions. The same operator set could support contextual-bandit, supervised, or rule-based meta-selection; these alternatives were not evaluated, so DQN is treated as one candidate controller rather than as the uniquely appropriate formulation.

Each action selects a deterministic ranking operator rather than a raw test identifier:

• $A_F$: Highest fused risk $p_{j,t}^F$;
• $A_H$: Highest history risk $p_{j,t}^H$;
• $A_S$: Highest semantic risk $p_{j,t}^S$;
• $A_{F/\widehat{\tau }}$: Highest $p_{j,t}^F/{\widehat{\tau }}_{j,t}$;
• $A_{\min \widehat{\tau }}$: Shortest predicted duration ${\widehat{\tau }}_{j,t}$;
• $A_U$: Highest fused-model uncertainty $u_{j,t}^F$.

Ties are resolved deterministically by archived original execution index. Candidate-first gating is an environment rule independent of state construction: when enabled, the chosen operator searches the unexecuted candidate subset first and then falls back to all remaining tests. Thus, the $2\times 2$ design separates candidate information in the policy state from candidate-first action execution.

Let $a_{j,k}\in \mathcal{A}$ be the DQN action and ${\pi }_{j,k}$ the selected test. After execution, $y_{j,{\pi }_{j,k}}$ and ${\tau }_{j,{\pi }_{j,k}}$ become observable. The implemented immediate reward is

$$\begin{array}{ll}{r}_{j,k}=& y_{j,{\pi }_{j,k}}+0.5\mathbb{I}\left\{y_{j,{\pi }_{j,k}}=1,F_{j,k}=0\right\}-0.05{\displaystyle \frac{{\tau }_{j,{\pi }_{j,k}}}{B_j\left(\beta \right)}}\\ & +0.5\mathbb{I}\{k=K_j\left(\beta \right)\}{\displaystyle \frac{F_{j,k+1}}{m_j}}.\end{array}$$

(30)

The terminal recall term is available to the offline simulator from the complete logged job but is not part of the pre-execution state. The objective is

$$\underset{\theta }{\max }\mathbb{E}\left[{\displaystyle \sum _{k=1}^{K_j\left(\beta \right)}}0.{95}^{k-1}{r}_{j,k}\right].$$

(31)

The reward is not a weighted sum of APFD, APFDc, or APFDk. The coefficients in Equation (30) and the discount factor in Equation (31) are fixed shaping choices. They keep direct failure discovery as the dominant immediate term, assign smaller bonuses to first-failure exposure and terminal recall, and normalize the execution-cost penalty by the nominal budget. Their optimality was not established, and no reward-coefficient sensitivity analysis was performed. Consequently, the policy comparison provides evidence about this particular reward design rather than about all possible reinforcement-learning objectives.

3.6. Admissible Information and No-Leakage Constraints

Before selecting the k-th test, an admissible policy may depend on exact current commits and hunks, current test identifiers or source snapshots, strictly prior-build history, fitted risk scores, conformal candidate status, the nominal budget, historical duration estimates, and outcomes and durations of tests already executed in the same episode. Formally, the selected action must be measurable with respect to

$${\mathcal{I}}_{j,k}=\sigma \left({\mathcal{C}}_j,{\left\{{\mathcal{H}}_c\right\}}_{c\in {\mathcal{C}}_j},{\{{\mathbf{h}}_{j,t},{\mathbf{s}}_{j,t},p_{j,t}^H,p_{j,t}^S,p_{j,t}^F,u_{j,t}^F,g_{j,t},{\widehat{\tau }}_{j,t}\}}_{t\in {\mathcal{T}}_j},B_j,{\{y_{j,{\pi }_{j,\mathsf{\ell }}},{\tau }_{j,{\pi }_{j,\mathsf{\ell }}}\}}_{\mathsf{\ell }<k}\right).$$

(32)

For every unexecuted test $t\in {\mathcal{R}}_{j,k}$, the current $y_{j,t}$ and ${\tau }_{j,t}$ are excluded from ${\mathcal{I}}_{j,k}$. At the dataset level, real builds are assigned wholly to one chronological split; training risk for RL is produced by expanding-window out-of-fold models fitted only on strictly earlier training builds; validation labels calibrate the conformal procedure; and test-build calibration pools and alpha states are updated only after the complete build has been predicted. These constraints define the leakage-safe decision problem evaluated in the remainder of this paper.

4. Materials and Methods

4.1. Study Workflow and Data Scope

The implemented study followed a staged, audit-oriented workflow rather than treating the CI table as an isolated classification dataset. Figure 1 summarizes the proposed workflow. RTPTorrent Version 1.1 was the primary empirical dataset [9]. The downloaded archive was rtp-torrent-v11.zip; its MD5 checksum was verified as 1f7fa822b0cf155bd007a94d1a24a336. The primary analysis covered Buck, HikariCP, OkHttp, and Deeplearning4j. For the supplementary passing-job operational sensitivity, historical job–build metadata were recovered from the archived TravisTorrent snapshot and validated against the exact RTPTorrent job–build mapping on overlapping jobs [29]. No external defect corpus was used.

The raw project tables contained 1,261,459 test-execution rows and 13,620 Travis jobs. Historical baselines were retained over this full raw population, whereas semantic and budgeted experiments required an exact failure-aware job–build–commit bridge.

Table 3. Audited data flow from raw CI executions to the final job–test modeling table. Counts refer to the four projects jointly.

Stage	Audited Size	Interpretation
Raw RTPTorrent tables	1,261,459 rows; 13,620 jobs	Full historical population before failure-aware provenance filtering.
Exact evaluation bridge	1811 jobs; 1781 builds	Failure-bearing jobs with exact build identity and validated job–commit relations.
Validated provenance	2321 relations; 2064 processed commits	Many-to-many job–commit links were retained, and mapped commits were checked against archived repositories.
Source-change extraction	12,696 changed files; 34,512 hunks	Merge commits used first-parent diffs; root commits were compared with the empty tree.
Semantic relation index	516,242 initial; 516,183 retained	Fifty-nine later-split relations from two repeated commits were purged.
Deduplicated semantic corpus	29,396 hunk texts; 5373 test texts	Each distinct text was embedded once.
Final modeling table	258,646 rows; 3645 failures	Chronological rows: 170,838 train, 42,034 validation, and 45,774 test.

reports the audited size and interpretation of each processing stage.

4.2. Exact Job–Build–Commit Reconstruction

The global archive member rtp-torrent/tr_all_built_commits.csv was used to connect Travis jobs to build and commit identifiers. The relation was preserved as many-to-many, as formalized in Equation (3); no rule selected a single presumed primary commit. Commit identifiers were validated against the archived bare repositories. Null build groups were excluded from the exact evaluation bridge, and consistency checks required every retained job to have a build mapping and at least one validated commit relation.

Splitting was performed at the real-build level independently within each project. Builds were ordered by Travis build number and then by build identifier, and contiguous chronological partitions of 70%, 15%, and 15% were assigned to training, validation, and test data, respectively. All concurrent jobs belonging to one build were therefore assigned to the same partition. This decision prevents job-level splitting from transferring information between parallel jobs of the same build.

4.3. Test-Execution Aggregation and Source Artifact Extraction

Raw execution rows were aggregated by project, Travis job, and test name. The archived original index was represented by the minimum row index, while duration, execution count, failures, errors, and skips were summed. A job–test row was labeled as failing when the sum of archived failures and errors was positive. These labels were retained only as retrospective outcomes and were excluded from predictive inputs.

For every validated mapped commit, the repository snapshot was queried directly. Merge commits were represented by first-parent diffs, while root commits were compared with the Git empty tree. Unified diffs were parsed into changed-code files and hunks; a single retained hunk was capped at 50,000 characters. Repository blobs larger than 512,000 bytes were not decoded as semantic source text.

Test-source resolution used the repository-wide historical path universe only as a lookup index. Candidate paths were generated from the archived test identifier, with at most 25 candidates examined, but source content was always read from the exact mapped commit blob. An ambiguous path was never forced. When no unique source path could be resolved, the test identifier itself was retained as a documented fallback representation. This policy produced 511,661 resolved source relations among 516,242 test–commit relations; unresolved relations remained in the study through the identifier fallback rather than being silently discarded.

4.4. Semantic Corpus, Embedding, and Alignment

The semantic corpus was constructed after provenance and split assignment. Hunk text and versioned test text were deduplicated, so identical text was embedded only once. Two commits appeared in more than one chronological partition; ownership was assigned to the earliest partition, and 59 later-partition relations were removed before embedding. The implicit hunk–test Cartesian products were evaluated in batches and were not materialized as a full table.

Embeddings were generated with the frozen jinaai/jina-embeddings-v2-base-code encoder at revision 516f4baf13dec4ddddda8631e019b5737c8bc250  [30,31]. Model parameters were not fine-tuned, and test outcomes were not used during representation learning. Each document was tokenized without truncation and divided into windows of at most 1024 model tokens, including special tokens, with a 96-token overlap and a maximum of 16 selected windows. Token embeddings were mean-pooled using the attention mask. Window vectors were normalized, aggregated with content-token-count weights, and normalized again to a 768-dimensional document vector.

A test representation was compared only with changed-code hunks belonging to its exact mapped commit. Cosine similarities were summarized within each commit by their maximum, mean, and mean of the three largest values, as defined in Equation (9). For a job associated with multiple commits, the implementation retained the maximum and arithmetic mean of each commit-level summary, semantic-availability fractions, source-resolution fractions, numbers of mapped commits and test-text versions, and candidate-hunk counts. Missing hunk evidence was represented explicitly rather than replaced by a fabricated similarity value.

4.5. Lexical TF–IDF Control

A lightweight lexical control based on the classical TF–IDF term-weighting family [18] was added to isolate whether the observed change–test signal requires a pretrained code-language representation. A separate TF–IDF vectorizer was fitted for each project using training-partition documents only. Tokenization retained identifier-like tokens matching [A-Za-z_][A-Za-z0-9_]*, lowercase text, used unigram and bigram features, sublinear term frequency, ${\mathsf{\ell }}_2$ normalization, minimum document frequency 2, and at most 150,000 features. Hunk and test vectors were compared by cosine similarity, and the hunk-to-test similarities were summarized with the same maximum, mean, and top-three aggregation pattern used for the pretrained representation. The resulting TF–IDF semantic features were evaluated both alone and concatenated with the 17 historical predictors. No validation or test document was used to fit the TF–IDF vocabulary or inverse-document-frequency weights.

4.6. Leakage-Safe Historical Features and Risk Models

Historical features were computed in project-specific build order and satisfied the strictly prior-build information restriction in Equation (5). All jobs in a build were scored from the same pre-build state. Only after the entire build was observed were test-level outcomes aggregated across parallel jobs: failure and error were aggregated by logical maximum, skip rate by summed skips divided by summed executions, duration by the median, and count by the sum. The state was then updated once per test and build.

The history-only representation contained 17 predictors: log-transformed prior build count; prior failure, error, and skip rates; previous result; log-transformed last, mean, rolling-median, and exponentially weighted duration; log-transformed mean execution count; failure rates over the previous five and ten observed builds; exponentially weighted failure rate; log-transformed recency since last failure and last observation; and indicators for never-failed and cold-start cases. The exponential weighting coefficient was 0.20, and the rolling duration window contained at most 20 observations.

The semantic-only representation contained 14 predictors. These comprised six cosine summaries; score-availability and source-resolution fractions; complete-missing and partial-coverage indicators; and log-transformed mapped-commit, test-text-version, and candidate-hunk counts. The fusion representation concatenated the 17 historical and 14 semantic predictors. Current failure label, current realized duration, current counts, current skips, original index, identifiers, and split labels were explicitly prohibited as model inputs.

Each project and feature group was modeled independently for seeds 42, 45, and 2026, yielding 36 risk models. The estimator was a pipeline comprising median imputation with missingness indicators, standardization, and an averaged stochastic-gradient logistic classifier. The classifier used an ${\mathsf{\ell }}_2$ penalty, balanced class weights, regularization parameter ${10}^{-4}$, a maximum of 3000 iterations, and tolerance ${10}^{-5}$. Models were fitted only on the training partition; validation scores were reserved for calibration and were not used to choose a preferred model or seed. For conformal calibration and policy inputs, the mean probability across the three seeds was used as the risk score in Equation (13), and the sample standard deviation across fused-model seeds supplied the heuristic variability proxy in Equation (14). Because only three risk-model seeds were used, this quantity is not presented as a calibrated epistemic-uncertainty estimate. Complete-ranking RQ1 metrics were first computed for each seed and then averaged within job and project for descriptive and paired analyses.

For cost-aware operators, predicted duration was selected in the following order: the prior-duration exponentially weighted mean, prior-duration arithmetic mean, and the median of the previous 20 durations. When all three test-specific estimates were unavailable, the fallback was computed strictly before the current real build from positive durations observed in earlier builds of the same project. The primary fallback was the pre-build project median; a pre-build first quartile and a one-second cold-start value were retained for sensitivity analysis. Current-build durations were appended only after every row of the build had been assigned a prediction. The confirmatory benchmark retained the original 0.001-duration floor for realized zero durations, while alternative analyses treated zeros as missing and replaced them with strictly causal project-level median or first-quartile values.

4.7. Class-Conditional Conformal Calibration

Static and adaptive class-conditional conformal procedures were applied to the history-only, semantic-only, and fused risk scores at target error rates $\alpha \in \{0.05,0.10,0.20\}$. The static procedure used the validation partition as a Mondrian split-conformal calibration set and the finite-sample quantile in Equation (23). The adaptive procedure initialized class-specific pools with validation nonconformity scores and processed test builds chronologically. All rows of a build used the same pre-build thresholds and class-specific alpha states. Only after every row in the build had been predicted were the observed class errors used to update Equation (26), and the completed build’s nonconformity scores were appended to the calibration pools.

The pre-specified primary configuration was fused risk with target $\alpha =0.10$. The resulting failure-inclusion indicator in Equation (25) was passed to the budgeted policy as a soft candidate flag. Its operational role is uncertainty communication and candidate-set control: it defines a coverage–efficiency trade-off, but it is not assumed to improve the underlying risk ranking. Non-candidates were not deleted or permanently pruned; a gated ranking operator searched candidate tests first and then used all remaining tests as fallback. Static class-conditional conformal retains its usual exchangeability-based interpretation; the adaptive procedure was assessed prequentially and was not treated as possessing an arbitrary-distribution-shift finite-sample guarantee [24,25].

4.8. Budget-Conditioned DQN Meta-Policy

The agent selected among the six deterministic operators in Equation (29), rather than selecting directly from a variable number of raw test identifiers. Training risk scores were generated by expanding-window out-of-fold models on the training partition. For each project, the earliest floor of 35% of training builds formed a warm-up window, and the remaining builds were divided into five chronological scoring blocks. Each block was scored only by models fitted on earlier builds, producing 818 failure-bearing out-of-fold training jobs.

The DQN experiment used a clean $2\times 2$ factorial ablation. Four project-specific policies were trained independently for every seed: DQN-pure received neither conformal candidate information nor candidate-first gating; DQN-candidate-state received the fraction of remaining candidate tests but did not gate selection; DQN-gate-only omitted candidate information from the state but searched candidate tests before non-candidates; and DQN-full-conformal combined candidate-state information and candidate-first gating. The label “full conformal” is configuration shorthand only: it indicates that conformal candidate information and gating are both supplied to the policy, not that the DQN itself inherits a conformal coverage guarantee. No gate indicator was included in any state. Consequently, the state dimension was 15 without candidate-state information and 16 with it. Candidate flags for training and validation episodes were generated prequentially and updated only after a complete real build; the held-out test partition used the exact adaptive fused $\alpha =0.10$ candidate indicator from the conformal stage.

Each configuration was trained in a separate network for the projects Buck, HikariCP, OkHttp, and Deeplearning4j and seeds 42, 45, and 2026. Every training job was replayed at nominal budget fractions of 0.10, 0.25, and 0.50 for 12 epochs. The network contained two hidden layers of 64 rectified linear units. A double-DQN target selected the next action with the online network and evaluated it with a separate target network [32]. Optimization used Adam [33] with learning rate ${10}^{-3}$, Huber loss, mini-batches of 256, replay capacity 100,000, replay warm-up of 512 transitions, gradient-norm clipping at 5.0, discount factor 0.95, and target synchronization every 500 optimization steps. Epsilon-greedy exploration decreased linearly from 1.00 to 0.05 over the first 70% of training episodes. The implemented reward remained the failure reward, first-failure bonus, normalized realized-cost penalty, and terminal recall bonus in Equation (30); APFD, APFDc, and APFDk were evaluation concepts rather than reward terms.

At decision time, the policy could use the nominal budget, already observed elapsed cost and failures, summaries of remaining risk and uncertainty, historical duration estimates, and candidate fraction only in the two candidate-state configurations. The current label and realized duration of an unexecuted test were unavailable. The selected test was executed before its duration became observable, and the episode stopped when cumulative cost reached or crossed the nominal threshold. The budget-crossing test therefore remained in the selected prefix. The fixed architecture, optimization, and exploration settings are summarized in

Table 4. Clean DQN ablation and fixed training configuration.

Parameter	Value	Parameter	Value
Configurations	Pure, candidate-state, gate-only, full conformal	Networks	Separate by project, seed, and configuration
Seeds/epochs	42, 45, 2026/12	State/action dimensions	15 or 16/6
Hidden layers	64, 64 with ReLU	Replay capacity/warm-up	100,000/512
Optimizer/rate	Adam/${10}^{-3}$	Batch/loss	256/Huber
Discount/target update	0.95/500 steps	Exploration	$1.00\to 0.05$ over 70%
Gradient clipping	Norm 5.0	Budget fractions	0.10, 0.25, 0.50
Failure/first-failure	1.0/0.5	Cost/terminal coefficients	0.05/0.5

.

4.9. Protocol-Aligned RETECS Baseline and DQN Sensitivities

A representative external learning-based TCP baseline was implemented from the published RETECS network configuration [14]. The project-specific state used strictly causal predicted duration, build recency, and four previous verdicts. The network contained one hidden layer with 12 units, replay capacity 10,000, replay batches up to 1000, Gaussian exploration standard deviation 0.20, learning rate ${10}^{-3}$, one training epoch per completed build, and the test case failure reward. Training used a 50% predicted-duration budget. To preserve the stricter leakage contract of this study, all jobs in the same real build were scored before any update from that build. Because the original RETECS software, industrial datasets, and present RTPTorrent schema are not identical, the result is reported as a protocol-aligned reimplementation, not an exact reproduction.

DQN robustness was assessed for the candidate-informed gated configuration with a limited one-factor-at-a-time design rather than a full hyperparameter search. The base $64\times 64$ architecture at learning rate ${10}^{-3}$ was evaluated with five seeds (42, 45, 2026, 31,415 and 27,182) and checkpoints after 6 and 12 epochs. Two single-seed diagnostics used learning rate $3\times {10}^{-4}$ with the $64\times 64$ architecture and a $128\times 64$ architecture with learning rate ${10}^{-3}$, both at 12 epochs. These diagnostics test whether the negative result is fragile to a few reasonable settings; they are not presented as an optimized Cartesian search.

The pre-execution-budget experiment retrained the candidate-informed gated DQN with the original three seeds using Equation (17) for both the nominal budget and reward cost normalization. The corresponding deterministic fused-risk baseline used the same predicted budget. The 5% budget was evaluated as an additional short-budget sensitivity. Buck-excluded and OkHttp-excluded analyses recomputed equal-project summaries from the retained projects to test whether the main ordering was driven by Buck’s duration quality or OkHttp’s provenance attrition.

4.10. Auditability and Reproducibility Controls

Every stage emitted a machine-readable manifest, quality tables, and an explicit readiness status before the next stage could run. The embedding model was fixed by repository revision. Primary risk and DQN stages used seeds 42, 45, and 2026; the RETECS and base DQN sensitivity analyses additionally used seeds 31,415 and 27,182, as specified above. All chronological split, fit/score overlap, class-presence, finite-score, and candidate-source checks were executed programmatically. Validation was used for diagnostics and conformal calibration, whereas the test partition was evaluated only after the project-specific risk models and DQN policies had been trained. The final statistical analysis averaged DQN seeds within each job before paired inference, preventing repeated stochastic runs from being treated as independent CI jobs. Software versions, file inventories, and final readiness checks were retained with the generated artifacts to support exact reconstruction of the reported analysis.

5. Experimental Setup

This section defines the research questions, evaluation populations, comparison methods, outcome measures, aggregation rules, and inferential procedure used in the final experiments. The complete-ranking, conformal-calibration, and budgeted-policy experiments answer different questions and are therefore reported with separate metric families rather than being collapsed into one composite score.

5.1. Research Questions

The empirical evaluation was organized around four research questions:

• RQ1—Historical, Pretrained Semantic, and Lexical Signals: What incremental value is provided by change–test similarity, how does history–semantic fusion compare with history alone, and does a pretrained code-language representation outperform a lightweight TF–IDF lexical control under the same temporal protocol?
• RQ2—Conformal Coverage and Candidate Efficiency: How do static and build-wise adaptive class-conditional conformal procedures trade failing-test coverage against candidate-set size, precision, and realized selected cost, and what candidate burden is observed when passing jobs are included where archival provenance permits?
• RQ3—Budget-Conditioned Sequential Control: Under nominal budgets of 10%, 25%, and 50% of full realized job cost, what are the separate and combined effects of conformal candidate-state information and candidate-first gating, and do any of the four independently trained DQN policies improve on strong deterministic fusion rankings? A 5% budget, RETECS comparison, hyperparameter diagnostics, and a pre-execution budget/reward normalization are evaluated as additional sensitivities.
• RQ4—Heterogeneity and Stability: How stable are the final budgeted results across projects, budget levels, and DQN seeds, and do Buck- or OkHttp-excluded analyses alter the principal method ordering?

RQ1 isolates representation effects without assuming that a pretrained encoder is uniquely beneficial. RQ2 treats conformal prediction as a candidate-set and uncertainty-control layer rather than a ranking method. RQ3 tests whether sequential operator selection adds value beyond deterministic operators and distinguishes the implemented DQN result from claims about RL in general. RQ4 is a robustness and heterogeneity analysis; it is not a leave-one-project-out generalization experiment, because all risk models and policies are trained separately within project.

5.2. Evaluation Population and Temporal Protocol

Figure 2 summarizes the audited reduction from the complete RTPTorrent tables to the final modeling population. The full historical population contained 13,620 jobs, while repository-aware experiments required the exact failure-aware provenance subset. Repository-availability checks, relation construction, and the cross-split commit purge yielded a final table of 258,646 job–test rows from 1800 failure-bearing jobs. The final table contained 3645 failing-test labels.

The chronological partition was formed independently within each project at the real-build level. The first 70% of builds formed the training partition, the following 15% formed the validation partition, and the final 15% formed the held-out test partition. All jobs belonging to the same build remained in one partition.

Table 5. Final chronological evaluation population by project. Job counts refer to failure-bearing jobs retained in the final modeling table.

Project	Train Jobs	Validation Jobs	Test Jobs	Test Rows	Test Failures
Buck	237	50	52	39,531	59
HikariCP	87	19	19	399	118
OkHttp	530	123	118	4890	118
Deeplearning4j	396	85	84	954	106
Total	1250	277	273	45,774	401

reports the final failure-bearing job counts and the held-out row population. The test partition contained 273 jobs, 45,774 job–test rows, and 401 failing-test labels.

The 45,774 rows describe scored job–test observations, whereas the inferential unit was the CI job; uncertainty analyses additionally retained real-build clustering. The effective held-out evaluation population should therefore be understood as 273 failure-bearing jobs rather than as 45,774 independent observations.

The primary ranking, conformal, and DQN analyses remain defined on failure-bearing jobs because exact repository-aware semantic provenance was constructed for that population. To assess operational candidate burden beyond failure-bearing cycles, an additional history-only conformal sensitivity reconstructed job–build metadata from TravisTorrent for Buck, HikariCP, and Deeplearning4j and validated 1032 overlapping archived job–build mappings with 100% build-ID agreement. The held-out operational population comprised 600 jobs and 173,561 job–test rows, including 156 failure-bearing and 444 passing jobs. OkHttp was excluded from this sensitivity because none of its RTPTorrent job identifiers matched the available TravisTorrent archival snapshot. This three-project analysis is therefore reported separately and is not labeled as a fused semantic result.

The role of each partition is shown in Figure 3. Risk models were fitted on training data. Expanding-window out-of-fold prediction within the training partition produced 818 failure-bearing jobs for DQN training after the 35% warm-up window. Validation data were used for diagnostics and conformal calibration, but not to choose a preferred model, seed, or budget. The test partition was evaluated only after all project-specific risk models and DQN policies had been trained. During adaptive test-time conformal evaluation, every row of a build was predicted before that build updated the class-specific calibration pools and alpha states.

5.3. Compared Methods and Pre-Specified Configurations

All primary learned risk models and DQN policies were project-specific. The original history-only, pretrained semantic-only, and pretrained fusion risk models used seeds 42, 45, and 2026; ties were resolved by archived original execution index. RQ1 retains these confirmatory models and adds a project-specific train-only TF–IDF semantic model and a history+TF–IDF fusion model as lexical controls. The lexical comparison uses the seed-averaged score representation and is reported as a diagnostic alongside, rather than as a replacement for, the pre-specified three-seed RQ1 analysis.

RQ2 compares static and build-wise adaptive class-conditional conformal calibration for the pre-specified pretrained fused $\alpha =0.10$ configuration, with the wider model–alpha grid retained as supporting analysis. A separate three-project history-only operational sensitivity includes passing jobs to estimate candidate fraction, row-level false-positive rate, passing-job candidate burden, and candidate execution cost under a more natural mixture of passing and failure-bearing CI jobs.

RQ3–RQ4 compare the archived original order, pretrained fused risk, pretrained fused risk divided by strictly causal predicted duration, conformal-first fused risk, and the four independently trained DQN configurations defined in Section 4.8. A protocol-aligned RETECS network is added as a representative external TCP baseline. The confirmatory DQN budget semantics use realized duration with the 0.001 floor. Strictly causal median and first-quartile zero-duration imputations and count-based budgets remain robustness analyses; the original DQN policies are transferred without retraining in those stress tests. The additional pre-execution-budget experiment instead retrains the candidate-informed gated DQN with predicted-duration budgets and reward normalization. Limited learning-rate, architecture, epoch, seed, 5% budget, Buck-excluded, and OkHttp-excluded analyses are reported as diagnostics rather than as a broad hyperparameter optimization exercise.

The comparison set is still not a comprehensive state-of-the-art benchmark. RETECS is a protocol-aligned reimplementation, and TF–IDF is a representative lexical control; DeepOrder, BM25, contextual-bandit controllers, and other published TCP systems are not reimplemented. Consequently, claims are restricted to the evaluated methods and protocol.

5.4. Evaluation Measures

The metric families in

Table 6. Outcome measures used in the three experimental stages.

Stage	Measure	Operational Definition
Complete ranking	FTE-APFD surrogate	Equation (15); failing test entities are the detection targets, not distinct fault identities.
	TTFF rank fraction	One-based rank of the first failing test divided by the number of tests in the job.
	TTFF cost	Cumulative realized duration through the first failing test.
	Count-prefix recall	Fraction of failing tests contained in the first $\lceil q{n}_j\rceil$ tests. Confirmatory values use $q\in \{0.10,0.25,0.50\}$; the extended sensitivity also reports $q=0.05$.
	Cost-prefix recall	Fraction of failing tests in the realized-cost prefix associated with a fraction q of full job cost. Confirmatory values use $q\in \{0.10,0.25,0.50\}$; the extended sensitivity also reports $q=0.05$. If the first test alone exceeded the target prefix, that first test was retained.
Conformal calibration	Failure coverage	Fraction of failing job–test rows whose conformal prediction set contained label 1.
	Candidate efficiency	Failure-candidate fraction, candidate precision, selected realized-cost fraction, mean set size, and empty/ambiguous/failure-only set rates.
	Job-level capture	Fractions of jobs with at least one or all failing tests included, and mean within-job failure recall.
Budgeted execution	Failure recall	Failures found in the selected prefix divided by all failing tests in the job, as in Equation (20). This was the primary endpoint.
	Any/all failure found	Indicators that at least one or all failing tests were found; project means are proportions of jobs.
	Realized effort	Selected-test fraction, realized cost fraction relative to full job cost, and relative budget excess.
	TTFF cost fraction	Realized cost through the first selected failing test divided by nominal budget. If no failure was found, the penalty was $\max (1,C_j/B_j)$.
	Failure precision	Number of failures found divided by number of selected tests.

were selected to preserve the distinction among complete-ranking quality, set-valued calibration, and realized budgeted execution. Higher values are preferred for recall, coverage, and precision measures; lower values are preferred for time-to-first-failure and cost measures.

The complete-ranking metrics were computed only for failure-bearing jobs. Phase 4 coverage metrics were computed over all 45,774 rows within the retained failure-bearing held-out jobs, with job-level capture summaries computed over the same 273 held-out jobs. Phase 5 budgeted outcomes used the corrected execution protocol in Equations (18)–(21): the current realized duration was not inspected before selection, the budget-crossing test was executed, and realized cost and excess were reported separately from the nominal budget.

5.5. Aggregation and Statistical Analysis

Primary descriptive summaries use equal-project macro aggregation because project sizes are highly imbalanced. Risk-model seeds were first averaged within job for paired comparisons, and DQN seeds were likewise averaged within job so that repeated stochastic runs were not treated as independent CI observations. The statistical unit was a failure-bearing CI job; jobs were paired by project, real build, and job identifier, and budget fraction was additionally included for budgeted comparisons.

Differences were oriented so that positive values favored the first method. Two-sided paired Wilcoxon signed-rank tests were calculated after removing exact zero differences, and paired rank-biserial correlation was reported as an effect size [34]. Holm’s sequential procedure controlled multiplicity within the RQ1, RQ2, and DQN comparison families [35]. Two complementary 95% bootstrap intervals were estimated with 5000 replicates following standard resampling principles [36]. The project-stratified bootstrap resampled jobs within each project and then averaged the project means equally. The build-cluster bootstrap resampled real builds within each project, retaining all jobs in a sampled build, before equal-project aggregation. The signed-rank test and equal-project bootstrap intervals target related but not identical weighting schemes; therefore, statistical interpretation considered multiplicity-adjusted p-values, effect sizes, and cluster intervals jointly rather than declaring superiority from one statistic alone.

The confirmatory RQ1 family retained fusion–history, fusion–semantic, and history–semantic comparisons for FTE-APFD surrogate, count-prefix recall at 25%, cost-prefix recall at 25%, TTFF rank fraction, and TTFF cost. The TF–IDF and RETECS comparisons are reported descriptively as additional sensitivities rather than folded post hoc into the original multiplicity family. RQ2 compared adaptive and static fused conformal calibration for failure coverage, candidate fraction, candidate precision, selected-cost fraction, and job-level any/all-failure capture. The DQN analysis compared the candidate-informed gated policy (the configuration labeled DQN-full-conformal) with each clean ablation and with the two strongest deterministic fusion rankings, and additionally estimated candidate-state, gating, and interaction contrasts from the $2\times 2$ design.

6. Results

The held-out population contained 273 failure-bearing jobs and 45,774 job–test rows, including 401 failing-test rows. Results are reported with equal project weight. Paired inference averages stochastic seeds within job and uses both project-stratified and real-build-cluster bootstrap intervals.

6.1. RQ1: Historical, Pretrained Semantic, and Lexical Signals

Table 7. Equal-project macro complete-ranking performance for RQ1. Upward and downward arrows indicate whether larger or smaller values are preferred. Bold values indicate the best results.

Method	FTE-APFD ↑	Recall@25 Count ↑	Recall@25 Cost ↑	TTFF Rank ↓	TTFF Cost ↓
Fusion	0.8840	0.8671	0.8404	0.0954	45.7518
History-only	0.8689	0.8517	0.8303	0.1080	39.2650
Semantic-only	0.7023	0.5411	0.5089	0.3367	88.6774

reports the principal complete-ranking results. Fusion produced the highest descriptive complete-ranking performance: equal-project macro FTE-APFD surrogate was 0.8840, compared with 0.8689 for history-only and 0.7023 for semantic-only. Count-prefix recall at 25% was 0.8671, 0.8517, and 0.5411, respectively; cost-prefix recall at 25% was 0.8404, 0.8303, and 0.5089. Fusion also improved the TTFF rank fraction relative to history-only (0.0954 versus 0.1080), but history-only retained a lower absolute TTFF cost (39.2650 versus 45.7518).

The paired analysis qualifies the descriptive ordering. Fusion did not significantly dominate history-only on any of the five RQ1 metrics after Holm correction. For FTE-APFD surrogate, the equal-project difference was 0.0151 and the build-cluster interval was positive (0.0010 to 0.0363), but the signed-rank comparison contained only 54 nonzero job pairs and did not survive correction ($p_{\mathrm{Holm}}=0.4253$). By contrast, fusion and history-only each outperformed semantic-only on all five metrics after Holm correction, with large rank-biserial effects. Figure 4 visualizes the oriented paired effects and their 95% build-cluster bootstrap intervals. These results support semantic information as a complementary signal but do not establish that fusion uniformly improves upon a strong historical model.

• Answer to RQ1: History was the dominant individual signal in the pre-specified analysis. Pretrained fusion was descriptively strongest there, but its advantage over history-only was not statistically uniform.

The lexical control materially qualifies the interpretation of the pretrained representation.

Table 8. Seed-averaged diagnostic comparison of pretrained and TF–IDF change–test representations. Values are equal-project macro summaries. This diagnostic is separate from the pre-specified three-seed RQ1 inferential family. Upward and downward arrows indicate whether larger or smaller values are preferred. Bold values indicate the best results.

Method	FTE-APFD ↑	TTFF Rank ↓	Recall@25Count ↑	Recall@25Cost ↑
History-only	0.8686	0.1084	0.8571	0.8593
Pretrained semantic	0.7027	0.2741	0.5407	0.6064
TF–IDF semantic	0.7502	0.2207	0.6113	0.6242
Pretrained fusion	0.8816	0.0980	0.8527	0.8778
TF–IDF fusion	0.8871	0.0943	0.8772	0.8666

uses seed-averaged risk scores for a common diagnostic comparison. TF–IDF semantic-only ranking reached an FTE-APFD surrogate of 0.7502, above 0.7027 for pretrained semantic-only ranking. History+TF–IDF fusion reached 0.8871, compared with 0.8816 for pretrained fusion in the same diagnostic. The methods traded places on individual prefix metrics: TF–IDF fusion was higher on 25% and 50% count-prefix recall, whereas pretrained fusion was higher on the corresponding cost-prefix recalls. The benchmark therefore supports change–test textual similarity as useful context but does not establish unique superiority of the pretrained code-language encoder over a simpler lexical representation.

The protocol-aligned RETECS network provides a representative external TCP comparison. Across five seeds, RETECS achieved mean FTE-APFD surrogate $0.8296\pm 0.0111$, with count-prefix recall of 0.7119, 0.7438, 0.7897, and 0.8677 at 5%, 10%, 25%, and 50%, respectively. For context, the seed-averaged pretrained-fusion diagnostic produced FTE-APFD surrogate 0.8816 and count-prefix recall 0.7601, 0.7905, 0.8527, and 0.9302. These values support the competitiveness of the deterministic fusion ranking within this protocol, but they are not presented as an exact reproduction of RETECS or as evidence of superiority over all external TCP systems.

6.2. RQ2: Conformal Coverage and Candidate Efficiency

At the row level, static fused calibration at $\alpha =0.10$ achieved failure coverage of 0.9244 with candidate fraction 0.3171 and candidate precision 0.3179. Adaptive calibration achieved coverage of 0.9058, candidate fraction 0.2598, and candidate precision 0.3915. Thus, the adaptive procedure moved coverage closer to the nominal 0.90 target while reducing the pooled candidate set and increasing pooled precision.

The paired job-level analysis gave equal-project mean within-job failure coverage of 0.9228 for adaptive and 0.9373 for static calibration. Mean within-job candidate fraction fell from 0.3274 to 0.2695, and mean within-job candidate precision increased from 0.4248 to 0.5316.

Table 9. Equal-project job-level RQ2 summary for adaptive and static fused conformal calibration at $\alpha =0.10$. Holm-adjusted p-values are from paired job-level Wilcoxon tests; bold values indicate the descriptively preferred method for each metric. Upward (↑) and downward (↓) arrows indicate that higher and lower metric values are preferred, respectively.

Metric	Static	Adaptive	${\mathit{p}}_{\mathbf{Holm}}$
Failure coverage ↑	0.9373	0.9228	1.0000
Candidate fraction ↓	0.3274	0.2695	0.0115
Candidate precision ↑	0.4248	0.5316	4.0 × 10−7
Selected cost fraction ↓	0.4085	0.3685	0.3593
Any failure captured ↑	0.9436	0.9470	1.0000
All failures captured ↑	0.9310	0.9003	1.0000

summarizes these job-level results and the Holm-adjusted paired tests. The candidate-fraction reduction and precision increase were significant after Holm correction ($p_{\mathrm{Holm}}=0.0115$ and $4.0\times {10}^{-7}$, respectively). Failure coverage, any-failure capture, all-failure capture, and selected-cost fraction did not differ significantly after correction. Project behavior was heterogeneous: HikariCP produced broad candidate sets, whereas Buck produced very small candidate sets, and adaptive calibration increased rather than reduced the candidate fraction for OkHttp.

• Answer to RQ2: Adaptive calibration improved candidate-set efficiency and within-job precision without a statistically confirmed loss of failure coverage, but the trade-off was strongly project-dependent. Its contribution is therefore candidate-set and uncertainty control rather than a guaranteed ranking gain.

A separate operational sensitivity included passing jobs for the three projects with recoverable archival build metadata.

Table 10. Three-project history-only operational sensitivity including passing jobs. Fractions are project-level means; the final row is an equal-project macro average. OkHttp is excluded because no RTPTorrent job identifiers matched the available TravisTorrent archival snapshot.

Project	Passing Jobs	Failure Coverage	Row FPR	Passing-Job Candidate Frac.	Passing-Job Cost Frac.
Buck	163	0.8644	0.0018	0.0018	0.0230
Deeplearning4j	6	0.9252	0.0344	0.1389	0.2921
HikariCP	275	0.9831	0.9267	0.9259	0.9161
Equal-project macro	444	0.9242	0.3210	0.3555	0.4104

reports history-only conformal results over 600 held-out jobs, including 444 passing jobs. Equal-project macro failing-row coverage was 0.9242, but candidate burden varied sharply by project. HikariCP marked most rows and most passing-job cost as candidates, whereas Buck produced a very small candidate set. Thus, high coverage did not imply uniformly compact or inexpensive candidate sets. This analysis is intentionally not labeled as fused semantic calibration because the retained semantic provenance pipeline was constructed for the failure-bearing population.

6.3. RQ3: Clean DQN Ablation and Deterministic Baselines

Table 11. Equal-project macro failure recall and realized cost fraction in the clean DQN ablation. Each entry is recall/cost.

Method	10%	25%	50%
Original order	0.0599/0.2386	0.3708/0.3691	0.7568/0.6272
Fusion risk	0.8114/0.1994	0.8778/0.3467	0.9347/0.6308
Fusion/predicted cost	0.8118/0.1980	0.8823/0.3383	0.9130/0.6371
DQN-pure	0.7521/0.1998	0.8192/0.3473	0.8886/0.6323
DQN-candidate-state	0.7614/0.1947	0.8363/0.3483	0.8965/0.6336
DQN-gate-only	0.7315/0.1994	0.8177/0.3523	0.8751/0.6309
DQN-full-conformal	0.7439/0.1986	0.8457/0.3494	0.9016/0.6334

reports the confirmatory time-budget results. The best deterministic methods reached failure recall of 0.8118, 0.8823, and 0.9347 at nominal 10%, 25%, and 50% budgets. Among DQN variants, candidate-state was best at 10% (0.7614), whereas the candidate-informed gated configuration was best at 25% and 50% (0.8457 and 0.9016). No DQN configuration was macro-best at any budget.

Figure 5 provides the corresponding recall trajectories across nominal budgets. It makes the consistent separation between the two deterministic fusion rankings and the four independently trained DQN policies visible without replacing the exact values in Table 11.

Across all 819 job–budget pairs, the candidate-informed gated configuration exceeded DQN-pure by 0.0120 mean recall, and the corrected signed-rank test was significant; however, the build-cluster interval included zero ($-0.0146$ to 0.0342). Therefore, the small average gain was not robust to build-cluster uncertainty. Candidate-state without gating also showed a small positive recall effect in the factorial signed-rank analysis, but its cluster interval crossed zero. No reliable recall interaction between candidate-state information and candidate-first gating was established.

The candidate-informed gated configuration remained below fusion risk by 0.0675, 0.0321, and 0.0331 macro recall at the three budgets. The aggregate paired comparison favored fusion risk with a build-cluster interval entirely below zero for full conformal ($-0.0795$ to $-0.0122$; globally Holm-adjusted $p=0.0497$). The comparison with fusion/predicted cost also favored the deterministic method in the signed-rank test, although its cluster interval crossed zero. The clean ablation therefore confirms that the earlier negative RL result was not an artifact of mixing gated and ungated episodes in one network.

• Answer to RQ3: Candidate information and gating changed policy behavior and precision but did not produce a robust recall advantage over the other DQN variants. Strong deterministic fusion rankings remained more effective than the candidate-informed gated DQN configuration.

6.4. RQ4: Heterogeneity, Stability, and Learned Actions

The principal weakness remained HikariCP. Full-conformal recall on this project was 0.3452, 0.6223, and 0.7389 across the three budgets, compared with fusion-risk recall of 0.6223, 0.7477, and 0.8452. Full-conformal seed ranges were 0.1610, 0.1455, and 0.2508 for HikariCP, whereas most project–budget ranges were below 0.04. DQN-pure was also unstable for HikariCP and Deeplearning4j at larger budgets, showing that instability was not exclusive to conformal gating.

All four agents used multiple actions. The candidate-informed gated configuration assigned the largest equal-project, equal-seed action share to fused-model uncertainty (0.2880), followed by shortest predicted duration (0.1578), history risk (0.1577), fusion/predicted cost (0.1507), semantic risk (0.1389), and fused risk (0.1069). The absence of action collapse did not translate into superior recall.

• Answer to RQ4: Sequential-control performance was project- and seed-dependent, particularly for HikariCP. Diverse operator use was learned, but operator diversity alone did not yield an advantage over deterministic ranking.

6.5. Robustness and Sensitivity Analyses

Duration quality was the most important data sensitivity. In the Buck test partition, 78.65% of durations were zero and 54.21% of rows required a causal fallback estimate. Replacing zero durations with strictly prior-build project medians or first quartiles did not change the main ordering: deterministic fusion methods remained best at all confirmatory time budgets. Under count-based budgets, fusion risk was best at 10% and 50%, while history risk was marginally best at 25%; no DQN was best. Figure 6 shows that this ordering is preserved across the alternative duration and budget definitions. Realized cost and budget-excess magnitudes were sensitive to duration treatment, so the retrospective time-budget results should not be interpreted as hard wall-clock guarantees.

The 5% budget sensitivity exposed stronger project dependence without reversing the aggregate conclusion. The five-seed base DQN reached equal-project recall 0.7081 at 5%, whereas pretrained fusion risk reached 0.7426 under the corresponding retrospective cost-prefix definition. Project-level DQN recall at 5% ranged from 0.3133 for HikariCP to 0.9187 for Deeplearning4j, emphasizing that very short CI windows can amplify project-specific behavior.

The limited DQN sensitivity in

Table 12. Limited sensitivity of the candidate-informed gated DQN. Base rows report the mean ± standard deviation across five seeds; (^†) daggered alternatives are single-seed diagnostics (seed 42), not a full Cartesian hyperparameter search. Pretrained fusion risk is shown as a deterministic reference. Bold values indicate the highest score in each column.

Setting	5%	10%	25%	50%
64 × 64, ${10}^{-3}$, 6 ep.	$0.7009\pm 0.0109$	$0.7553\pm 0.0308$	$0.8326\pm 0.0259$	$0.9046\pm 0.0231$
64 × 64, ${10}^{-3}$, 12 ep.	$0.7081\pm 0.0265$	$0.7737\pm 0.0190$	$0.8371\pm 0.0168$	$0.8910\pm 0.0164$
64 × 64, $3\times {10}^{-4}$, 12 ep. †	0.7195	0.7623	0.8434	0.9274
128 × 64, ${10}^{-3}$, 12 ep. †	0.6978	0.7112	0.8104	0.8609
Pretrained fusion risk	0.7426	0.8114	0.8778	0.9347

did not identify a setting that consistently surpassed pretrained fusion risk. The five-seed $64\times 64$ base configuration showed modest between-seed variability at 12 epochs. The lower learning rate improved the single-seed 50% result but remained below fusion risk; the larger $128\times 64$ architecture was weaker across all four budgets. Six versus twelve epochs did not produce a uniform direction of change. Figure 7 visualizes these one-factor-at-a-time diagnostics.

The pre-execution-budget experiment removed the future full-job realized-duration total from both the budget and the DQN cost-normalization denominator. As shown in

Table 13. Pre-execution budget and reward-normalization sensitivity. Budgets are fractions of the sum of strictly prior-history duration predictions. Values report equal-project macro failure recall and realized cost fraction.

Budget	DQN Recall	Fusion Recall	DQN Cost	Fusion Cost
5%	0.7641	0.8192	0.2441	0.2506
10%	0.8302	0.8487	0.2969	0.2971
25%	0.9000	0.9038	0.4812	0.4623
50%	0.9374	0.9341	0.7598	0.7562

, deterministic fusion remained descriptively higher at 5%, 10%, and 25%, while DQN was marginally higher at 50%. Paired Wilcoxon tests across the four budget levels did not provide Holm-adjusted evidence of DQN superiority (smallest adjusted $p=0.0712$). The experiment also revealed imperfect duration calibration: at a nominal 5% predicted-duration budget, realized cost averaged approximately 24–25% of the retrospective full-job cost, so pre-execution deployability introduces a separate duration-forecasting challenge.

Buck- and OkHttp-excluded summaries also preserved the principal deterministic-versus-DQN ordering. Excluding Buck, full-conformal DQN recall was 0.6937, 0.8188, and 0.8848 at 10%, 25%, and 50%, compared with fusion-risk recall of 0.7848, 0.8617, and 0.9247; the corresponding fusion-risk realized cost fractions were 0.2238, 0.3722, and 0.6687. Excluding OkHttp, DQN recall was 0.7197, 0.8300, and 0.8867, compared with fusion-risk recall of 0.8107, 0.8682, and 0.9327. Thus, Buck’s zero-duration concentration and OkHttp’s low provenance retention do not explain the overall method ordering, although both remain important threats to absolute cost estimates and project-level generality.

Semantic source and truncation sensitivity retained the same broad pattern. This diagnostic re-ranking used the seed-averaged probability assigned to each row, whereas the confirmatory RQ1 summary averages seed-specific job metrics. Over the full 45,774-row test table, the diagnostic macro FTE-APFD surrogate values were 0.8816 for pretrained fusion, 0.8686 for history-only, and 0.7027 for pretrained semantic-only. Restricting the analysis to resolved, non-truncated relations reduced the failing-row population from 401 to 264; pretrained fusion retained the highest FTE-APFD surrogate (0.8593 versus 0.8521 for history-only), although history-only was better on some count- and cost-prefix outcomes.

Provenance attrition was uneven. The final modeling retention was 70.77% for Buck, 94.70% for HikariCP, 39.60% for OkHttp, and 96.25% for Deeplearning4j. Retained OkHttp jobs were significantly smaller, shorter, and contained fewer failing rows than excluded jobs after Holm correction. No corresponding Holm-significant retained–excluded differences were detected for the other projects. The exclusion sensitivity shows that the aggregate method ordering is not driven by OkHttp, but it does not remove the selection bias within the retained OkHttp estimate.

6.6. Comparative Interpretation of Experimental Methods and Findings

The experimental comparisons were designed to answer complementary questions rather than to rank all methods on a single common axis. History-only, semantic-only, and fused models isolate the contribution of different information sources; the TF–IDF control tests whether the benefit of change–test similarity is specific to the pretrained representation; conformal calibration evaluates uncertainty and candidate-set efficiency; RETECS provides an external learning-based TCP reference; and the DQN ablation evaluates whether adaptive sequential operator selection improves upon fixed deterministic rankings. The additional short-budget, pre-execution-budget, duration, seed, hyperparameter, and project-exclusion analyses assess whether the observed method ordering is sensitive to particular experimental assumptions.

The evidence supports four main conclusions: First, historical risk was the strongest individual signal, while change–test similarity provided complementary information. However, the TF–IDF control showed that this benefit cannot be attributed uniquely to the pretrained code-language encoder. Second, adaptive conformal calibration improved candidate-set efficiency and precision, although its operational burden was strongly project-dependent. Third, the protocol-aligned RETECS comparison supported the competitiveness of deterministic fusion within the present benchmark without establishing general state-of-the-art superiority. Fourth, the evaluated operator-level DQN did not provide a consistent advantage over the deterministic fusion rankings after the clean ablation, five-seed analysis, short-budget evaluation, limited hyperparameter diagnostics, pre-execution budget experiment, and project-exclusion analyses.

Table 14. Comparative interpretation of the evaluated experimental methods.

Method	Experimental Role	Main Comparative Finding	Practical Interpretation
History-only	Leakage-safe deterministic baseline	Strongest individual signal in the pre-specified analysis; FTE-APFD surrogate = 0.8689	Strong low-complexity reference point
Pretrained fusion	Integration of history and semantic context	Highest descriptive primary FTE-APFD surrogate (0.8840), but not significantly better than history-only after Holm correction	Semantic information can complement history, but uniform superiority was not established
TF–IDF-based models	Lightweight lexical control	TF–IDF fusion reached 0.8871 versus 0.8816 for pretrained fusion in the separate seed-averaged diagnostic	Pretrained representation was not uniquely superior in this benchmark
Adaptive conformal calibration	Uncertainty and candidate-set control	Reduced candidate fraction and increased precision without a statistically confirmed loss of failure coverage	Useful for candidate-set efficiency rather than as an independent ranking improvement
RETECS	External learning-based TCP reference	Reached $0.8296\pm 0.0111$ FTE-APFD surrogate under the present protocol	Supports within-protocol competitiveness of deterministic fusion without implying universal superiority
DQN variants	Adaptive sequential operator selection	Did not consistently outperform the strongest deterministic fusion rankings across the evaluated budgets	Additional complexity did not yield a consistent held-out benefit in the evaluated design

highlights the resulting complexity–benefit trade-off. History-only ranking provided a strong low-complexity baseline, while textual context was useful but did not require a uniquely superior pretrained representation. Conformal calibration contributed primarily to uncertainty-aware candidate control, and the external RETECS comparison provided an additional learning-based reference under the same chronological protocol. The evaluated DQN added adaptive sequential control, but this additional complexity did not translate into a consistent held-out advantage over the strongest deterministic rankings. Overall, the comparisons indicate that additional model complexity should be justified by incremental benefit under matched temporal and budget conditions.

7. Discussion

The results sharpen the practical interpretation of the framework. The primary contribution is not that additional machine-learning components automatically improve TCP. Rather, this study provides a leakage-safe way to test which information and control layers add value under chronological CI constraints. Historical risk remained the strongest individual signal. Pretrained change–test similarity was useful as context, but the TF–IDF experiment shows that a lightweight lexical representation can be at least as competitive on this benchmark. The appropriate conclusion is therefore that textual change–test alignment can complement history; the present evidence does not establish that the pretrained encoder is uniquely necessary.

This interpretation is especially relevant to agile/CI practice. Early failure exposure is a technical feedback objective, not a direct measure of business value. Regression testing helps preserve delivered value by identifying changes that may degrade existing behavior, and prioritization can shorten the time to that evidence when a complete suite does not fit the immediate feedback window [1]. Prioritization also differs from permanent test selection in the present design: candidate information changes ordering, but non-candidate tests remain available as fallback. The added 5% analysis shows why ordering can matter even in very short windows, while the strong HikariCP variation cautions against assuming that one budget or policy behaves uniformly across projects.

The conformal layer should likewise be interpreted narrowly. It supplies set-valued failure inclusion and an explicit coverage–efficiency trade-off. It is not a proof that the fused ranking improves, and candidate-first gating often preserves much of the underlying risk order. The passing-job sensitivity further shows that candidate burden can be substantial even when failing-row coverage is high: HikariCP produced broad candidate sets, while Buck produced very small ones. Operational use therefore requires project-level monitoring of candidate fraction, false-positive burden, and selected cost rather than relying on coverage alone.

The RL result is a local negative result. The clean $2\times 2$ ablation already separated candidate-state information from gating; the new five-seed and limited hyperparameter diagnostics reduce the likelihood that the conclusion is caused by one random seed, one training duration, one learning rate, or one network width. The pre-execution-budget retraining also removes the most direct future-duration dependence from the budget and cost-normalization denominator. Even so, these experiments cannot distinguish a limitation of reinforcement learning as a broad paradigm from limitations of this particular operator-level design. The available training population contains 818 failure-bearing out-of-fold jobs, the state is an aggregate summary, the action space contains six deterministic operators, and the reward includes fixed shaping terms. A different state, direct test-level action space, contextual bandit, supervised selector, reward, or substantially larger training population could behave differently. This paper therefore does not claim evidence against RL in general.

The protocol-aligned RETECS result adds an external point of reference without removing this caution. RETECS was reimplemented under the present chronological split and leakage restrictions, not reproduced on its original industrial data and software stack. Its lower descriptive FTE-APFD surrogate and prefix recall therefore support competitiveness of the present deterministic fusion ranking within this protocol, but they do not establish universal superiority over RETECS, DeepOrder, or other published systems.

Duration remains the principal operational uncertainty. Buck contains many zero-duration rows, and pre-execution duration estimates can substantially overshoot their nominal budget when measured against realized full-job cost. The oracle-free experiment removes dependence on future total duration from the budget definition, but it also exposes the need for better duration calibration if a deployment requires a strict wall-clock limit. Conservative upper bounds, quantile duration prediction, cancellation, or admission control would be required for a hard deadline.

For practice, the evidence supports a staged adoption strategy. A project should first establish correct temporal provenance and a strong history-only baseline. Lexical or pretrained change–test context can then be added only if a chronological ablation demonstrates value for that project. Conformal candidate sets can be useful when their project-specific coverage and cost burden are monitored. An adaptive RL controller should be retained only if it improves the simpler deterministic operators on held-out temporal data. This ordering keeps the engineering objective–earlier regression feedback under constrained CI resources—ahead of model complexity.

8. Threats to Validity

8.1. Construct Validity

RTPTorrent identifies failing tests but does not provide distinct fault identities. The FTE-APFD surrogate and recall therefore measure early exposure of failing-test entities rather than classical fault detection, and several failing tests caused by one latent defect can contribute more than once. The confirmatory nominal time budget is a fraction of the retrospectively observed full-job cost and the budget-crossing test is executed, so it is an evaluation stopping rule rather than a hard deployment deadline. The pre-execution sensitivity removes the future full-job duration from the budget and DQN cost normalization, but historical duration estimates can be poorly calibrated. Zero durations, especially in Buck, still affect realized-cost interpretation. The across-seed fused-risk standard deviation uses only three risk-model seeds and is treated as a heuristic variability proxy, not calibrated epistemic uncertainty. Conformal prediction is the separate set-valued calibration mechanism. The adaptive conformal update controls and DQN reward coefficients are fixed implementation choices whose optimality is not established.

8.2. Internal Validity

The pipeline reconstructs archived many-to-many job–build–commit provenance and applies build-wise chronological updates. Errors in archived identifiers, source resolution, or test-name mapping could nevertheless affect semantic alignment. The TF–IDF vocabulary and inverse-document-frequency weights are fitted only on project-specific training text, and the pretrained encoder is frozen, reducing direct label leakage but not eliminating representation mismatch. RETECS is protocol-aligned rather than an exact reproduction of the original software. The DQN sensitivity is intentionally limited: the five-seed analysis applies to the base configuration, while the alternative learning rate and architecture are single-seed one-factor diagnostics. The aggregate DQN observation is not guaranteed to be Markov-sufficient; different remaining test sets can share the same summary while supporting different optimal actions.

8.3. Conclusions Validity

Project sizes and failure frequencies are imbalanced, and HikariCP contains relatively few failure-bearing jobs. Equal-project macro summaries, paired signed-rank tests, Holm correction, project-stratified bootstrap, and build-cluster bootstrap reduce but do not eliminate uncertainty. Signed-rank tests and cluster bootstrap intervals use different weighting estimands and can disagree, particularly when many paired differences are exactly zero. Non-significant comparisons are interpreted as insufficient evidence of superiority rather than equivalence. The TF–IDF and RETECS experiments are reported as additional sensitivities and are not retroactively inserted into the original RQ1 multiplicity family. Their role is to bound interpretation, not to enlarge the confirmatory claim set post hoc.

8.4. External Validity

The primary study covers four open-source Java projects; all risk models and policies were trained within project, so cross-project generalization is not established. Exact-provenance retention is particularly low for OkHttp, and retained OkHttp jobs differ from excluded jobs on several characteristics. The OkHttp-excluded sensitivity preserves the main aggregate ordering but does not remove selection bias from the project-specific retained estimate. Buck’s duration recording is also atypical; Buck-excluded results preserve the ordering, but absolute cost estimates remain uncertain.

Primary ranking outcomes were evaluated on failure-bearing jobs because repository-aware semantic provenance was constructed for that population. The supplementary operational analysis includes 444 passing test jobs across Buck, HikariCP, and Deeplearning4j and reports candidate and cost burden, but it is history-only and excludes OkHttp because matching TravisTorrent archival build metadata were unavailable. It therefore improves, but does not fully resolve, fleet-level external validity. Results may not transfer to other languages, industrial repositories, substantially larger suites, or natural CI populations with different failure prevalence. The comparison set now includes one lexical control and one protocol-aligned external TCP baseline, but it remains incomplete: BM25, DeepOrder, contextual-bandit selectors, and other modern systems are not reimplemented.

9. Conclusions

This study evaluated a context-aware sequential ranking framework for dynamic TCP under a strict chronological CI protocol. Exact archived provenance was combined with strictly prior-build execution history, versioned change–test similarity, class-conditional conformal calibration, and independently trained operator-level DQN policies. The contribution is the integration, formalization, and empirical evaluation of these components, rather than a claim of new algorithmic primitives.

The primary held-out results show that history is the strongest individual signal. Pretrained history–semantic fusion reached FTE-APFD surrogate 0.8840 versus 0.8689 for history-only and 0.7023 for pretrained semantic-only ranking, but the fusion–history comparison did not provide Holm-adjusted evidence of uniform superiority. The TF–IDF control further limits the representation claim: TF–IDF semantic-only ranking reached 0.7502 and history+TF–IDF fusion reached 0.8871 in the seed-averaged diagnostic. Thus, the benchmark supports textual change–test context but does not establish unique superiority of the pretrained encoder. A protocol-aligned RETECS reimplementation reached $0.8296\pm 0.0111$ FTE-APFD surrogate across five seeds, providing an external reference without supporting a universal state-of-the-art claim.

Adaptive conformal calibration reduced the candidate fraction and increased precision while retaining approximately nominal failing-test coverage, but the passing-job operational analysis showed that candidate efficiency is strongly project-dependent. The candidate-informed gated DQN remained below strong deterministic fusion rankings in the original confirmatory budgets. Five-seed stability, 5% short-budget evaluation, limited learning-rate/architecture/training-duration diagnostics, a pre-execution budget and reward normalization, and Buck- and OkHttp-excluded analyses did not establish a consistent DQN advantage. The negative result is therefore specific to the evaluated operator-level state/action/reward design and training data; it is not evidence that RL is ineffective for TCP in general.

Taken together, the experimental comparisons show that the evaluated methods serve different roles: history and semantic models assess information sources, TF–IDF provides a lexical control, conformal calibration evaluates uncertainty-aware candidate control, RETECS provides an external learning-based reference, and the DQN ablation evaluates adaptive sequential control. Under matched chronological and budget conditions, the results indicate that additional model complexity should be retained only when it provides measurable held-out benefit over strong deterministic baselines.

The practical implication is conservative. Regression-test ordering should first rely on correct temporal provenance and strong low-complexity baselines. Lexical or pretrained context should be retained only when project-specific chronological evidence justifies it, conformal candidate sets should be monitored for both coverage and burden, and RL complexity should be adopted only when it demonstrates held-out benefit. Future work should evaluate better-calibrated pre-execution duration budgets, broader passing-job populations, cross-project transfer, additional lexical and external baselines, simpler adaptive selectors, reward and conformal-control sensitivity, richer action spaces, other programming languages, and datasets with explicit fault identities.