Abstract
DevOps toolchains record 164 metrics across eight software development lifecycle (SDLC) phases, yet no public dataset links these pre-deployment signals to production runtime outcomes. We present PRESTO, a framework for predicting software system performance from multi-phase SDLC metrics, plus a Gaussian copula-based synthetic data generator—both evaluated as a proof of concept, not a validated prediction method. Under nested cross-validation that selects the algorithm and hyperparameters within the training partition, non-autoregressive SDLC process features reach to across three synthetic domains; a target-correlation ablation shows that this is largely recovery of the generator’s specified structure, shared identically across all three domains, so the results show that the framework recovers relationships embedded in the generator. Across four real-world datasets, the primary validation analyses are Mozilla Perfherder’s autoregressive persistence on a real production-performance signal ( to ) and SQuaD’s CVE-count () and static-analysis-enriched defect-fix () results; the confidence intervals are wide throughout, and most remaining comparisons include zero. The central limitation is that no available dataset joins eight-phase SDLC telemetry to production uptime, so each real dataset tests only a narrower proxy outcome (build duration, browser-performance regression, CI workflow duration, defect and CVE counts). PRESTO is therefore a framework and benchmark whose central proposition remains untested.
1. Introduction
Enterprise software teams ship code faster than ever, yet performance validation remains the slowest gate in the deployment pipeline. Traditional performance testing slows deployment pipelines because full-cycle load, stress, and endurance testing takes time that delivery cannot spare. This paper’s central thesis is that shift-left testing principles extend beyond functional testing to performance prediction: SDLC metrics, machine learning, and historical performance data together can forecast performance before deployment, without full-cycle testing on every change.
1.1. Problem Context
As DevOps adoption accelerates and microservices replace monolithic architectures, enterprise systems generate rich telemetry across every phase of the software lifecycle, from requirements traceability through build pipeline metrics to production uptime. Performance prediction remains largely disconnected from this telemetry: most approaches rely on post-deployment monitoring or pre-release load testing, leaving the SDLC metrics pipeline untapped. This costs teams in three ways: feedback is delayed, conflicting with agile principles; full testing is impractical for every minor change, leaving deployment decisions under-informed; and teams discover SLO breaches reactively, after the blast radius has already reached customer-facing traffic—a posture that is especially costly in microservice architectures where one degraded service cascades across dependent consumers.
1.2. Opportunity and Approach
Modern SDLC toolchains capture detailed metrics across eight phases: requirements analysis, code development, build processes, testing execution, performance validation, user acceptance testing, chaos engineering, and production deployment. Our analysis of enterprise-scale development projects identified 164 distinct metrics routinely collected across these phases. This volume is what makes machine learning performance prediction viable: by tracking SDLC patterns across releases, models can flag early indicators of potential problems before deployment, tying feedback to specific code changes rather than a downstream load test.
1.3. Research Questions
Two scope conditions bound these questions: First, no real-world dataset available to this study combines full multi-phase SDLC telemetry with production uptime as its target, so PRESTO’s central proposition is not tested end to end anywhere in this paper. Second, “system performance” in RQ1 therefore spans two different kinds of target: a synthetic System Uptime variable on the three copula-generated domains, and four real-world proxy outcomes (build duration, browser-performance regression, CI workflow duration, and defect/CVE counts) on the real datasets, none of which is production uptime. We accordingly present PRESTO as a framework and a proof-of-concept evaluation rather than a validated prediction method, and the four research questions below should be read in that light:
RQ1: To what extent can SDLC metrics predict system performance? (Section 4.1.2).
- Operationalization: We assess prediction accuracy using , MAE, and RMSE, distinguishing SDLC-only predictive signal from temporal autocorrelation of the target variable.
RQ2: Which SDLC phases and metrics are most predictive of system performance? (Section 4.1.7).
- Operationalization: We analyze feature-importance rankings and phase-level contribution percentages.
RQ3: How do different machine learning algorithms compare for SDLC-based performance prediction? (Section 4.1.2 and Section 4.1.3).
- Operationalization: We evaluate five algorithms (Linear Regression, Ridge, Lasso, Random Forest, Gradient Boosting) under both a fixed-hyperparameter baseline and a nested cross-validation protocol that tunes every algorithm equivalently.
RQ4: What factors influence the accuracy of SDLC-based performance predictions? (Section 4.1.9).
- Operationalization: We examine temporal dependencies, cross-phase interactions, release characteristics, and system-evolution impacts.
1.4. Contributions
We propose PRESTO (Performance REgression from Software development Telemetry and Operations), a performance prediction framework driven by multi-phase SDLC metrics, evaluated on three synthetic enterprise domains (a five-year cloud platform, 170 releases; a CRM/SaaS system, 200 releases; a payment processor, 150 releases) and four independent real-world datasets. Cross-domain results appear in Section 4.1.10.
Our contributions are as follows:
- Operationalizing PRESTO, a unified SDLC metric collection and standardization framework enabling full-lifecycle analysis that integrates development process indicators with performance prediction.
- Shifting prediction left: using 241 features engineered from pre-deployment SDLC metrics (test pass rates, build stability, requirements traceability) to predict a production-uptime target before deployment, which would provide risk signals earlier than canary deployments can surface them. The qualifier matters: the uptime target is realized only in this paper’s three synthetic domains. No real-world dataset available to us pairs pre-deployment SDLC features with a production-uptime outcome, so this contribution is a framework design demonstrated on synthetic data, not a capability validated on production systems.
- A reproducible Gaussian copula-based synthetic data generator producing 164 SDLC metrics across eight phases, calibrated to DORA 2024 benchmarks, released as a public benchmark. Cross-domain consistency in this generator’s output reflects a single shared correlation specification applied under three different marginal/temporal configurations, not three independent confirmations of the same finding (Section 5.1).
- Establishing transparent baselines under a nested-cross-validation model-selection protocol (Section 3.2.5 and Section 4.1.2): the nested-CV-selected without-AR model achieves to across the three enterprise domains, beating naive baselines on the reference domain; the wide bootstrap intervals reported alongside are untuned-baseline intervals on fixed-hyperparameter Random Forest (Section 4.1.5) and do not cover these nested-CV-selected figures.
- Quantifying autoregressive-feature dependence in SDLC performance prediction (Section 4.1.2): an algebraic-recoverability audit identifies a same-row leakage mode in two of the four AR feature families (Section 3.1.3); under that leaky construction, approximately 88% of the with-AR gain appears attributable to autoregressive persistence, an artifact of the leakage rather than genuine persistence (Section 5.1). Under the corrected construction, the picture is domain-specific: the reference domain’s with-AR advantage all but disappears (its bootstrap CI includes zero, exactly like the without-AR result), and only one of three domains (XYZ Sales Force) has a with-AR result whose own bootstrap CI excludes zero under fixed hyperparameters; even there, a paired-difference test cannot distinguish that result from the same domain’s without-AR result (Section 4.3), and a fairly regularized nested-CV-selected model does not use the AR features at all.
- Identifying that testing and build infrastructure metrics (Section 4.1.7) are the strongest SDLC predictors of production performance in this synthetic design (Build-phase 32.1%, Test-phase 21.1%, versus Code-phase 4.9%), a hypothesis that the real-world TravisTorrent validation corroborates only weakly and only in feature-ranking direction—not in predictive strength—once a definitionally circular feature is excluded (Section 4.2.1).
- Proposing a continuous learning framework with drift detection (PSI, KS tests) and automated retraining triggers. This framework is an architectural proposal only; as Section 3.2.6 note, it has not been experimentally validated, and we do not claim that it is ready for production deployment as it stands.
- Proposing PA-RFE (Phase-Aware Recursive Feature Elimination) (Algorithm 1, Section 4.1.8), which guarantees cross-phase feature representation: a structural, mechanical property that holds on all three domains under both validation-based and nested-CV stopping-point selection. Its test-set predictive accuracy does not reliably exceed simpler feature-selection baselines, and under nested selection it varies in direction across the three domains (Section 4.1.8). We do not claim that PA-RFE delivers a general predictive advantage; the phase-coverage guarantee is the contribution.
- Evaluating on four independent real-world datasets (Section 4.2) of deliberately different shapes: TravisTorrent (within-project temporal), Mozilla Perfherder (within-signature temporal, genuine AR persistence), GHALogs (cross-sectional, with a full-feature result carried largely by a near-tautological feature, Section 4.2.3), and SQuaD (cross-sectional, defect and vulnerability prediction). This precisely scopes what each dataset does and does not confirm about PRESTO’s central proposition, rather than treating any one of them as a full validation of multi-phase SDLC metrics predicting production uptime.
| Algorithm 1 Phase-Aware Recursive Feature Elimination (PA-RFE) | |
| |
| 1: ; | |
| 2: while do | |
| 3: Train model M on | |
| 4: PermutationImportance(M, ) | |
| 5: Record in | |
| 6: Sort F by ascending (least important first) | |
| 7: | ▹ elimination batch |
| 8: for f in sorted F do | |
| 9: if then | |
| 10: break | |
| 11: end if | |
| 12: if then | |
| 13: | |
| 14: end if | |
| 15: end for | |
| 16: if then | |
| 17: break | ▹ no features removable |
| 18: end if | |
| 19: | |
| 20: end while | |
| 21: | |
| 22: return , | |
1.5. Paper Organization
Section 2 reviews related work. Section 3 covers the methodology, including the nested cross-validation protocol and the per-dataset split construction. Section 4 presents the results. Section 5 addresses threats to validity. Section 6 discusses practical implications and future work. Section 7 concludes the paper. Appendix A presents generator calibration verification, defect analysis, and secondary robustness diagnostics, and Appendix B defines methodological terms and the four real-world datasets—both kept out of the main body so that the paper stays focused on its research questions, methodology, main analyses, and conclusions; a separate accompanying Supplementary Materials file holds the per-family paired-difference interpretation and the full secondary robustness diagnostics.
2. Related Work
Software performance prediction has progressed through three eras, each solving a different bottleneck: queuing theory for static architectures (1980s to 1990s), component-level modeling for design-time prediction (2000s), and machine learning for configuration optimization (2010s). None of these approaches use SDLC process metrics as predictive inputs. We organize prior work into seven areas and identify the gap that PRESTO targets.
2.1. Traditional Performance Modeling and Prediction
Software performance prediction originated with queuing network models [1,2] and foundational evaluation methodologies [3,4,5]. Smith [6] established Software Performance Engineering (SPE) as a discipline focused on early lifecycle performance considerations, later demonstrating practical applications with object-oriented methodologies [7]. Balsamo et al. [8] and Cortellessa et al. [9] surveyed model-based prediction approaches, while Jiang and Hassan [10] addressed load-testing scalability challenges. Performance regression detection methods include statistical process control [11], automated repository analysis [12], and early distributed application prediction [13]. Weyuker and Vokolos [14] established practical testing methodologies for industrial systems.
2.2. Design-Level and Component-Based Performance Prediction
Component-based performance prediction emerged through Liu et al. [15] for design-level analysis and Pooley and King [16] for UML integration. Menasce et al. [17] advanced capacity-planning methodologies, while Kant [18] addressed Internet application challenges. Woodside et al. [19] developed layered queuing networks for multi-tier systems, Koziolek [20] surveyed component-based evaluation methods, and Becker et al. [21] introduced the Palladio Component Model (PCM) for model-driven prediction. Jin et al. [22] and Chen et al. [23] examined configuration effects on testing and variability management, respectively.
2.3. Software Metrics and Quality Prediction
Software metrics research has established foundations for using process indicators as quality predictors, from the Chidamber–Kemerer OO metrics suite [24] and McCabe’s cyclomatic complexity [25] to COCOMO II [26] for cost estimation. Fault prediction evolved through historical defect patterns [27], complexity–defect correlations at Microsoft [28], classification benchmarks [29], and just-in-time quality assurance [30]. Related work on code smell longevity [31] and technical debt [32] provides context for understanding how code quality indicators evolve. A recent business-driven systematic mapping study of industrial ML defect-prediction practice [33] reached a conclusion consonant with this paper’s own emphasis on practical, decision-relevant framing over raw accuracy: predictive power alone rarely drives industrial adoption without a clear link to business cost and process context. While these studies demonstrate the predictive value of development metrics for quality attributes, performance prediction has remained largely outside their scope.
2.4. DevOps and Continuous Integration Metrics
The DORA metrics framework [34] established four key delivery performance indicators; the 2024 report [35] confirms that elite teams achieve 127× faster lead times. CI metrics research has explored adoption patterns [36], quality improvements [37], CI feature usage [38], build duration [39], test-build relationships [40], deep learning for build failure prediction [41], DevOps adoption [42], metrics reviews [43], performance-oriented DevOps [44], and parallel CI acceleration [45]. Existing research primarily applies these metrics to deployment success, delivery velocity, and defect detection rather than runtime performance forecasting.
2.5. Issue-Handling Processes and Project-Actor Cooperation
None of the metric families surveyed above capture a distinct source of process signal: how a project’s actors (developers, maintainers, reviewers, and reporters) cooperate and exchange information while resolving issues, and how that cooperation pattern shifts as a project encounters unexpected problems. This literature typically represents issue-handling activity as a graph over tracker events (assignments, comment threads, reopens, cross-issue references) rather than as the throughput counts (ticket volume, mean time to triage) that PRESTO’s Requirements and Production phases currently capture. Graph-driven analysis of issue-handling schemes [46] shows that structural properties of this cooperation graph (actor centrality, information-exchange latency between the events that make up an issue’s lifecycle, and the shape of reopen/escalation subgraphs) carry signal about project health distinct from simple ticket counts, extending a broader line of work on issue–reopen rates and triage time as reliability indicators [27].
This connects directly to two limitations that this paper acknowledges (Section 6.3): PRESTO’s current 164-metric registry omits issue-tracking process signals and does not model project-actor cooperation at all. A ninth, cross-cutting metric family (an “Issue-Handling Graph” phase, alongside PRESTO’s existing eight lifecycle phases) would add graph-derived features (actor-cooperation density, information-exchange latency, reopen/escalation subgraph patterns) computed from issue-tracker event logs; the same rolling-window and lag feature-engineering pipeline already used throughout this paper (Section 3.1.3) would apply to these features unchanged, since they would be release-aggregated time series like every other PRESTO metric. This is a proposed extension, not one that this paper implements or evaluates: none of the four real-world datasets used here (Section 4.2) provide issue-tracker event logs at the granularity that a cooperation graph requires, and adding this family without evaluating it would repeat the kind of unvalidated-but-asserted claim that this paper otherwise avoids (compare Contributions item 7’s continuous-learning framework, similarly flagged as architectural-only). We return to this in Section 6.4.
2.6. Machine Learning in Software Engineering
ML adoption in software engineering is well established for defect prediction [47,48], with surveys identifying performance prediction as under-explored [49,50]. Performance-related ML work includes build outcome prediction [51], the detection and prevention of temporal data leakage in build prediction [52] (a companion paper by two of the present authors that develops and validates a general taxonomy of temporal-leakage failure modes in ML-based build prediction; PRESTO applies that taxonomy’s shift-fix methodology to a different task and contributes a further, previously uncatalogued leakage mode specific to this setting: the target-derived difference/percentage-change same-row leakage documented in Section 3.1.3), change-induced incident analysis [53], and memory-pattern-based detection [54]. For configurable systems, approaches range from performance-influence models [55] and deep sparse networks (DeepPerf) [56] to sequential optimization (FLASH) [57], data-efficient learning [58], and transfer learning [59,60]. Additional work addresses Spark performance [61], validation methodology [62], early regression detection [63], continuous prediction [64], ML in DevOps [65], ML-SDLC integration [66], reliability prediction [67], and microservice fault localization [68]. AIOps and MLOps have accelerated operational ML adoption [69,70,71]. The integration of SDLC process metrics with performance prediction remains a gap.
2.7. Research Gaps and Contributions
The review of the existing literature reveals three gaps that our research addresses: an Integration Gap (traditional performance prediction relies on design models, historical performance data, or experience-based forecasting, missing the development process metrics that contemporary toolchains provide), a Scope Gap (existing SDLC metrics research focuses on quality and effort prediction rather than runtime performance forecasting), and a Coverage Gap (ML applications in software engineering have not used the multi-phase metric collection capabilities that modern toolchains provide, including the issue-handling cooperation signals discussed in Section 2.5).
Table 1 positions PRESTO relative to the most relevant prior approaches.
Table 1.
PRESTO positioned against the most relevant prior performance-prediction approaches.
As the table shows, prior performance prediction methods operate on configuration spaces (DeepPerf, Siegmund, FLASH), architectural models (Palladio), or aggregate delivery metrics (DORA). None use the fine-grained, multi-phase SDLC process metrics that modern CI/CD toolchains generate. Each study also measures and validates its own target differently: DeepPerf and Siegmund’s runtime-performance figures come from benchmarking the configured system directly, FLASH measures its performance metric the same way, through direct configuration sampling, DORA’s delivery-performance figures come from a 31,000-respondent self-report survey rather than direct instrumentation, Palladio’s response-time predictions are validated against measured response times in case-study systems, and the defect-prediction baseline’s target is a binary defect label taken from issue-tracker and version-control history. PRESTO’s own target, System Uptime, is defined and measured as described in Section 3.2.4.
3. Materials and Methods
This section presents the PRESTO framework methodology, covering data intake, feature engineering, model architecture, and validation strategy, including the nested cross-validation protocol.
3.1. Data Intake Pipeline
3.1.1. Multi-Source Data Ingestion Challenges
PRESTO ingests SDLC metrics from disparate source systems (issue trackers, version-control platforms, CI/CD servers, test frameworks, monitoring tools), each with its own schema, sampling cadence, and identifier scheme. Reconciling 164 metrics across eight phases into one release-aligned table requires resolving inconsistent release identifiers across systems, handling systems that report on different cadences (per-commit vs. per-deployment vs. daily), and tolerating instrumentation gaps that differ by phase and by organization.
3.1.2. Data Harmonization and Standardization
Before any normalization happens, each of the 164 metrics has to come from somewhere. In a real deployment of PRESTO, the eight phases draw from different source systems, and Table 2 gives representative examples of how each phase’s metrics would actually be extracted, rather than leaving that step implicit.
Table 2.
Representative examples of how each SDLC phase’s metrics would be extracted in a real deployment.
This paper’s own three synthetic domains do not go through this extraction pipeline. Because no organization publicly releases the raw Jira, GitHub, CI, and monitoring exports needed to build a real SDLC-to-performance dataset (Section 3.1.4 explains why), the synthetic generator produces already-harmonized metric values directly from calibrated distributions, skipping the extraction step described above.
Converting 164 distinct metrics into consistent ML features requires per-metric normalization strategies: logarithmic scaling for heavy-tailed metrics (), standardization for bounded-distribution metrics (), and one-hot, target, or ordinal encoding, depending on categorical cardinality and structure.
3.1.3. Temporal Alignment and Feature Engineering
SDLC metrics have complex temporal relationships that must be preserved during feature engineering. Code metrics measured at commit-level granularity must align with build metrics captured per deployment cycle, which, in turn, must synchronize with test metrics generated during release validation. Release is the main temporal unit; commit-level metrics are aggregated to release level, performance outcomes are aligned to the corresponding release window, and missing values are forward-filled for time-series continuity.
Feature engineering transforms raw metrics into predictive signals through rolling-window aggregations (moving averages, standard deviations, minima, maxima, and coefficients of variation across 3-, 5-, 7-, and 10-period windows), lag features (values from 1, 2, 3, and 5 previous releases), trend derivatives (rate-of-change calculations), and cross-phase interaction terms.
Two Same-Row Leakage Modes and Their Correction: In the rolling-window computation (add_rolling_features()), statistics computed inclusive of the current row, rather than over strictly-prior releases, leak the current target into its own predictors; the pipeline therefore shifts each series by one period before computing any rolling statistic.
A second mode arises in the lag features: To reconcile a weak raw autocorrelation in the target variable with a large apparent benefit from autoregressive (AR) features, we audit every AR-derived feature for algebraic recoverability of the current-row target, the same check applied to Mozilla Perfherder’s adapter (Section 4.2.2) and to TravisTorrent’s tr_log_testduration (Section 4.2.1). The audit identifies a construction defect in add_lag_features(): for the target column specifically, diff_k and pct_change_k were computed against the current, unshifted row (df_out[metric] - df_out[metric].shift(k)) rather than a prior-shifted one; only lag_k was genuinely prior-period. Across all three synthetic domains and all four lag steps, target_t = lag_k + diff_k and target_t = lag_k \cdot (1 + \text{pct\_change}_k) hold exactly in 97% to 99% of rows (the only exceptions are the handful of NaN leading rows that each lag step introduces). Under this leaky construction, 8 of the 32 AR features (the target’s own diff_k and pct_change_k across four lag steps) let a model algebraically reconstruct the current-row target rather than predict it, contrary to the assumption that all 12 lag/diff/pct-change features were correctly shifted.
The corrected construction mirrors the rolling-window shift: add_lag_features() computes the target’s own diff_k and pct_change_k against a series shifted by one period first, so every target-derived AR feature at row t describes strictly-prior releases only. The correction is scoped to the target column specifically; for the three non-target metrics in the same feature family (build success rate, Customer Satisfaction Score, throughput), the current row’s own value is a legitimate contemporaneous feature, already used directly elsewhere in the feature set, so an unshifted diff_k/pct_change_k for those metrics is not a leakage problem and is left unchanged. We verified the scoping: the without-AR feature matrix is byte-identical under the leaky and corrected constructions (confirmed via hash comparison), since without-AR results exclude the target-derived AR columns entirely, regardless of how they were computed.
The effect on with-AR results is large: on the reference domain, the with-AR Random Forest holdout is 0.914 under the leaky construction and approximately 0.11 to 0.19 under the corrected one, with a bootstrap confidence interval that includes zero. Section 5.1 gives the full disclosure, and every with-AR result in this paper uses the corrected construction.
3.1.4. Synthetic Data Generation Methodology
No public dataset combines SDLC process metrics with correlated runtime system performance measurements. We confirmed this through an exhaustive search of major ML data repositories (UCI Machine Learning Repository, Kaggle, Zenodo, Papers with Code Datasets) and software engineering benchmarks (TravisTorrent [72], GHTorrent, PROMISE repository). This absence of public benchmarks is itself a contribution gap that our synthetic generator addresses, providing the first reproducible dataset for this prediction task.
The core idea is simple even though the implementation has several moving parts: pick realistic distributions for each of the 164 metrics individually, then wire in the correlations that we believe should hold between them, and finally layer on realistic time dynamics so that consecutive releases are not independent draws. Section 5.1 discusses candidly what this approach can and cannot demonstrate. The generator employs a four-stage pipeline: (1) marginal-distribution specification, calibrated against the DORA 2024 State of DevOps report [35] and published enterprise project statistics [34]; (2) a Gaussian-copula correlation structure [73] over 84 specified non-zero pairs, with the correlation matrix ensured positive semi-definite via Higham’s alternating-projections algorithm [74] and the prediction target generated as a copula-correlated metric with its own marginal distribution rather than a deterministic function of the features, so the model must discover latent correlations rather than recover a known function; (3) temporal dynamics (AR(1) rank-preserving autoregression, a maturity trend, release-type effects, and stochastic shock events); and (4) statistical validation against the generator’s own specifications.
Metric Registry (Representative Excerpt): Table 3 gives a representative excerpt of the full 164-metric registry (src/metric_registry.py in the released code), spanning the distribution families used across all eight phases; the complete registry, including every metric’s definition, distribution, and parameters, is in the reproducibility package rather than reproduced in full here.
Table 3.
Representative excerpt of the 164-metric registry.
Table 4 gives the domain-specific System Uptime overrides, as applied by the corrected generator (defect analysis below).
Table 4.
Domain-specific System Uptime overrides.
Generator Calibration Defects: An audit of the generator code against its documentation identifies three implementation defects: 48 of 84 documented correlation pairs were silently discarded by an iteration-logic error (only 36 ever reached the copula); a per-domain distribution override for System Uptime was documented but never applied by any code path; and two metrics documented as negative-binomial counts were silently sampled from a continuously valued Gamma distribution instead, never rounded to integers. We corrected all three, regenerated all three synthetic domains, and reran the full downstream pipeline; every synthetic-domain number in this paper reflects the corrected generator. Full detail (the exact code paths, the metadata verifying the fix, and the before/after correlation-matrix distance and goodness-of-fit figures) is presented in Appendix A.1 rather than reproduced here.
One related issue was identified and deliberately left unfixed: System Uptime’s realized lag-1 autocorrelation remains far below its specified target (0.031 vs. a specified 0.55 to 0.65) because its distribution is heavily right-skewed toward the upper bound (roughly 82% of releases pin at the ceiling after bounds enforcement), leaving the rank-based AR(1) transform little room to express persistence for this specific variable. This is a structural property of pairing a ceiling-concentrated marginal with a rank-based AR(1) transform applied before bounds clipping, not an implementation defect; we did not redesign the target’s marginal distribution or the temporal-generation procedure to force a higher realized ACF, since doing so in direct response to this observation would be exactly the kind of results-contingent design choice that Section 5.1 already flags as a threat when discussing the copula’s hand-specified correlations elsewhere. All 163 other metrics recover their specified AR(1) structure within tolerance; this is an acknowledged limitation of the target variable’s temporal realism specifically, discussed further alongside the AR-feature audit in Section 5.1.
Domain Profiles: Three domain-specific configurations: ABC Cloud Provider (170 releases over 5 years, 200+ microservices, DORA “High” tier, reference dataset), XYZ Sales Force (200 releases over 5.5 years, aggressive release cadence, DORA “High” tier), and Card Payment Processor (150 releases over 4 years, ~one release every 9.7 days, PCI-DSS compliance, DORA “High” tier).
Reproducibility: All generation parameters are published as YAML configuration files; the generator is deterministic given a fixed seed. The complete generator code and configuration are archived at Zenodo (DOI: 10.5281/zenodo.22005439).
Limitations of Synthetic Data: Cannot capture emergent phenomena in real distributed systems; copula-specified correlations may be simpler than real-world dependencies; temporal dynamics are parametric approximations of complex organizational processes; real-world validation with production SDLC data remains essential future work.
3.1.5. Data Quality and Anomaly Detection
The pipeline implements range validation, statistical outlier detection (z-score thresholds), cross-metric consistency checks, and missing-value handling (forward-fill for time series, mean imputation for cross-sectional data).
3.2. Proposed Approach and Models
PRESTO employs a multi-layered architecture that transforms heterogeneous SDLC metrics into performance forecasts. Figure 1 presents the complete system workflow across five architectural tiers.
Figure 1.
PRESTO system architecture: proposed end-to-end pipeline from SDLC metric sources through feature engineering and ML model training to prediction output.
3.2.1. Feature Engineering Pipeline
Figure 2 summarizes the six-stage pipeline detailed below.
Figure 2.
PRESTO Feature Engineering Pipeline. Six-stage transformation from raw SDLC metrics to a reduced feature set via PA-RFE (241 SDLC features before reduction; 273 including autoregressive temporal features).
The pipeline generates candidate features in five stages, applies variance-based filtering, and then performs phase-aware dimensionality reduction (Stage 6): (1) rolling-window aggregations across four window sizes with five statistics each; (2) lag features from 1, 2, 3, and 5 previous releases; (3) multiplicative cross-phase interaction terms; (4) categorical encodings; (5) removal of near-zero-variance or highly collinear () features, leaving 273 features, of which 241 are SDLC process features and 32 are autoregressive (AR) temporal features, analyzed separately in Section 4.1.11; and (6) Phase-Aware Recursive Feature Elimination (PA-RFE) on the 241 SDLC features.
Stage 6: Phase-Aware Recursive Feature Elimination (PA-RFE): Unlike standard RFE [75], which eliminates features globally by importance and can remove entire SDLC phases, PA-RFE enforces a phase-balance constraint: each of the eight SDLC phases retains at least features throughout elimination (). Algorithm 1 presents the procedure. Selecting the feature subset by maximizing validation-set across a single fixed 60/20/20 temporal split overfits severely on the reference domain (validation vs. test ), so PA-RFE’s stopping point is instead selected under the nested cross-validation protocol described in Section 3.2.5: the stopping point is chosen by mean inner-CV across 2 inner temporal folds within the 80% outer-training partition—never by a single validation split—and the resulting feature subset is scored once on the untouched 20% outer holdout. Results are presented in Section 4.1.8.
Before the pseudocode, here is what each symbol means: X is the feature matrix (n rows p columns); y is the target vector; maps each feature name to its SDLC phase; is the minimum features that PA-RFE must keep per phase; s is the step size (features removed per iteration); F is the current working feature set; C is the ablation curve, a running log of pairs; is the permutation-importance vector for features in F; E is the current iteration’s elimination batch; is the final answer: whichever F in C scored best by the selection criterion in force (mean inner-CV under the nested protocol; validation on a fixed split for the comparison reported in Section 4.1.8).
Each feature is mapped to its SDLC phase via the metric registry; derived features inherit the phase of their parent metric. Computational complexity is , where K is iterations (∼), n samples, p current features, and r permutation repeats.
3.2.2. Machine Learning Model Architecture
The prediction system evaluates five regression algorithms selected to span the complexity spectrum: Linear Regression (interpretable baseline), Ridge Regression (L2 regularization; appropriate given correlated SDLC metrics), Lasso Regression (L1 regularization; implicit feature selection), Random Forest [76] (bagged decorrelated trees; handles high-dimensional correlated features via random subspace selection), and Gradient Boosting [77] (sequential additive ensemble; captures cross-phase interactions without explicit interaction terms).
3.2.3. Hyperparameter Specifications
Table 5 reports the fixed hyperparameter configuration used for this paper’s fixed-hyperparameter baseline results (Section 4.1.2, Section 4.1.3, Section 4.1.4, Section 4.1.5, Section 4.1.6, Section 4.1.7, Section 4.1.8 and Section 4.1.9 unless otherwise noted); Section 3.2.5 describes the nested-cross-validation protocol that tunes every algorithm’s hyperparameters; where the two disagree, this paper reports the nested-CV-selected result as the main result (per the model-selection rule, Section 3.2.4) while retaining the fixed-hyperparameter tables as an explicit baseline comparison.
Table 5.
Fixed hyperparameter configuration used for the fixed-hyperparameter baseline results.
3.2.4. Experimental Setup and Validation Strategy
Dataset Characteristics: Reference dataset (ABC Cloud Provider): 170 releases, 5 years, 200+ microservices. Further Domains: XYZ Sales Force, 200 releases, 5.5 years; Card Payment Processor, 150 releases, 4 years. Feature Space: 273 engineered features (241 without AR features) from 8 SDLC phases. Target Variable: System Uptime (%); in a live deployment, this would be computed per release window as over a 30-day post-release window, sourced from whatever monitoring platform the organization runs. In this study’s synthetic domains, System Uptime is instead drawn directly from a calibrated marginal distribution inside the copula generator (Section 3.1.4).
Validation Protocol: Because PRESTO forecasts performance from pre-deployment signals, respecting chronological order is essential to avoid temporal data leakage, a failure mode characterized for build prediction in prior work [52]. We employ 5-fold expanding-window time-series cross-validation (TimeSeriesSplit(n_splits = 5)) on the 80% training partition. Final holdout evaluation uses the remaining 20% (the most recent releases), never seen during training or, under the nested protocol described next, during model or hyperparameter selection either.
Per-Dataset Split Construction: This paper evaluates seven datasets that differ in whether they carry a genuine time index at all, so a single global statement of the validation protocol would be misleading. Table 6 states, per dataset, what the training/validation/test partitions were built from and whether TimeSeriesSplit is functioning as temporal validation or only as a deterministic resampling scheme over a fixed row order. The distinction matters because the same code path is used in both situations: in every dataset, the rows are first sorted by the stated ordering variable, and the partitions are then taken as contiguous positional slices (no dataset uses a shuffled or randomly partitioned split), so what the folds mean depends entirely on whether that ordering is chronological within an entity. The two cross-sectional datasets (GHALogs, SQuaD) retain TimeSeriesSplit for pipeline uniformity with the five temporal datasets, not because it validates them temporally; Section 4.2.3 and Section 4.2.4 restate this where those results are reported, and Section 4.1.5 explains why neither is a candidate for the moving-block bootstrap.
Table 6.
Train/validation/test construction per dataset, and whether TimeSeriesSplit performs temporal validation on it. Inner-fold counts drop from 5 to 3 for real-world outer-training partitions exceeding 20,000 rows (Section 3.2.5).
Model-Selection Rule: Model selection follows a nested design, described in full in Section 3.2.5: model and hyperparameter selection happens entirely within the 80% outer-training partition via inner TimeSeriesSplit(5), and the untouched 20% outer holdout is scored exactly once, by the already-selected configuration. This is the paper’s main reported protocol for every “best model” claim: per synthetic domain and AR condition (Section 4.1.2), per real-world dataset that involves a model-selection step at all (Section 4.2; GHALogs is the exception, reporting all five candidates rather than selecting one; see Section 4.2.3), and for PA-RFE’s feature-count stopping point (Section 4.1.8). We also report fixed-hyperparameter results throughout as a secondary, explicitly labeled fixed-hyperparameter baseline, because the contrast between the two protocols is itself informative (Ridge and Lasso are the nested-CV-selected winner in most comparisons where Random Forest or Gradient Boosting won under fixed hyperparameters). Nested selection is not a bias-free procedure in every case: Section 4.2.1 reports a case (TravisTorrent’s largest project) where the nested inner-CV score misleadingly favors a catastrophically unstable linear model over a reasonable tree-ensemble alternative—a genuine limitation of inner-CV-based selection under severe multicollinearity.
Feature-Availability Boundary: Every feature used to predict release t’s System Uptime must be genuinely knowable before release t is deployed. Table 7 states this boundary per feature category.
Table 7.
Feature-availability boundary per feature category: what is genuinely knowable before release t is deployed.
Every feature category above is measured during or before release t’s own pre-deployment pipeline; release t’s own production metrics are never used to predict release t’s own uptime. AR features are built exclusively from releases strictly before t, per the corrected lag construction in Section 3.1.3.
Software Environment: Python 3.14.5 (Python Software Foundation, Wilmington, DE, USA), scikit-learn 1.9.0, pandas 3.0.5, numpy 2.5.2, scipy 1.18.0, PyYAML 6.0.3, and Matplotlib 3.11.1 (all open-source packages). The SQuaD static-analysis aggregation (Section 4.2.4) used DuckDB (DuckDB Foundation, Amsterdam, The Netherlands; https://duckdb.org); the SonarQube, PMD, and CK metrics were computed by the SQuaD dataset authors and are used as distributed with that dataset. The tools named in Table 2 (e.g., Jira and Jenkins) are illustrative data sources for a future deployment and were not used in this study. Pinned versions recorded in ENVIRONMENT.txt (Data Availability Statement). Random seed: 42.
3.2.5. Nested Cross-Validation Protocol
A shared nested-CV harness (nested_cv_harness.py) is applied to every “best model” and “best feature subset” claim in this paper.
Protocol: The outer split is the identical fixed 80/20 temporal split used throughout this paper. Within the 80% outer-training set, an inner TimeSeriesSplit(5) grid search selects, for each of the 5 candidate algorithms, its best hyperparameters and records its best inner-CV (Linear Regression, having none, uses a plain 5-fold TimeSeriesSplit score for comparability). The algorithm and hyperparameter combination with the highest inner-CV —never the highest holdout —is selected. The selected configuration is refit once on the full outer-training set and scored once on the untouched outer holdout; that score is the only holdout that participates in any downstream claim. We also record the configuration that selection directly by holdout would pick; this holdout-selected comparison is optimistic by construction, because it selects by the same score that it reports, and it is shown beside nested results to illustrate selection optimism.
Hyperparameter grids are kept modest relative to the 120 to 160 outer-training rows in this paper’s synthetic domains (a large grid search would itself risk overfitting inner folds as small as ~20 to 30 rows): Ridge/Lasso alpha, 19 log-spaced points in ; Random Forest, n_estimators , max_depth , min_samples_split (18 combinations, min_samples_leaf fixed at 2); Gradient Boosting, n_estimators, max_depth , learning_rate (27 combinations). For real-world datasets whose outer-training partition exceeds 5000 rows, a reduced grid is used to keep runtime tractable (7-point alpha grid; 4-combination RF/GB grids), and for datasets exceeding 20,000 rows, a further-reduced grid with bounded tree depth and 3-fold inner CV is used, because unbounded tree growth on outer-training partitions of this size (up to 115,000 rows) is computationally disproportionate; every such reduction is disclosed per result row rather than applied silently.
What This Protocol Does and Does Not Fix: It removes the winner’s-curse-style optimism of selecting a model by the same score later reported as its result. It does not, by itself, guarantee a numerically stable result—Section 4.2.1 reports a case where the inner-CV score favors a linear model that is numerically catastrophic on the actual holdout, because severe multicollinearity in a wide, correlated real-world feature space makes a small inner fold’s an unreliable proxy for outer-holdout stability—nor does it correct for the comparison-family size across the roughly 150 selections to which this protocol is applied; Section 5.4 addresses that separately.
3.2.6. Continuous Learning and Model Evolution
For production deployment, PRESTO’s design includes a continuous learning architecture to address concept drift [78]. Figure 3 illustrates the proposed feedback loop (not experimentally validated in this study), monitoring feature-distribution shifts via Kolmogorov–Smirnov tests, chi-squared tests, and Population Stability Index (PSI), triggering retraining when PSI exceeds 0.2, monthly drops by more than 5%, or a major architecture change is deployed.
Figure 3.
Proposed continuous learning framework (not experimentally validated): feedback loop from production prediction through drift detection to automated retraining triggers.
4. Results and Discussion
This section presents the experimental results, analyzing model performance, feature importance, and factors influencing prediction accuracy.
This Paper’s Primary Real-World Validation Set, Stated Once: Given approximately 150 reported model/feature comparisons across this paper (Section 5.4), we designate a small, explicit primary set here. “Primary” denotes the analyses that this manuscript emphasizes in interpreting PRESTO’s real-world evidence. The designation was made after the results were available; it is not a prospective pre-registration, and these analyses carry no error-rate guarantee beyond that of the remaining, exploratory comparisons. Statistical significance, by contrast, is a factual descriptor of a bootstrap CI and is reported for every comparison in this paper, primary or exploratory alike: what is restricted to the primary set is any claim that a significant result validates PRESTO’s central proposition, not the word “significant” itself. The primary real-world validation analyses are exactly two: Mozilla Perfherder’s with-AR results (Section 4.2.2), and SQuaD’s CVE-count and static-analysis-enriched defect-fix-rate results (Section 4.2.4). GHALogs is deliberately not in this set: its full-feature significance does not survive the mean_n_steps ablation reported in Section 4.2.3. The nested-CV-selected result per synthetic domain is reported with its bootstrap CI but read as diagnostic of the corrected methodology rather than as validation, since the synthetic domains’ own target-correlation ablation shows that their signal is recovery of specified generator structure (Section 4.1.5). Every other comparison is exploratory or diagnostic, including TravisTorrent’s per-project results, GHALogs both before and after its ablation, SQuaD’s plain defect-fix-rate result, the nested-CV-tuning grid results, and PA-RFE’s per-domain figures. We avoid the word “confirmatory” for any analysis in this paper, including the two primary ones, because a label implying pre-specification would misdescribe how they were chosen.
4.1. Synthetic Domain Results
This subsection reports results on the three fully synthetic, copula-generated enterprise domains (ABC Cloud Provider, Card Payment Processor, and XYZ Sales Force). Section 4.2 reports results on four independent real-world datasets, kept separate so that synthetic and real findings are never conflated.
4.1.1. Experimental Setup
We report results on ABC Cloud (170 releases, reference dataset) unless otherwise stated; cross-domain results appear in Section 4.1.10. The dataset characteristics and validation protocol are reported in Section 3.2.4 and Section 3.2.5. Of the 273 engineered features, 32 are derived from the target variable itself. Of these 32, 24 (the 20 rolling-window features and 4 lag_k features) are genuinely autoregressive under the shifted construction; the remaining 8 (diff_k/pct_change_k) are same-row leakage under the leaky construction and genuinely prior-period under the corrected one (Section 3.1.3). We report all results both with and without these 32 AR features throughout.
4.1.2. Model Performance Comparison (RQ1 and RQ3)
Answer to RQ1: The answer is bounded by what this study is able to test. SDLC metrics predict a synthetic uptime target to a modest degree, and four real-world proxy outcomes to a variable degree; no dataset used here tests SDLC metrics predicting real production uptime, so RQ1 is answered for a framework under proof-of-concept evaluation, not for a validated prediction method (Section 1.3 and Section 4.2). Under nested-CV selection, SDLC process metrics (without AR features) achieve to across three synthetic enterprise domains. No bootstrap interval is computed on these nested-CV-selected fits; the untuned-baseline 95% CIs of Section 4.1.5 (fixed-hyperparameter Random Forest) are wide and, for most individual domains, include zero. With AR features included, the reference domain’s result is (nested-CV-selected) or (fixed-hyperparameter) under the corrected AR-feature construction (Section 3.1.3; Section 5.1 reports the leakage effect), barely above the without-AR figure and not statistically distinguishable from it. Card Payment shows the same pattern. Only XYZ Sales has a with-AR result whose own bootstrap CI excludes zero under fixed hyperparameters, and even there the paired-difference test finds its with-AR and without-AR results indistinguishable from each other (Section 4.3); under nested selection, the two conditions receive an identical in that domain anyway, since Lasso’s L1 regularization zeroes the AR coefficients when they are not worth their regularization cost. The real-world validation results in Section 4.2, not the synthetic domains alone, are what should be read as evidence of transferable signal, and then only for the narrower proxy outcomes that each of those datasets actually measures.
Answer to RQ3: Under equivalent nested-CV tuning effort (Section 3.2.5), Ridge or Lasso is the selected model in all six synthetic domain/AR-condition combinations (reported below). This reverses the fixed-hyperparameter baseline picture (Table 8 and Table 9 below), where tree ensembles appear to dominate. “Trees are necessary” is a property of the fixed-hyperparameter baseline specifically, not a general finding about algorithm choice for this task.
Table 8.
ABC Cloud, fixed-hyperparameter baseline, with AR features. Random Forest is the best fixed-hyperparameter model ().
Table 9.
ABC Cloud, fixed-hyperparameter baseline, without AR features.
Table 8 and Table 9 report the fixed-hyperparameter baseline for the ABC Cloud dataset; Table 10 reports the nested-CV-selected result for all three domains. Figure 4 plots the per-model with-AR and without-AR fixed-hyperparameter results.
Table 10.
All 3 domains, nested-CV-selected result (selection never uses the holdout score).
Figure 4.
Impact of AR-feature removal on holdout (ABC Cloud, fixed-hyperparameter baseline). The with-AR advantage is small.
Key Findings for RQ1:
- Without AR features, the nested-CV-selected best model (Ridge on ABC Cloud and Card Payment, Lasso on XYZ Sales) achieves to ; the reference domain’s figure (0.108) is nearly identical to the fixed-hyperparameter Random Forest figure (0.106) reported alongside it—a robustness finding independent of model-selection protocol.
- With AR features, the reference domain’s fixed-hyperparameter Random Forest achieves ; for Random Forest specifically, the fraction of this with-AR result attributable to AR features, computed the same way as the 88% figure obtained under the leaky construction (Section 5.1), , is on the unrounded fits, itself a share of a much smaller total effect; under nested selection, the equivalent figure is close to zero (0.111 vs. 0.108). This is the one place that the 45% figure is derived; later mentions (Section 4.1.9 and Section 5.1) cross-reference it.
- Card Payment’s with-AR advantage is similarly small, and its bootstrap CI includes zero (Section 4.1.5); XYZ Sales is the one domain whose with-AR result has a bootstrap CI excluding zero under fixed hyperparameters (Section 4.1.5), although a paired-difference test cannot separate it from that domain’s without-AR result (Section 4.3), and nested/regularized selection does not use the AR features at all.
Note on CV-Holdout Discrepancy: Several models show negative CV despite a positive or less-negative holdout . This occurs because TimeSeriesSplit with on an 80%-training partition as small as ~136 samples creates early folds with as few as ~23 training samples for 241 features—an unreliable model-selection signal in this small-n, high-p regime. The same phenomenon is why nested inner-CV selection is itself not immune to picking an unstable model: TravisTorrent’s dd-agent case (Section 4.2.1) shows an inner-CV score favoring a numerically catastrophic linear model over a workable tree ensemble.
Key Findings for RQ3: Under fixed hyperparameters, tree ensembles outperform linear models when AR features are excluded, but Gradient Boosting collapses on the reference domain (), while Random Forest does not. Under the nested-CV protocol (Section 3.2.5), Ridge or Lasso is the selected model in all six domain/condition combinations; this is the main basis for this paper’s answer to RQ3, with the fixed-hyperparameter baseline comparison retained as a secondary, explicitly scoped finding.
Interpretation: This paper distinguishes two distinct predictive signals: autoregressive persistence of the target variable, and SDLC process metrics. Both signal sources are legitimately available in practical deployment. Under the corrected AR-feature construction, the first signal is weak and domain-dependent on synthetic data; the second signal’s range does not depend on the AR construction (without-AR results exclude the target-derived AR columns entirely), although the model that best exploits it differs once every algorithm is tuned equally.
4.1.3. Model Characterization
The best-performing model without AR features, under fixed hyperparameters, remains a non-parametric ensemble (Random Forest) that does not yield a closed-form equation; under nested selection, it is instead a regularized linear model (Ridge or Lasso) in all six cases. The failure of unregularized/lightly regularized linear models in the fixed-hyperparameter no-AR setting is expected given the problem geometry: with features and training samples, ordinary least squares requires inverting a rank-deficient matrix, and the minimum-norm solution extrapolates wildly on test data (). Ridge’s fixed alpha = 1.0 default is insufficiently regularized for this regime (); Lasso’s L1 sparsity avoids collapse by effectively reducing p below n. Random Forest handles this through random feature subsampling across decorrelated trees. Once alpha is properly tuned via nested cross-validation, however, Ridge recovers entirely (Table 10): the practical conclusion is that a linear-model baseline at default hyperparameters is not a fair comparison, and neither is an untested assumption that a tree ensemble will work.
4.1.4. Statistical Validation and Baseline Comparison
We conducted four statistical validation diagnostics on the ABC Cloud dataset (without AR features, Gradient Boosting): a naive-baseline comparison, a reimplemented Siegmund-style performance-influence-model baseline [55], multi-seed stability, and a permutation test. We summarize the headline conclusion here and move the full per-seed and per-baseline tables to Appendix A.2, since these diagnostics characterize Gradient Boosting’s specific failure mode on this domain, not this paper’s actual without-AR headline result (which rests on Random Forest under fixed hyperparameters, or Ridge under nested selection): Gradient Boosting is negative on every seed tested (mean ) and does not beat any naive baseline; the reimplemented Siegmund baseline collapses similarly on the reference domain (), though not cross-domain (it reaches on Card Payment); a 1000-permutation test does not support Gradient Boosting’s without-AR signal being real (). All four diagnostics concern the without-AR condition, so they do not depend on the AR-feature construction. A fifth check, residual diagnostics, is reported with the others in Appendix A.2.
4.1.5. Bootstrap Confidence Intervals, Block Bootstrap, and Target-Correlation Ablation
Answer: Bootstrap 95% CIs on holdout are wide and, for most individual (domain, model, condition) combinations, include zero; this holds for the reference domain’s with-AR result under the corrected AR construction, as well as for its without-AR result. A target-correlation ablation confirms that, on synthetic data, the without-AR signal is generator-prior recovery, not independent discovery. A moving-block bootstrap, which tests whether the standard pairs bootstrap’s exchangeability assumption is appropriate for temporally ordered holdouts, covers the headline condition of every temporally ordered dataset family in this paper, rather than the synthetic domains alone, and is consistent with the pairs bootstrap on significance status in all 16 cases: 6 synthetic domain/condition combinations here, TravisTorrent’s 7 per-project full-feature holdouts (Section 4.2.1), and Perfherder’s 3 with-AR per-signature holdouts (Section 4.2.2), the last being the set that carries this paper’s primary real-world validation designation. The secondary conditions on those two real-world datasets are not block-bootstrapped: Perfherder’s alert-history-only condition (Section 4.2.2) and TravisTorrent’s testduration-excluded condition carry pairs intervals only, and their with-versus-without comparisons are instead tested as paired differences (Section 4.3).
Method: (a) A pairs bootstrap (2000 resamples per condition) resamples each holdout’s (prediction, actual) pairs with replacement and recomputes per resample. (b) A moving-block bootstrap (2000 resamples, block length via the rule, floored at 2) resamples contiguous blocks of the same pairs instead, to account for autocorrelated errors in temporally ordered holdouts; it is applied to all six synthetic domain/condition combinations in Table 11, and to TravisTorrent’s seven per-project holdouts and Perfherder’s three with-AR per-signature holdouts as well, reported where each dataset’s other results are (Section 4.2.1 and Section 4.2.2). Both cross-sectional datasets are exempt for a different and more basic reason, and we state it for each rather than once collectively: GHALogs’ rows are ordered by each repository’s most recent push date but carry no within-repository time series, and SQuaD’s rows carry no date ordering at all, following alphabetical aggregation order instead (Table 6, Section 3.2.4). In neither case is there within-entity temporal dependence for a moving-block bootstrap to preserve, so both remain pairs-bootstrap-only by design rather than by omission. (c) The target-correlation ablation regenerates ABC Cloud under two variants of correlation_templates.yaml’s 21-entry target_correlations section: zeroed (all 21 entries removed) and permuted (same 21 values, shuffled). Both bootstrap methods and the ablation use the corrected AR-feature construction.
Table 11.
Random Forest holdout (fixed-hyperparameter baseline) with 95% pairs- and moving-block-bootstrap CIs, all 6 domain/condition combinations. Mean CI width change (block minus pairs): units. Untuned-baseline intervals: computed on fixed-hyperparameter Random Forest; they do not quantify uncertainty around the nested-CV-selected models (Table 10, Section 4.2.1, Section 4.2.2 and Section 4.2.4).
Model Protocol for Both Bootstraps: Every interval in this subsection, in the two real-world block-bootstrap tables of Section 4.2.1 and Section 4.2.2, and in the paired-difference analysis of Section 4.3, is computed on fixed-hyperparameter Random Forest fits—this paper’s secondary, explicitly labeled fixed-hyperparameter baseline (Section 3.2.4)—and not on the nested-CV-selected protocol that Section 3.2.5 establishes as the main reported protocol. Holding the model fixed is what makes these analyses interpretable: they ask whether an interval survives a change in resampling scheme, as well as whether a gap between two feature sets is distinguishable from zero, and letting nested selection change the algorithm per case or per condition as well would confound each question with a second one. The cost is that these intervals attach to Random Forest specifically, which on some datasets is neither the nested-CV-selected model nor the best of the five fixed-hyperparameter candidates reported for the same case elsewhere in this section; where that gap is large, we say so in the table. Captions call these untuned-baseline intervals: they do not quantify uncertainty around the nested-CV-selected models. The 95% CI columns of the nested-CV result tables in Section 4.2.1, Section 4.2.2 and Section 4.2.4 are a different kind, conditional holdout intervals: a pairs bootstrap (2000 resamples) of the refit nested-CV-selected model’s holdout predictions, which reflects holdout sampling variability given the selected model, not the variability in the selection step. The GHALogs results (Section 4.2.3) report pairs-bootstrap intervals on five fixed-hyperparameter models, with no selection step involved.
Under the corrected AR construction, the reference domain’s with-AR result () has a CI that includes zero, exactly like the without-AR result; under the leaky construction, it appeared cleanly significant (Section 5.1). Card Payment’s with-AR result likewise includes zero under both the block and the pairs bootstrap. The XYZ Sales with-AR result is the only with-AR condition that is significant, and it is the single cell in this table most worth having a dependence-aware check on for exactly that reason: its block-bootstrap CI excludes zero just as its pairs-bootstrap CI does, so the one surviving synthetic-domain AR advantage is not an artifact of the pairs bootstrap’s exchangeability assumption. Across the table, there are 0 of 6 block-versus-pairs significance flips. Whether that AR advantage is distinguishable from the corresponding without-AR result is a different question, which two overlapping marginal CIs cannot answer, and which Section 4.3 tests directly.
Ablation Result: Zeroing the 21 target correlations moves ABC Cloud’s without-AR Random Forest from 0.106 to ; permuting them moves it to . Because System Uptime (%) is linked to the rest of the copula only through these 21 entries, this remains close to a controlled null, confirming that the without-AR predictive signal on synthetic data is recovery of hand-specified generator structure, not something that the models discovered independently. We note an unexpected coupling: the with-AR condition also moves under both ablations (0.195 baseline zeroed), traced to the copula’s global positive semi-definite correction perturbing System Uptime’s realized samples, even though it is not specified to correlate with anything else in the zeroed variant—a reminder that the copula’s variables are not as cleanly separable as the configuration file’s section structure suggests. Full detail: R1_BOOTSTRAP_AND_ABLATION_EVIDENCE.md, BLOCK_BOOTSTRAP_EVIDENCE.md.
4.1.6. Hyperparameter Sensitivity: Nested-CV Alpha Tuning
This subsection reports a narrower nested-CV cross-check alongside the full nested-CV harness (Section 3.2.5): a nested-CV pass tuning only Ridge/Lasso’s alpha (not algorithm choice or Random Forest/Gradient Boosting’s hyperparameters). Its conclusion (that the default alpha = 1.0 substantially understates linear-model performance, and that tuned linear models are competitive with or exceed Random Forest’s fixed-hyperparameter holdout on all three domains without AR) is consistent with the full nested-CV harness’s results in Table 10 above, which give the main comparison and extend the same logic to Random Forest and Gradient Boosting’s own hyperparameters, as well as to algorithm choice itself. We retain this narrower analysis because its without-AR tuned results (Ridge 0.108 reference domain, 0.608 Card Payment; Lasso 0.620 XYZ Sales) match the full nested-CV harness’s results almost exactly—a cross-check that the two independently run analyses agree. Full detail: M3_NESTED_CV_TUNING_EVIDENCE.md.
4.1.7. Feature Importance Analysis (RQ2)
Answer: Build-phase metrics carry the largest aggregate share of feature importance (32.1%), with Test-phase metrics close behind (21.1%), together outweighing Code-phase metrics (4.9%) by roughly 6.6:1 and 4.3:1, respectively. Test Environment Availability (%) remains the single most important individual feature (20.6%). These shares must be read for what they are: they describe which features a model leans on inside a single synthetic domain whose inter-metric correlations we specified ourselves in the copula generator (Section 3.1.4). They are evidence that the framework recovers relationships embedded in the generator, and a hypothesis worth testing on real data, not a confirmed finding about which SDLC phases drive production uptime; the caveats at the end of this subsection develop the point. This analysis uses the without-AR feature set, so it does not depend on the AR-feature construction.
We analyze feature importance using permutation importance on the Random Forest model for the ABC Cloud dataset without AR features ( repeats on the test set). Table 12 and Figure 5 present the top 15 features.
Table 12.
Top 15 Random Forest feature importances (without AR features), color-coded by SDLC phase.
Figure 5.
Top 15 Random Forest feature importances (without AR features), color-coded by SDLC phase. Build and test infrastructure metrics dominate; Code-phase metrics are a small contributor.
Phase-Level Contribution Summary: Table 13 aggregates these importances by SDLC phase.
Table 13.
Phase-level aggregate feature-importance contribution summary.
Two caveats limit how far this ranking should be pushed: First, this is a predictive association learned inside one synthetic domain whose inter-metric correlations were specified by us in the copula generator (Section 5.1); it tells us which features the model leans on to reconstruct a hand-specified structure, not which SDLC activities causally drive uptime, and it should not be read as investment guidance on its own (see Section 6.2). Second, permutation importance is not immune to multicollinearity: correlated Build-phase features (raw rate, difference terms, percent-change term, environment-consistency measure) may be substituting for one another rather than each carrying independent signal.
4.1.8. Phase-Aware Feature Reduction
Stopping-Point Selection: PA-RFE’s feature-count selection is nested (Section 3.2.1 and Section 3.2.5): a 2-fold inner TimeSeriesSplit within the 80% outer-training partition selects the stopping point by mean inner-CV (permutation-importance for the coarser inner-fold curves, reduced from 10 for tractability; the final subset at the selected stopping point is evaluated at full fidelity), and the resulting subset is scored once on the untouched outer holdout. Table 14 compares this with selection by validation- maximization on a single fixed split.
Table 14.
PA-RFE stopping point selected by validation- maximization on a single fixed split vs. by nested CV.
Relative to validation-based selection, nested selection changes the picture in a non-uniform way: ABC Cloud’s catastrophic overfitting is substantially reduced (albeit still negative); Card Payment flips from positive to negative, since its validation-selected “win” depended on a feature count that did not generalize under nested selection; XYZ Sales more than doubles, the one domain where nested selection improves the result. We do not read this as PA-RFE reliably achieving competitive predictive accuracy under any selection protocol tested. What holds under both selection rules is the structural guarantee: all eight real SDLC phases retain representation at the selected subset in every domain—a mechanical consequence of the constraint, not an empirical finding. PA-RFE’s contribution is this structural guarantee. We do not claim that it delivers a general predictive advantage over simpler feature-selection baselines.
4.1.9. Factors Influencing Prediction Accuracy (RQ4)
Answer: Under the corrected AR-feature construction (Section 3.1.3), temporal autocorrelation’s contribution to apparent predictive power is small and domain-specific: approximately 45% on the reference domain under fixed hyperparameters (~88% under the leaky construction, Section 5.1), and negligible under nested/regularized selection. PA-RFE reliably guarantees cross-phase coverage; its raw predictive accuracy is domain-dependent and, under nested selection, ranges from a severe collapse (ABC Cloud) through a reversal to negative (Card Payment) to a substantial improvement (XYZ Sales) (Section 4.1.8).
Temporal Autocorrelation
System Uptime’s realized lag-1 autocorrelation is weak in all three domains (0.031, , and for ABC Cloud, Card Payment, and XYZ Sales respectively), yet the fixed-hyperparameter with-AR advantage is not uniformly zero; this apparent tension, along with its resolution via the AR-feature audit, is discussed in full in Section 5.1.
Feature Dimensionality Effects
The feature-to-sample ratio (241 features against 120 to 160 training rows, depending on domain) exceeds the classical 10-samples-per-feature guideline by 15 to 20 times (Section 5.4), the dominant driver of the fixed-hyperparameter linear models’ instability characterized in Section 4.1.3.
Cross-Phase Interaction Effects
Multiplicative cross-phase interaction terms (Stage 3, Section 3.2.1) allow tree-based and nested-CV-selected linear models alike to capture non-additive relationships between phases (e.g., build stability × test coverage) without manual specification.
Release-Type Effects
The generator’s release-type modulation (major releases show 3 to 8% quality-metric degradation; patch releases show 2 to 5% improvement; see Section 3.1.4) is a specified, not emergent, property of the synthetic domains, consistent with this paper’s broader caution about reading synthetic-domain patterns as independently discovered.
4.1.10. Cross-Domain Validation
The nested-CV-selected without-AR result per domain is the one already given in Table 10 (Section 4.1.2) and is not repeated as a second table here: Ridge at on ABC Cloud, Ridge at on Card Payment, and Lasso at on XYZ Sales. No bootstrap interval is computed on the synthetic nested-CV-selected fits themselves. Every pairs, block, and paired-difference interval on the synthetic domains attaches to the fixed-hyperparameter Random Forest protocol (Section 4.1.5 and Section 4.3), so the uncertainty statement available per domain is that protocol’s comparison point rather than a nested one. On that protocol, the ABC Cloud and Card Payment intervals are wide and include zero in both AR conditions, while the XYZ Sales Random Forest interval excludes zero in both conditions (without-AR ; with-AR , Table 11), the only domain where the untuned-baseline interval separates from zero at all. The paired-difference test nonetheless finds the two XYZ Sales conditions indistinguishable from each other (Section 4.3).
Figure 6 shows the full with-AR/without-AR nested-CV-selected comparison across all three domains.
Figure 6.
Cross-domain validation, nested-CV-selected: holdout R2 per domain, with and without AR features. Bars are labeled with the nested-CV-selected algorithm; no tree ensemble is selected in any of the six domain/condition combinations.
As Section 5.1 discusses in detail, this 0.108-to-0.620 spread reflects one correlation specification’s signal recovered under three different marginal/temporal configurations, not three independent confirmations of the same finding.
4.1.11. Feature Selection and Autoregressive-Feature Analysis
The 32 AR-derived features and their prior-period status under the corrected construction are itemized in Section 4.1.1. The algebraic-recoverability audit that identifies the lag-feature leakage is reported in Section 3.1.3 and Section 5.1.
4.2. Real-World Validation
This subsection reports PRESTO’s methodology applied to four independent real-world datasets: within-project temporal (TravisTorrent), within-signature temporal (Mozilla Perfherder), cross-sectional CI performance (GHALogs), and cross-sectional defect-proneness (SQuaD). None of these four provides the full 241-feature, eight-phase SDLC coverage that the synthetic domains do, and none of them—individually or together—directly tests PRESTO’s full proposition that multi-phase SDLC process metrics predict production System Uptime. Each tests a narrower, real piece of that proposition: TravisTorrent and GHALogs test whether process/repository characteristics predict CI-pipeline timing outcomes (build duration, CI-run duration), not production performance; Perfherder tests the genuine autoregressive persistence of a real production-performance signal, with no SDLC process features involved; SQuaD tests whether process and code-quality metrics predict defect and vulnerability outcomes, not uptime. Closing the gap to a direct test (a real dataset joining full eight-phase SDLC telemetry to a genuine production-uptime outcome) remains an avenue for future work (Section 6.4).
4.2.1. TravisTorrent (Build Duration, 2017-Vintage)
We evaluated PRESTO on real continuous integration data from TravisTorrent [72] (2.64 million builds, 1359 projects), using seven projects (DataDog/dd-agent, bundler/bundler, getsentry/sentry, gonum/matrix, mongodb/mongoid, rg3/youtube-dl, rspec/rspec-core). We cannot reconstruct a formal, reproducible selection rule behind this specific set of seven (the selection log was not retained); we flag this as a limitation (Section 6.3) rather than retrofitting one.
TravisTorrent covers four of PRESTO’s eight phases (Code, Test, Build, Project); the regression target is build duration, not System Uptime, since no production-performance metric is available in CI-only datasets. The split is per project and genuinely chronological, and we name the ordering variable so that the claim is auditable rather than asserted: each project’s builds are sorted ascending by gh_build_started_at, the build’s own start timestamp; the most recent 20% of that project’s builds form its holdout, and an inner TimeSeriesSplit runs entirely inside the remaining 80% (Table 6, Section 3.2.4). Table 15 reports the nested-CV-selected results (full-feature condition), alongside the model selected by holdout among the same five inner-CV-tuned candidates—an optimistic comparison (Section 3.2.5).
Table 15.
TravisTorrent, full-feature condition: nested-CV-selected model vs. model selected by holdout (optimistic comparison). 95% CI: conditional holdout interval on the refit nested-CV-selected model; it does not include model-selection variability.
Two findings stand out: First, youtube-dl’s positive result survives nested selection almost exactly (0.471 vs. 0.475 under holdout selection) and remains this paper’s best real-world single-project result, still exceeding the synthetic baseline. Second, nested selection fails catastrophically on dd-agent, the largest project (~144,000 builds). The inner-CV procedure selects Ridge Regression because its inner-CV score () is less bad than Gradient Boosting’s (), even though Ridge’s actual outer-holdout is approximately million (a numerically unstable fit under severe multicollinearity at this scale) versus Gradient Boosting’s 0.392 under holdout selection. We do not discard this result or substitute a better-behaved model: it is a genuine limitation of nested inner-CV selection itself (Section 3.2.5). Excluding tr_log_testduration and its derivatives (the robustness check described below) does not rescue dd-agent’s nested result, and under nested selection every project’s holdout is negative once the feature is excluded, youtube-dl’s included (); full detail can be found in TRAVISTORRENT_NESTED_SELECTION_EVIDENCE.md.
Test-phase metrics (test duration and derivatives, test counts, assert density) lead in five of seven projects, directionally consistent with the synthetic domains’ testing-infrastructure-dominance finding (Section 4.1.7), but a robustness check excluding the definitionally circular tr_log_testduration shows that every project’s holdout turns negative under both protocols (fixed-hyperparameter Random Forest, Section 4.3; nested-CV selection, above); this dataset’s quantitative signal was substantially carried by a feature partially circular with the target, and TravisTorrent should be read as weak, feature-ranking-only corroboration, not confirmation.
Dependence-Aware Bootstrap: TravisTorrent’s holdouts are genuinely build-ordered within each project, so a pairs bootstrap that treats consecutive builds as exchangeable draws is an assumption worth testing rather than asserting. Table 16 reports the moving-block bootstrap of Section 4.1.5 alongside the pairs bootstrap on the same fitted predictions, per project. The model is fixed-hyperparameter Random Forest throughout—this evidence family’s convention so that the pairs-versus-block comparison is not confounded by a change of algorithm—which means that the point column below is the Random Forest fit specifically and not the nested-CV-selected figure in Table 15’s old-protocol column; the comparison of interest is between the two intervals on a given row, not between these point estimates and the nested ones.
Table 16.
TravisTorrent moving-block versus pairs bootstrap, fixed-hyperparameter Random Forest, full-feature condition, per project. Block length follows the same rule and is reported per project because holdout size varies by two orders of magnitude. Untuned-baseline intervals: computed on fixed-hyperparameter Random Forest; they do not quantify uncertainty around the nested-CV-selected models (Table 10 and Table 15, Section 4.2.2 and Section 4.2.4).
Block intervals are uniformly wider than pairs intervals here (mean width change units)—the direction expected when residuals within a project’s build sequence are positively autocorrelated—and the widening is proportionally largest on the projects with the narrowest pairs intervals: mongoid’s interval roughly quadruples in width. Zero of seven projects change significance status between the two schemes, and none of the seven is part of this paper’s primary real-world validation set in either case (Section 4), so this table adds dependence-aware coverage for a temporal holdout without altering any designation. It does not rescue the substantive picture: five of the seven point estimates remain below zero, a sixth (matrix, 0.005) is indistinguishable from it under either scheme, and five of the six intervals that exclude zero do so from below.
Figure 7 shows the full-feature nested-CV-selected results across all seven projects, alongside the holdout-selected comparison.
Figure 7.
TravisTorrent build-duration prediction, full-feature condition, 7 projects: Nested-CV-selected vs. Selected by holdout R2 (optimistic comparison). The dd-agent (true ) and bundler (true ) nested bars are capped at for display; true values are annotated.
4.2.2. Mozilla Perfherder (Real Runtime Performance)
Mozilla’s Perfherder/Treeherder system records production-facing runtime performance measurements cross-referenced with human-confirmed regression alerts [79]. This validation exercises PRESTO’s feature-engineering and model-comparison methodology against a genuine real-world target, rather than the SDLC-process-features half of PRESTO’s claim, since Perfherder provides no upstream SDLC process metrics.
We use three signatures: ESPN page-load time (3548 pushes), Instagram Perceptual Speed Index (3177 pushes), and NYTimes Perceptual Speed Index (3225 pushes). As with TravisTorrent, the split is genuinely chronological, and we name the ordering variable: each signature’s pushes are sorted ascending by push_timestamp, the most recent 20% form that signature’s holdout, and the inner TimeSeriesSplit runs inside the remaining 80%, with its fold count reduced below five only where a signature’s training partition is too short to support five folds (Table 6, Section 3.2.4). Table 17 reports the nested-CV-selected results, and Figure 8 plots them against the alert-history-only condition.
Table 17.
Perfherder, with-AR condition: nested-CV-selected vs. selected by holdout among the same inner-CV-tuned candidates (optimistic comparison; Lasso, Ridge, Ridge). 95% CI: conditional holdout interval on the refit nested-CV-selected model; it does not include model-selection variability.
Figure 8.
Real-data validation: nested-CV-selected holdout R2, with-AR vs. alert-history-only features, across three Mozilla Perfherder signatures.
This is the most robust real-world result in this paper. All three with-AR results are part of this paper’s primary real-world validation set: real production performance metrics are strongly, if variably, predictable from their own recent history, on real data with real noise and no copula prior to fall back on. Without historical values, collapses to strongly negative for every model and signature. The bug-triage-metadata enrichment finding (see the “Does Bug-Triage Metadata Beat a Bare Alert Flag?” section: both feature sets remain firmly -negative on two build-artifact-size signatures, though bug-enriched features cut holdout MAE by roughly 80%) is a secondary, exploratory analysis rather than part of that primary set.
Dependence-Aware Bootstrap, and Why It Matters Most Here: Because Perfherder’s with-AR results are the real-world half of this paper’s primary validation set, they are the results whose confidence intervals most need to survive a resampling scheme that does not assume that consecutive pushes are exchangeable. We therefore give all three signatures the moving-block bootstrap of Section 4.1.5, ordered by the same push_timestamp that the split uses. Two framing points apply to Table 18: First, the fitted model is fixed-hyperparameter Random Forest—this evidence family’s convention so that pairs-versus-block is the only thing changing; on Perfherder, that model is substantially weaker than the Lasso that nested-CV selection picks for all three signatures, so the point values below sit well beneath Table 17’s figures for the same signatures and are fixed-hyperparameter baseline fits, not the headline numbers. Second, what transfers from one protocol to the other is the significance status, not the point estimate, and that status is what the primary-set designation rests on.
Table 18.
Perfherder moving-block versus pairs bootstrap, fixed-hyperparameter Random Forest, with-AR condition, per signature. The alert-history-only condition is not bootstrapped here; the with-AR-versus-without-AR gap is tested directly as a paired difference in Section 4.3. Untuned-baseline intervals: computed on fixed-hyperparameter Random Forest; they do not quantify uncertainty around the nested-CV-selected models (Table 10, Table 15 and Table 17, Section 4.2.4).
Zero of three signatures change significance status between the two schemes: all three with-AR intervals exclude zero under the dependence-aware bootstrap as well as the pairs bootstrap. This is the result in this family where dependence-aware inference matters most, since a flip here would directly cost Perfherder its place in the primary set. The block intervals are wider, as expected (mean width change units), and the widening is largest on instagram-speedindex, whose lower bound falls from 0.108 to 0.011, close enough to zero that we would not describe its margin as comfortable, only as intact.
Does Bug-Triage Metadata Beat a Bare Alert Flag?
Two Perfherder signatures with rich bug-linked alert history (installer size on osx-cross, 17,217 pushes, 23 bug-linked alerts; a build-size metric on osx-aarch64-shippable, 11,119 pushes, 11 bug-linked alerts) show that both alert-flag-only and bug-metadata-enriched feature sets remain firmly -negative, but bug-enriched features reduce holdout MAE by roughly 80% on both signatures—a modest, bounded-metric finding, not a demonstrated predictive win. Nested selection (PERFHERDER_NESTED_SELECTION_EVIDENCE.md) does not change this qualitative picture.
4.2.3. GHALogs (Modern CI Cohort, Cross-Sectional)
GHALogs provides 513k GitHub Actions workflow runs with per-step timing across 116k workflows [80]. We use the 28,443 repositories with both real CI performance data and real repository-level process/popularity characteristics, predicting mean CI run duration.
Split Construction, and Why TimeSeriesSplit Is Not Temporal validation Here: GHALogs is cross-sectional. After aggregating run_features.csv to one row per repository and joining repository-level process and popularity metrics, each of the 28,443 rows is a distinct repository, and no repository contributes a time series of its own. The 80/20 split is nevertheless not random: rows are sorted ascending by each repository’s pushedAt timestamp (its most recent push), so the holdout is the 20% of repositories that were most recently active. We prefer this to a shuffled split because it keeps the training partition from being interleaved in calendar time with the holdout. The inner TimeSeriesSplit(5) applied within the 80% outer-training partition should be read accordingly: it is a deterministic, non-overlapping resampling of a recency-ordered cohort, not temporal cross-validation, because an expanding-window fold carries no forecasting semantics when each successive row is a different repository rather than a later observation of the same one. Table 6 (Section 3.2.4) records this alongside the other six datasets. One consequence is worth naming explicitly: because the ordering variable is repository recency rather than anything causally upstream of CI duration, a distribution shift between older and newer repositories would show up as reduced holdout , not as leakage. No historical target values are available per repository to leak from in the first place.
Full-Feature Result (Table 19, Figure 9): Gradient Boosting achieves the best holdout (0.113, 95% CI [0.050, 0.356]), and all five models’ CIs exclude zero. Unlike the other three real-world datasets, GHALogs reports all five candidate models rather than selecting one, by nested inner CV or by any other procedure, so there is no selection step here to nest. The conclusion that this dataset supports below—that none of the five ablated models’ intervals exclude zero—therefore does not depend on how a model would have been chosen. The top feature by Random Forest importance is mean_n_steps (48.6%), which is largely definitional (workflows with more steps mechanically take longer), analogous to the circularity already conceded for TravisTorrent’s tr_log_testduration.
Table 19.
GHALogs, full-feature vs. mean_n_steps-excluded. All five models use fixed hyperparameters, with no selection step; 95% CIs are pairs-bootstrap intervals on those fits.
Figure 9.
GHALogs: holdout R2 for five fixed-hyperparameter models (no selection step), full feature set (all 5 pairs-bootstrap CIs exclude zero) versus mean_n_steps-excluded (none do).
The mean_n_steps ablation changes the interpretation substantially. With mean_n_steps removed, the best model’s holdout drops to 0.007 (Lasso), and none of the five models’ 95% bootstrap CIs exclude zero. GHALogs’ significance does not survive removing the definitionally circular feature, the same fate as the TravisTorrent robustness check. GHALogs’ full-feature result is therefore substantially carried by a near-tautological feature rather than by independent process/popularity signal, so despite having no historical target values to leak, GHALogs is not a clean test of PRESTO’s headline claim. GHALogs is accordingly excluded from this paper’s primary real-world validation set (Section 4), and that set’s weight falls on Perfherder and SQuaD instead; see Section 4.2.4. The argument above compares two marginal intervals across a significance threshold, so we also test the drop directly: the paired-difference bootstrap of Section 4.3 puts the full-minus-ablated gap for fixed-hyperparameter Random Forest (0.096 to ) at , with a CI of excluding zero, which makes the downgrade a measured effect of removing the feature rather than an inference from two intervals happening to fall on opposite sides of zero.
4.2.4. SQuaD (Real Process Metrics, Cross-Sectional Defect-Proneness)
SQuaD [81] provides 62,449 release-level process metrics across 440 open-source projects. We aggregate to 408 projects with ≥5 observed releases, predicting mean per-release defect-fix count. Table 20 reports the nested-CV-selected results for all three SQuaD variants.
Table 20.
SQuaD, all three variants: nested-CV-selected vs. selected by holdout (optimistic comparison). 95% CI: conditional holdout interval on the refit nested-CV-selected model; it does not include model-selection variability.
Split Construction, and Why TimeSeriesSplit Is Not Temporal Validation Here: We aggregate to one row per project rather than using all 62,449 release-level rows, because a single project (apache#servicemix-bundles) accounts for 27% of the raw rows and would dominate a release-level split. Unlike TravisTorrent, Perfherder, and GHALogs, this dataset’s process_metrics.csv carries no per-project release-date column, so there is no date available by which to order the rows at all. The 80/20 split therefore follows groupby-aggregation row order, which is alphabetical by project name rather than temporal. We checked what confound that arbitrary ordering could admit: 406 of the 408 qualifying projects are Apache-hosted, which bounds how far an organizational difference could separate the training and holdout partitions, but the split is genuinely not chronological, unlike the other three real-world validations in this paper. The TimeSeriesSplit-wrapped inner CV reported alongside the holdout should accordingly be read as plain resampling of an arbitrary but fixed ordering, not as genuine temporal cross-validation. No historical target values are used as features in any of the three SQuaD variants, so the absence of a temporal split is not itself a leakage risk; it is a limit on what the inner folds mean, and we report it as such rather than letting TimeSeriesSplit’s name imply a temporal guarantee that this dataset cannot support. Table 6 (Section 3.2.4) records this alongside the other six datasets.
There are three distinct outcomes: Defect-fix rate, which scores 0.450 under holdout selection, is non-significant under nested selection (, CI includes zero)—the nested-CV-selected model (Lasso) is far more conservative than the holdout-selected Random Forest, and the more conservative choice matters here. CVE-count is part of this paper’s primary real-world validation set, identical under both selection rules (both select Random Forest, , significant). The static-analysis-enriched defect-fix result is significant under nested selection (CI ), whereas the fixed-hyperparameter Random Forest’s interval (, ) narrowly includes zero. SQuaD’s CVE-count and enriched-defect-fix results, alongside Perfherder’s with-AR results, are this paper’s primary real-world validation analyses (Section 4).
From Defects to Vulnerabilities: A Harder Security Target
Method: cve_data.csv, 5294 CVE-linkage rows, 1479 distinct CVE IDs, log1p(cve_count) target; results per Table 20 above. Every linear model fails on this target under both protocols.
Enrichment with Real Static-Analysis Code-Quality Features
Method: SonarQube/PMD/CK features aggregated via DuckDB, 48 features up from 13; coverage: 311 to 352 of 408 projects, depending on source; results per Table 20 above. Under nested selection, this is a significant result (a conditional holdout interval on the nested-CV-selected model; see Table 20). Being significant on its own terms is not the same as being better than the plain feature set, however, and we separate these two claims rather than letting the first imply the second: the paired-difference bootstrap of Section 4.3 puts the enriched-minus-plain gap at with a CI of that includes zero, so the enrichment does not significantly outperform the 13-feature process-metric baseline on this 82-project holdout. The enriched result’s own significance is unaffected by that; what is not established is that enrichment is why it is significant. The paired difference is an untuned-baseline interval on fixed-hyperparameter Random Forest, whose own enriched interval narrowly includes zero (Section 4.2.4); the significance claim rests on the nested-CV-selected model.
4.3. Paired-Difference Confidence Intervals for Feature-Set Comparisons
Answer: Four families of comparison reported above are argued by setting two marginal confidence intervals side by side, which is not a test of whether the difference between the two conditions differs from zero. Tested directly, as a bootstrap interval on the paired difference in holdout , the four families separate cleanly. Perfherder’s with-AR advantage and GHALogs’ mean_n_steps ablation are both large and unambiguously significant. The synthetic domains’ with-AR advantage is noise-level in all three domains. TravisTorrent’s circular-feature ablation excludes zero in all seven projects, but two of the seven do so only because the holdout is large, at a magnitude that carries no practical meaning. SQuaD’s static-analysis enrichment does not significantly outperform the plain feature set, even though the enriched result is itself significant.
Method: For each comparison, we resample bootstrap indices once per resample and apply the same indices to both conditions’ (prediction, actual) pairs, recompute under each condition, and take the 2.5/97.5 percentiles of the resulting distribution of (2000 resamples, seed 20260810). Pairing the indices is what makes the interval a statement about the difference rather than about two separate quantities: a resample that happens to draw an unusually easy or unusually hard subset of the holdout moves both conditions in the same direction, and that shared movement cancels out the difference instead of inflating its variance. Row alignment was asserted before pairing in every comparison, with both conditions’ actual target vectors verified to be identical in value and order. As set out in Section 4.1.5, the fitted model is fixed-hyperparameter Random Forest, held identical across the two conditions of every comparison so that each difference reflects the feature-set change alone; these are therefore untuned-baseline intervals, not intervals on the nested-CV-selected figures of Section 3.2.5. Table 21 reports the with-AR-versus-without-AR family, and Table 22 reports the three feature-ablation and feature-enrichment families. Differences are computed on unrounded values, so a row’s rounded difference can differ by 0.001 from the difference of its two rounded per-condition columns (ABC Cloud: ). Full detail: PAIRED_DIFFERENCE_BOOTSTRAP_EVIDENCE.md.
Table 21.
Paired-difference bootstrap on the with-AR-minus-without-AR gap in holdout . Without-AR: the 241-feature SDLC-only set (synthetic) or the alert-history-only set (Perfherder). Synthetic per-condition values are the Table 11 fits; differences use unrounded . Untuned-baseline intervals: computed on fixed-hyperparameter Random Forest; they do not quantify uncertainty around the nested-CV-selected models (Table 10, Table 15, Table 17 and Table 20).
Table 22.
Paired-difference bootstrap on the three feature-ablation and feature-enrichment comparisons, fixed-hyperparameter Random Forest in both conditions. TravisTorrent excludes tr_log_testduration and derivatives. Sentry and mongoid rows use four decimals (Supplementary Materials S1). Untuned-baseline intervals: computed on fixed-hyperparameter Random Forest; they do not quantify uncertainty around the nested-CV-selected models (Table 10, Table 15, Table 17 and Table 20).
The per-family interpretation behind these four results—including why TravisTorrent’s two smallest, most negligible gaps still technically exclude zero and why SQuaD’s enriched result is significant on its own terms without significantly beating the plain feature set—is given in full in Supplementary Materials S1.
4.4. Deployment Considerations
PRESTO’s trained models are lightweight (under 50 MB) and produce predictions in under 100 ms per release, making them suitable for integration into CI/CD pipelines as a pre-merge gate or canary promotion criterion. We emphasize that the reported values were obtained on synthetic data (largely recovery of specified structure) or on real datasets testing narrower propositions than production-uptime prediction; site-specific validation against a real eight-phase SDLC-to-uptime dataset is required before any operational reliance.
4.5. Potential Applications
Based on the predictive signal observed in Section 4.2 (scoped per dataset, not treated as uniform confirmation of the full PRESTO proposition), we identified potential applications warranting real-world investigation: release risk scoring, test-environment investment prioritization (using the feature-importance rankings of Section 4.1.7, with the caveats stated there), early warning for deployment decisions, and process-metric benchmarking across teams. We decline to quantify dollar-value business impact, since such estimates require real-world deployment data that this study does not provide.
5. Threats to Validity
Following established guidelines for empirical software engineering research, we discuss threats to validity, organized by category.
5.1. Internal Validity
Synthetic-Generator Calibration Defects: An audit of the generator code against its documentation identifies three implementation defects (dropped correlation pairs, unapplied domain overrides, a substituted distribution family); all three are corrected (Section 3.1.4; full detail in Appendix A.1), and every synthetic-domain number in this paper reflects the corrected generator. The target’s low realized lag-1 autocorrelation remains as a structural limitation.
Autoregressive-Feature Construction Leakage: This is the paper’s central internal-validity finding. To reconcile System Uptime’s weak realized lag-1 autocorrelation (0.031, , across the three domains) with a large apparent AR benefit, we audited all 32 AR-derived features for algebraic recoverability of the current-row target, extending the acf()/lag-1-autocorrelation diagnostic (Section 3.1.4) from the target alone to every AR-derived predictor. In the leaky construction, add_lag_features() computes the target’s own diff_k and pct_change_k against the current, unshifted row rather than a prior-shifted one; target_t = lag_k + diff_k and target_t = lag_k \cdot (1+\text{pct\_change}_k) hold exactly in 97% to 99% of rows across all three domains and all four lag steps. Removing just these 8 (of 32) features accounts for 97.8% of the reference domain’s entire with-AR gain over the without-AR baseline (104% and 102% on Card Payment Processor and XYZ Sales Force, respectively, i.e., the entire gain and then some, with the remainder slightly negative). Under the leaky construction, the reference domain’s with-AR is 0.914 (CI , excluding zero), and roughly 88% of the with-AR gain appears attributable to AR persistence; under the corrected construction, which shifts the series before differencing as the rolling-window features do (Section 3.1.3), it is 0.195 (fixed-hyperparameter)/≈ (nested-CV-selected), its CI includes zero, and the AR-persistence share is approximately 45% of a much smaller total effect (derivation in Section 4.1.2). Every with-AR result in this paper uses the corrected construction. The general lesson is that target-derived differences and ratios must be computed on a prior-shifted series, and an algebraic-recoverability check detects violations that a target-only autocorrelation diagnostic misses. Full Audit Detail: AR_ACF_DIAGNOSTICS_EVIDENCE.md.
Nested Selection Is Not a Complete Fix for Model-Selection Uncertainty: Nested cross-validation (Section 3.2.5) is used for model selection throughout this paper, removing the winner’s-curse-style optimism of selecting by the reported holdout score. It is not, however, immune to failure: TravisTorrent’s largest project (Section 4.2.1) shows nested inner-CV selection choosing a numerically catastrophic linear model over a reasonable tree-ensemble alternative, because a small inner fold’s is an unreliable proxy for outer-holdout stability under severe multicollinearity. We report this as a genuine internal-validity concern rather than treating nested selection as a solved problem.
Feature Engineering Bias: Feature engineering was informed by domain knowledge; feature logic is identical across all three domains and was defined before observing results.
Hyperparameter Selection: The nested-CV harness (Section 3.2.5) tunes every algorithm, including Random Forest and Gradient Boosting, so that trees are not compared at fixed defaults against tuned linear models.
Copula Correlation Specification: The 84 inter-metric correlations were specified by the authors; all 84 are applied by the corrected generator. If these correlations encode assumptions that align with the prediction models, results may be optimistic. Correlation specifications are published in full for independent review.
Cross-Domain Correlation Non-Independence: The same 84-pair correlation configuration is applied to all three domains, varying only marginals, temporal dynamics, and release cadence. The cross-domain validation result (Section 4.1.10) is one correlation specification’s signal recovered under three configurations, not three independent confirmations—a qualification also stated in the Abstract and Contributions (Section 1.4, item 3).
Near-Tautological Features Ablation: System availability (%) ranks #4 in Gini-based feature importance and, when removed, collapses the reference domain’s Random Forest below zero while improving Card Payment’s; its contribution is domain-specific and real rather than uniformly tautological. Full Detail: M9_SYSTEM_AVAILABILITY_ABLATION_EVIDENCE.md.
5.2. External Validity
Synthetic Data (Critical): Exclusive reliance on synthetic data for the core validation remains the most critical limitation; no public dataset combining SDLC process metrics with runtime performance outcomes exists. Partial Mitigation: Four real-world datasets are used, but, as Section 4.2 states, none directly tests the full proposition. The primary real-world validation set is Perfherder’s with-AR results and SQuaD’s CVE-count and enriched defect-fix results (Section 4); GHALogs (significance does not survive the mean_n_steps ablation), TravisTorrent (weak, feature-ranking-only corroboration), and SQuaD’s plain defect-fix rate (non-significant under nested selection) are exploratory.
Domain Coverage: Three synthetic domains, seven TravisTorrent projects, three main and two bug-enriched Perfherder signatures, and 408 SQuaD projects. Findings may not transfer to embedded, mobile, real-time, or safety-critical software.
5.3. Construct Validity
Target Variable: System Uptime (%) captures availability but not latency, throughput, error rates, or resource efficiency. SDLC Metric Coverage: 164 metrics may omit architectural complexity, dependency health, technical debt, and, as Section 2.5 discusses, issue-handling process and project-actor cooperation signals. Temporal Granularity: Release-level aggregation may mask within-release dynamics.
5.4. Conclusion Validity
Sample Size: Feature-to-sample ratios (1.51 to 2.01 features per training sample) fall 15 to 20 times short of the classical 10-samples-per-feature guideline. PA-RFE’s nested-selection results (Section 4.1.8) show that this is a partial, domain-dependent mitigation at best, not a solved problem: the selected feature count under nested selection does not reliably improve test-set .
Comparison-Family Size and Multiple Comparisons: This paper reports on the order of 150 comparisons: three synthetic domains × two AR conditions under both fixed-hyperparameter and nested-CV protocols, PA-RFE’s nested and non-nested stopping-point selections, seven TravisTorrent projects × two conditions under both protocols, five Perfherder signature/condition combinations under both protocols, and three SQuaD variants under both protocols, plus the nested-CV alpha-tuning cross-check and the target-correlation ablation. We do not apply a formal multiple-comparisons correction across this family, because the comparisons are largely exploratory and diagnostic rather than a single pre-registered hypothesis-testing exercise; the primary designation of Section 4 is likewise not a pre-registration and carries no error-rate guarantee beyond that of the exploratory comparisons. The language consequence of that choice is handled by the primary-set declaration of Section 4, which names two analyses, restricts “primary” to them throughout the paper, describes every other comparison as exploratory or diagnostic at first mention, and keeps statistical significance a factual descriptor reported for every comparison regardless of that status. “Genuine” is likewise restricted to the primary set wherever it characterizes a result’s evidentiary status; the word is also used descriptively elsewhere in this paper for structural and temporal properties (a genuinely chronological split, a genuine time index, or a genuine limitation of inner-CV-based selection), where that restriction does not apply and was never intended to. We do not use “confirmatory” as a label for any analysis here, since the primary set was designated after the results were known. Separately from family size, the form of inference matters for this paper’s four feature-set comparisons: each is tested as a bootstrap interval on the paired difference (Section 4.3) rather than by placing the two conditions’ marginal CIs side by side, which materially changes the reading of two of the four families and adds to a third a practical-significance caveat that a marginal-CI comparison would hide.
Transparent Reporting: Unlike prior synthetic-data studies reporting , we separate temporal autocorrelation from genuine SDLC prediction throughout, and we extend that separation to the lag-feature leakage mode identified by the algebraic-recoverability audit (Section 5.1).
5.5. Summary of Validity Assessment
Table 23 consolidates severity and mitigation status per individually named threat.
Table 23.
Threats to validity: severity and mitigation status per individually named threat.
Critical Message: The most consequential validity threat is internal, not external: under the leaky AR-feature construction, most of the apparent autoregressive persistence on synthetic data is an artifact of same-row leakage rather than a phenomenon (Section 5.1). Readers should weigh the corrected, domain-specific AR-persistence picture and the primary real-world validation set (Perfherder’s with-AR results; SQuaD’s CVE-count and enriched defect-fix results) over any single synthetic-domain point estimate when judging how much confidence this paper’s findings deserve.
6. Discussion and Future Work
One scope condition governs everything below, and we name it directly rather than leaving it to be inferred from the threats section: no dataset available to this study joins multi-phase SDLC telemetry to production uptime as its target. The three domains that carry an uptime target are synthetic, and the four datasets that carry real data carry proxy outcomes instead (build duration, browser-performance regression, CI workflow duration, and defect and CVE counts). PRESTO’s central proposition is therefore not tested end to end anywhere in this paper, and what follows is the discussion of a framework’s proof-of-concept evaluation, not of a validated prediction result.
6.1. Summary of Findings
Build- and Test-phase infrastructure metrics carry more feature importance than code quality in this synthetic design (Section 4.1.7)—a pattern that we flag as a hypothesis worth testing on real data, not a confirmed finding, since the underlying correlations are hand-specified. Across the four research questions, SDLC process metrics provide modest predictive signal on synthetic data, which the target-correlation ablation attributes largely to recovery of the generator’s hand-specified structure rather than to independently discovered relationships (RQ1), narrowly corroborated in feature-ranking direction by TravisTorrent and more substantively by SQuaD’s CVE-count and enriched-defect-fix results, the two SQuaD analyses in the primary set (the other primary analysis, Perfherder’s with-AR results, uses no SDLC process features); Build- and Test-phase metrics dominate feature importance (RQ2); under fair, nested-CV tuning, regularized linear models are competitive with or exceed tree ensembles in most comparisons, reversing the fixed-hyperparameter baseline picture (RQ3); and under the corrected lag-feature construction of Section 3.1.3, autoregressive persistence on synthetic data is a small, domain-specific factor (RQ4).
6.2. Practical Implications
We separate practical implications into three categories:
Empirically Supported: Real-world process metrics carry some predictive value for specific, narrowly scoped outcomes on specific datasets: SQuaD’s CVE-count and static-analysis-enriched defect-fix results—both significant under nested selection (Section 4.2.4)—and, most robustly, historical performance values are a genuine, strong predictor of near-future performance on Mozilla Perfherder’s real production data. Two results do not belong in this category: GHALogs’ CI-run-duration result does not survive its mean_n_steps ablation (Section 4.2.3), and SQuaD’s plain defect-fix rate is non-significant under nested selection (, CI ).
Synthetic-Design Observations: The build/test-infrastructure-over-code-quality ranking (Section 4.1.7) originates in hand-specified copula correlations, is sensitive to how faithfully the generator implements them (Section 3.1.4), and is not independently validated anywhere in this paper. This is a hypothesis that the synthetic design surfaces, not a measured effect; it should not inform investment decisions on its own.
Propositions for Future Industrial Validation: That even modest predictive accuracy () can provide operational value for release risk assessment when combined with other decision inputs is a proposition that this paper does not itself test, not a finding. That teams should prioritize build/test infrastructure investment over code-quality investment is likewise a hypothesis requiring real-data confirmation before being treated as operational guidance, doubly so given that both the feature-importance ranking and the AR-persistence share are sensitive to generator and feature-construction defects (Section 3.1.3 and Section 3.1.4).
6.3. Limitations
The main limitations are synthetic-only core validation, limited domain coverage, author-specified copula correlations, a single target variable, high feature-to-sample ratios, and an AR-feature leakage mode (Section 3.1.3) that is corrected here but is easy to miss without an algebraic-recoverability check. Section 5 provides detailed threat analysis. We did not model development methodology, third-party library/framework mix, automatic code generation, or CI/CD tooling choice [26,42]; we also did not incorporate issue-tracking process metrics or logging/observability signals [27,69], although Section 2.5 discusses a concrete extension path for the former via issue-handling-process research. Real-world validation datasets in this paper carry no user-facing signals (star counts, sentiment, adoption trends); whether these add predictive value beyond process metrics is an open question.
6.4. Future Work
The most critical next step remains real-world validation through organizational partnerships providing production performance metrics alongside full eight-phase SDLC data. Beyond that, ordered roughly by tractability, the following directions should be pursued:
- Extend nested cross-validation to remove its own remaining failure mode. Section 4.2.1 shows that nested inner-CV selection can itself pick a catastrophically unstable model under severe multicollinearity; a follow-up could add a stability penalty or a robust-scoring inner criterion (e.g., median rather than mean inner-fold , or an explicit variance penalty) to the nested-CV harness.
- Extend the SQuaD static-analysis enrichment. The enriched defect-fix result is significant, so a natural next step is to extend the same static-analysis enrichment to the CVE-count target and improve SonarQube/PMD/CK coverage beyond the 311 to 352 of 408 projects achieved here.
- Resolve TravisTorrent’s target-circularity cleanly, ideally via a dataset whose target is a genuine production-performance signal, not a build-duration proxy structurally entangled with Test-phase timing.
- Build the issue-handling-graph metric family proposed in Section 2.5, once a dataset providing issue-tracker event logs at sufficient granularity is identified, and evaluate it rather than merely propose it.
- Apply a formal multiple-comparisons correction to a pre-registered subset of this paper’s claims; the explicit primary set named in Section 4 is a natural starting candidate.
- Longer-Horizon Directions: Multi-target prediction, neural architectures, transfer learning across organizations, and causal inference.
7. Conclusions
This paper investigates whether SDLC process metrics can predict software system performance, introducing PRESTO and a Gaussian copula-based synthetic data generator. One limitation frames every conclusion below: no dataset available to this study joins multi-phase SDLC telemetry to production uptime as its target, so PRESTO is offered as a framework and a proof-of-concept evaluation, not as a validated prediction method. Within that scope, the findings are as follows:
- SDLC process metrics show modest predictive signal on synthetic data ( to under nested-CV selection across three enterprise domains), which a target-correlation ablation shows is largely recovery of the copula’s hand-specified structure. No single real-world dataset, and no combination of the four used here, directly confirms the full proposition that multi-phase SDLC metrics predict production uptime; SQuaD’s CVE-count and static-analysis-enriched defect-fix results, along with Perfherder’s autoregressive-persistence result on real production data, are this paper’s primary real-world validation analyses, and they validate narrower, real pieces of that proposition rather than the proposition itself; GHALogs’ full-feature result does not survive the mean_n_steps ablation of Section 4.2.3.
- Autoregressive persistence on synthetic data is a small, domain-specific factor. Under the leaky construction, a same-row leakage mode (two of four AR feature families computed against the unshifted current-row target) accounts for approximately 98% of the reference domain’s entire with-AR predictive advantage (Section 5.1); under the corrected construction, that domain’s with-AR result is not distinguishable from its without-AR result, and only XYZ Sales Force has a with-AR result whose own bootstrap CI excludes zero—a result that a paired-difference test cannot distinguish from its without-AR counterpart (Section 4.3), and one that a fairly regularized nested-CV-selected model does not need.
- Tree-based ensembles outperform linear models at fixed default hyperparameters, but this is not a general property: under equivalent nested-CV tuning effort, Ridge or Lasso wins all six synthetic domain/condition comparisons. “Trees are necessary” should be read as specific to the fixed-hyperparameter baseline.
- Testing and build infrastructure stability are the strongest predictors of the synthetic uptime target (Build 32.1%, Test 21.1%, vs. Code 4.9%). Because those correlations were hand-specified in the generator, this is evidence that the framework recovers structure that the generator embedded, not evidence about real production uptime; TravisTorrent corroborates it only weakly and only in feature-ranking direction once a circular feature is excluded, so it remains a hypothesis pending real-data confirmation.
- PA-RFE reliably guarantees cross-phase feature coverage (a structural property that holds under both validation-based and nested-CV stopping-point selection) but does not reliably deliver competitive predictive accuracy: under nested selection, its test-set result varies in direction across the three domains, from severely negative on ABC Cloud Provider to a substantial improvement on XYZ Sales (Section 4.1.8).
- The copula-based data generator, released as a reusable benchmark (Zenodo DOI: 10.5281/zenodo.22005439), enables future research without requiring access to proprietary enterprise data; the corrected add_lag_features() implementation and the algebraic-recoverability audit that motivated it are included for anyone extending it.
Full validation with production performance metrics across multiple organizations, joined to complete eight-phase SDLC telemetry, remains the essential next step that this paper’s four real-world datasets each partially, but never fully, substitute for.
Supplementary Materials
The following supporting information can be downloaded at https://www.mdpi.com/article/10.3390/technologies14100657/s1: the per-family interpretation of Section 4.3’s paired-difference results (Supplementary Materials S1), and the full secondary robustness diagnostics behind Section 4.1.4’s headline conclusions (Supplementary Materials S2), both in Supplementary-Materials.pdf.
Author Contributions
Conceptualization, A.J.R. and L.N.M.; methodology, A.J.R. and L.N.M.; software, A.J.R.; validation, A.J.R.; formal analysis, A.J.R.; investigation, A.J.R.; data curation, A.J.R.; writing (original draft preparation), A.J.R.; writing (review and editing), L.N.M. and B.S.; visualization, A.J.R.; supervision, L.N.M. and B.S.; project administration, A.J.R. All authors have read and agreed to the published version of the manuscript.
Funding
This research received no external funding.
Institutional Review Board Statement
Not applicable.
Informed Consent Statement
Not applicable.
Data Availability Statement
All code and generated data are archived at Zenodo, DOI: 10.5281/zenodo.22005439 (source repository: https://github.com/amitrangari/presto-data (accessed on 2 October 2026)), including (1) the Gaussian copula-based synthetic data generator with YAML configuration files; (2) the three regenerated synthetic domain datasets (170, 200, 150 releases); (3) feature engineering, ML training, PA-RFE, and autoregressive feature analysis pipelines, including the shared nested-cross-validation model-selection harness applied to every algorithm and feature-selection comparison in the paper; (4) the bootstrap-confidence-interval and target-correlation ablation analysis, extended with a moving-block bootstrap for temporally ordered holdouts, covering the synthetic domains as well as the TravisTorrent and Mozilla Perfherder holdouts, plus the paired-difference bootstrap script covering all four feature-set comparison families; (5) the nested-CV hyperparameter tuning and Siegmund-style baseline reimplementation; (6) the system availability ablation; (7) real-world validation adapters for TravisTorrent (including its permutation-importance companion script, the test-duration-exclusion robustness check, and the nested-selection rerun), Mozilla Perfherder, GHALogs (including the mean_n_steps ablation), and SQuaD, plus the bootstrap-CI extension, SQuaD CVE-count/Perfherder bug-enrichment scripts, and the SQuaD static-analysis (SonarQube/PMD/CK) enrichment pipeline; (8) the autoregressive-feature algebraic-recoverability audit and the corrected add_lag_features() implementation that it motivated; and (9) all statistical validation and figure-generation scripts. Per-analysis evidence files in the archive document each reported result: AR_ACF_DIAGNOSTICS_EVIDENCE.md, BLOCK_BOOTSTRAP_EVIDENCE.md, GHALOGS_ABLATION_EVIDENCE.md, M3_NESTED_CV_TUNING_EVIDENCE.md, M8_SIEGMUND_BASELINE_RESULTS.md, M9_SYSTEM_AVAILABILITY_ABLATION_ EVIDENCE.md, NESTED_MODEL_SELECTION_EVIDENCE.md, PAIRED_DIFFERENCE_BOOTSTRAP_EVIDENCE.md, PA_RFE_NESTED_SELECTION_EVIDENCE.md, PERFHERDER_BLOCK_BOOTSTRAP_EVIDENCE.md, PERFHERDER_NESTED_SELECTION_EVIDENCE.md, R1_BOOTSTRAP_AND_ABLATION_EVIDENCE.md, R2_LAG_SHIFT_FIX_EVIDENCE.md, SQUAD_NESTED_SELECTION_EVIDENCE.md, TRAVISTORRENT_BLOCK_BOOTSTRAP_EVIDENCE.md, TRAVISTORRENT_NESTED_SELECTION_EVIDENCE.md. To keep the archive lightweight, bulk third-party real-world raw data (TravisTorrent, Mozilla Perfherder, GHALogs) is not re-hosted; each dataset’s original DOI is linked as a related identifier, and each real-data/<source>/DOWNLOAD.md file in the archive documents exact retrieval instructions. Software: Python 3.14.5, scikit-learn 1.9.0, numpy 2.5.2, pandas 3.0.5, scipy 1.18.0 (pinned versions recorded in ENVIRONMENT.txt, matching Section 3.2.4). Random seed 42 fixed throughout for reproducibility. The synthetic generator is deterministic and designed as a reusable benchmark, allowing researchers to modify distribution parameters and domain profiles via YAML configuration.
Acknowledgments
During the preparation of this manuscript, the authors used Claude Code (Anthropic PBC, San Francisco, CA, USA; Claude Sonnet 4.5/5 models) as an AI-assisted research and writing tool, for the following purposes: (1) drafting and revising manuscript prose across all sections; (2) writing the Python analysis scripts referenced throughout this paper (feature engineering, model training/evaluation, bootstrap confidence intervals, the target-correlation and system availability ablations, the nested cross-validation model-selection harness, the autoregressive-feature algebraic-recoverability audit that identified the lag-feature leakage, permutation importance, and the real-world dataset adapters for TravisTorrent, Mozilla Perfherder, GHALogs, and SQuaD), including re-implementing the PA-RFE algorithm and the TravisTorrent adapter from the authors’ own pseudocode and methodology descriptions after the originals were found missing from the reproducibility package; (3) generating the figure-plotting scripts used to produce the paper’s charts. All quantitative results reported in this paper were produced by executing this code against the underlying data (synthetic and real) with standard scientific Python libraries (scikit-learn, pandas, NumPy, SciPy); no numerical result, table entry, or figure value was generated or fabricated directly by the AI without underlying code execution. All AI-assisted code and prose were reviewed, verified, and edited by the authors, who take full responsibility for the content, correctness, and conclusions of this publication, including its methodological limitations and the honest reporting of findings that complicate rather than support the paper’s own headline claims—most notably the disclosure that a lag-feature leakage mode accounts for most of the apparent autoregressive persistence on synthetic data.
Conflicts of Interest
Author Amit J. Rangari was employed by JPMorgan Chase, and author Lalit Narayan Mishra was employed by Lowe’s Companies Inc. The remaining author declares that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest. No proprietary data, internal metrics, or confidential process/correlation analysis from either employer was used to inform this paper’s 164-metric registry, the copula generator’s 84 hand-specified correlation coefficients, or any other aspect of the synthetic data design. All metric definitions and correlation assumptions are grounded in publicly available DORA benchmarks and general SDLC domain knowledge, not employer-internal data.
Abbreviations
The following abbreviations are used in this manuscript:
| ACF | Autocorrelation Function |
| API | Application Programming Interface |
| APM | Application Performance Monitoring |
| AR | Autoregressive |
| AR(1) | First-Order Autoregressive (process) |
| CI | Confidence Interval |
| CI/CD | Continuous Integration/Continuous Delivery |
| CK | Chidamber and Kemerer (object-oriented metrics suite) |
| CRM | Customer Relationship Management |
| CV | Cross-Validation |
| CVE | Common Vulnerabilities and Exposures |
| CVSS | Common Vulnerability Scoring System |
| CWE | Common Weakness Enumeration |
| DORA | DevOps Research and Assessment |
| GB | Gradient Boosting |
| JSON | JavaScript Object Notation |
| KS | Kolmogorov–Smirnov (test) |
| MAE | Mean Absolute Error |
| MAPE | Mean Absolute Percentage Error |
| ML | Machine Learning |
| MLOps | Machine Learning Operations |
| MTTR | Mean Time to Recovery |
| NN | Neural Network |
| PA-RFE | Phase-Aware Recursive Feature Elimination |
| PCI-DSS | Payment Card Industry Data Security Standard |
| PCM | Palladio Component Model |
| PRESTO | Performance REgression from Software development Telemetry and Operations |
| PSI | Population Stability Index |
| RF | Random Forest |
| RFE | Recursive Feature Elimination |
| RMSE | Root-Mean-Square Error |
| SaaS | Software as a Service |
| SDLC | Software Development Lifecycle |
| SLO | Service Level Objective |
| SPE | Software Performance Engineering |
| SQuaD | Software Quality Dataset |
| UAT | User Acceptance Testing |
| UML | Unified Modeling Language |
| YAML | YAML Ain’t Markup Language |
Appendix A. Generator Calibration and Secondary Robustness Diagnostics
This appendix holds material kept out of the main body so that the paper stays focused on its research questions, methodology, main analyses, and conclusions. Two further items are reported in the accompanying Supplementary Materials file rather than inline here: the per-family interpretation of Section 4.3’s paired-difference results (Supplementary S1), and the full secondary robustness diagnostics behind Section 4.1.4 (Supplementary S2).
Appendix A.1. Generator Calibration Verification and Defect Analysis
Regeneration Provenance: Every synthetic-domain number in this paper reflects data regenerated with three corrections in place. First, the rolling-window statistics of add_rolling_features() are shifted to strictly-prior releases rather than computed inclusive of the current row (Section 3.1.3). Second, the pipeline runs under a pinned numpy/scipy/scikit-learn environment (ENVIRONMENT.txt); the environment behind the earliest, unpinned runs was not recorded—a reproducibility gap that we disclose. Third, an audit of the generator code identifies the three implementation defects detailed below, unrelated to the rolling-window shift, and all synthetic domains are regenerated with them corrected. With-AR figures additionally use the corrected lag-feature construction of Section 3.1.3.
Defect 1—Dropped Correlation Pairs: intra_phase_correlations in the correlation configuration is stored as a dictionary keyed by SDLC phase, but the code that collects correlation pairs iterated it the same way as the two flat-list sections, silently discarding all 48 intra-phase pairs. Only 36 of the 84 documented pairs (the target and cross-phase pairs) ever reached the copula. We corrected the iteration logic so that all 84 pairs were applied (verified: n_correlation_pairs in each domain’s generation metadata rose from 38 to 93 phase-qualified pairs).
Defect 2—Domain-Specific Overrides Never Applied: domain_profiles.yaml documents a per-domain override for System Uptime’s distribution (for example, ABC Cloud Provider: Beta (a = 15, b = 1.2, loc = 92, scale = 8), bounds [92, 100]), but no code path ever read it; every domain silently drew System Uptime from the same shared base distribution. We wired the override through the marginal-construction and bounds-enforcement steps. One subtlety affects the correction: the override key is a plain metric name with no phase qualifier, and “Mean Time to Recovery (MTTR) (Minutes)” exists in both the production and chaos phases as separately calibrated metrics, so unrestricted matching would let a production-intended override overwrite the unrelated chaos-phase metric too; a bounds-validation check detects this, and override matching is restricted to the production phase specifically.
Defect 3—A Documented Distribution Family Was Never Actually Used: Two metrics (code smells, static analysis issues) are documented in the registry as negative-binomial counts, but the copula engine had no negative-binomial sampler and silently substituted a moment-matched, continuously valued Gamma distribution, which was also never rounded to an integer despite being a count metric (92.4% of generated code smells values were non-integer). We added a negative-binomial sampler and added both metrics to the integer-rounding step at export time.
Two related issues were identified but deliberately not changed: dora_benchmarks.yaml’s tier-specific benchmark values are documented but never loaded by any code path; wiring them in would require deciding how a DORA tier’s benchmark values should map onto specific distribution parameters—a design decision rather than a plumbing fix—so we left this as a disclosed limitation. Separately, System Uptime’s realized lag-1 autocorrelation remains far below its specified target even after the domain-override fix, for the structural reason given in Section 3.1.4: a ceiling-concentrated marginal paired with a rank-based AR(1) transform applied before bounds clipping.
Calibration Diagnostics with and without the Defect 1 Correction: Correcting the dropped correlation pairs changes calibration fidelity in both directions at once: the full 84-pair correlation-matrix distance improved (Frobenius norm 11.81 to 11.17, tolerance pass rate 25.0% to 53.6%, since far more of the documented structure is now attempted), but the original 36 target and cross-phase pairs individually fit slightly worse than before (Frobenius norm on that subset: 6.15 to 7.01), because they now share one global positive semi-definite correction step with 48 additional constraints competing for the same projection budget. The marginal-distribution goodness-of-fit gap between the pre-temporal-transform draw (92% Kolmogorov–Smirnov pass rate) and the final, temporally transformed data on which the models actually train (37.2% with the corrections, materially unchanged from 36.0% without them) is unaffected by these corrections, since it originates in the temporal-dynamics stage rather than the correlation or distribution-family code.
Appendix A.2. Secondary Robustness Diagnostics
This subsection provides the full detail behind Section 4.1.4’s headline conclusions, all on the ABC Cloud dataset, without AR features, using Gradient Boosting (these diagnostics target Gradient Boosting’s failure mode; the without-AR headline result rests on Random Forest under fixed hyperparameters, or Ridge under nested selection, per Section 4.1.2). A naive-baseline comparison, a reimplementation of Siegmund et al.’s [55] performance-influence method, multi-seed stability, residual diagnostics, and a permutation test are reported in full in Supplementary Materials S2; the summary is that none of these five diagnostics changes the conclusions of Section 4.1.4, and two of them (the naive-baseline comparison and multi-seed stability) show a qualitative reversal relative to the same diagnostics on the defective generator’s data, reported there in full rather than only asserted here.
Appendix B. Glossary
This glossary defines methodological terms and the four real-world datasets as used throughout this paper, distinct from the acronym-expansion list in the Abbreviations section.
Methodology and Evaluation Terms:
- Autoregressive (AR) Features: Rolling-window and lag statistics computed from the target variable’s own prior values. Legitimate predictors under the shifted construction of Section 3.1.3 (rolling-window and diff_k/pct_change_k alike), which ensures that only strictly-prior observations are used; they encode history-based persistence rather than SDLC process signal. This paper reports results with and without them throughout.
- Holdout : Coefficient of determination computed once on a test partition held out from all training and model selection, never reused. Under the nested-CV protocol (Section 3.2.5), the outer holdout is touched exactly once by an already-selected configuration.
- Nested Cross-Validation: A two-level validation design where model/hyperparameter selection happens via an inner cross-validation loop entirely within a training partition, and a held-out outer partition is scored exactly once by the already-selected configuration (Section 3.2.5), used throughout this paper in place of direct holdout-score selection.
- Cross-Validation (CV) : Mean across the folds of a TimeSeriesSplit, used for tuning/selection (inner loop, under the nested protocol) and reported for transparency.
- Same-Row (target) Leakage: A leakage class where a feature computed for a given row inadvertently includes information from that same row’s target value. This paper documents and corrects two such modes, in the rolling window and in the diff_k/pct_change_k construction, both in Section 3.1.3.
- Gaussian Copula: The statistical technique used to generate PRESTO’s synthetic SDLC metrics (Section 3.1.4).
- Target-Correlation Ablation: An experiment that zeroes or permutes the copula’s hand-specified target correlations to test whether predictive signal reflects genuine structure or recovery of specified structure (Section 4.1.5).
- Permutation Importance: A model-agnostic feature-importance method measuring the drop in a fitted model’s performance when a feature’s values are randomly shuffled.
- Bootstrap Confidence Interval: A resampling-based uncertainty interval. This paper reports both a pairs bootstrap and a moving-block bootstrap for temporally ordered holdouts (Section 4.1.5).
- Moving-Block Bootstrap: A bootstrap that resamples contiguous blocks of consecutive observations rather than individual rows, so that local dependence within a temporally ordered holdout is preserved instead of assumed away. Applied to the headline condition of each of this paper’s five temporally ordered dataset families (the three synthetic domains, TravisTorrent, and Mozilla Perfherder) (Section 4.1.5, Section 4.2.1 and Section 4.2.2); not applicable to the two cross-sectional datasets, which have no within-entity ordering to block over.
- Paired-Difference Bootstrap: A bootstrap interval computed on the difference in holdout between two conditions, using the same resample indices for both, so the interval tests whether the gap between the conditions differs from zero. This is distinct from, and not substitutable by, comparing the two conditions’ separate marginal intervals (Section 4.3).
- Cross-Sectional Validation: An evaluation design with one row per entity and no within-entity time series (GHALogs, SQuaD). TimeSeriesSplit applied to such a design resamples a fixed row ordering rather than validating temporally; see Table 6 and Section 3.2.4.
- Within-Project/Within-Signature Temporal Validation: A chronologically ordered split within a single project or signature’s own history (TravisTorrent, Mozilla Perfherder).
- Selection (winner’s-curse) Bias: The optimism from choosing the best of several candidates using the same metric later reported as the headline result. The nested-CV protocol addresses this directly, although Section 5.1 discusses a remaining failure mode.
- DORA Metrics: Key software-delivery performance metrics used to calibrate the synthetic data generator.
- CVE/CWE/CVSS. Common Vulnerabilities and Exposures, Common Weakness Enumeration, and the Common Vulnerability Scoring System, used in the SQuaD CVE-count analysis (see the “From Defects to Vulnerabilities: A Harder Security Target” section).
Real-World Validation Datasets:
- TravisTorrent: Travis CI build logs joined with GitHub project metadata, 2.64 million builds across 1359 projects; within-project build-duration prediction (Section 4.2.1).
- Mozilla Perfherder: Mozilla’s performance-regression tracking system, per-push runtime measurements cross-referenced with human-confirmed regression alerts; within-signature temporal validation; this paper’s most robust real-world result.
- GHALogs: GitHub Actions workflow-run logs with per-step timing across 116k workflows; cross-sectional CI-duration prediction, whose full-feature result is carried largely by the near-tautological mean_n_steps feature (Section 4.2.3).
- SQuaD (Software Quality Dataset): Process and static-analysis metrics from 440 mature open-source projects; cross-sectional defect-fix-rate and CVE-count prediction.
References
- Lazowska, E.D.; Zahorjan, J.; Graham, G.S.; Sevcik, K.C. Quantitative System Performance: Computer System Analysis Using Queueing Network Models; Prentice-Hall: Englewood Cliffs, NJ, USA, 1984. [Google Scholar]
- Reiser, M.; Lavenberg, S.S. Mean-value analysis of closed multichain queuing networks. J. ACM 1980, 27, 313–322, Correction in J. ACM 1981, 28, 629. [Google Scholar] [CrossRef] [Scilit]
- Ferrari, D. Computer Systems Performance Evaluation; Prentice-Hall: Englewood Cliffs, NJ, USA, 1978. [Google Scholar]
- Sauer, C.H.; Chandy, K.M. Computer Systems Performance Modeling; Prentice-Hall: Englewood Cliffs, NJ, USA, 1981. [Google Scholar]
- Jain, R. The Art of Computer Systems Performance Analysis: Techniques for Experimental Design, Measurement, Simulation, and Modeling; Wiley: Hoboken, NJ, USA, 1991. [Google Scholar]
- Smith, C.U. Performance Engineering of Software Systems; Addison-Wesley: Reading, MA, USA, 1990. [Google Scholar]
- Smith, C.U.; Williams, L.G. Performance Solutions: A Practical Guide to Creating Responsive, Scalable Software; Addison-Wesley: Reading, MA, USA, 2002. [Google Scholar]
- Balsamo, S.; Di Marco, A.; Inverardi, P.; Simeoni, M. Model-based performance prediction in software development: A survey. IEEE Trans. Softw. Eng. 2004, 30, 295–310. [Google Scholar] [CrossRef] [Scilit]
- Cortellessa, V.; Di Marco, A.; Inverardi, P. Model-Based Software Performance Analysis; Springer: Berlin/Heidelberg, Germany, 2011. [Google Scholar] [CrossRef] [Scilit]
- Jiang, Z.M.; Hassan, A.E. A Survey on Load Testing of Large-Scale Software Systems. IEEE Trans. Softw. Eng. 2015, 41, 1091–1118. [Google Scholar] [CrossRef] [Scilit]
- Nguyen, T.H.D.; Adams, B.; Jiang, Z.M.; Hassan, A.E.; Nasser, M.; Flora, P. Automated Detection of Performance Regressions Using Statistical Process Control Techniques. In Proceedings of the 3rd ACM/SPEC International Conference on Performance Engineering (ICPE); Association for Computing Machinery: New York, NY, USA, 2012; pp. 299–310. [Google Scholar] [CrossRef] [Scilit]
- Foo, K.C.; Jiang, Z.M.; Adams, B.; Hassan, A.E.; Zou, Y.; Flora, P. Mining Performance Regression Testing Repositories for Automated Performance Analysis. In Proceedings of the 10th International Conference on Quality Software (QSIC); IEEE: Piscataway, NJ, USA, 2010; pp. 32–41. [Google Scholar] [CrossRef] [Scilit]
- HP Labs. Predicting Performance in Distributed Enterprise Applications; Technical Report HPL-2006-76; HP Labs: Palo Alto, CA, USA, 2006. [Google Scholar]
- Weyuker, E.J.; Vokolos, F.I. Experience with Performance Testing of Software Systems: Issues, an Approach, and Case Study. IEEE Trans. Softw. Eng. 2000, 26, 1147–1156. [Google Scholar] [CrossRef] [Scilit]
- Liu, Y.; Fekete, A.; Gorton, I. Design-level performance prediction of component-based applications. IEEE Trans. Softw. Eng. 2005, 31, 928–941. [Google Scholar] [CrossRef] [Scilit]
- Pooley, R.J.; King, P. The Unified Modelling Language and performance engineering. IEE Proc.-Softw. 1999, 146, 2–10. [Google Scholar] [CrossRef] [Scilit]
- Menascé, D.A.; Dowdy, L.W.; Almeida, V.A.F. Performance by Design: Computer Capacity Planning by Example; Prentice Hall PTR: Upper Saddle River, NJ, USA, 2004. [Google Scholar]
- Kant, K. Introduction to Computer System Performance Evaluation; McGraw-Hill: Columbus, OH, USA, 1992. [Google Scholar]
- Woodside, C.M.; Franks, G.; Petriu, D.C. The future of software performance engineering. In Proceedings of the Future of Software Engineering (FOSE); IEEE: Piscataway, NJ, USA, 2007; pp. 171–187. [Google Scholar] [CrossRef] [Scilit]
- Koziolek, H. Performance evaluation of component-based software systems: A survey. Perform. Eval. 2010, 67, 634–658. [Google Scholar] [CrossRef] [Scilit]
- Becker, S.; Koziolek, H.; Reussner, R. The Palladio Component Model for Model-Driven Performance Prediction. J. Syst. Softw. 2009, 82, 3–22. [Google Scholar] [CrossRef] [Scilit]
- Jin, D.; Qu, X.; Cohen, M.B.; Robinson, B. Configurations everywhere: Implications for testing and debugging in practice. In Proceedings of the International Conference on Software Engineering (ICSE); Association for Computing Machinery: New York, NY, USA, 2014. [Google Scholar] [CrossRef] [Scilit]
- Chen, L.; Babar, M.A.; Cawley, C. A status report on the evaluation of variability management approaches. In Proceedings of the International Conference on Evaluation and Assessment in Software Engineering, Durham, UK, 20–21 April 2009. [Google Scholar] [CrossRef] [Scilit]
- Chidamber, S.R.; Kemerer, C.F. A metrics suite for object oriented design. IEEE Trans. Softw. Eng. 1994, 20, 476–493. [Google Scholar] [CrossRef] [Scilit]
- McCabe, T.J. A complexity measure. IEEE Trans. Softw. Eng. 1976, SE-2, 308–320. [Google Scholar] [CrossRef] [Scilit]
- Boehm, B.W.; Abts, C.; Brown, A.W.; Chulani, S.; Clark, B.K.; Horowitz, E.; Madachy, R.; Reifer, D.J.; Steece, B. Software Cost Estimation with COCOMO II; Prentice Hall: Upper Saddle River, NJ, USA, 2000. [Google Scholar]
- Ostrand, T.J.; Weyuker, E.J. The distribution of faults in a large industrial software system. In Proceedings of the International Symposium on Software Testing and Analysis (ISSTA); Association for Computing Machinery: New York, NY, USA, 2002; pp. 55–64. [Google Scholar] [CrossRef] [Scilit]
- Nagappan, N.; Ball, T.; Zeller, A. Mining metrics to predict component failures. In Proceedings of the International Conference on Software Engineering (ICSE); Association for Computing Machinery: New York, NY, USA, 2006; pp. 452–461. [Google Scholar] [CrossRef] [Scilit]
- Lessmann, S.; Baesens, B.; Mues, C.; Pietsch, S. Benchmarking Classification Models for Software Defect Prediction: A Proposed Framework and Novel Findings. IEEE Trans. Softw. Eng. 2008, 34, 485–496. [Google Scholar] [CrossRef] [Scilit]
- Kamei, Y.; Shihab, E.; Adams, B.; Hassan, A.E.; Mockus, A.; Sinha, A.; Ubayashi, N. A Large-Scale Empirical Study of Just-in-Time Quality Assurance. IEEE Trans. Softw. Eng. 2013, 39, 757–773. [Google Scholar] [CrossRef] [Scilit]
- Arcoverde, R.; Garcia, A.; Figueiredo, E. Understanding the longevity of code smells: Preliminary results of an explanatory survey. In Proceedings of the International Workshop on Refactoring Tools; Association for Computing Machinery: New York, NY, USA, 2011; pp. 33–36. [Google Scholar] [CrossRef] [Scilit]
- Li, Z.; Avgeriou, P.; Liang, P. A systematic mapping study on technical debt and its management. J. Syst. Softw. 2015, 101, 193–220. [Google Scholar] [CrossRef] [Scilit]
- Stradowski, S.; Madeyski, L. Machine learning in software defect prediction: A business-driven systematic mapping study. Inf. Softw. Technol. 2023, 155, 107128. [Google Scholar] [CrossRef] [Scilit]
- Forsgren, N.; Humble, J.; Kim, G. Accelerate: The Science of Lean Software and DevOps; IT Revolution Press: Portland, OR, USA, 2018. [Google Scholar]
- Google Cloud. Accelerate State of DevOps Report 2024; Technical report; DORA: Mountain View, CA, USA, 2024. [Google Scholar]
- Hilton, M.; Tunnell, T.; Huang, K.; Marinov, D.; Dig, D. Usage, costs, and benefits of continuous integration in open-source projects. In Proceedings of the International Conference on Automated Software Engineering (ASE); Association for Computing Machinery: New York, NY, USA, 2016; pp. 426–437. [Google Scholar] [CrossRef] [Scilit]
- Vasilescu, B.; Yu, Y.; Wang, H.; Devanbu, P.; Filkov, V. Quality and productivity outcomes relating to continuous integration in GitHub. In Proceedings of the Joint Meeting on Foundations of Software Engineering (ESEC/FSE); Association for Computing Machinery: New York, NY, USA, 2015; pp. 805–816. [Google Scholar] [CrossRef] [Scilit]
- Gallaba, K.; McIntosh, S. Use and misuse of continuous integration features: An empirical study of projects that (mis)use Travis CI. IEEE Trans. Softw. Eng. 2020, 46, 33–50. [Google Scholar] [CrossRef] [Scilit]
- Ghaleb, T.A.; da Costa, D.A.; Zou, Y. An Empirical Study of the Long Duration of Continuous Integration Builds. Empir. Softw. Eng. 2019, 24, 2102–2139. [Google Scholar] [CrossRef] [Scilit]
- Beller, M.; Gousios, G.; Zaidman, A. Oops, My Tests Broke the Build: An Explorative Analysis of Travis CI with GitHub. In Proceedings of the 14th International Conference on Mining Software Repositories (MSR); IEEE: Piscataway, NJ, USA, 2017; pp. 356–367. [Google Scholar] [CrossRef] [Scilit]
- Saidani, I.; Ouni, A.; Mkaouer, M.W. Improving the Prediction of Continuous Integration Build Failures Using Deep Learning. Autom. Softw. Eng. 2022, 29, 21. [Google Scholar] [CrossRef] [Scilit]
- Riungu-Kalliosaari, L.; Mäkinen, S.; Lwakatare, L.E.; Tiihonen, J.; Männistö, T. DevOps adoption benefits and challenges in practice: A case study. In Proceedings of the International Conference on Product-Focused Software Process Improvement (PROFES); Springer: Cham, Switzerland, 2016. [Google Scholar] [CrossRef] [Scilit]
- Amaro, R.M.D.; Pereira, R.; Mira da Silva, M. DevOps Metrics and KPIs: A Multivocal Literature Review. ACM Comput. Surv. 2024, 56, 1–41. [Google Scholar] [CrossRef] [Scilit]
- Brunnert, A.; van Hoorn, A.; Willnecker, F.; Danciu, A.; Hasselbring, W.; Heger, C.; Herbst, N.; Kounev, S.; Koziolek, H.; Kroß, J.; et al. Performance-Oriented DevOps: A Research Agenda; Technical Report SPEC-RG-2015-01; SPEC Research Group–DevOps Performance Working Group: Gainesville, VA, USA, 2015. [Google Scholar]
- Fallahzadeh, E.; Bavand, A.H.; Rigby, P.C. Accelerating Continuous Integration with Parallel Batch Testing. In Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE); Association for Computing Machinery: New York, NY, USA, 2023. [Google Scholar] [CrossRef] [Scilit]
- Dobrzyński, B.; Sosnowski, J. Graph-Driven Exploration of Issue Handling Schemes in Software Projects. Appl. Sci. 2024, 14, 4723. [Google Scholar] [CrossRef] [Scilit]
- Menzies, T.; Greenwald, J.; Frank, A. Data mining static code attributes to learn defect predictors. IEEE Trans. Softw. Eng. 2007, 33, 2–13. [Google Scholar] [CrossRef] [Scilit]
- Hall, T.; Beecham, S.; Bowes, D.; Gray, D.; Counsell, S. A systematic literature review on fault prediction performance in software engineering. IEEE Trans. Softw. Eng. 2012, 38, 1276–1304. [Google Scholar] [CrossRef] [Scilit]
- Zhang, J.; Wang, X.; Zhang, H.; Sun, H.; Wang, K.; Liu, X. A Novel Neural Source Code Representation Based on Abstract Syntax Tree. In Proceedings of the 41st International Conference on Software Engineering (ICSE); IEEE: Piscataway, NJ, USA, 2019; pp. 783–794. [Google Scholar] [CrossRef] [Scilit]
- Allamanis, M.; Barr, E.T.; Devanbu, P.; Sutton, C. A survey of machine learning for big code and naturalness. ACM Comput. Surv. 2018, 51, 1–37. [Google Scholar] [CrossRef] [Scilit]
- Chen, B.; Chen, L.; Zhang, C.; Peng, X. BuildFast: History-Aware Build Outcome Prediction for Fast Feedback and Reduced Cost in Continuous Integration. In Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering (ASE); Association for Computing Machinery: New York, NY, USA, 2020; pp. 42–53. [Google Scholar] [CrossRef] [Scilit]
- Mishra, L.N.; Rangari, A.; Nagrare, S.; Nayak, S.K. A Taxonomy for Detecting and Preventing Temporal Data Leakage in Machine Learning-Based Build Prediction: A Dual-Platform Empirical Validation. PLoS ONE 2026, 21, e0340167. [Google Scholar] [CrossRef] [Scilit] [PubMed]
- Wu, Y.; Chai, B.; Li, Y.; Liu, B.; Li, J.; Yang, Y.; Jiang, W. An Empirical Study on Change-Induced Incidents of Online Service Systems. In Proceedings of the IEEE/ACM 45th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP); IEEE: Piscataway, NJ, USA, 2023; pp. 234–245. [Google Scholar] [CrossRef] [Scilit]
- Nistor, A.; Song, L.; Marinov, D.; Lu, S. Toddler: Detecting performance problems via similar memory-access patterns. In Proceedings of the International Conference on Software Engineering (ICSE); IEEE: Piscataway, NJ, USA, 2013; pp. 562–571. [Google Scholar] [CrossRef] [Scilit]
- Siegmund, N.; Grebhahn, A.; Apel, S.; Kästner, C. Performance-Influence Models for Highly Configurable Systems. In Proceedings of the Joint Meeting on Foundations of Software Engineering (ESEC/FSE); Association for Computing Machinery: New York, NY, USA, 2015; pp. 284–294. [Google Scholar] [CrossRef] [Scilit]
- Ha, H.; Zhang, H. DeepPerf: Performance Prediction for Configurable Software with Deep Sparse Neural Network. In Proceedings of the 41st International Conference on Software Engineering (ICSE); IEEE: Piscataway, NJ, USA, 2019; pp. 1095–1106. [Google Scholar] [CrossRef] [Scilit]
- Nair, V.; Yu, Z.; Menzies, T.; Siegmund, N.; Apel, S. Finding Faster Configurations Using FLASH. IEEE Trans. Softw. Eng. 2020, 46, 794–811. [Google Scholar] [CrossRef] [Scilit]
- Guo, J.; Yang, D.; Siegmund, N.; Apel, S.; Sarkar, A.; Valov, P.; Czarnecki, K.; Wasowski, A.; Yu, H. Data-efficient Performance Learning for Configurable Systems. Empir. Softw. Eng. 2018, 23, 1826–1867. [Google Scholar] [CrossRef] [Scilit]
- Jamshidi, P.; Velez, M.; Kästner, C.; Siegmund, N.; Kawthekar, P. Transfer Learning for Improving Model Predictions in Highly Configurable Software. In Proceedings of the 12th International Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS); IEEE: Piscataway, NJ, USA, 2017. [Google Scholar] [CrossRef] [Scilit]
- Jamshidi, P.; Velez, M.; Kästner, C.; Siegmund, N. Learning to sample: Exploiting similarities across environments to learn performance models for configurable systems. In Proceedings of the Joint Meeting on Foundations of Software Engineering (ESEC/FSE); Association for Computing Machinery: New York, NY, USA, 2018; pp. 71–82. [Google Scholar] [CrossRef] [Scilit]
- Maros, A.; Murai, F.; da Silva, A.P.C.; Almeida, J.M.; Lattuada, M.; Gianniti, E.; Hosseini, M.; Ardagna, D. Machine Learning for Performance Prediction of Spark Cloud Applications. In Proceedings of the IEEE 12th International Conference on Cloud Computing (CLOUD); IEEE: Piscataway, NJ, USA, 2019; pp. 99–106. [Google Scholar] [CrossRef] [Scilit]
- Tantithamthavorn, C.; McIntosh, S.; Hassan, A.E.; Matsumoto, K. An Empirical Comparison of Model Validation Techniques for Defect Prediction Models. IEEE Trans. Softw. Eng. 2017, 43, 1–18. [Google Scholar] [CrossRef] [Scilit]
- Liao, L.; Eismann, S.; Li, H.; Bezemer, C.P.; Costa, D.E.; van Hoorn, A.; Shang, W. Early Detection of Performance Regressions by Bridging Local Performance Data and Architectural Models. In Proceedings of the 47th IEEE/ACM International Conference on Software Engineering (ICSE); IEEE: Piscataway, NJ, USA, 2025. [Google Scholar] [CrossRef] [Scilit]
- Biringa, C.; Kul, G. PACE: A Program Analysis Framework for Continuous Performance Prediction. ACM Trans. Softw. Eng. Methodol. 2024, 33, 1–23. [Google Scholar] [CrossRef] [Scilit]
- Mboweni, T.; Masombuka, T.; Dongmo, C. A Systematic Review of Machine Learning DevOps. In Proceedings of the International Conference on Electrical, Computer and Energy Technologies (ICECET); IEEE: Piscataway, NJ, USA, 2022. [Google Scholar] [CrossRef] [Scilit]
- Mohapatra, H.; Pramanik, S.; Mishra, S.R. Revolutionizing Software Development: The Transformative Influence of Machine Learning Integrated SDLC Model. In Boosting Software Development Using Machine Learning; Artificial Intelligence-Enhanced Software and Systems Engineering; Springer: Cham, Switzerland, 2025; Volume 7. [Google Scholar] [CrossRef] [Scilit]
- Behera, A.K.; Chaudhury, P.; Dash, C.S.K. A Comprehensive Survey on Intelligent Software Reliability Prediction. Discov. Comput. 2025, 28, 90. [Google Scholar] [CrossRef] [Scilit]
- Zhou, X.; Peng, X.; Xie, T.; Sun, J.; Ji, C.; Liu, D.; Xiang, Q.; He, C. Latent Error Prediction and Fault Localization for Microservice Applications by Learning from System Trace Logs. In Proceedings of the 27th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE); Association for Computing Machinery: New York, NY, USA, 2019; pp. 683–694. [Google Scholar] [CrossRef] [Scilit]
- Dang, Y.; Lin, Q.; Huang, P. AIOps: Real-World Challenges and Research Innovations. In Proceedings of the 41st IEEE/ACM International Conference on Software Engineering: Companion (ICSE-Companion); IEEE: Piscataway, NJ, USA, 2019; pp. 4–5. [Google Scholar] [CrossRef] [Scilit]
- Notaro, P.; Cardoso, J.; Gerndt, M. A Survey of AIOps Methods for Failure Management. ACM Trans. Intell. Syst. Technol. 2021, 12, 1–45. [Google Scholar] [CrossRef] [Scilit]
- Diaz-De-Arcaya, J.; Torre-Bastida, A.I.; Zárate, G.; Miñón, R.; Almeida, A. A Joint Study of the Challenges, Opportunities, and Roadmap of MLOps and AIOps: A Systematic Survey. ACM Comput. Surv. 2024, 56, 1–30. [Google Scholar] [CrossRef] [Scilit]
- Beller, M.; Gousios, G.; Zaidman, A. TravisTorrent: Synthesizing Travis CI and GitHub for Full-Stack Research on Continuous Integration. In Proceedings of the 14th International Conference on Mining Software Repositories (MSR); IEEE: Piscataway, NJ, USA, 2017; pp. 447–450. [Google Scholar] [CrossRef] [Scilit]
- Nelsen, R.B. An Introduction to Copulas, 2nd ed.; Springer: New York, NY, USA, 2006. [Google Scholar] [CrossRef] [Scilit]
- Higham, N.J. Computing the Nearest Correlation Matrix—A Problem from Finance. IMA J. Numer. Anal. 2002, 22, 329–343. [Google Scholar] [CrossRef] [Scilit]
- Guyon, I.; Weston, J.; Barnhill, S.; Vapnik, V. Gene Selection for Cancer Classification using Support Vector Machines. Mach. Learn. 2002, 46, 389–422. [Google Scholar] [CrossRef] [Scilit]
- Breiman, L. Random Forests. Mach. Learn. 2001, 45, 5–32. [Google Scholar] [CrossRef] [Scilit]
- Friedman, J.H. Greedy Function Approximation: A Gradient Boosting Machine. Ann. Stat. 2001, 29, 1189–1232. [Google Scholar] [CrossRef] [Scilit]
- Gama, J.a.; Žliobaitė, I.; Bifet, A.; Pechenizkiy, M.; Bouchachia, A. A Survey on Concept Drift Adaptation. ACM Comput. Surv. 2014, 46, 1–37. [Google Scholar] [CrossRef] [Scilit] [PubMed]
- Besbes, M.B.; Damasceno Costa, D.E.; Mujahid, S.; Castelluccio, M.; Mierzwinski, G. Mozilla Performance Time Series Dataset. 2025. Zenodo Dataset. Available online: https://doi.org/10.5281/zenodo.14642238 (accessed on 2 October 2026).
- Moriconi, F.; Durieux, T.; Falleri, J.R.; Troncy, R.; Francillon, A. GHALogs: Large-Scale Dataset of GitHub Actions Runs. 2024. Zenodo Dataset. Available online: https://doi.org/10.5281/zenodo.10154920 (accessed on 2 October 2026).
- Robredo, M.; Esposito, M.; Taibi, D.; Peñaloza, R.; Lenarduzzi, V. SQuaD: The Software Quality Dataset. 2025. Zenodo Dataset. Available online: https://doi.org/10.5281/zenodo.17566690 (accessed on 2 October 2026).
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2026 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license.








