Instance-Adaptive Lazy Inference for Tree Ensembles via Stability Surrogates and Residual-Capacity Certificates

Abstract

Treeensembles usually evaluate every tree or boosting stage for every input, even when the partial prediction has already stabilized. We study inference-time lazy evaluation for pre-trained random forests and gradient-boosted decision trees. LazyRF stops random-forest evaluation using a Bayesian vote-stability score for the current leading class. LazyGBM uses a deterministic residual-capacity certificate: if the current class separation exceeds twice the maximum possible contribution of later stages, the final predicted class is guaranteed unchanged. At the default LazyRF threshold, LazyRF reduces evaluated trees by 60–90% across four benchmarks and 30 seeds while passing a paired-seed equivalence check with a 0.002 accuracy margin against full random-forest evaluation. For LazyGBM, the certified rule is conservative: certificate-only savings are 1–12% in the reported ablations, while more aggressive heuristic exits are reported separately and treated as empirically audited prototypes. We, therefore, use evaluated trees or stages as the main cost measure. Runtime measurements provide Python-prototype context rather than hardware-independent latency claims.

1. Introduction

Tree ensembles, especially Random Forests (RF) and Gradient Boosted Decision Trees (GBDT), remain core building blocks in computational machine learning because they provide strong accuracy and robust behavior on structured data, and are widely supported by mature training toolchains [1,2,3,4]. Standard inference evaluates every tree or boosting stage for every input, giving a fixed per-example cost even when the partial ensemble prediction has already stabilized. This evaluate-all protocol motivates lazy evaluation: for easy inputs, a method may be able to stop after a prefix of the ensemble while retaining the full-model decision. The practical relevance of this reduction depends on the cost profile of the surrounding pipeline. Lazy evaluation is useful only when the ensemble-evaluation loop is a measured cost after data access, preprocessing, feature construction, distributed scheduling, and I/O have been accounted for. This condition appears in latency-sensitive services (e.g., fraud screening, ranking, and monitoring pipelines), in high-volume batch scoring, and in always-on services where repeated full-ensemble evaluation contributes to compute or energy use [5]. When pipeline overhead dominates, fewer evaluated trees need not reduce end-to-end latency. This paper, therefore, studies component-count reduction as a bounded algorithmic objective for fixed tree ensembles, not as a universal deployment-acceleration claim.

1.1. Ensemble Inference as a Sequential Process

Although an ensemble is typically presented as a static mapping $f:{\mathbb{R}}^d\to {\mathbb{R}}^K$, the inference procedure is naturally sequential and evolves as the individual base learners are evaluated and aggregated. Let $S_t\left(x\right)$ denote a partial state after evaluating t components on input x. For RFs, $S_t\left(x\right)$ can be the class-vote count vector, and for GBDTs, it can be the partial class-score (logit) vector. The state evolves according to:

$$S_{t+1}\left(x\right)=S_t\left(x\right)+u_{t+1}\left(x\right),t=0,1,\dots ,T-1,$$

(1)

where $u_{t+1}\left(x\right)$ is the contribution of the $(t+1)$-th component (a one-hot vote for RFs, an additive score increment for GBDTs), and T denotes the total number of components (trees or boosting stages). Conventional inference fixes the horizon to $t=T$ for every input, regardless of how quickly the partial state stabilizes.

1.2. Gap: Reducing Expected Work While Preserving Full-Model Behavior

Previous studies have accelerated ensemble inference in two ways. First, systems-level methods optimize tree traversal and memory layout to reduce constant factors (e.g., QuickScorer-style bitvector interleaving), but typically preserve evaluate-all-component semantics [6]. Second, conditional computation approaches, notably cascades and early-exit models, stop computation early based on confidence heuristics at pre-defined checkpoints [7,8]. Although effective in specific settings, these strategies often require additional training (for staged models or auxiliary exits) or impose coarse-grained checkpoints. More importantly, they do not directly provide a unified, model-preserving mechanism to quantify and tune fidelity relative to the deterministic prediction of a fixed, fully evaluated ensemble.

1.3. Our Approach: Fidelity-Targeted Stopping Rules with Explicit Stability Targets

This study targets inference-time lazy evaluation for pre-trained ensembles: evaluate components sequentially and stop early on an instance x once the current prediction is sufficiently stable with respect to the unevaluated remainder. The mathematical object of interest is a stopping rule whose cost is the number of evaluated ensemble components and whose fidelity target is agreement with a fixed full model. We use the fidelity-constrained stopping-rule objective formalized in Section 3.1 as a design lens: reduce the expected evaluated components while controlling disagreement with the full model under an explicit uncertainty model for the unseen suffix. We do not claim to compute the exact constrained optimum at serving time. Instead, the objective defines the fidelity target, and the implemented algorithms provide two types of practical evidence: a deterministic argmax-preservation certificate for boosting and empirically audited stability surrogates for RF and heuristic GBM exits.

Specifically, we develop two inference-time wrappers: (i) LazyRF for random forests, which uses a Dirichlet vote model together with a posterior-dominance score for the current leader; and (ii) LazyGBM for gradient boosting, whose mathematically certified component is a residual-capacity argmax certificate, with optional margin-based prototype heuristics reported separately and controlled at inference time.

1.4. Research Questions and Contributions

Guided by the above framework, we investigate three questions:

1.

Can we derive instance-adaptive stopping rules for tree ensembles that substantially reduce the expected computation while providing explicit, quantitative stability surrogates tied to agreement with a full evaluation?

2.

Can such stopping rules be applied as inference-time wrappers around the same fitted RF/GBDT models without modifying the learning procedure?

3.

Empirically, how do the resulting methods compare to static cascades and early-exit baselines in the accuracy–efficiency trade-off across diverse datasets and random seeds?

In this paper, fidelity means matching the deterministic full-ensemble prediction, not necessarily the ground-truth label. A model-preserving wrapper is a test-time procedure that uses the same fitted ensemble rather than retraining or replacing it. This paper makes four contributions.

1.

Full-model fidelity objective for lazy ensemble inference. We formulate partial ensemble evaluation through a constrained stopping objective: minimize expected work while bounding disagreement with the deterministic full-model prediction (Equation (2)). This formulation differs from standard early-exit and cascade objectives, which target ground-truth accuracy, confidence thresholds, or cost–accuracy Pareto curves. Our target is instead the reproducible output of a fixed, fully evaluated ensemble, giving the paper a narrow fidelity-to-full-model framing rather than a generic confidence-thresholding objective.

2.

Random-forest vote-stability stopping. We introduce a Dirichlet posterior-dominance score ${\tilde{\rho }}_t\left(x\right)$ (Equation (3)) that estimates, under an exchangeable Bayesian vote model, whether the current RF leader is stable. This score exploits the discrete vote structure of bagged forests, operates at tree-level granularity rather than fixed checkpoints, and provides an assumption-aware, continuously tunable stopping statistic. Prior cascade and early-exit methods for tree ensembles use confidence thresholds on raw vote fractions or fixed-checkpoint rules; the posterior-dominance score instead provides a conservative risk-screening statistic whose empirical accuracy–efficiency frontier is strong, without a claim of superior ordering over vote-share heuristics.

3.

Certified GBM stopping with separated heuristics. We derive a deterministic early-stopping condition for additive boosting that preserves the full-model prediction (Proposition 1): if the current top-1 vs. top-2 margin exceeds twice the remaining ${\mathcal{\ell }}_{\infty }$ capacity, the final argmax is guaranteed unchanged. This certificate is the main mathematical guarantee for LazyGBM and follows from additive score accumulation and explicit per-stage output bounds. We also give an expected-work bound (Proposition 2) showing that certificate savings depend on the distribution of the partial margin relative to the remaining stage capacity. We then evaluate optional margin-based heuristics as a separate prototype layer, providing practitioners with a transparent spectrum from certified full-model agreement to more aggressive empirically audited stopping.

4.

Reproducible evaluation of work, fidelity, and calibration. Across four benchmarks and 30 random seeds, we report accuracy and work-unit reductions and characterize accuracy–efficiency trade-offs via threshold sweeps, within-family comparisons to static cascades, contextual early-exit neural references, runtime-context measurements, surrogate-calibration audits, stopping-time/work-quantile analyses, and component-level ablations that separate guaranteed from heuristic savings.

The resulting thesis is deliberately positive but bounded: the paper contributes a provably exact, label-free full-model agreement certificate for additive boosting, a strong reproducible empirical accuracy–work frontier for random forests, and an explicit separation between deterministic guarantees, assumption-aware stopping statistics, and heuristic stopping statistics.

1.5. Summary of Empirical Findings (And How to Interpret Them)

At the default LazyRF operating point ${\alpha }_{\mathrm{stop}}=0.95$, LazyRF reduces average evaluated trees from 100 to $10.02$–$39.60$ (relative work reduction $0.60$–$0.90$) while keeping the displayed main-table accuracy change close to Full RF (rounded range $[-0.0013,0.0000]$; Section 5.2); the paired-seed analysis reports a Higgs mean difference of $-0.0014$ while still passing the $\pm 0.002$ equivalence check. Aggregated over datasets, LazyRF evaluates $21.57$ work units on average (work reduction $0.78$) with a mean accuracy change of $-0.0005$ relative to Full RF (Section 5.2). LazyGBM should be read in three layers. Certificate-only is the certified guarantee. Certificate-plus-flip applies that certificate first and then permits an audited flip-score heuristic, so it is empirically audited rather than fully certified; Section 5.8 shows zero disagreement for both variants on Covertype and Higgs, and certificate-plus-flip on Higgs matches the current prototype at $80.29$ average work units with $0.0000$ disagreement. The more aggressive default $\gamma =3.0$ prototype reduces work by $0.19$–$0.68$ and matches Full GBM accuracy on Credit Card and Higgs, but its Covertype and MNIST accuracy decreases ($-0.0049$ and $-0.0040$) arise from exploratory heuristic behavior rather than from the certificate alone and fail the paired equivalence check in Section 5.3. In aggregate, the current LazyGBM prototype reduces work by $0.32$ with a mean accuracy change of $-0.0022$ relative to Full GBM (Section 5.2).

1.6. Scope of Claims

The central claim is an algorithmic reduction in the expected number of evaluated ensemble components under a stopping rule whose worst-case work remains bounded by full evaluation within the same model family. This form of lazy truncation is orthogonal to systems-level accelerators such as QuickScorer- or Treelite-style compiled traversal: those methods reduce the cost of evaluating a tree or a full ensemble, whereas the present rules decide how many components to evaluate for a given instance. For submission, this manuscript is intended as an original, non-concurrent article: no prior LazyRF/LazyGBM manuscript is reused here, and the contribution claimed in this paper is the combination of a fixed-ensemble fidelity formulation, a residual-capacity certificate for preserving the full-model prediction in additive boosting, and an explicit separation between deterministic certificates, assumption-aware RF stopping statistics, and optional GBM heuristics.

1.7. Paper Organization

The remainder of this paper is organized as follows. Section 2 reviews the relevant literature on cascades/early exits, systems-level ensemble inference accelerators, and sequential decision-making foundations. Section 3 presents the theoretical foundations and stability surrogates underlying stopping rules. Section 4 details the materials and methods, including the cost model, algorithms, and implementation choices. Section 5 reports the experimental results and threshold-sweep trade-offs. Section 6 discusses the implications, limitations, and threats to validity. Finally, Section 7 concludes the paper.

2. Related Work

This study lies at the intersection of conditional computation (stopping early on “easy” inputs), efficient execution of tree ensembles (reducing the cost per evaluated tree), production serving systems (meeting end-to-end latency/throughput constraints), and sequential decision-making (formalizing when additional evidence is no longer worth acquiring). We position our instance-adaptive lazy inference wrappers in these directions.

2.1. Static Cascades and Early-Exit Mechanisms

Cascaded classification originated in early vision systems, in which a sequence of progressively more expensive classifiers quickly filters out easy negatives [7]. The cascade paradigm has since been generalized to modern ML pipelines via learned two-stage cascades and anytime prediction frameworks [9], checkpointed ensembles, and routing rules, which allow inputs to terminate early when predictions are already confident. In deep networks, early-exit architectures attach auxiliary heads to intermediate layers. These heads apply entropy- or confidence-based exit criteria, enabling significant savings with limited accuracy loss [8]. Related studies have also explored specialized inference engines and compression-aware execution for neural models [10].

Our work shares the goal of reducing the expected inference cost, but differs in emphasis and deployment constraints. First, we targeted tree ensembles directly and preserved the trained model. LazyRF and LazyGBM are inference-time wrappers that do not introduce additional student models, exit heads, or gating networks. Second, instead of stopping at a small set of fixed checkpoints, we adapt the computation to a fine granularity (tree-by-tree for RF, stage-by-stage for boosting). Third, we explicitly tie early stopping to a stability surrogate—a computable certificate or bound/approximation to determine whether the current prediction can change when the remaining components are evaluated. This is a different design goal than standard confidence heuristics: the surrogate is intended to track the agreement with the full-model decision under the stated modeling assumptions (RF) or via deterministic residual-capacity reasoning (boosting).

2.2. Efficient Inference Algorithms and Compilation for Tree Ensembles

A complementary line of work reduces the constant factors of tree evaluation without changing the semantics of the full inference. QuickScorer accelerates scoring for additive ensembles via bitset-based traversal and cache-efficient rule evaluation [6]. Subsequent work further improves data-level parallelism and layout-aware execution; for example, RapidScorer maximizes compactness in data-level parallelization to speed up evaluation of large ensembles in latency-sensitive settings [11]. At the compilation level, Treelite converts tree ensembles (e.g., from XGBoost or LightGBM) into optimized C code by compiling decision rules into nested conditionals [12], and newer compilers like SilvanForge further optimize these schedules for modern hardware [13].

These methods primarily target the per-tree execution efficiency and typically evaluate all trees for every input. By contrast, our lazy inference schemes aim to reduce the number of evaluated components by stopping once the prediction is sufficiently stable with respect to the unevaluated remainder. Therefore, the two approaches are largely orthogonal and potentially composable. A production implementation can combine low-level tree execution accelerators with instance-adaptive truncation.

2.3. Fast Forest/GBDT Inference Libraries and Prediction Serving Systems

Recent work has emphasized end-to-end inference performance and deployability for decision forests in production. Guan et al. compared multiple end-to-end decision forest inference pipelines (e.g., ONNX-based runtimes, library-specific engines, and GPU-oriented backends) and highlighted that performance depends not only on kernel efficiency but also on data access patterns and surrounding data-management layers [14]. On the library side, Yggdrasil Decision Forests provides a C++-centric framework designed for both research and production, including serving and interoperability across language bindings [15].

Beyond tree-specific runtimes, prediction-serving systems address deployment concerns, such as batching, caching, model selection, and tail-latency control. Clipper introduced a general-purpose architecture for low-latency, high-throughput prediction serving across heterogeneous model frameworks [16], whereas TensorFlow-Serving provides a widely deployed serving stack with optimized model lookup and versioning mechanisms [17]. Recent work also explores compiling tree ensembles into tensor computations to leverage unified neural serving stacks [18]. At the pipeline level, InferLine focuses on provisioning and autoscaling multi-stage inference pipelines to meet stringent end-to-end latency objectives under varying loads [19]. These systems reduce end-to-end latency through batching, scheduling, compilation, layout, and hardware-aware execution. Our stopping rules address a different factor: for each instance, they decide how many components of the already-trained ensemble to evaluate while preserving a worst-case work bound equal to full evaluation within the same model family. In a production implementation, the realized latency would depend on both terms—the per-component cost supplied by the backend and the number of components retained by the stopping rule—as well as the overhead of evaluating the rule itself. In response to this deployment question, Section 5.6 adds native LightGBM/XGBoost predictor measurements as optimized full-model context, while Section 5.7 adds full-size targeted checks for TL2cgen compilation of the fitted scikit-learn GBM, an optimized LazyGBM traversal, compact retrained ensembles, and shallow-tree retrained ensembles under the four-dataset, 30-seed protocol. The native rows quantify what is available when model/backend substitution is allowed; the targeted compiled/compact/shallow rows answer fidelity and retraining questions, but they are still controlled CPU timing checks rather than deployed service benchmarks.

2.4. Sequential Analysis, Optimal Stopping, and Adaptive Computation

Sequential analysis provides classical tools for deciding, based on partial evidence, whether additional samples are needed to reach a decision with a controlled error. Wald’s sequential probability ratio test (SPRT) formalizes optimal (in an expected-sample sense) testing between simple hypotheses under type-I/II constraints [20]. More recent work revisits learning stopping policies for early classification under finite horizons and connects uncertainty quantification to stopping decisions [21].

More broadly, adaptive inference has been studied under adaptive computation and budgeted prediction, in which computation (or feature acquisition) is allocated per instance under explicit cost constraints. Adaptive Computation Time (ACT) learns a halting policy that dynamically allocates iterative computation to harder inputs [22]. Budgeted prediction frameworks learn gating or model-selection rules that approximate a high-cost model using cheaper surrogates on easy instances [23]; related cost-aware tree-ensemble variants have been studied for feature-acquisition budgets [24]. Our contribution is complementary: we keep a fixed, pre-trained ensemble and adapt inference by truncating the evaluation of its existing components, using stability surrogates designed to track agreement with the full-model decision.

Our setting differs from classical sequential testing in an important way: the “ground truth” decision we aim to match is the deterministic prediction of a fixed, fully evaluated ensemble. The uncertainty that enables probabilistic reasoning arises from modeling the unobserved suffix of the computation given the observed prefix (e.g., posterior-predictive uncertainty over unseen RF votes or bounded residual capacity for unseen boosting stages). Accordingly, we use the sequential perspective primarily as a design lens for stopping-time rules and thresholding rather than claiming to implement an SPRT-optimal procedure in the classical hypothesis-testing sense.

2.5. Positioning of the Present Work

Table 1 provides a structured comparison of the present work against the most directly related paradigms. To summarize the key differentiators, three axes jointly distinguish the present contribution from prior work:

Table 1. Positioning of the present work against related adaptive-inference paradigms.

Paradigm	Model Family	Training Change	Granularity	Fidelity Target	Stopping Mechanism
Static cascades [7]	Any (staged)	Yes (staged training)	Fixed checkpoints	Ground-truth accuracy	Confidence threshold at each stage
Early-exit networks [8]	Neural	Yes (auxiliary heads)	Fixed layer exits	Ground-truth accuracy	Entropy/confidence at exit heads
Budgeted prediction [23]	Any	Yes (learned gating)	Model-selection level	Cost–accuracy trade-off	Learned routing policy
Adaptive Computation Time [22]	RNN	Yes (halting unit)	Per-step	Learned halting	Cumulative halting probability
Anytime ensembles [9]	Boosting	Yes (reordered)	Stage-level	Ground-truth accuracy	Anytime loss bound
LazyRF (ours)	Random forest	No	Tree-by-tree	Full-model agreement	Bayesian posterior-dominance score
LazyGBM (ours)	GBM	No	Stage-by-stage	Full-model agreement	Guaranteed GBM stop rule plus audited heuristic exits

Note. Granularity is the resolution at which stopping decisions are made. Fidelity target is what the stopping rule is designed to preserve. Training change indicates whether the method requires modifying the training procedure.

(i) No training modification. Cascades, early-exit networks, budgeted prediction, and anytime ensembles all require changes to the training procedure—staged training, auxiliary exit heads, learned gating networks, or reordered boosting sequences. By contrast, LazyRF and LazyGBM operate as post-hoc inference-time wrappers around an already-trained ensemble. This provides a model-preservation advantage for already-trained scikit-learn-style RF or GBM models because the wrappers do not require retraining, revalidation of a new model, or access to the training data.

(ii) Fine-grained, per-component stopping. Cascade and early-exit methods stop at a small, fixed set of pre-defined checkpoints (e.g., after 10, 50, or 100 trees). Our methods check the stopping condition at every block boundary (LazyRF, typically every $B=10$ trees) or every stage (LazyGBM, $B=1$), enabling the stopping time $\tau \left(x\right)$ to adapt to each instance at the natural granularity of the ensemble. This fine granularity is what enables LazyRF to achieve 78% mean work reduction (Section 5.2) compared to 55% for the fixed cascade and 50% for the two-stage cascade at the same operating point.

(iii) Fidelity to the deterministic full-model prediction. Standard early-exit and cascade criteria target ground-truth accuracy (or a proxy thereof), meaning that the stopping rule is designed to be correct with respect to the true label. Our stopping criteria instead target agreement with the full-model prediction ${\widehat{y}}_{\mathrm{full}}\left(x\right)$—a strictly different objective. This distinction matters because it enables a well-defined, verifiable fidelity guarantee: disagreement with the full model can be measured exactly on any dataset without labels, and the LazyGBM certificate (Proposition 1) provides a deterministic, zero-disagreement guarantee when triggered. Margin-vs.-remaining-budget arguments also appear in anytime boosting, including SpeedBoost [9]; the narrower contribution here is a full-model agreement certificate for fixed, pre-trained GBM inference, framed as fidelity to the full-model prediction rather than target-label loss.

This positioning is important for interpreting the experiments: the strongest fairness claims are within-family comparisons to fixed-evaluation or RF-cascade baselines, whereas cross-family comparisons to neural early exits are contextual rather than like-for-like.

3. Theoretical Foundations

This section makes the mathematical content explicit before the algorithms and experiments. The stopping-rule formulation in Equation (2) places the work–fidelity trade-off in the language of sequential analysis (MSC 62L10), while the LazyRF posterior-dominance score uses a Bayesian vote model (MSC 62F15). For additive boosting, Proposition 1 gives the main full-model agreement certificate, Corollary 1 gives its binary specialization, and Proposition 2 bounds expected certificate work through the distribution of partial margins relative to residual capacity. These statements connect martingale/filtration-style stopping-time notation (MSC 60G40) with algorithmic inference for machine learning (MSC 68T05).

3.1. Stopping-Rule View of Lazy Inference

Lazy inference treats test-time evaluation as a sequential decision process with an explicit stopping rule. We summarize the key notation used throughout this section in Table 2. Let $\left\{X_t\left(x\right)\right\}$ denote the partially evaluated state after t components on input x (e.g., vote counts for RFs or partial logits for GBMs), and let $\left\{{\mathcal{F}}_t\left(x\right)\right\}$ be the natural filtration generated by $\left\{X_t\left(x\right)\right\}$. A stopping rule is a measurable mapping $\tau \left(x\right)$ such that $\{\tau \left(x\right)\le t\}\in {\mathcal{F}}_t\left(x\right)$ for all t.

Table 2. Notation used in Section 3.

Symbol	Meaning
x	Input instance.
K	Number of classes.
T	Number of trees in RF.
S	Number of stages in GBM.
$\tau \left(x\right)$	(Instance-dependent) stopping time; number of components evaluated before predicting.
${\mathcal{F}}_t\left(x\right)$	Information revealed by evaluating the first t components on input x (filtration).
${\widehat{y}}_t\left(x\right)$/${\widehat{y}}_{\mathrm{full}}\left(x\right)$	Predicted label after t components/after full evaluation.
$n_{t,c}\left(x\right)$	RF vote count for class c after t trees.
$c_t^{★}\left(x\right)$	Current RF leader (argmax of $n_{t,c}\left(x\right)$) after t trees.
$c_t^{\left(2\right)}\left(x\right)$	Current RF runner-up under the observed votes.
${\alpha }_{t,c}\left(x\right)$	Dirichlet posterior parameter for class c after t trees (prior + counts).
${\alpha }_{t,0}\left(x\right)$	Sum of Dirichlet posterior parameters: ${\sum }_{c=1}^K{\alpha }_{t,c}\left(x\right)$.
$p\left(x\right)$	Latent RF class-probability vector under the Dirichlet posterior.
${\mu }_t^{\left(p\right)}\left(x\right)$/${({\sigma }_t^{\left(p\right)}\left(x\right))}^2$	Mean/variance of the posterior best-vs-runner-up probability gap.
${\rho }_t\left(x\right)$	Conceptual full-model agreement probability after t components.
${\tilde{\rho }}_t\left(x\right)$	Computable RF posterior-dominance score used for stopping (Equation (3)).
$F_t\left(x\right)$	GBM class-score (logit) vector after t boosting stages.
$g_s\left(x\right)$/$f_s\left(x\right)$	GBM stage output (pre-/post-shrinkage): $f_s\left(x\right)=\nu g_s\left(x\right)$.
$\nu$	GBM learning rate (shrinkage).
$k_1\left(t\right)$ / $k_2\left(t\right)$	Current top-1/runner-up class at stage t (fixed tie-break).
${\Delta }_t\left(x\right)$	Top-1 vs. top-2 margin at stage t: $F_{t,k_1\left(t\right)}\left(x\right)-F_{t,k_2\left(t\right)}\left(x\right)$.
$w_s$	Per-stage ${\mathcal{\ell }}_{\infty }$ bound on $f_s\left(x\right)$: $\parallel f_s{\left(x\right)\parallel }_{\infty }\le w_s$.
$R_t={\sum }_{s=t+1}^S{w}_s$	Remaining ${\mathcal{\ell }}_{\infty }$ capacity (suffix sum).
${\alpha }_{\mathrm{stop}}$/$\gamma$	User thresholds: RF stop confidence/GBM stability/flip-score threshold.
$t_{min}$	Minimum evaluated components before a stop check is permitted (trees for RF; stages for GBM).

A canonical soundness objective is to minimize the expected work subject to a fidelity constraint relative to the full evaluation.

$$\underset{\tau (·)}{min}\mathbb{E}\left[\tau \left(x\right)\right]\mathrm{s}.\mathrm{t}.Pr\left({\widehat{y}}_{\tau \left(x\right)}\left(x\right)\ne {\widehat{y}}_{\mathrm{full}}\left(x\right)\right)\le \delta ,$$

(2)

where $\delta \in (0,1)$ is user-chosen. Because the full ensemble is deterministic, the probability in (2) is induced by an explicit uncertainty model for the unseen suffix conditional on the observed prefix. Solving (2) exactly is generally infeasible at serving time; instead, we derive computable stability quantities (probabilistic surrogates or deterministic certificates) whose thresholding yields practical, instance-adaptive stopping rules.

3.2. Random Forests as a Bayesian Vote Process

3.2.1. Posterior-Dominance Score Used by the Implementation

Consider a T-tree RF classifier, where tree i casts vote $h_i\left(x\right)\in \{1,\dots ,K\}$. After evaluating t trees, let $n_{t,c}\left(x\right)={\sum }_{i=1}^t\mathbb{I}\{h_i\left(x\right)=c\}$ be the vote count for class c, let $c_t^{★}\left(x\right)=arg{max}_c{n}_{t,c}\left(x\right)$ be the current leader, and define Dirichlet posterior parameters ${\alpha }_{t,c}\left(x\right)=\beta +n_{t,c}\left(x\right)$ with ${\alpha }_{t,0}\left(x\right)={\sum }_c{\alpha }_{t,c}\left(x\right)$. The experimental LazyRF implementation reasons about a latent class-probability vector $p\left(x\right)\sim \mathrm{Dirichlet}\left({\alpha }_t\left(x\right)\right)$ and uses the posterior-dominance score

$${\tilde{\rho }}_t\left(x\right)\triangleq Pr\left(arg\underset{c}{max}{p}_c\left(x\right)=c_t^{★}\left(x\right)\mid {\mathcal{F}}_t\left(x\right)\right)$$

(3)

as its stopping statistic.

3.2.2. How ${\tilde{\rho }}_t\left(x\right)$ Is Approximated in Practice

Code-specific quantities used only by the optional GBM flip-score heuristic, including the squared suffix proxy $Q_t$, the auxiliary denominator $V_t^{\mathrm{A}1}$, and the implementation score $s_{\mathrm{flip}}^{\left(\mathrm{code}\right)}(t;x)$, are introduced later in the heuristic flip-score paragraph, Section 4.3, and Appendix A.

The quantity in (3) is not evaluated exactly for all K in the prototype. For $K=2$, it reduces to the exact Beta posterior tail $Pr(p_{c_t^{★}}>1/2\mid {\mathcal{F}}_t)$. For $K>2$, the implementation uses Monte Carlo Dirichlet samples when the total posterior mass is small (${\alpha }_{t,0}<30$), and otherwise uses a best-vs-runner-up Gaussian approximation. Let $c_t^{\left(2\right)}\left(x\right)$ denote the current runner-up class. Under the Dirichlet posterior,

$${\mu }_t^{\left(p\right)}\left(x\right)\triangleq {\displaystyle \frac{{\alpha }_{t,c_t^{★}}\left(x\right)-{\alpha }_{t,c_t^{\left(2\right)}}\left(x\right)}{{\alpha }_{t,0}\left(x\right)}},$$

(4)

and

$$\begin{array}{cc}\hfill {\left({\sigma }_t^{\left(p\right)}\left(x\right)\right)}^2\triangleq & {\displaystyle \frac{1}{{\alpha }_{t,0}{\left(x\right)}^2\left({\alpha }_{t,0}\left(x\right)+1\right)}}\times [{\alpha }_{t,c_t^{★}}\left(x\right)\left({\alpha }_{t,0}\left(x\right)-{\alpha }_{t,c_t^{★}}\left(x\right)\right)\hfill \\ \hfill & +{\alpha }_{t,c_t^{\left(2\right)}}\left(x\right)\left({\alpha }_{t,0}\left(x\right)-{\alpha }_{t,c_t^{\left(2\right)}}\left(x\right)\right)+2{\alpha }_{t,c_t^{★}}\left(x\right){\alpha }_{t,c_t^{\left(2\right)}}\left(x\right)].\hfill \end{array}$$

(5)

It is the same below: large-count approximation then becomes

$${\tilde{\rho }}_t\left(x\right)\approx \mathsf{\Phi }\left({\displaystyle \frac{{\mu }_t^{\left(p\right)}\left(x\right)}{{\sigma }_t^{\left(p\right)}\left(x\right)}}\right),$$

where $\mathsf{\Phi }(·)$ is the standard normal CDF. This score is monotone in posterior dominance, but it is not the exact finite-horizon probability that the partially evaluated forest will match the deterministic full-forest decision. In the Gaussian branch it is also a runner-up approximation rather than a full multiclass argmax probability. The Dirichlet vote model also treats the unevaluated votes through an exchangeable latent-probability view; correlated trees reduce the effective amount of independent vote evidence and can weaken the literal posterior interpretation. Accordingly, we interpret ${\tilde{\rho }}_t\left(x\right)$ as an assumption-aware posterior-confidence score whose relation to actual full-model disagreement and forest diversity must be validated empirically. Section 5.9 provides that calibration audit.

3.3. Boosting as Additive Score Accumulation

A K-class gradient-boosted tree ensemble (with shrinkage) produces additive class scores

$$F_t\left(x\right)=F_0\left(x\right)+{\displaystyle \sum _{s=1}^t}{f}_s\left(x\right),f_s\left(x\right)=\nu g_s\left(x\right),$$

(6)

and predicts ${\widehat{y}}_t\left(x\right)=arg{max}_c{F}_{t,c}\left(x\right)$. Assume that a per-stage bound $\parallel f_s{\left(x\right)\parallel }_{\infty }\le w_s$ holds for all x (computable from tree leaf ranges). Define the remaining ${\mathcal{\ell }}_{\infty }$ capacity,

$$R_t\triangleq {\displaystyle \sum _{s=t+1}^S}{w}_s.$$

Then any class score can increase by at most $R_t$ and decrease by at most $R_t$ over the unseen remainder, implying that

$${\Delta }_S\left(x\right)\ge {\Delta }_t\left(x\right)-2R_t.$$

(7)

Proposition 1

(Lossless early stopping for boosting). If ${\Delta }_t\left(x\right)>2R_t$, then ${\widehat{y}}_S\left(x\right)={\widehat{y}}_t\left(x\right)$; i.e., stopping at t gives the same label as full evaluation.

Proof. 

Let $r_c={\sum }_{s=t+1}^S{f}_{s,c}\left(x\right)$. By the definition of $R_t$, $|r_c|\le R_t$ for every class c. For the current leader $k_1\left(t\right)$ and any competitor $c\ne k_1\left(t\right)$,

$$F_{S,k_1\left(t\right)}\left(x\right)-F_{S,c}\left(x\right)=F_{t,k_1\left(t\right)}\left(x\right)-F_{t,c}\left(x\right)+r_{k_1\left(t\right)}-r_c\ge {\Delta }_t\left(x\right)-2R_t.$$

If ${\Delta }_t\left(x\right)>2R_t$, this difference is strictly positive for every competitor, so the stage-t leader remains the unique full-model argmax. Hence ${\widehat{y}}_S\left(x\right)={\widehat{y}}_t\left(x\right)$.    □

Corollary 1

(Binary boosting certificate). For binary additive boosting, let $g_t\left(x\right)$ denote the signed scalar margin after t stages, with prediction determined by the sign of $g_t\left(x\right)$ under the fixed tie-break rule, and let $R_t$ bound the absolute remaining scalar contribution. If $|g_t\left(x\right)|>R_t$, then $\mathrm{sign}\left(g_S\left(x\right)\right)=\mathrm{sign}\left(g_t\left(x\right)\right)$; stopping at t gives the same label as full evaluation.

This is the binary specialization of Proposition 1: the unseen suffix can perturb the scalar binary decision margin by at most $R_t$, so a margin whose absolute value already exceeds $R_t$ cannot cross zero. The multiclass certificate uses $2R_t$ because a pairwise class margin can decrease by $R_t$ through the current leader and by another $R_t$ through a competitor.

Proposition 2

(Expected work of certificate stopping). Let X be drawn from the deployment distribution, and define the certificate stopping time

$${\tau }_{\mathrm{cert}}\left(X\right)\triangleq min\left(\{t\in \{1,\dots ,S\}:{\Delta }_t\left(X\right)>2R_t\}\cup \left\{S\right\}\right).$$

Then

$$\mathbb{E}\left[{\tau }_{\mathrm{cert}}\left(X\right)\right]\le 1+{\displaystyle \sum _{t=1}^{S-1}}Pr\left({\Delta }_t\left(X\right)\le 2R_t\right).$$

Proof. 

For any integer-valued stopping time supported on $\{1,\dots ,S\}$, $\mathbb{E}\left[\tau \right]=1+{\sum }_{t=1}^{S-1}Pr(\tau >t)$. If ${\tau }_{\mathrm{cert}}\left(X\right)>t$, the certificate has not fired by stage t; in particular, the stage-t certificate condition is false, so ${\Delta }_t\left(X\right)\le 2R_t$. Therefore, $Pr({\tau }_{\mathrm{cert}}\left(X\right)>t)\le Pr({\Delta }_t\left(X\right)\le 2R_t)$ for each $t<S$, which gives the bound.    □

This bound connects the deterministic certificate to expected component work without adding an experimental assumption. Large savings are possible when the margin law rapidly places most examples in the region ${\Delta }_t\left(X\right)>2R_t$ for small t; savings are limited when the distribution of ${\Delta }_t\left(X\right)$ remains close to or below the residual-capacity threshold until late boosting stages. This formalizes the empirical observation that boosting saves less when later stages remain corrective.

Only this residual-capacity certificate is deterministic; the RF posterior-dominance score and the non-certificate LazyGBM exits are surrogates or heuristics whose fidelity must be assessed empirically.

3.3.1. Heuristic Flip Score Used in the Prototype

When $w_s$ are loose, the deterministic certificate can be conservative. The non-certificate LazyGBM branch, therefore, uses a monotone margin–capacity score that decreases as the current top-class margin grows relative to a squared suffix-capacity proxy,

$$Q_t\triangleq {\displaystyle \sum _{s=t+1}^S}{w}_s^2.$$

The implementation score is

$$s_{\mathrm{flip}}^{\left(\mathrm{code}\right)}(t;x)\triangleq exp\left(-{\displaystyle \frac{{\Delta }_t{\left(x\right)}^2}{2(Q_t+\epsilon )}}\right),$$

(8)

with a small numerical $\epsilon$ in the denominator. This score is not a probabilistic guarantee for deterministic GBM; it is an empirical margin–capacity heuristic whose behavior is audited in the experiments. The auxiliary Azuma–Hoeffding derivation that motivated its monotone form is given in Appendix A, but Proposition 1 remains the sole mathematical guarantee for LazyGBM.

The implementation uses the following stack:

1.

Lossless certificate: stop if ${\Delta }_t\left(x\right)>2R_t$.

2.

Multiclass ratio heuristic: in multiclass settings, stop if ${\Delta }_t\left(x\right)/\left(2R_t\right)>\eta$ with $\eta =1.5$.

3.

Flip-score heuristic: if $t\ge t_{min}$, stop when $s_{\mathrm{flip}}^{\left(\mathrm{code}\right)}(t;x)<\lambda e^{-\gamma }$, with $\lambda =2$ in multiclass and $\lambda =1$ in binary.

4.

Late-stage fallback: in multiclass settings only, once more than $80\%$ of stages have been evaluated, stop if the current top-1 margin is simply positive.

Only the first rule guarantees the same prediction as the full model. Rules (ii)–(iv) are prototype heuristics included to study a broader deployment frontier, and they are ablated explicitly in Section 5.8. Accordingly, the user parameter $\gamma$ is an aggressiveness parameter for the heuristic flip score and not a certified or calibrated bound on $Pr({\widehat{y}}_S\left(x\right)\ne {\widehat{y}}_t\left(x\right)\mid {\mathcal{F}}_t\left(x\right))$ in the deterministic model.

3.3.2. Summary

For RFs, the evaluated implementation uses a Dirichlet posterior-dominance score over the current leader rather than an exact finite-horizon full-forest agreement probability. For boosting, the additive structure provides a deterministic residual-capacity certificate together with a stack of margin-based heuristics whose aggressiveness is exposed to the user. Together, these mechanisms yield lightweight, instance-adaptive inference rules that leave training unchanged but must be interpreted through their empirical agreement with full evaluation.

3.3.3. Mathematical Scope and Assumptions

The formulation in (2) is a fidelity target for designing and evaluating stopping rules, not an online algorithm for solving the constrained optimization problem exactly. LazyRF instantiates this target through a Dirichlet-model posterior-dominance surrogate; the score is useful for thresholding but is not an exact full-forest agreement probability. LazyGBM contributes a deterministic full-model agreement certificate only when the residual-capacity condition is satisfied. Its ratio, flip-score, and late-stage exits remain empirical heuristics whose effect is quantified by the disagreement, sweep, and ablation results.

4. Materials and Methods

This section specifies the computational cost model and presents the inference-time procedures for LazyRF and LazyGBM. Both methods act as wrappers around a pre-trained ensemble: training is unchanged, and only test-time evaluation is truncated via an instance-adaptive stopping rule. Table 3 summarizes the guarantee status of each stopping quantity before the algorithmic details are introduced.

Table 3. Guarantee and heuristic status of the stopping quantities used in this manuscript.

Component	Role	Status	Interpretation
Fidelity-constrained objective	Design and evaluation target	Formulation, not an exact online solver	Defines the desired full-model-agreement constraint that the stopping rules approximate or certify.
LazyRF posterior-dominance score	Stopping statistic	Probabilistic surrogate under a Dirichlet vote model	Tunable and empirically audited; not an exact finite-horizon agreement probability.
LazyGBM residual-capacity certificate	Stopping condition	Deterministic full-model agreement certificate	When triggered, the final full-model argmax cannot change.
LazyGBM ratio, flip-score, and late exits	Optional prototype stopping rules	Heuristics	Used only when small full-model disagreement is acceptable; interpreted through disagreement rates and ablations.

4.1. Cost Model and Work Units

We consider multiclass classification with input $x\in {\mathbb{R}}^d$ and label $y\in \{1,\dots ,K\}$. The inference cost is measured in work units:

• Random Forest (RF): One work unit is one tree evaluation (one root-to-leaf traversal).
• Gradient Boosting (GBM): One work unit is a boosting stage (one boosting round added to the additive score).

Clarification (multiclass GBM): In scikit-learn, GradientBoostingClassifier treats one “stage” as one boosting round: a single tree for binary classification or K trees for K-class problems. Accordingly, the reported GBM work units should be interpreted as rounds. In the multiclass case, each round entails K tree traversals. Therefore, a full evaluation costs T (RF) or S (GBM rounds) work units per example. The lazy method evaluates the stopping time $\tau \left(x\right)\le T$ (RF) or $\tau \left(x\right)\le S$ (GBM), thereby incurring the expected cost $\mathbb{E}\left[\tau \right(x\left)\right]$. We report work units as the primary cost proxy because they are independent of a specific software backend; wall-clock timings additionally depend on language/runtime, vectorization, batching, and wrapper overhead. Comparability note: Work units are intended for within-family comparisons (RF vs. RF; GBM vs. GBM). Cross-family comparisons (e.g., RF vs. MLP) are qualitative.

4.2. LazyRF: Block-Wise Posterior-Dominance Stopping

LazyRF targets RF majority-vote inference. Let $h_t\left(x\right)\in \{1,\dots ,K\}$ be the class vote from tree t. After evaluating t trees, we define the vote counts

$$n_{t,c}\left(x\right)={\displaystyle \sum _{i=1}^t}\mathbb{I}\{h_i\left(x\right)=c\},{\displaystyle \sum _{c=1}^K}{n}_{t,c}\left(x\right)=t,$$

and the current leader $c_t^{★}\left(x\right)=arg{max}_c{n}_{t,c}\left(x\right)$ using a fixed deterministic tie-break rule.

4.2.1. Stopping Statistic

As derived in Section 3, the implementation computes the posterior-dominance score ${\tilde{\rho }}_t\left(x\right)$ from Equation (3). For binary tasks, this is the exact Beta posterior probability that the current leader remains above $1/2$. For multiclass tasks, the prototype uses Monte Carlo Dirichlet sampling while ${\alpha }_{t,0}<30$, and otherwise switches to the Gaussian approximation based on (4) and (5). We stop once ${\tilde{\rho }}_t\left(x\right)>{\alpha }_{\mathrm{stop}}$ for a user-chosen threshold ${\alpha }_{\mathrm{stop}}\in (0,1)$. Because ${\tilde{\rho }}_t\left(x\right)$ is a posterior-confidence proxy rather than an exact finite-horizon agreement probability, its fidelity is assessed empirically rather than claimed analytically.

4.2.2. Block-Wise Evaluation

To amortize the decision overhead, we evaluate trees in blocks of size B and check the stopping condition only at the block boundaries. Thus, the admissible LazyRF stopping times lie on the fixed grid $\{B,2B,\dots ,T\}$; if $t_{min}$ falls between two boundaries, the first eligible check occurs at the next boundary. This preserves the worst-case cost T while reducing the per-tree control-flow overhead. The complete LazyRF inference procedure is detailed in Algorithm 1.

Algorithm 1 LazyRF block-wise posterior-dominance inference
Require: Input x; trained RF ${\left\{h_t\right\}}_{t=1}^T$; block size B;
	threshold ${\alpha }_{\mathrm{stop}}$; min trees $t_{min}$; prior $\beta >0$ (default $\beta =1$).
1:	Initialize counts $n_c\leftarrow 0$ for $c=1,\dots ,K$.
2:	for $t\in \{B,2B,\dots ,T\}$ do
3:	for $i=t-B+1$ to t do
4:	Evaluate vote $v\leftarrow h_i\left(x\right)$ and update $n_v\leftarrow n_v+1$.
5:	end for
6:	$c^{★}\leftarrow arg{max}_c{n}_c$	▹ fixed tie-break
7:	if $t\ge t_{min}$ then
8:	$a_c\leftarrow \beta +n_c$ for all c; $a_0\leftarrow {\sum }_{c=1}^K{a}_c$.
9:	if $K=2$ then
10:	Compute exact Beta posterior:
11:	$\tilde{\rho }\leftarrow Pr(p_{c^{★}}>1/2\mid a)$.
12:	else if $a_0<30$ then
13:	Estimate $\tilde{\rho }$ via Monte Carlo Dirichlet samples.
14:	else
15:	$c^{\left(2\right)}\leftarrow arg{max}_{c\ne c^{★}}{a}_c$
16:	Compute ${\mu }^{\left(p\right)}$ and ${\left({\sigma }^{\left(p\right)}\right)}^2$ via (4) and (5).
17:	$\tilde{\rho }\leftarrow \mathsf{\Phi }({\mu }^{\left(p\right)}/{\sigma }^{\left(p\right)})$
18:	end if
19:	if $\tilde{\rho }>{\alpha }_{\mathrm{stop}}$ then return $\widehat{y}\leftarrow c^{★}$ and $\tau \left(x\right)\leftarrow t$
20:	end if
21:	end if
22:	end forreturn $\widehat{y}\leftarrow arg{max}_c{n}_c$ and $\tau \left(x\right)\leftarrow T$

4.2.3. Complexity

Updating the vote counts is $O\left(B\right)$. For binary tasks, the stop check is an exact Beta tail evaluation. For multiclass tasks, the small-count branch uses a fixed Monte Carlo budget, while later checkpoints use only runner-up selection and a Gaussian CDF evaluation. Thus, the per-checkpoint overhead is modest and independent of T.

4.2.4. Implementation Notes

(i) The Gaussian branch uses a fast tanh approximation to the normal CDF $\mathsf{\Phi }(·)$. (ii) The Monte Carlo branch is used only for small posterior masses; later checkpoints default to the cheaper Gaussian approximation. (iii) The method retains the fitted RF: the underlying forest is unchanged; LazyRF only adaptively chooses $\tau \left(x\right)$.

4.3. LazyGBM: Residual-Capacity Certificate with a Prototype Heuristic Stack

LazyGBM targets additive-score ensembles (GBMs). Its method has two explicitly separated layers: a deterministic certificate that preserves the full-model prediction when triggered, and optional prototype heuristics that trade additional work reduction for empirically measured disagreement risk. Let $F_t\left(x\right)\in {\mathbb{R}}^K$ be the class-score (logit) vector after t stages, with full model $F_S\left(x\right)$. We define the current top-1 and top-2 classes

$$k_1\left(t\right)=arg\underset{c}{max}{F}_{t,c}\left(x\right),k_2\left(t\right)=arg\underset{c\ne k_1\left(t\right)}{max}{F}_{t,c}\left(x\right),$$

and margin ${\Delta }_t\left(x\right)=F_{t,k_1\left(t\right)}\left(x\right)-F_{t,k_2\left(t\right)}\left(x\right)$.

4.3.1. Precomputable Per-Stage Bounds

For stage s, we define an ${\mathcal{\ell }}_{\infty }$ bound on its (shrinkage-scaled) score contribution as

$$w_s\triangleq \underset{\mathcal{\ell }\in \mathrm{leaves}\left(s\right)}{max}\parallel \nu v_{s,\mathcal{\ell }}{\parallel }_{\infty },$$

(9)

where $v_{s,\mathcal{\ell }}\in {\mathbb{R}}^K$ is the leaf value vector for stage s and $\nu$ is the learning rate (absorbed into $v_{s,\mathcal{\ell }}$ if the library stores shrinkage-scaled leaves). We precompute the suffix sums:

$$R_t\triangleq {\displaystyle \sum _{s=t+1}^S}{w}_s,Q_t\triangleq {\displaystyle \sum _{s=t+1}^S}{w}_s^2,$$

(10)

which can be stored in arrays and accessed in $O\left(1\right)$ time per step.

4.3.2. Lossless Stopping (Deterministic Certificate)

As shown in Proposition 1, if ${\Delta }_t\left(x\right)>2R_t$ then the final argmax cannot change and stopping gives the same prediction as the full model.

4.3.3. Additional Heuristics Used in the Prototype

When $w_s$ are loose, the lossless certificate can be conservative. The reported prototype, therefore, layers several margin-based heuristics on top of the certificate; these heuristic exits account for any savings beyond the certificate-only mode and are not part of the deterministic guarantee. First, it computes the implementation flip-score heuristic from Section 3,

$$s_{\mathrm{flip}}^{\left(\mathrm{code}\right)}(t;x)=exp\left(-{\displaystyle \frac{{\Delta }_t{\left(x\right)}^2}{2(Q_t+\epsilon )}}\right),$$

and evaluates the stopping condition

$$s_{\mathrm{flip}}^{\left(\mathrm{code}\right)}(t;x)<\lambda e^{-\gamma },$$

where $\gamma >0$ is the user-chosen stability threshold (implementation parameter spr_threshold) and $\lambda$ is the flip-score scale (multiclass_flip_scale; default $\lambda =2$ in multiclass and $\lambda =1$ in binary). For the binary case ($\lambda =1$), this simplifies to

$${\displaystyle \frac{{\Delta }_t{\left(x\right)}^2}{2(Q_t+\epsilon )}}>\gamma ,$$

(11)

whereas in the multiclass prototype with $\lambda =2$ the equivalent algebraic condition is

$${\displaystyle \frac{{\Delta }_t{\left(x\right)}^2}{2(Q_t+\epsilon )}}>\gamma -ln2.$$

This implementation score is a heuristic margin–capacity rule; its behavior is validated empirically in the ablation tables. In the multiclass prototype, this score is combined with a margin-to-bound ratio heuristic and a late-stage positive-margin fallback. Under the reported default $\eta =1.5$ and the certificate-first control flow of Algorithm 2, the ratio branch is inactive in the reported experiments because any example satisfying ${\Delta }_t\left(x\right)/\left(2R_t\right)>\eta$ would already satisfy the certificate. None of these heuristics is lossless; they are exposed and ablated because they delineate a broader deployment frontier, even though the reported default operating point is shaped by the flip-score heuristic and late-stage fallback rather than by the ratio branch. The full prototype procedure is summarized in Algorithm 2.

Algorithm 2 LazyGBM prototype inference (certificate + heuristic stack)
Require: Input x; trained GBM stages ${\left\{f_s\right\}}_{s=1}^S$ with initializer $F_0\left(x\right)$; precomputed $(R_t,Q_t)$; minimum stages $t_{min}$; ratio threshold $\eta$; stability threshold $\gamma$; flip-score scale $\lambda$; late-stage fraction $\kappa$.
1:	$F\leftarrow F_0\left(x\right)$.
2:	for $t=1$ to S do
3:	$F\leftarrow F+f_t\left(x\right)$.
4:	$k_1\leftarrow arg{max}_c{F}_c$;    $k_2\leftarrow arg{max}_{c\ne k_1}{F}_c$.
5:	$\Delta \leftarrow F_{k_1}-F_{k_2}$.
6:	if $\Delta >2R_t$ then return $\widehat{y}\leftarrow k_1$ and $\tau \left(x\right)\leftarrow t$	▹ lossless certificate
7:	end if
8:	if $K>2$ and $R_t>0$ and $\Delta /\left(2R_t\right)>\eta$ then return $\widehat{y}\leftarrow k_1$ and $\tau \left(x\right)\leftarrow t$	▹ ratio heuristic
9:	end if
10:	if $t\ge t_{min}$ then
11:	if $Q_t=0$ or $s_{\mathrm{flip}}^{\left(\mathrm{code}\right)}(t;x)<\lambda e^{-\gamma }$ then return $\widehat{y}\leftarrow k_1$ and $\tau \left(x\right)\leftarrow t$	▹ flip-score heuristic
12:	end if
13:	end if
14:	if $K>2$ and $t>\kappa S$ and $\Delta >0$ then return $\widehat{y}\leftarrow k_1$ and $\tau \left(x\right)\leftarrow t$	▹ late-stage fallback
15:	end if
16:	end forreturn $\widehat{y}\leftarrow arg{max}_c{F}_c$ and $\tau \left(x\right)\leftarrow S$

4.3.4. Binary Case

For $K=2$, Corollary 1 gives the scalar lossless certificate $|g_t\left(x\right)|>R_t$ (with a consistent definition of $R_t$ from (10)), and the implemented flip-score heuristic reduces to the binary simplification in Equation (11), namely $g_t{\left(x\right)}^2/\left(2(Q_t+\epsilon )\right)>\gamma$. The factor differs from the multiclass rule because a binary decision is a single scalar margin perturbable by at most $R_t$, whereas a multiclass pairwise margin can move by $2R_t$.

4.4. Shared Implementation Choices

4.4.1. Minimum Evaluation ($t_{min}$)

Both algorithms enforce a minimum number of evaluated components before checking the stopping conditions to avoid brittle early exits when the evidence is insufficient. For LazyRF, this acts only as an eligibility gate on the fixed block-boundary checkpoint grid; it does not offset the grid itself.

4.4.2. Deterministic Tie-Breaking

All $argmax$ operations use a fixed, deterministic tie-break rule to ensure reproducibility and avoid oscillatory behavior near ties.

4.4.3. What Is (And Is Not) Guaranteed

LazyGBM in lossless mode is certified by Proposition 1. All other LazyGBM exits (ratio heuristic, flip-score heuristic, and late-stage fallback) are prototype heuristics rather than deterministic or calibrated probabilistic guarantees for deterministic GBM inference. LazyRF uses a posterior-dominance score ${\tilde{\rho }}_t\left(x\right)$ under a Dirichlet vote model; this score is a tunable fidelity mechanism rather than an exact full-model agreement probability.

4.4.4. Recorded Outputs

For each inference method, we recorded (i) predictive metrics, (ii) average work units $\mathbb{E}\left[\tau \right(x\left)\right]$, and (iii) when enabled, per-example stopping traces such as $\tau \left(x\right)$ and the stop reason. These recorded summaries support the stopping-time/work-quantile and ablation analyses reported later. Work units are emphasized because they abstract away library/runtime constant factors and isolate algorithmic reductions in the expected evaluation.

5. Experimental Results

This section empirically evaluates the proposed instance-adaptive inference wrappers for tree ensembles (LazyRF and LazyGBM) and compares them to representative baselines across four benchmark classification tasks. The results should be read as two evidence classes: certified component reduction when the LazyGBM residual-capacity certificate fires, and empirically audited heuristic stopping behavior when LazyRF surrogates or optional LazyGBM heuristic exits are enabled. We focus on (i) the accuracy–computation trade-off induced by adaptive stopping, (ii) comparisons to static cascades and early-exit neural baselines, and (iii) robustness across random seeds and stopping thresholds.

5.1. Experimental Setup

5.1.1. Datasets and Preprocessing

We evaluated four publicly available classification benchmarks: MNIST (10 classes, 70,000 images; flattened to 784 features), Covertype (7 classes, 581,000 source instances, 54 features), Higgs (binary, ∼11 M source events, 28 features), and Credit Card Fraud (binary, highly imbalanced, 284,000 source instances, 30 features). Dataset provenance, access URLs, and licensing notes are provided in Section 5.1.3. For MNIST, we use the standard train/test split and scale pixel intensities to $[0,1]$. For tabular datasets, we use stratified random train/test splits (80/20) per seed and standardize features using statistics computed on the corresponding training split only. The reported experiments use the full MNIST train/test split. To keep runtimes manageable, the implemented loaders cap the three large tabular datasets before the 80/20 split: Covertype is randomly capped at 200,000 examples using the experiment seed, while Higgs and Credit Card are read with a 200,000-row cap from the source CSV before stratified splitting. Secondary train/test caps are supported by the harness for smoke tests but were not used for the main reported results.

5.1.2. Model Configurations

To make the empirical evaluation self-contained and reproducible, Table 4 summarizes the concrete model hyperparameters, early-exit settings, and baseline configurations used in all experiments. Unless otherwise stated, we used library defaults and varied only the random seed (which controls the tabular train/test split and model randomness). All models were trained once per dataset/seed, and no post-hoc tuning was performed to favor the proposed methods.

Table 4. Model and baseline configurations used in the experiments. “Full” models define the same-family reference for work normalization. “Lazy” methods wrap the fitted full model without retraining. The “Role in evidence” column states the question each row answers; only the full-model references, model-preserving stopping rules, and same-family RF stopping baselines support the central stopping-rule comparison. (default) indicates library defaults.

Component	Role in Evidence	Implementation	Key Hyperparameters/Training Details
Full RF	Full-model reference	sklearn.ensemble. RandomForest Classifier	$n_{\mathrm{estimators}}=100$; criterion=gini; max_depth=None; max_features=sqrt; bootstrap=True; min_samples_split=2; min_samples_leaf=1; class_weight=None (all default); random_state set to seed; n_jobs=-1.
LazyRF	Model-preserving stopping rule	Wrapper around Full RF	Stopping confidence ${\alpha }_{\mathrm{stop}}=0.95$; minimum evaluated trees $t_{min}=10$; block size $B=10$ (check every B trees); other parameters same as Full RF.
Fixed Cascade	Same-family RF stopping baseline	Static checkpoints on Full RF	Evaluate Full RF at fixed checkpoint set $C=\{c_1<\cdots <c_L=T\}$; exit if ${max}_c{\widehat{p}}_c\left(x\right)\ge {\tau }_{\mathcal{\ell }}$ at checkpoint $c_{\mathcal{\ell }}$; Checkpoints $C=\{10,50,100\}$; thresholds $\tau =(0.9,0.9,0.0)$.
Two-stage Cascade RF	Same-family RF stopping baseline	Two RFs	Stage-1 RF ($T_1=10$ trees); exit if ${max}_c{\widehat{p}}_c^{\left(1\right)}\left(x\right)\ge 0.9$; else defer to Full RF.
Compact RF-50	Retrained size alternative	sklearn.ensemble. RandomForest Classifier	Same configuration as Full RF except $n_{\mathrm{estimators}}=50$; retrained compact alternative, not a model-preserving wrapper.
Shallow RF-d10	Retrained depth alternative	sklearn.ensemble. RandomForest Classifier	Same configuration as Full RF except max_depth=10; retrained shallow-tree alternative, not a model-preserving wrapper.
QuickScorer	Full-evaluation runtime baseline	Feature-ordered tree evaluation	Constant-factor timing baseline; evaluates all trees (no work reduction).
Full GBM	Full-model reference	sklearn.ensemble. GradientBoosting Classifier	loss=log_loss; $n_{\mathrm{estimators}}=100$; learning_rate $\nu =0.1$; max_depth=3; subsample=1.0; random_state set to seed.
LazyGBM	Model-preserving stopping rule	Wrapper around Full GBM	Certified mode: residual-capacity certificate. Tunable mode: stop when stability threshold $\gamma$ is met; $t_{min}=10$; block size $B=1$.
LazyGBM Optimized	Optimized same-rule implementation	OptimizedLazyGBM wrapper around Full GBM	Same stopping semantics and defaults as LazyGBM; uses packed scikit-learn tree arrays and Numba-compiled traversal. Model-preserving only after equivalence checks against the Python LazyGBM wrapper pass.
TL2cgen compiled Full GBM	Exact-compilation check	treelite.sklearn import plus tl2cgen.Predictor	The fitted scikit-learn Full GBM is imported and compiled; called exact fitted-model compilation only when test-set disagreement against the fitted scikit-learn model is zero.
Compact GBM-50	Retrained size alternative	sklearn.ensemble. GradientBoosting Classifier	Same configuration as Full GBM except $n_{\mathrm{estimators}}=50$; retrained compact alternative, not a model-preserving wrapper.
Shallow GBM-d2	Retrained depth alternative	sklearn.ensemble. GradientBoosting Classifier	Same configuration as Full GBM except max_depth=2; retrained shallow-tree alternative, not a model-preserving wrapper.
LightGBM native	Replacement backend/model	lightgbm. LGBMClassifier	$n_{\mathrm{estimators}}=100$; learning_rate=0.1; max_depth=3; num_leaves=8; CPU prediction timed with n_jobs=1. This is a native full-model alternative, not an exact compilation of the fitted scikit-learn GBM and not a LazyGBM wrapper.
XGBoost native	Replacement backend/model	xgboost. XGBClassifier	$n_{\mathrm{estimators}}=100$; learning_rate=0.1; max_depth=3; tree_method=hist; device=cpu; CPU prediction timed with n_jobs=1. This is a native full-model alternative, not an exact compilation of the fitted scikit-learn GBM and not a LazyGBM wrapper.
Early-exit “BranchyNet”	Cross-family early-exit reference	Three-block MLP with exits	EarlyExitMLP (hidden width 128, exits after blocks 1, 2). Training: Adam (lr=${10}^{-3}$), batch size 64, 5 epochs; loss $\mathcal{L}={\mathcal{L}}_{\mathrm{final}}+0.3{\mathcal{L}}_{\mathrm{exit}2}+0.2{\mathcal{L}}_{\mathrm{exit}1}$. Inference: entropy thresholds $({\tau }_1,{\tau }_2)=(0.5,0.3)$.

5.1.3. Code, Data, and Environment

Code

We released the source code for data loading, model training, evaluation, backend comparisons, and table/figure generation at https://github.com/pz1004/lazy_inference (accessed on 26 June 2026). The experimental environment used Python 3.11.8 (Python Software Foundation, Wilmington, DE, USA). The Python packages used were NumPy 1.26.4, pandas 2.3.3, scikit-learn 1.8.0, SciPy 1.16.3, Matplotlib 3.10.8, PyTorch 2.9.0+cu128, Numba 0.63.1, LightGBM 4.6.0 (Microsoft Corporation, Redmond, WA, USA), XGBoost 2.1.4, Treelite 4.7.0, and TL2cgen 1.0.0. These are open-source software packages; no chemicals, reagents, devices, instruments, commercial cell lines, samples, or biological materials were used in this computational study.

Data

All datasets are public benchmarks. Covertype and HIGGS are distributed through the UCI Machine Learning Repository under CC BY 4.0; Credit Card Fraud is distributed on Kaggle under ODbL v1.0; MNIST is freely available for research and educational use, although no formal license is specified by the original distributors. The repository provides the code needed to reproduce the preprocessing and evaluation pipeline, and the source sizes and experimental caps are reported here.

• MNIST: Released by Yann LeCun, Corinna Cortes, and Christopher J.C. Burges; freely available for research and educational purposes, with no formal license specified by the original distributors.
• Covertype: UCI Machine Learning Repository. Donated by Jock A. Blackard and licensed under CC BY 4.0.
• HIGGS: UCI Machine Learning Repository. Donated by Daniel Whiteson and licensed under CC BY 4.0.
• Credit Card Fraud: Kaggle “Credit Card Fraud Detection.” Originally released by the Machine Learning Group at Université Libre de Bruxelles (ULB) and licensed under the Open Database License (ODbL) v1.0.

5.1.4. Models and Baselines

We use two evidence tiers. The first tier tests the central stopping-rule claim: if the fitted model family is held fixed, how many ensemble components can be skipped while retaining audited agreement or accuracy relative to the full model?

• RF stopping comparisons: Full RF, LazyRF, Fixed Cascade, and Two-Stage Cascade RF compare stopping policies within the RF family.
• GBM stopping comparisons: Full GBM and LazyGBM compare full boosting evaluation with lazy stopping on the same fitted GBM; certificate-only, certificate-plus-flip, and prototype LazyGBM variants are separated in the ablation analysis.

These rows are the fairness anchors because they preserve the model family and mainly change the inference-time stopping policy.

The second tier answers practical alternatives that a reader may reasonably consider before adopting lazy stopping:

• If all trees are still evaluated, QuickScorer-style evaluation indicates constant-factor runtime context rather than work-unit reduction.
• If the model class can change, BranchyNet gives a cross-family early-exit reference, but it changes the architecture, training procedure, and unit of computation.
• If the backend or implementation can change, native LightGBM/XGBoost predictors test replacement models, TL2cgen tests exact fitted-model compilation, and Optimized LazyGBM tests whether the Python LazyGBM stopping semantics survive a lower-overhead traversal.
• If retraining is allowed, Compact RF-50, Shallow RF-d10, Compact GBM-50, and Shallow GBM-d2 measure the speed–accuracy trade-off of smaller or depth-limited ensembles.

These second-tier rows are not fairness anchors for the central stopping-rule claims. They change the intervention being tested: the model family, training procedure, backend, implementation target, or deployment objective.

5.1.5. Stopping-Rule Configurations

Unless otherwise stated, all ensembles used the same base size within each family to isolate the inference-time control-flow effects. For LazyRF, we set the confidence threshold to ${\alpha }_{\mathrm{stop}}=0.95$, enforce a minimum number of evaluated trees $t_{min}$ to avoid pathological early exits, and apply the stopping check every B trees (block-wise inference), so the first eligible stop check is the first block boundary with $t\ge t_{min}$. The default values are ${\alpha }_{\mathrm{stop}}=0.95$, $t_{min}=10$, and $B=10$. For LazyGBM, the main tables report the current prototype configuration described in Algorithm 2: deterministic certificate, multiclass ratio heuristic ($\eta =1.5$; inactive under the reported control flow), flip-score threshold $\gamma =3.0$, multiclass flip-scale $\lambda =2$, late-stage fallback after $0.8S$, and $t_{min}=10$. Section 5.8 additionally evaluates certificate-only and certificate-plus-flip variants, where certificate-plus-flip means certificate first, then the audited flip-score heuristic.

5.1.6. Evaluation Metrics and Cost Reporting

Predictive Performance

We report the test accuracy for all datasets. For Credit Card Fraud (highly imbalanced), we additionally compute AUROC and AUPRC from positive-class scores during evaluation and report the resulting 30-seed audit in Table 5. To keep the main accuracy–work tables uniform across datasets and model families, Table 6, Table 7, Table 8, Table 9 and Table 10 report accuracy only. The primary fidelity target is agreement with the full model’s predicted class; preserving probability-score rankings or calibrated scores is evaluated separately only for the imbalanced Credit Card Fraud audit.

Table 5. Credit Card Fraud score-metric audit over 30 seeds. AUROC and AUPRC are reported as mean ± 95% CI from positive-class scores. QuickScorer preserves the Full RF scoring semantics in this audit and, therefore, matches the Full RF score metrics.

Method	AUROC (Mean ± 95% CI)	AUPRC (Mean ± 95% CI)
Full RF	0.9526 ± 0.0056	0.8624 ± 0.0118
LazyRF	0.9356 ± 0.0056	0.8370 ± 0.0119
Fixed Cascade	0.9356 ± 0.0056	0.8393 ± 0.0121
Cascade RF (Two-Stage)	0.9356 ± 0.0056	0.8394 ± 0.0121
QuickScorer	0.9526 ± 0.0056	0.8624 ± 0.0118
Full GBM	0.6295 ± 0.0794	0.4227 ± 0.0777
LazyGBM	0.6289 ± 0.0796	0.4219 ± 0.0774
Full BranchyNet	0.9750 ± 0.0052	0.8459 ± 0.0132
Early Exit BranchyNet	0.9763 ± 0.0052	0.8483 ± 0.0131

Table 6. Main results by dataset and method. We report mean test accuracy (mean ± 95% CI), mean work units, and relative work reduction (Equation (12)) with respect to the corresponding full-model reference within the same family.

Dataset	Method	Accuracy (Mean ± 95% CI)	Average Work Units	Relative Work Reduction
Covertype	Full RF	0.9271 ± 0.0006	100.00	0.00
Covertype	Fixed Cascade	0.9271 ± 0.0006	44.70	0.55
Covertype	Cascade RF (Two-Stage)	0.9271 ± 0.0006	50.66	0.49
Covertype	LazyRF	0.9267 ± 0.0006	19.16	0.81
Covertype	QuickScorer	0.9271 ± 0.0006	100.00	0.00
Covertype	Full GBM	0.7721 ± 0.0007	100.00	0.00
Covertype	LazyGBM	0.7672 ± 0.0006	81.00	0.19
Covertype	Full BranchyNet	0.8419 ± 0.0016	3.00	0.00
Covertype	Early Exit BranchyNet	0.8376 ± 0.0015	2.07	0.31
Credit Card	Full RF	0.9995 ± 0.0000	100.00	0.00
Credit Card	Fixed Cascade	0.9995 ± 0.0000	10.08	0.90
Credit Card	Cascade RF (Two-Stage)	0.9995 ± 0.0000	10.11	0.90
Credit Card	LazyRF	0.9995 ± 0.0000	10.02	0.90
Credit Card	QuickScorer	0.9995 ± 0.0000	100.00	0.00
Credit Card	Full GBM	0.9987 ± 0.0002	100.00	0.00
Credit Card	LazyGBM	0.9987 ± 0.0002	31.67	0.68
Credit Card	Full BranchyNet	0.9993 ± 0.0001	3.00	0.00
Credit Card	Early Exit BranchyNet	0.9993 ± 0.0000	1.00	0.67
Higgs	Full RF	0.7218 ± 0.0007	100.00	0.00
Higgs	Fixed Cascade	0.7217 ± 0.0007	82.13	0.18
Higgs	Cascade RF (Two-Stage)	0.7217 ± 0.0007	90.69	0.09
Higgs	LazyRF	0.7205 ± 0.0007	39.60	0.60
Higgs	QuickScorer	0.7218 ± 0.0007	100.00	0.00
Higgs	Full GBM	0.7120 ± 0.0008	100.00	0.00
Higgs	LazyGBM	0.7120 ± 0.0008	80.29	0.20
Higgs	Full BranchyNet	0.7304 ± 0.0009	3.00	0.00
Higgs	Early Exit BranchyNet	0.7304 ± 0.0009	2.48	0.17
MNIST	Full RF	0.9695 ± 0.0002	100.00	0.00
MNIST	Fixed Cascade	0.9695 ± 0.0002	44.14	0.56
MNIST	Cascade RF (Two-Stage)	0.9695 ± 0.0002	49.57	0.50
MNIST	LazyRF	0.9694 ± 0.0003	17.50	0.83
MNIST	QuickScorer	0.9695 ± 0.0002	100.00	0.00
MNIST	Full GBM	0.9459 ± 0.0000	100.00	0.00
MNIST	LazyGBM	0.9419 ± 0.0000	81.00	0.19
MNIST	Full BranchyNet	0.9741 ± 0.0010	3.00	0.00
MNIST	Early Exit BranchyNet	0.9746 ± 0.0009	1.12	0.63

Table 7. Aggregate performance pooled over all datasets and seeds. We report mean test accuracy (mean ± 95% CI), mean work units, and relative work reduction (Equation (12)) relative to the within-family full reference.

Method	Accuracy (Mean ± 95% CI)	Avg. Work Units	Relative Work Reduction
Full RF	0.9045 ± 0.0197	100.00	0.00
Fixed Cascade	0.9045 ± 0.0197	45.26	0.55
Cascade RF (Two-Stage)	0.9045 ± 0.0197	50.26	0.50
LazyRF	0.9040 ± 0.0198	21.57	0.78
QuickScorer	0.9045 ± 0.0197	100.00	0.00
Full GBM	0.8572 ± 0.0215	100.00	0.00
LazyGBM	0.8550 ± 0.0215	68.49	0.32
Full BranchyNet	0.8864 ± 0.0196	3.00	0.00
Early Exit BranchyNet	0.8855 ± 0.0197	1.67	0.44

Table 8. Paired seed-level accuracy and TOST equivalence reanalysis from existing summaries. Mean diff. denotes the lazy/reference accuracy difference. Disagree. denotes the logged disagreement rate against the full-model reference. TOST uses an absolute-accuracy margin of $\pm 0.002$ at $\alpha =0.05$.

Dataset	Comparison	n	Mean Diff.	Disagree.	Paired p	Equiv.
Covertype	LazyRF vs. Full RF	30	$-0.0004$	$0.0025$	$7.1\times {10}^{-11}$	Pass
Credit Card	LazyRF vs. Full RF	30	$-0.0000$	$0.0000$	$0.136$	Pass
Higgs	LazyRF vs. Full RF	30	$-0.0014$	$0.0139$	$3.1\times {10}^{-12}$	Pass
MNIST	LazyRF vs. Full RF	30	$-0.0002$	$0.0007$	$7.1\times {10}^{-5}$	Pass
Covertype	LazyGBM current vs. Full GBM	30	$-0.0048$	$0.0131$	$2.0\times {10}^{-25}$	Fail
Credit Card	LazyGBM current vs. Full GBM	30	$-0.0000$	$0.0000$	$0.326$	Pass
Higgs	LazyGBM current vs. Full GBM	30	$0.0000$	$0.0000$	$1.000$	Pass
MNIST	LazyGBM current vs. Full GBM	30	$-0.0040$	$0.0088$	<${10}^{-300}$	Fail
Covertype	LazyGBM certificate only vs. Full GBM	30	$0.0000$	$0.0000$	$1.000$	Pass
Higgs	LazyGBM certificate only vs. Full GBM	30	$0.0000$	$0.0000$	$1.000$	Pass
Covertype	LazyGBM certificate-plus-flip vs. Full GBM	30	$0.0000$	$0.0000$	$1.000$	Pass
Higgs	LazyGBM certificate-plus-flip vs. Full GBM	30	$0.0000$	$0.0000$	$1.000$	Pass

Table 9. LazyRF threshold sweep. $\alpha$ controls the stopping aggressiveness; larger $\alpha$ is more conservative. We report mean accuracy (mean ± 95% CI), average work units, and relative work reduction for $\alpha \in \{0.90,0.95,0.97,0.99\}$. Mean ± 95% CI over 30 seeds.

Dataset	$\mathit{\alpha }$	Accuracy (Mean ± 95% CI)	Average Work Units	Relative Work Reduction
Covertype	0.90	0.9262 ± 0.0006	16.53	0.83
Covertype	0.95	0.9267 ± 0.0006	19.16	0.81
Covertype	0.97	0.9270 ± 0.0006	21.48	0.79
Covertype	0.99	0.9271 ± 0.0006	25.71	0.74
Credit Card	0.90	0.9995 ± 0.0000	10.02	0.90
Credit Card	0.95	0.9995 ± 0.0000	10.02	0.90
Credit Card	0.97	0.9995 ± 0.0000	10.03	0.90
Credit Card	0.99	0.9995 ± 0.0000	10.04	0.90
Higgs	0.90	0.7193 ± 0.0007	32.98	0.67
Higgs	0.95	0.7205 ± 0.0007	39.60	0.60
Higgs	0.97	0.7216 ± 0.0006	48.34	0.52
Higgs	0.99	0.7217 ± 0.0007	56.53	0.43
MNIST	0.90	0.9690 ± 0.0002	15.42	0.85
MNIST	0.95	0.9694 ± 0.0003	17.50	0.83
MNIST	0.97	0.9695 ± 0.0003	18.95	0.81
MNIST	0.99	0.9695 ± 0.0003	22.27	0.78

Table 10. LazyGBM stability-threshold sweep ($\gamma$). Here $\gamma$ is the threshold applied to the heuristic margin–capacity score rather than a classical sequential-test statistic. The flat rows on Covertype, Credit Card, and MNIST are interpreted as a negative/robustness result: savings there arise primarily from non-$\gamma$ exits, while Higgs shows the main $\gamma$-controlled work trade-off. We report mean accuracy (mean ± 95% CI), average work, and relative work reduction for $\gamma \in \{2.0,3.0,4.0\}$. Mean ± 95% CI over 30 seeds.

Dataset	$\mathit{\gamma }$	Accuracy (Mean ± 95% CI)	Average Work Units	Relative Work Reduction
Covertype	2.0	0.7672 ± 0.0006	81.00	0.19
Covertype	3.0	0.7672 ± 0.0006	81.00	0.19
Covertype	4.0	0.7672 ± 0.0006	81.00	0.19
Credit Card	2.0	0.9987 ± 0.0002	31.67	0.68
Credit Card	3.0	0.9987 ± 0.0002	31.67	0.68
Credit Card	4.0	0.9987 ± 0.0002	31.68	0.68
Higgs	2.0	0.7120 ± 0.0008	74.74	0.25
Higgs	3.0	0.7120 ± 0.0008	80.29	0.20
Higgs	4.0	0.7120 ± 0.0008	83.42	0.17
MNIST	2.0	0.9419 ± 0.0000	81.00	0.19
MNIST	3.0	0.9419 ± 0.0000	81.00	0.19
MNIST	4.0	0.9419 ± 0.0000	81.00	0.19

Work-Unit Proxy

We report a backend-independent work-unit proxy: one work unit corresponds to evaluating one RF tree or one GBM boosting stage. For BranchyNet, it is the number of blocks executed. Let ${\overline{w}}_{\mathrm{method}}$ be the mean work unit per test example and ${\overline{w}}_{\mathrm{full}}$ the corresponding full-model value in the same family. We define relative work reduction as

$$\mathrm{WorkRed}=1-{\displaystyle \frac{{\overline{w}}_{\mathrm{method}}}{{\overline{w}}_{\mathrm{full}}}}.$$

(12)

We intentionally avoid labeling (12) as hardware-energy savings because hardware energy depends on implementation details (vectorization, caching, batching, etc.) not captured by tree/block counts.

Tail Behavior and Worst-Case Bound

Tail latency is a predictable concern in early-exit methods. In this paper, we report mean work units as a backend-independent proxy and make the worst-case explicit. For all lazy methods, the stopping time is capped by the ensemble size, so the worst-case work equals the corresponding full model, up to wrapper overhead. To complement the mean work proxy, Table 11 reports 30-seed means of capped p50/p90/p95 stopping-time work quantiles for the lazy methods together with mean runtime and the corresponding full-model latency baseline. These are work-quantile statistics, not latency-quantile measurements; we interpret the runtime measurements as implementation context rather than as deployment-time latency evidence.

Table 11. Stopping-time work quantiles and Python-prototype runtime context under the 30-seed manuscript protocol.

Dataset	Method	Accuracy	Disagreement	Work p50	Work p90	Work p95	Mean	Full-Model
							Runtime (s)	Latency (s)
Covertype	LazyGBM	0.7672	0.0131	81.00	81.00	81.00	0.9106	0.2558
Covertype	LazyRF	0.9267	0.0025	10.00	40.00	88.67	1.6236	0.0694
Credit Card	LazyGBM	0.9987	0.0000	31.70	31.70	31.70	0.0827	0.0181
Credit Card	LazyRF	0.9995	0.0000	10.00	10.00	10.00	0.0471	0.0373
Higgs	LazyGBM	0.7120	0.0000	87.62	98.40	99.23	0.1721	0.0510
Higgs	LazyRF	0.7205	0.0139	20.00	100.00	100.00	0.4856	0.0666
MNIST	LazyGBM	0.9419	0.0088	81.00	81.00	81.00	2.7224	0.1895
MNIST	LazyRF	0.9694	0.0007	10.00	30.00	59.35	0.5528	0.0450

5.1.7. Random Seeds and Statistical Protocol

All reported values were averaged over $n=30$ random seeds. Each seed induces (i) a train/test split (for tabular datasets), (ii) model training randomness, and (iii) evaluation. For each metric, we report the sample mean and standard deviation across the seeds. We computed 95% confidence intervals using the Student-t critical value:

$$\overline{z}\pm t_{0.975,n-1}{\displaystyle \frac{s}{\sqrt{n}}}.$$

(13)

When comparing a method to its full-model reference, we compute per-seed differences (e.g., in accuracy) to reduce the variance attributable to the split and training seeds; the main tables report absolute accuracies (mean ± 95% CI) alongside work measures. These intervals are descriptive summaries across random seeds and are not used as formal equivalence tests or as proof of no difference.

5.2. Overall Performance

Table 6 reports, for each dataset and method, the mean test accuracy (mean ± 95% CI), mean work units, and relative work reduction with respect to the corresponding full-model reference within the same family (Equation (12)). Table 11 complements these averages with stopping-time quantiles and runtime context for the lazy methods.

Across datasets, LazyRF consistently reduces expected work, achieving relative work reductions of $0.60$–$0.90$ while its displayed main-table accuracy changes remain in the rounded range $[-0.0013,0.0000]$ relative to the corresponding Full RF (Table 6); the paired analysis below reports the exact seed-level differences. Relative to the two RF-cascade baselines, it usually delivers the largest within-family work reduction at the chosen operating point. For LazyGBM, Section 5.8 gives the certificate/heuristic split: the certificate-based variants are the safer reading, while the current prototype recovers extra work at the observed cost of larger disagreement on Covertype and MNIST (see Figure 1 and Figure 2).

Figure 1. Test accuracy by method on the four benchmarks, visualizing Table 6. Bars show mean accuracy over $n=30$ random seeds for tree-ensemble methods (Full RF, static cascades, LazyRF; Full GBM, LazyGBM) and neural baselines (Full and Early-Exit BranchyNet). Numerical values and confidence intervals are provided in Table 6.

Figure 2. Average work units by method on the four benchmarks, visualizing Table 6. For tree ensembles this corresponds to the average number of trees evaluated per test example (normalized so that Full RF/GBM use 100); for BranchyNet it is the average depth in blocks (1–3). LazyRF and LazyGBM consistently reduce work relative to their full counterparts, with LazyRF providing larger savings than LazyGBM.

LazyGBM reduces the number of evaluated boosting stages (averaging 32% reduction for the current prototype; Table 7), but the achievable reduction is generally more modest than that for RFs, reflecting the sequential error-correction structure of boosting.

Compared with the early-exit neural baseline, the proposed lazy schemes provide an instance-adaptive mechanism for tree ensembles that preserves a strict worst-case upper bound equal to full inference within the same model family. Cross-family comparisons (e.g., RF vs. BranchyNet) should be interpreted cautiously because their work units correspond to different computational primitives.

For the highly imbalanced Credit Card Fraud task, the AUROC/AUPRC audit in Table 5 provides a complementary view to accuracy. LazyRF has the same rounded accuracy as Full RF in Table 6, but its positive-class score ranking is lower (AUROC/AUPRC $0.9356/0.8370$ vs. $0.9526/0.8624$ for Full RF) because its score is derived from the partial posterior state at stopping. This distinction is expected because the stopping objective targets full-model label agreement rather than preservation of the full-model score vector. The RF cascades show a similar score-metric trade-off, whereas QuickScorer matches Full RF because it evaluates the complete forest. LazyGBM closely tracks Full GBM in both AUROC and AUPRC, indicating that the LazyGBM early exits do not materially change minority-class ranking relative to the GBM reference at this operating point.

5.3. Paired Accuracy and Equivalence Reanalysis

To strengthen the conclusion-validity evidence without running new experiments, we reanalyzed the existing 30-seed summaries by pairing each lazy method with its same-seed full-model reference. For each dataset–method pair, we computed the signed accuracy difference (lazy minus full), a paired t-test against zero mean difference, and two one-sided tests (TOST) for equivalence with a margin of $\pm 0.002$ absolute accuracy. Table 8 reports the resulting seed-level reanalysis. This is a reanalysis of stored summaries, not a new model run. McNemar tests were not run because the retained summaries contain seed-level accuracies and disagreement rates, but not per-example full/lazy prediction pairs.

The paired analysis sharpens, rather than broadens, the claims. LazyRF satisfies the $\pm 0.002$ equivalence criterion on all four datasets, even where the paired t-test detects a small nonzero mean difference. This combination is not contradictory: with 30 paired seeds, a tiny mean difference can be statistically detectable while still lying inside the pre-specified practical equivalence margin. The current LazyGBM prototype satisfies the equivalence criterion on Credit Card and Higgs but fails it on Covertype and MNIST, matching the interpretation that those two accuracy decreases come from heuristic exits beyond the certificate. In the Covertype/Higgs ablation summaries, both certificate-only and certificate-plus-flip LazyGBM preserve zero logged disagreement and pass the equivalence check; this supports the certificate-first reading while leaving per-example McNemar analysis to future retained-prediction analyses.

5.4. Threshold Sweeps

For LazyRF, we sweep ${\alpha }_{\mathrm{stop}}\in \{0.90,0.95,0.97,0.99\}$ to trace the empirical accuracy–efficiency frontier (Table 9). For LazyGBM in tunable mode, we sweep $\gamma \in \{2.0,3.0,4.0\}$ (Table 10). With all other hyperparameters and data splits held fixed, these sweeps provide two distinct checks: LazyRF shows the intended threshold-controlled frontier, whereas the LazyGBM sweep is mainly a robustness/negative result because several datasets are governed by non-$\gamma$ exits.

The LazyGBM sweep should, therefore, not be read as a uniformly smooth frontier. Covertype, Credit Card, and MNIST are effectively flat over the swept $\gamma$ range, indicating that their work savings under the current prototype come from exits outside the flip-score threshold. Higgs is the informative $\gamma$-controlled case: increasing $\gamma$ makes the flip-score rule more conservative, moving average work from $74.74$ to $83.42$ while leaving mean accuracy unchanged at the displayed precision.

5.5. Stopping-Time Distribution and Runtime Context

The stopping-time quantiles sharpen the interpretation of the work proxy. Because Table 11 reports 30-seed means of per-seed quantiles, we use it to complement rather than replace the mean work results in Table 6. For easy datasets such as Credit Card and MNIST, LazyRF exits early for most examples, so the p50 and p90 work are far below the full-forest budget. Higgs is more difficult: the LazyRF p90 and p95 work remain at the full-forest budget, consistent with the smaller mean work reduction in Table 6. For LazyGBM, Covertype and MNIST consistently exit at 81 stages under the current multiclass prototype, Credit Card exits around 32 stages, and Higgs retains a high-work tail (p90 $98.40$, p95 $99.23$), indicating that a substantial subset of Higgs examples still requires most boosting stages.

The runtime columns in Table 11 also explain why work reduction alone is not enough to claim deployment latency improvement. In this Python prototype, library-backed full-model inference is often faster in wall-clock time than the lazy wrappers despite their lower work counts, because stopping decisions introduce Python-level control flow and bookkeeping overhead. This is why we treat work units as the primary algorithmic evidence and runtime as contextual, implementation-dependent evidence.

5.6. Native Optimized-Backend Context

Native LightGBM and XGBoost answer the replacement-model question: if the fitted scikit-learn GBM does not have to be preserved, how fast are optimized full-model predictors under the same four-dataset, 30-seed protocol? Table 12 reports this context. These rows quantify the option available when replacing and validating the model/backend is acceptable; they do not compile the fitted scikit-learn GradientBoostingClassifier, preserve that fitted object, or implement LazyGBM stopping.

Table 12. Native optimized-backend runtime context over the same four-dataset, 30-seed protocol.

Dataset	Method	Accuracy	Mean Runtime (s)	Speedup vs. Full GBM	Speedup vs. LazyGBM	Model-Preserving Status
Covertype	Full GBM (sklearn)	0.7721 ± 0.0007	0.2558 ± 0.0009	1.00×	3.56×	Yes
Covertype	LazyGBM (Python wrapper)	0.7672 ± 0.0006	0.9106 ± 0.0075	0.28×	1.00×	Yes
Covertype	LightGBM native	0.7612 ± 0.0009	0.5399 ± 0.0041	0.47×	1.69×	No
Covertype	XGBoost native	0.7444 ± 0.0008	0.1028 ± 0.0007	2.49×	8.86×	No
Credit Card	Full GBM (sklearn)	0.9987 ± 0.0002	0.0181 ± 0.0001	1.00×	4.57×	Yes
Credit Card	LazyGBM (Python wrapper)	0.9987 ± 0.0002	0.0827 ± 0.0103	0.24×	1.00×	Yes
Credit Card	LightGBM native	0.9944 ± 0.0014	0.0447 ± 0.0027	0.42×	1.93×	No
Credit Card	XGBoost native	0.9995 ± 0.0000	0.0169 ± 0.0002	1.07×	4.91×	No
Higgs	Full GBM (sklearn)	0.7120 ± 0.0008	0.0510 ± 0.0003	1.00×	3.37×	Yes
Higgs	LazyGBM (Python wrapper)	0.7120 ± 0.0008	0.1721 ± 0.0021	0.30×	1.00×	Yes
Higgs	LightGBM native	0.7112 ± 0.0007	0.1061 ± 0.0008	0.48×	1.62×	No
Higgs	XGBoost native	0.7114 ± 0.0007	0.0165 ± 0.0002	3.09×	10.43×	No
MNIST	Full GBM (sklearn)	0.9459 ± 0.0000	0.1895 ± 0.0031	1.00×	14.39×	Yes
MNIST	LazyGBM (Python wrapper)	0.9419 ± 0.0000	2.7224 ± 0.0137	0.07×	1.00×	Yes
MNIST	LightGBM native	0.9540 ± 0.0000	0.2110 ± 0.0012	0.90×	12.91×	No
MNIST	XGBoost native	0.9368 ± 0.0000	0.0576 ± 0.0003	3.29×	47.29×	No

Under this replacement-model interpretation, the native rows sharpen the decision boundary. XGBoost native prediction is faster than the scikit-learn Full GBM reference on all four datasets in this protocol ($1.07$–$3.29\times$) and faster than the Python LazyGBM wrapper by $4.91$–$47.29\times$. LightGBM native is faster than the Python LazyGBM wrapper ($1.62$–$12.91\times$) but slower than the scikit-learn Full GBM reference on these datasets ($0.42$–$0.90\times$). Accuracy also changes because the native rows are separately trained models: XGBoost improves Credit Card accuracy but is lower on Covertype and MNIST, while LightGBM improves MNIST but is lower on the three tabular datasets. Thus, if substitution and revalidation are acceptable, a native predictor may be the simpler deployment choice. If exact preservation of the fitted scikit-learn ensemble is required, these rows are not drop-in substitutes and are not evidence that LazyGBM itself is faster.

5.7. Compiled, Optimized-Lazy, and Retrained-Model Context

The next two context tables answer three follow-up questions that the native-backend table does not answer. First, can the fitted scikit-learn Full GBM be compiled without changing its test predictions? Second, can LazyGBM’s stopping semantics be preserved while reducing Python-wrapper overhead? Third, what speed–accuracy trade-off is obtained by retraining smaller or shallower ensembles? These rows follow the context-row interpretation in Section 5.1.4: they test fidelity or substitution under the manuscript’s dataset/seed protocol, not deployed-service latency. Table 13 is backed by 840 timing rows (four datasets × 30 seeds × seven selected methods) and 240 fidelity-check rows for the compiled and optimized-lazy checks. Table 14 summarizes a 720-row retrained-alternative analysis, combining verified full/compact rows with 240 additional shallow-tree rows for Shallow RF-d10 and Shallow GBM-d2. MNIST uses the full 60,000/10,000 train/test split; Covertype, HIGGS, and Credit Card use the existing 200,000-row caps before splitting, matching the manuscript data protocol. Training or compilation time is reported separately.

Table 13. Full-size targeted compiled and optimized-lazy context over the four-dataset, 30-seed manuscript protocol.

Dataset	Method	Accuracy	Mean Runtime (s)	Speedup vs. Full GBM	Speedup vs. LazyGBM Python	Model-Preserving
Covertype	Full GBM	0.7721 ± 0.0007	0.2560 ± 0.0066	1.00×	4.04×	Yes
Covertype	LazyGBM Python	0.7672 ± 0.0006	1.0304 ± 0.0197	0.25×	1.00×	Yes
Covertype	LazyGBM Optimized	0.7672 ± 0.0006	0.3336 ± 0.0057	0.77×	3.09×	Yes
Covertype	TL2cgen compiled Full GBM	0.7721 ± 0.0007	0.1432 ± 0.0009	1.79×	7.19×	Yes
Credit Card	Full GBM	0.9987 ± 0.0002	0.0171 ± 0.0004	1.00×	4.54×	Yes
Credit Card	LazyGBM Python	0.9987 ± 0.0002	0.0775 ± 0.0089	0.24×	1.00×	Yes
Credit Card	LazyGBM Optimized	0.9987 ± 0.0002	0.0276 ± 0.0093	1.15×	4.38×	Yes
Credit Card	TL2cgen compiled Full GBM	0.9987 ± 0.0002	0.0064 ± 0.0005	2.79×	12.74×	Yes
Higgs	Full GBM	0.7120 ± 0.0008	0.0506 ± 0.0005	1.00×	3.23×	Yes
Higgs	LazyGBM Python	0.7120 ± 0.0008	0.1635 ± 0.0012	0.31×	1.00×	Yes
Higgs	LazyGBM Optimized	0.7120 ± 0.0008	0.0779 ± 0.0010	0.65×	2.10×	Yes
Higgs	TL2cgen compiled Full GBM	0.7120 ± 0.0008	0.0300 ± 0.0003	1.69×	5.46×	Yes
MNIST	Full GBM	0.9459 ± 0.0000	0.2693 ± 0.0102	1.00×	13.20×	Yes
MNIST	LazyGBM Python	0.9419 ± 0.0000	3.5409 ± 0.0999	0.08×	1.00×	Yes
MNIST	LazyGBM Optimized	0.9419 ± 0.0000	0.7365 ± 0.0190	0.37×	4.81×	Yes
MNIST	TL2cgen compiled Full GBM	0.9459 ± 0.0000	0.1468 ± 0.0017	1.83×	24.10×	Yes

Table 14. Full-size targeted retrained model-alternative context over the four-dataset, 30-seed manuscript protocol.

Dataset	Method	Accuracy	Mean Runtime (s)	Work Units	Speedup vs. Family Full	Accuracy Delta
Covertype	Full RF	0.9271 ± 0.0006	0.7344 ± 0.0044	100.00	1.00×	0.0000
Covertype	Compact RF-50	0.9254 ± 0.0007	0.3717 ± 0.0027	50.00	1.98×	−0.0018
Covertype	Shallow RF-d10	0.7511 ± 0.0010	0.2044 ± 0.0020	100.00	3.60×	−0.1761
Covertype	Full GBM	0.7721 ± 0.0007	0.2560 ± 0.0066	100.00	1.00×	0.0000
Covertype	Compact GBM-50	0.7564 ± 0.0006	0.1322 ± 0.0016	50.00	1.94×	−0.0157
Covertype	Shallow GBM-d2	0.7465 ± 0.0007	0.2050 ± 0.0027	100.00	1.25×	−0.0256
Credit Card	Full RF	0.9995 ± 0.0000	0.1315 ± 0.0018	100.00	1.00×	0.0000
Credit Card	Compact RF-50	0.9995 ± 0.0000	0.0664 ± 0.0009	50.00	1.98×	−0.0000
Credit Card	Shallow RF-d10	0.9995 ± 0.0000	0.0752 ± 0.0005	100.00	1.75×	−0.0000
Credit Card	Full GBM	0.9987 ± 0.0002	0.0171 ± 0.0004	100.00	1.00×	0.0000
Credit Card	Compact GBM-50	0.9987 ± 0.0001	0.0091 ± 0.0002	50.00	1.89×	0.0000
Credit Card	Shallow GBM-d2	0.9986 ± 0.0002	0.0153 ± 0.0003	100.00	1.12×	−0.0001
Higgs	Full RF	0.7218 ± 0.0007	0.7475 ± 0.0030	100.00	1.00×	0.0000
Higgs	Compact RF-50	0.7166 ± 0.0007	0.3729 ± 0.0014	50.00	2.00×	−0.0052
Higgs	Shallow RF-d10	0.7081 ± 0.0007	0.2255 ± 0.0007	100.00	3.31×	−0.0138
Higgs	Full GBM	0.7120 ± 0.0008	0.0506 ± 0.0005	100.00	1.00×	0.0000
Higgs	Compact GBM-50	0.7021 ± 0.0007	0.0273 ± 0.0002	50.00	1.86×	−0.0099
Higgs	Shallow GBM-d2	0.6951 ± 0.0007	0.0352 ± 0.0002	100.00	1.44×	−0.0169
MNIST	Full RF	0.9695 ± 0.0002	0.2721 ± 0.0097	100.00	1.00×	0.0000
MNIST	Compact RF-50	0.9673 ± 0.0003	0.1365 ± 0.0048	50.00	1.99×	−0.0023
MNIST	Shallow RF-d10	0.9479 ± 0.0004	0.1349 ± 0.0048	100.00	2.04×	−0.0216
MNIST	Full GBM	0.9459 ± 0.0000	0.2693 ± 0.0102	100.00	1.00×	0.0000
MNIST	Compact GBM-50	0.9264 ± 0.0000	0.1261 ± 0.0048	50.00	2.14×	−0.0195
MNIST	Shallow GBM-d2	0.9238 ± 0.0000	0.1572 ± 0.0027	100.00	1.72×	−0.0221

The fidelity gates passed on all full-size targeted checks. TL2cgen compilation of the fitted scikit-learn Full GBM had zero test-set disagreement in 120/120 dataset-seed checks, so the corresponding rows in Table 13 can be read as exact fitted-model compilation for those test splits. The optimized LazyGBM implementation matched the Python LazyGBM wrapper in predictions, average work, per-sample evaluated-stage counts, and stop reasons in 120/120 checks. Its value is that the Python wrapper semantics can be preserved while reducing wrapper overhead: it is $2.10$–$4.81\times$ faster than the Python LazyGBM wrapper. Relative to the scikit-learn Full GBM reference, however, Optimized LazyGBM ranges from $0.37$–$1.15\times$, so it is faster on Credit Card but slower on Covertype, HIGGS, and MNIST. TL2cgen compiled Full GBM is $1.69$–$2.79\times$ faster than scikit-learn Full GBM and remains a full-model method, not a lazy stopping rule.

The retrained rows in Table 14 make the model-substitution alternatives explicit. Compact RF-50 reduces the estimator count by half and gives $1.98$–$2.00\times$ prediction speedups, with accuracy deltas from approximately $0.0000$ to $-0.0052$. Shallow RF-d10 keeps 100 estimators but limits depth; it is faster ($1.75$–$3.60\times$) but its accuracy cost is dataset-dependent, ranging from approximately $0.0000$ on Credit Card to $-0.1761$ on Covertype. Compact GBM-50 gives $1.86$–$2.14\times$ speedups, with accuracy deltas from approximately $0.0000$ to $-0.0195$. Shallow GBM-d2 gives smaller speedups ($1.12$–$1.72\times$) and accuracy deltas from $-0.0001$ to $-0.0256$. These rows do not use the same fitted model and should not be compared to LazyRF/LazyGBM as the same intervention: they answer what happens when a smaller or shallower model is retrained and revalidated, not whether an already validated full ensemble can be evaluated adaptively without retraining.

5.8. Ablation Studies

These 30-seed ablation tables (Table 15 and Table 16) clarify which design choices matter most. For LazyRF, smaller block sizes recover additional work savings because they allow more frequent stop checks. Under the fixed block-boundary implementation, $t_{min}$ acts only as an eligibility gate on that grid: with $B=10$, the $t_{min}=5$ and $t_{min}=0$ rows coincide because both first check at 10 trees, whereas $t_{min}=20$ delays the first eligible check to 20 trees and is, therefore, more conservative. Larger block sizes also matter substantially: moving from $B=10$ to $B=20$ skips the 10-tree checkpoint altogether and increases average work on both Covertype and Higgs.

Table 15. Mean over 30 seeds for LazyRF ablation results on Covertype and Higgs. Disagreement is measured against Full RF.

Dataset	Setting	Accuracy	Avg. Work	Disagreement
Covertype	Minimum trees $t_{min}=0$	0.9267	19.16	0.0025
Covertype	Minimum trees $t_{min}=5$	0.9267	19.16	0.0025
Covertype	Block size $B=1$	0.9263	16.45	0.0050
Covertype	Block size $B=5$	0.9266	17.87	0.0033
Covertype	Block size $B=10$	0.9267	19.16	0.0025
Covertype	Block size $B=20$	0.9270	27.80	0.0011
Covertype	Minimum trees $t_{min}=20$	0.9270	26.69	0.0014
Higgs	Minimum trees $t_{min}=0$	0.7205	39.60	0.0139
Higgs	Minimum trees $t_{min}=5$	0.7205	39.60	0.0139
Higgs	Block size $B=1$	0.7194	33.28	0.0233
Higgs	Block size $B=5$	0.7201	37.01	0.0169
Higgs	Block size $B=10$	0.7205	39.60	0.0139
Higgs	Block size $B=20$	0.7213	48.49	0.0058
Higgs	Minimum trees $t_{min}=20$	0.7213	45.14	0.0071

Table 16. Mean over 30 seeds for LazyGBM ablation results on Covertype and Higgs. Disagreement is measured against Full GBM; work units are boosting rounds.

Dataset	Setting	Accuracy	Avg. Work	Disagreement
Covertype	Certificate only	0.7721	99.09	0.0000
Covertype	Certificate + flip score	0.7721	98.60	0.0000
Covertype	Current, $t_{min}=0$	0.7672	81.00	0.0131
Covertype	Current prototype	0.7672	81.00	0.0131
Covertype	Current, $t_{min}=20$	0.7672	81.00	0.0131
Higgs	Certificate only	0.7120	88.41	0.0000
Higgs	Certificate + flip score	0.7120	80.29	0.0000
Higgs	Current, $t_{min}=0$	0.7120	80.29	0.0000
Higgs	Current prototype	0.7120	80.29	0.0000
Higgs	Current, $t_{min}=20$	0.7120	80.29	0.0000

For LazyGBM, the certificate-only variant is the most conservative by construction, but the effect of the additional heuristics is dataset dependent. On Covertype, certificate-plus-flip recovers only a small amount of work relative to certificate-only, and both remain much more conservative than the current prototype. The logged stop fractions show that the ratio branch is inactive at the reported default $\eta =1.5$ and that flip-score exits are negligible on Covertype, so the remaining gap to the current prototype is attributable almost entirely to the late-stage fallback. On Higgs, certificate-plus-flip already matches the current prototype, so the flip-score heuristic appears to account for nearly all of the observed savings. These results are important for interpretation: the main-table LazyGBM numbers are not attributable to the lossless certificate alone. The LazyGBM heuristic stack is, therefore, supported by threshold sweeps, full-model disagreement rates, and targeted Covertype/Higgs ablations rather than by a calibration-style probabilistic guarantee.

5.9. Surrogate Validation and Calibration

A natural question is whether the LazyRF stopping score is empirically predictive of full-model disagreement (i.e., the event that early stopping changes the deterministic full-forest prediction). To validate this, we record for each test example the stopping-time score $\tilde{\rho }$ used by LazyRF and mark an example as a flip if the lazy prediction disagrees with the corresponding Full RF prediction. Table 17 summarizes 30-seed mean calibration statistics across all four datasets, and Figure 3 reports a reliability-style diagram on Covertype using 20,000 randomly sampled test examples from representative seed 42.

Table 17. Mean over 30 seeds for cross-dataset LazyRF calibration summary. ECE is the expected calibration error between binned mean risk proxy and observed flip rate; Monotone seeds counts the seeds whose binned observed flip rates are nondecreasing with risk.

Dataset	Observed Flip Rate	Mean Risk Proxy	Spearman $\mathit{\rho }$	ECE	Monotone Seeds
Covertype	0.0025	0.0228	0.0514	0.0204	0/30
Credit Card	0.0000	0.0006	0.1046	0.0005	25/30
Higgs	0.0139	0.0661	0.0086	0.0534	0/30
MNIST	0.0007	0.0191	0.0294	0.0184	5/30

Figure 3. LazyRF surrogate calibration on Covertype (20,000 test examples, representative seed 42). We bin examples by the predicted flip-risk proxy $1-\tilde{\rho }$ (quantile bins) and plot the observed flip rate (disagreement with Full RF) in each bin. The surrogate is used as a practical monotone stopping statistic and should not be interpreted as a calibrated posterior probability.

The calibration evidence is informative but heterogeneous. On Covertype, the 30-seed mean risk proxy is $2.28\%$, whereas the observed flip rate is $0.25\%$; the proxy is, therefore, conservative on average, but the mean rank correlation is weak ($\rho =0.0514$) and no seed satisfies the monotone-seeds check in Table 17. MNIST shows the same low-disagreement pattern (mean flip rate $0.07\%$ vs. mean risk proxy $1.91\%$), but only $5/30$ seeds satisfy this check. Credit Card remains effectively degenerate at the chosen operating point: the mean observed flip rate rounds to zero, the mean proxy is $0.06\%$, and $25/30$ seeds satisfy the check, but the event rate is too small to support strong calibration claims. Higgs is the hardest case: the proxy is again conservative on average (mean proxy $6.61\%$ vs. flip rate $1.39\%$), but the mean Spearman correlation is extremely weak ($\rho =0.0086$) and no seed satisfies the monotone-seeds check. We, therefore, treat $\tilde{\rho }$ only as a dataset-dependent stopping statistic that can provide conservative risk screening, not as a uniformly calibrated or strongly ordered per-example flip-probability estimate.

5.10. Summary of Findings

Overall, the experiments support one central claim and two decision boundaries. The central claim is that lazy stopping within the same model family can reduce expected evaluated components while keeping the lazy prediction close to the full-model prediction; the strongest evidence is LazyRF under the paired-equivalence checks. The first boundary is LazyGBM: certificate-only is the certified floor, certificate-plus-flip is a certificate-first heuristic variant with zero logged disagreement in the Covertype/Higgs ablations, and the full prototype remains exploratory. The second boundary is deployment: if a user can replace, compile, optimize, or retrain the model, the context rows describe those alternatives rather than extending the central stopping-rule claim.

6. Discussion

This section synthesizes the empirical findings through the lens of the research questions, clarifies the scope of the results, and delineates limitations, threats to validity, and actionable directions for future work.

6.1. RQ1: How Much Work Can Be Eliminated Without Materially Degrading Accuracy?

Across the evaluated benchmarks, LazyRF provides the largest (or tied-largest) work savings among RF-family methods at the operating point used in Table 6, reducing expected work by $0.60$–$0.90$ while remaining inside the paired $\pm 0.002$ equivalence margin. The gains are largest on Credit Card and MNIST, where the stopping-time quantiles in Table 11 show that most examples exit very early, and smaller on Higgs, where a larger fraction of examples remain near the full-forest budget. For LazyGBM, the ablation rows separate the certified certificate-only rule from heuristic additions; the current prototype adds exploratory exits to recover more work. This pattern is consistent with later boosting stages remaining important on some examples even when the partial margin appears stable.

6.2. RQ2: How Do the Lazy Methods Compare to Static Cascades and Early-Exit Neural Baselines?

Relative to static cascades, LazyRF typically achieves larger work reductions at comparable accuracy; e.g., on Covertype and MNIST it reduces work by $0.81$ and $0.83$ versus $0.55$ and $0.56$ for the fixed cascade, while changing accuracy by at most $-0.0004$ and $-0.0001$ relative to the fixed cascade (Table 6). These are the fairest baseline comparisons in the paper because all methods remain in the RF family and differ mainly in inference policy rather than model class or training procedure.

The remaining rows answer different engineering choices, so they should not be read as additional stopping baselines for the same fitted ensemble. QuickScorer asks what constant-factor speedup is possible when all trees are still evaluated. BranchyNet asks how a cross-family early-exit neural model behaves under a different architecture and training objective. Native LightGBM/XGBoost ask whether backend/model replacement is preferable when revalidation is allowed. TL2cgen and Optimized LazyGBM ask whether exact compilation or lower-overhead same-rule traversal is feasible. Compact and shallow retrained ensembles ask whether a smaller validated model is a better latency option. These comparisons are useful precisely because they change the deployment decision; they are not evidence that lazy wrappers dominate optimized serving systems or separately tuned cross-family models.

6.3. What Is Technically Novel Beyond Prior Art?

The high-level intuition that stable predictions can stop early is not new. The narrower contribution of this paper is to make that intuition operational for fixed tree ensembles under a full-model-fidelity target. Rather than learning a new gate, retraining a cascade, or optimizing ground-truth accuracy under a budget, the proposed rules ask whether the current partial ensemble prediction is likely or guaranteed to match the deterministic prediction of the same fully evaluated model. Three elements carry this contribution:

6.3.1. The Posterior-Dominance Stopping Statistic for RF

Prior RF cascade and checkpoint methods use raw vote-fraction thresholds (e.g., stop if the top class exceeds 90% of votes so far). The posterior-dominance score ${\tilde{\rho }}_t\left(x\right)$ instead places a Dirichlet posterior over the latent class-probability vector and computes the posterior probability that the current leader is dominant under that posterior (Equation (3)). This makes ${\tilde{\rho }}_t\left(x\right)$ a tunable, conservative risk-screening statistic that incorporates posterior uncertainty in the latent vote probabilities. The calibration analysis (Table 17) shows that the score is conservative on average but only weakly ordered on some datasets; accordingly, the practical contribution is the tunable statistic and the empirically observed accuracy–efficiency frontier, not a claim of uniform calibration, superior ordering over vote-share, or universal superiority over simpler vote-share heuristics.

6.3.2. The Lossless Residual-Capacity Certificate for GBM

Proposition 1 provides a deterministic, zero-disagreement early-stopping condition: if ${\Delta }_t\left(x\right)>2R_t$, the final prediction is guaranteed identical to the full model. The certificate uses the additive score structure of a fixed pre-trained boosting ensemble together with explicit per-stage ${\mathcal{\ell }}_{\infty }$ contribution bounds; it is, therefore, a full-model agreement statement, not a confidence heuristic about the true label. This is related in spirit to margin-vs.-future-budget reasoning in anytime boosting [9], but the object certified here is agreement with the already-trained full ensemble. Proposition 2 further shows that expected certificate work is controlled by how quickly the partial-margin distribution moves above the residual-capacity threshold. Although the certificate is conservative when per-stage bounds are loose (as shown by the ablation in Table 16, where certificate-only yields only 1–12% work reduction), it provides a hard safety floor before any optional heuristic exits are considered.

6.3.3. The Fidelity-to-Full-Model Formulation

The constrained stopping objective in Equation (2) targets $Pr({\widehat{y}}_{\tau \left(x\right)}\left(x\right)\ne {\widehat{y}}_{\mathrm{full}}\left(x\right))\le \delta$, where ${\widehat{y}}_{\mathrm{full}}\left(x\right)$ is the deterministic prediction of the full ensemble—not the ground-truth label. This target differs from standard cascade, early-exit, and budgeted-prediction objectives that are usually expressed in terms of ground-truth accuracy or a cost–accuracy Pareto curve. The practical consequence is that fidelity can be audited without labels: any deployment can log the disagreement rate by periodically running the full model on a sample, providing a label-free monitoring signal. This formulation also decouples stopping-rule evaluation from the model’s intrinsic accuracy, making the framework applicable even when labels are delayed, unavailable, or noisy.

6.4. RQ3: Robustness Across Thresholds and Random Seeds

Threshold sweeps reveal smooth and largely monotone operating frontiers for LazyRF: increasing $\alpha$ reduces the aggressiveness of stopping, increasing work and slightly increasing accuracy (Table 9). The one-factor ablations in Section 5.8 sharpen this picture: smaller block sizes recover additional savings by checking more often, whereas $t_{min}$ changes the operating point only when it moves the first eligible checkpoint to a later block boundary. At $B=10$, the $t_{min}=5$ and $t_{min}=0$ rows coincide, while $t_{min}=20$ is more conservative because it defers the first stop check to 20 trees. For LazyGBM, the flat $\gamma$ rows on Covertype, Credit Card, and MNIST are a negative/robustness result: changing the flip-score threshold has no displayed effect because savings arise from non-$\gamma$ exits. Higgs is the meaningful $\gamma$-controlled case, with relative work reduction moving from $0.25$ to $0.17$ as $\gamma$ increases from $2.0$ to $4.0$ while accuracy remains unchanged at the displayed precision (Table 10). The ablation results further show that the certificate-only variant is the most conservative and that the extra heuristic savings are dataset dependent: on Covertype they come almost entirely from the late-stage fallback, whereas on Higgs the flip-score heuristic already recovers most of the gap to the current prototype.

6.5. Why Lazy Inference Helps (And When It Is Expected to Help Most)

The empirical pattern aligns with the margin-stability view of the ensemble prediction. For random forests, many instances become effectively “decided” early: as votes accumulate, posterior uncertainty about the latent dominant class contracts rapidly, so additional trees yield diminishing returns for a large fraction of the inputs. LazyRF operationalizes this by stopping when its posterior-dominance score exceeds a user-specified tolerance, thereby allocating more computation to ambiguous instances and less to easy ones. Consequently, datasets with a higher prevalence of high-margin instances are expected to admit larger savings, whereas datasets with substantial overlap between class-conditional distributions require more trees on average at a fixed fidelity target.

The RF interpretation depends on the vote model. The Dirichlet score is most natural when the remaining tree votes behave like exchangeable evidence about a stable class-probability vector. Real forests contain correlated trees because bootstrap samples, feature subsets, and shared predictive structure do not produce independent voters. Correlation effectively lowers the independent evidence content of the observed prefix, so the score should not be read as a calibrated posterior flip probability. This is why Table 17 reports both average conservativeness and weak rank ordering on some datasets. In practice, the statistic is expected to be more reliable for diverse, high-margin forests and less reliable when many correlated trees make similar errors or when class margins remain small.

For boosting, early termination is intrinsically more delicate because later stages are designed to correct earlier residual errors; therefore, informative contributions can persist deeper into the sequence. This explains why LazyGBM typically yields smaller and more variable work reductions than LazyRF under a global configuration (Table 6). It also explains why the most aggressive prototype settings are not uniformly benign: when later stages remain corrective on hard examples, heuristic exits can increase disagreement even if the mean work drops. In practice, boosting-oriented lazy inference benefits the most when the margin grows quickly for a substantial subset of examples or when tighter, instance-conditioned residual-capacity bounds can be obtained.

6.6. Practical Interpretation: Target Users and Alternatives

The deployment decision should start with a cost audit. Lazy evaluation is worth considering for teams that must preserve an already validated RF or GBM, can periodically audit full-model disagreement, and have measured ensemble evaluation as a material part of latency or batch-scoring cost after data access, preprocessing, feature construction, distributed scheduling, and I/O. A simple cost-decomposition rule is useful: if full ensemble evaluation is only a fraction f of end-to-end time, then even eliminating all ensemble work can reduce total time by at most f. Thus, when model evaluation accounts for only 5% of a pipeline, adaptive tree stopping cannot yield more than a 5% end-to-end gain; when model evaluation is a large batch-scoring cost and the fitted model cannot be replaced, the same work reduction can be operationally relevant. The present evidence is strongest for this model-preserving setting. It is weaker for pipelines where feature construction, network transfer, database access, or orchestration dominates, and it is unnecessary when an optimized full-model predictor already satisfies the required latency, throughput, and resource budget.

The effect sizes should, therefore, be read at the component-evaluation level. The reported savings are reductions in evaluated components (Equation (12)), not measured hardware energy consumption or hardware-independent latency gains. Wall-clock improvements in the present prototype can be attenuated, or even inverted, by Python-level control-flow and bookkeeping overhead, whereas the full-model baselines are implemented in highly optimized library code. The most reliable claim supported by the experiments is an algorithmic reduction in expected evaluated components, together with a worst-case component count bounded by full evaluation within each family.

Reducing ensemble size, limiting tree depth, retraining a compact model, or deploying optimized predictors are more direct choices when a new model or backend can be validated and the goal is end-to-end latency. The native-backend context supports this practical advice: XGBoost native is faster than the scikit-learn Full GBM reference by $1.07$–$3.29\times$ and faster than the Python LazyGBM wrapper by $4.91$–$47.29\times$ (Table 12), but the accuracy changes mean it is a replacement model, not a drop-in preservation mechanism. The full-size targeted checks sharpen the same point: TL2cgen preserves fitted Full GBM predictions in 120/120 checks, Optimized LazyGBM preserves Python LazyGBM semantics in 120/120 checks while still being slower than scikit-learn Full GBM on three datasets, and Compact RF-50/GBM-50 or Shallow RF-d10/GBM-d2 reduce runtime only by changing and retraining the model (Table 13 and Table 14). The shallow rows also show why this alternative cannot be assumed to be harmless: Shallow RF-d10 loses $0.1761$ accuracy on Covertype and $0.0216$ on MNIST, while Shallow GBM-d2 loses $0.0256$ on Covertype and $0.0221$ on MNIST. The proposed wrappers address the narrower case in which the existing fitted ensemble is the object to preserve. They expose an inference-time knob for trading average evaluated components against full-model disagreement. For RFs, the relevant knobs are ${\alpha }_{\mathrm{stop}}$, B, and $t_{min}$ (Table 9 and Table 15). For GBMs, the first decision is whether exact full-model agreement is mandatory: use certificate-only when it is; inspect certificate-plus-flip when the Covertype/Higgs ablation evidence is acceptable for the deployment; treat the full current prototype as exploratory because its additional savings come from heuristic exits.

6.7. Limitations and Threats to Validity

We grouped limitations using standard validity taxonomies:

Internal validity: (i) Timing comparisons are confounded by implementation asymmetry: optimized library inference vs. Python control flow in lazy wrappers. (ii) Any reimplementation-based baseline (e.g., QuickScorer in Python) should not be interpreted as representative of optimized systems. (iii) The native LightGBM/XGBoost measurements are separately trained full-model alternatives, while the TL2cgen, Optimized LazyGBM, compact-ensemble, and shallow-tree rows remain controlled CPU checks under the context-row interpretation defined in Section 5.1.4. (iv) The composability argument for LazyGBM is algorithmic: an optimized backend could exploit fewer evaluated stages, but the present evidence does not establish hardware-independent latency improvement. (v) LazyGBM behavior depends on the particular choice of residual-capacity bound and heuristic stack; different bounding strategies or heuristic thresholds may shift the trade-off.

Construct validity: (i) Accuracy alone may be insufficient for imbalanced tasks (e.g., fraud detection), where AUROC/AUPRC and cost-sensitive metrics better reflect utility. (ii) The work proxy (evaluated trees/stages) does not directly quantify hardware energy. (iii) We report stopping-time quantiles averaged over 30 seeds, but a fuller deployment study would still measure tail latency over repeated runs in an optimized implementation. (iv) The LazyRF posterior-dominance score is informative but not uniformly calibrated across datasets, so it should be interpreted as a practical stopping statistic rather than as a universal flip-probability estimator.

External validity: (i) The results are based on four datasets and may not generalize to other modalities (e.g., ranking, regression) or different data regimes (e.g., distribution shift). (ii) The experiments primarily used scikit-learn-style ensembles for the lazy methods that retain the fitted model; transferring the stopping rules to widely deployed toolchains (for example, XGBoost/LightGBM) would require adaptations to handle residual bounds, traversal overhead, and backend APIs. (iii) The context rows add native backend, compiled backend, optimized-lazy, compact-model, and shallow-model evidence, but they still do not evaluate tuned hyperparameter searches or deployed service benchmarks.

Conclusion validity: The paired seed-level tests and TOST equivalence reanalysis in Section 5.3 reduce uncertainty about practically negligible accuracy differences using the stored 30-seed summaries. The remaining limitation is per-example agreement testing: McNemar tests require retained full/lazy prediction pairs, which were not available for this revision.

6.8. Future Work

Several directions follow directly from the empirical findings and the above limitations: (i) extending the TL2cgen and Optimized LazyGBM checks to additional datasets, model sizes, tuned hyperparameters, and serving stacks; (ii) integrating LazyRF/LazyGBM into optimized inference stacks (vectorized/compiled implementations) to translate work savings into consistent wall-clock reductions; (iii) implementing the stopping rules inside modern boosted-tree toolchains, rather than only benchmarking native full-model alternatives, and investigating tighter, instance-conditioned residual-capacity bounds to improve LazyGBM savings while controlling disagreement risk; (iv) extending task-appropriate analyses for imbalanced problems beyond AUROC/AUPRC to expected cost and assessing whether early stopping disproportionately affects minority-class behavior; (v) measuring deployment tail latency in optimized implementations and relating latency quantiles to stopping-time and margin distributions; (vi) evaluating robustness under distribution shift and auditing whether adaptive stopping induces systematic differences in compute allocation across subpopulations.

7. Conclusions

We studied instance-adaptive lazy inference for tree ensembles as an inference-time wrapper that truncates the evaluation of a fixed, pre-trained model. The central idea is to treat ensemble inference as a sequential process and stop when the current prediction is sufficiently stable with respect to the unevaluated remainder. The stopping objective defines the fidelity target relative to the deterministic full model; the practical contribution is a set of computable stopping quantities that can be evaluated online with modest additional bookkeeping.

We presented two concrete instantiations:

• LazyRF models the observed RF votes with a Dirichlet posterior and stops according to a posterior-dominance score for the current leader. The resulting threshold ${\alpha }_{\mathrm{stop}}$ is an interpretable conservativeness knob, and the calibration audit clarifies its dataset-dependent behavior.
• LazyGBM uses the additive boosting structure to derive a deterministic residual-capacity certificate, whose binary specialization is given in Corollary 1 and whose expected-work implication is given in Proposition 2. The certificate-only mode is the conservative safety floor; certificate-plus-flip is a certificate-first, empirically audited variant that adds the flip-score heuristic; and the full current prototype is exploratory.

Across four classification benchmarks and $n=30$ random seeds, LazyRF reduced expected work by 60–90% at ${\alpha }_{\mathrm{stop}}=0.95$ (relative work reduction $0.60$–$0.90$), with displayed main-table accuracy changes in $[-0.0013,0.0000]$ and paired seed-level equivalence under the $\pm 0.002$ margin. For the current prototype at the default $\gamma =3.0$ operating point, LazyGBM reduced work by 19–68%, passing paired equivalence on Credit Card and Higgs but failing it on Covertype and MNIST; the stronger and cleaner LazyGBM claim is, therefore, the separated certificate-first evidence rather than the prototype alone.

The additional analyses clarify when this claim should and should not be used. Stopping-time quantiles show that LazyRF savings are concentrated on easy inputs, while harder settings such as Higgs retain a high-work tail. The calibration audit shows that the LazyRF stopping score is useful but dataset-dependent. The LazyGBM ablations separate certified stopping from more aggressive heuristic exits.

The backend and model-alternative analyses define the decision boundary. If deployment can replace the fitted model, native predictors or retrained compact/shallow ensembles may be simpler choices. If exact fitted-model behavior must be retained, TL2cgen gives a compiled full-model option when zero-disagreement checks pass. If the LazyGBM stopping rule itself must be retained, the optimized LazyGBM rows test whether the same semantics can be run with lower wrapper overhead. The lazy wrappers target the narrower case in which the existing fitted ensemble remains the object to preserve and average component evaluation is still a measured cost.

The methods, therefore, provide interpretable control knobs (${\alpha }_{\mathrm{stop}}$ for LazyRF and a certificate/variant/threshold configuration for LazyGBM) for settings in which average component evaluation remains a measured cost. The strongest supported claim is narrow: the methods reproducibly reduce the expected number of evaluated ensemble components, with explicit evidence about where each stopping rule is reliable, conservative, or heuristic. The RF score depends on modeling assumptions, the GBM prototype includes heuristic exits beyond the deterministic certificate, and the Python runtime context does not establish hardware-independent latency or energy reduction. Within those boundaries, the study establishes a practical route to adaptive computation when the fitted tree ensemble must be retained.