Noise-Dependent Robustness of XGBoost, LightGBM, and CatBoost

Abstract

Gradient-boosted decision trees (GBDTs) are among the leading methods for tabular data; however, their comparative robustness to corrupted labels remains unclear. This study resolves that uncertainty with a controlled benchmark protocol in which library identity is the only free variable, so that an observed difference can be attributed to the implementation rather than to tuning, data splits, or noise realizations, and in which capacity, noise floor, and dataset-selection confound checks are mandatory before any ranking is reported. We benchmarked Extreme Gradient Boosting (XGBoost), Light Gradient-Boosting Machine (LightGBM), and Categorical Boosting (CatBoost) under symmetric, asymmetric pair-flip, and instance-dependent label noise across 15 datasets from the OpenML Curated Classification benchmark suite 2018 (OpenML-CC18), four noise rates, and three random seeds. The clean data-tuned configurations were fixed across the noise conditions. Predictive performance is the macro-averaged F1 score (macro-F1) on a clean test set, and degradation is the absolute drop in macro-F1 relative to each library’s own clean-data score on the same dataset, so the recoveries reported below are absolute percentage points of macro-F1. The ranking depends on both the noise rate and the noise model: no library separated at a 10% rate under symmetric or asymmetric noise; separation emerged from 20% under symmetric noise and only at 40% under asymmetric noise; and under instance-dependent noise, it weakened as the rate rose. At 40% symmetric and asymmetric noise, CatBoost showed significantly lower per-dataset degradation than LightGBM (Friedman tests, both $p<0.001$), with mean ranks of 1.20 versus 2.47 and 1.20 versus 2.67, respectively, and with XGBoost intermediate in both cases (2.33 and 2.13) and separable from CatBoost under symmetric noise only. At 40% instance-dependent noise, the ranking disappeared: no significant library ranking was detected ($p=0.63$), and performance gaps were smaller than twice the pooled seed standard deviation, which shows that GBDT robustness conclusions are noise-model-dependent. Rankings also varied by evaluation dimension: LightGBM had the most stable feature importances in observed means, calibration rankings depended on the noise model, and CatBoost’s training loss was the strongest mislabel-detection signal. Under symmetric noise only, small-loss reweighting significantly improved all three libraries, while early stopping recovered up to 8.5 pp for LightGBM; mitigations were not evaluated under asymmetric or instance-dependent noise. For practitioners, this means the library and the mitigation should be chosen together with the noise process that is expected: prefer CatBoost when label-conditional noise is likely and accuracy is the objective; expect no library to buy robustness under feature-dependent noise at high rates; apply early stopping or small-loss reweighting under symmetric noise, where both give significant gains and early stopping helps LightGBM most; and select on calibration, mislabel detectability, or importance stability instead when the deployment depends on those, because the ranking differs by dimension.

1. Introduction

Gradient-boosted decision trees (GBDTs) are consistently among the strongest to perform well on tabular supervised learning tasks in both academic benchmarks and industrial practices [1,2]. The three most widely used open-source implementations are Extreme Gradient Boosting (XGBoost) [3], Light Gradient-Boosting Machine (LightGBM) [4], and Categorical Boosting (CatBoost) [5,6]. These libraries are frequently selected based on clean-data benchmarks [7] or practitioner familiarity, a selection pattern documented in analyses of Kaggle competition solutions, which found that gradient-boosting implementations were the default first choice of top-ranked practitioners, largely because they were the established, well-supported tools in the community [8], with little consideration of how they behave when training labels are corrupted.

Label noise is pervasive in real-world datasets. Crowdsourced annotations, automated labeling pipelines, and domain expert disagreement all introduce systematic or random mislabeling [9,10]. Label noise rates of 5–30% are common in practical applications [11], and recent large-scale audits have confirmed that even curated benchmarks contain non-trivial annotation errors [12,13]. For practitioners who rely on GBDTs for high-stakes decisions, such as credit scoring, medical triage, and fraud detection, understanding which library degrades most gracefully under label corruption is directly actionable. Evidence from one of those domains points the same way: a 2025 study of credit risk models reports that controlled degradation of training data quality measurably changes model performance; so, label quality is a deployment variable rather than a fixed property of the dataset [14].

Despite the practical importance of this question, the existing literature offers little guidance. The most closely related prior work studied the robustness of XGBoost to label noise on tabular data but did not compare across libraries [15]. An earlier study on the noise sensitivity of supervised learning techniques predates the maturity of both LightGBM and CatBoost [16]. The deep-learning literature on learning with noisy labels is extensive [12,17,18], but its solutions (robust loss functions, co-training, and meta-learning) are designed for neural architectures and do not directly transfer to tree ensembles. To the best of our knowledge, no prior study has systematically compared the three dominant GBDT libraries under controlled noise conditions with statistical testing across multiple datasets.

This study fills this gap with a controlled benchmark designed for practitioner relevance. Our contributions are fivefold, and they concern the comparison protocol and the scope of its evaluation; the findings that the protocol yields are reported in Section 4. First, we define a protocol that licenses a library robustness ranking under label-conditional noise. Its primary estimand is degradation relative to a per-(dataset, library) clean baseline, which measures what a practitioner loses by keeping their library and acquiring noisier labels rather than conflating robustness with clean-data skill; hyperparameters are tuned once on clean data and then frozen across every noise condition, and data splits and noise realizations are functions of the dataset and seed alone, so that library identity is the only free variable; and rankings are established with the Friedman test followed by the post hoc Nemenyi procedure across 15 datasets, the accepted framework for comparing classifiers over multiple datasets because it aggregates by rank and does not require accuracy scales to be commensurable across datasets [19,20]. Second, we specify the confound checks that a robustness claim must pass before reporting it, covering capacity, noise floor, and dataset selection. These checks are the reason we report a null result rather than a weak ranking at the highest rate of instance-dependent noise, where the mislabeling probability correlates with sample difficulty. Third, we extend the evaluation beyond predictive accuracy to probability calibration, mislabel detectability from the training loss, and feature-importance stability, so that library choice can be assessed against the dimension on which a given deployment depends rather than accuracy alone. Fourth, we evaluate two library-agnostic and computationally inexpensive mitigations, early stopping and small-loss reweighting, within the same protocol, which turns the comparison into guidance for practitioners who have already run one of these libraries in production. Fifth, we developed a config-driven benchmarking harness and released it with the per-unit results. Its contribution is one of engineering rather than of algorithms, and it consists of four capabilities that ad hoc benchmark scripts typically lack: the experimental grid is declared in the configuration, so adding a library, a noise model, or a mitigation is a configuration change rather than new code; a ledger-based resume lets a 2529-unit grid survive interruption and re-run only what is missing; per-unit artifact storage makes every number in this paper traceable to the individual fit that produced it; and the statistical reporting is regenerated from those artifacts rather than transcribed by hand, which removes a common source of error in benchmark papers. Together these make the comparison auditable and extensible instead of a one-off experiment.

The remainder of this paper is organized as follows. Section 2 reviews the related work on GBDT comparison, label-noise models, and noise-robust learning. Section 3 details the experimental methodology, including the datasets, noise-injection protocols, and statistical testing. Section 4 presents the results organized around the five claims. Section 5 discusses the mechanisms, practitioner implications and threats to validity. Finally, Section 6 concludes with actionable recommendations for future research.

2. Related Work

This section reviews the work on which our design and claims depend: the architectural differences between the three libraries that could plausibly produce different noise behaviors (Section 2.1), the noise models under which prior robustness conclusions were obtained (Section 2.2), what is already known about label-noise tolerance in tree learners (Section 2.3), and the statistical machinery we adopt for multi-dataset comparison (Section 2.4). Section 2.5 presents the resulting gap that this study aims to address.

2.1. Gradient-Boosted Decision Trees

Gradient boosting [21,22] constructs an additive ensemble of decision trees by iteratively fitting new trees to the negative gradient of the loss function. XGBoost introduced system-level optimizations (sparsity-aware split finding and column block structure) and a regularized objective that penalizes tree complexity [3]. LightGBM further improves efficiency through Gradient-based One-Side Sampling (GOSS) and Exclusive Feature Bundling (EFB), enabling faster training on high-dimensional data [4]. CatBoost contributed ordered boosting, a permutation-based scheme that reduces prediction shift (a form of target leakage during training) and native support for categorical features via ordered target statistics [5,6]. Benchmarking published since has kept these implementations at the center of tabular practice while narrowing their margin: a continuously maintained benchmark over curated tabular tasks reports that gradient-boosted trees remain strong contenders once deep learners are granted comparable tuning and ensembling budgets, with cross-model ensembles ahead of any single family [23], and tabular foundation models pretrained on synthetic priors now match or exceed tuned GBDTs on small datasets [24]. Neither line of work varies in label quality, so which of these implementations degrades most gracefully under corrupted labels remains open independently of them. A 2026 robustness analysis does vary it, reporting that a tabular foundation model’s ranking performance is largely preserved when up to 35% of binary labels are randomly flipped [25]; that finding concerns one in-context learner under symmetric corruption, and it does not order the gradient-boosting implementations against each other.

These architectural differences suggest potentially different robustness profiles in the presence of label corruption, because the mechanisms that distinguish the libraries determine how cheaply an ensemble can isolate a single mislabeled point: leaf-wise growth with unconstrained leaf counts can carve out a small, low-support region around an individual corrupted example, whereas level-wise oblivious trees cannot express such a region without applying the same split across the entire level. For example, CatBoost differs from the other two libraries in its use of oblivious (symmetric) decision trees, ordered target statistics for categorical features, and ordered boosting (when enabled), which is a permutation-based scheme designed to reduce prediction shift [5,6]. It remains unknown whether any of these differences translate into noise robustness, which has not been empirically tested in controlled multi-library comparisons. We note that in our benchmark, CatBoost runs in its default plain boosting mode (Section 5.1); our results speak to the libraries as practitioners use them out of the box, not to order boosting.

2.2. Label-Noise Models

The label-noise literature distinguishes three progressively realistic noise models [11,26,27]. In symmetric (uniform) noise, each label is independently flipped to a uniformly random different class with probability $\eta$. This is the simplest and most studied model; however, it is also the least realistic, as real annotation errors are rarely class-independent [27]. In asymmetric (class-conditional) noise, the flip probability depends on the true class: class $c$ flips only to a designated target class, typically $(c + 1)\mathrm{m}\mathrm{o}\mathrm{d}K$ in the standard pair-flip protocol. This model systematically confuses visually or semantically similar classes [26,28]. Estimating that class-conditional structure from corrupted labels alone remains an active problem; a 2026 treatment recovers the confusion matrix and the clean class proportions jointly through an exponential-tilting density ratio, with consistency guarantees but without assuming the noise rates are known in advance [29]. In instance-dependent noise, the flip probability depends on the features of the individual sample; typically, samples near the decision boundary are more likely to be mislabeled [30,31,32]. Of the three synthetic models, this one most directly emulates ambiguity-dependent labeling errors, since human annotators disagree the most on ambiguous instances [10]; it nonetheless remains a synthetic proxy rather than a fully realistic noise process.

A key critique of the label-noise robustness literature is that the conclusions drawn under symmetric noise may not be applicable to more realistic noise models [30,33,34]. Recent evidence sharpens that critique by showing that noise structure, and not the noise rate alone, governs benchmark difficulty: generating instance-dependent corruption through controlled input perturbations on image and tabular data yields label distributions closer to human uncertainty than rater-based synthetic noise, and it exposes failure modes of co-training and semi-supervised corrections that symmetric benchmarks leave hidden [35]. Work on tabular deep learning arrives at a compatible conclusion from the modeling side, arguing that several of the design choices that make recent tabular methods effective succeed because they implicitly manage data uncertainty rather than treat it as uniform across the feature space [36]. Our benchmark explicitly tests all three models to characterize where the conclusions generalize and where they fail to do so.

2.3. Noise-Robust Learning for Tree Ensembles

The noise tolerance of tree learners predates the current GBDT generation. Quinlan’s decision tree induction work treated noisy training data as a first-order concern and motivated pruning explicitly as a noise-tolerance device, on the grounds that an unpruned tree accommodates corrupted examples by creating small, low-support leaves for them [37]. Later studies separated class noise from attribute noise in C4.5-style learners and found that label corruption was substantially more damaging than comparable levels of feature corruption, with damage growing faster than linearly in the noise rate [38]. Subsequent work argued that classifiers under noise should be ranked by loss relative to their own clean baseline rather than by noisy accuracy alone, formalizing this with C4.5 as a reference learner [39]. Two conclusions from that literature carry into our design: robustness is properly expressed as degradation relative to a per-learner clean baseline, which is our primary estimand (Section 3.6), and the mechanism by which tree learners resist label noise is structural regularization, which is pruning in the C4.5 setting, and the growth policies and regularization defaults that differentiate the three libraries in the modern setting (Section 2.1). However, the literature does not provide a controlled comparison of modern implementations.

Most current noise-robust learning research targets deep neural networks, including robust loss functions [40,41], sample selection via small-loss criteria [42], co-training [17,43], and meta-learning-based reweighting [44]. The literature on tree ensemble methods is sparse. XGBoost was studied under symmetric and asymmetric noise on tabular data, and significant performance degradation above 20% noise was found, but without a cross-library comparison [15]. Nettleton et al. (2010) examined the noise sensitivity of various classifiers, including early boosting methods; however, their study predates the current generation of GBDT libraries [16]. Random classification noise can theoretically defeat convex potential boosters [45], although modern GBDT implementations incorporate regularization mechanisms that are not present in the early boosting algorithms.

Solutions published since this benchmark was designed have begun to close the tree-ensemble gap, although none of them is a controlled comparison of the three libraries. A 2025 survey of learning under label noise catalogues the loss-adjustment, regularization, and sample-selection families but still draws its evidence almost entirely from deep architectures [46]. Two proposals instead modify the boosting procedure itself: Robust-GBDT replaces the convex training objective with a nonconvex robust focal loss and reports gains over bagging and robust-boosting baselines when label noise and class imbalance occur together [47], while ITBoost scores each sample by the description length of its residual trajectory across boosting iterations and down-weights samples whose residuals fluctuate irregularly, rather than trusting the instantaneous gradient [48]. Both introduce a new learner, so they change the algorithm rather than hold it fixed, and neither reports how the unmodified XGBoost, LightGBM, and CatBoost defaults rank against one another, which is the question a practitioner faces before adopting any modification. In parallel, the detection literature has moved toward systematic evaluation: a decomposition of noisy-label detectors into gathering, disagreement, and aggregation components was benchmarked across vision and tabular data [49], and post hoc calibration of the scoring model was shown to improve both the accuracy and the robustness of mislabeled-instance detection [50]. A complementary 2025 proposal scores a label against the representation-space neighborhood of its own class rather than against the training loss, although it reads that representation from a network’s penultimate layer and so does not transfer directly to tree ensembles [51]. Both results bear on our beyond-accuracy analysis, in which each library’s training loss is used as a mislabel detection signal and its calibration is measured under the same noise conditions (Section 4.4).

Two computationally inexpensive mitigations are directly applicable to any GBDT library without architectural modification: (1) early stopping on a validation set, which limits overfitting to noisy labels by halting training before the ensemble memorizes corrupted samples, and (2) small-loss sample reweighting, which down-weights training samples with high loss (likely mislabeled) in a second training pass [52]. Recent theory supports both: an ICML 2025 analysis of the feature learning process shows that training passes through two stages, first fitting the clean signal and only later letting the gradient along the noise direction dominate, which is precisely the window that early stopping and small-loss selection exploit [53]. That result is derived for two-layer convolutional networks rather than for boosted trees, so we treat it as motivation for testing the two mitigations here, not as evidence that they transfer. We benchmarked both mitigations across all three libraries to determine whether the optimal mitigation strategy was library-dependent.

2.4. Statistical Comparison of Classifiers

The standard methodology for comparing classifiers across multiple datasets involves the Friedman test for omnibus differences, followed by the post hoc Nemenyi test with critical difference diagrams [19]. This framework was extended to all pairwise comparisons [20]. We adopted this methodology throughout, complemented by Wilcoxon signed-rank tests [54] for targeted pairwise comparisons and confounder checks that required observed effects to exceed twice the pooled seed standard deviation.

2.5. Research Gap

Four specific gaps emerge from the literature reviewed above. First, no controlled comparison exists among XGBoost, LightGBM, and CatBoost under label noise; the closest study covers a single library [15], the broadest classifier panel predates two of the three libraries [16], and the noise-robust boosting methods proposed since introduce modified learners rather than rank the unmodified implementations against each other [47,48]. Second, existing tree-learner robustness evidence is concentrated on label-conditional noise; therefore, it cannot establish whether a robustness ranking survives feature-dependent corruption, which is the noise process closest to real annotator behavior [30,31,32]. Third, robustness is almost always reported as accuracy alone, leaving unanswered whether the most accurate library under noise is also the best calibrated, most stable in its feature attributions, or most useful for locating mislabeled records. Fourth, the mitigation literature is written for neural architectures [17,40,41,42,43,44], so practitioners already running a GBDT library have no evidence on which cheap, library-agnostic mitigation to pair with the library. Our protocol addresses all four within a single design, in which library identity is the only free variable.

3. Materials and Methods

3.1. Datasets

We drew the datasets from the OpenML Curated Classification benchmark suite 2018 (OpenML-CC18) [55,56], which contains 72 classification datasets, and screened them in the order shown in Table 1.

Table 1. Dataset screening from OpenML-CC18.

Step	Criterion	Excluded	Remaining
0	Full OpenML-CC18 suite	-	72
1	Outside 1000–70,000 samples	19	53
2	More than 200 features	12	41
3	Flattened image–pixel representation	1	40
4	Final selection for joint coverage and native categorical and missing-value handling	25	15

Note: Steps 1 and 2 keep the full grid computationally feasible on a single workstation. Step 2 already removes the large image-pixel datasets (CIFAR_10, Fashion-MNIST, mnist_784, mfeat-pixel, and semeion), and step 3 removes optdigits, the one flattened-pixel dataset that falls within the feature cap; the exclusion is applied because this study is scoped to tabular data, and a flattened-pixel representation is not a tabular feature space. Step 4 selects from the 40 remaining candidates for successful native categorical and missing-value handling in all three libraries and for joint coverage of binary and multiclass targets; balanced and imbalanced class distributions; and numeric, categorical, and mixed feature types.

The resulting selection spanned 1000–48,842 samples, 2–26 classes, and majority class shares of 0.04–0.94. Table 2 summarizes the characteristics of the dataset.

Table 2. Dataset characteristics.

-	OpenML Identifier	n	Features	Classes	Majority Share
adult	1590	48,842	14	2	0.761
electricity	151	45,312	8	2	0.575
bank-marketing	1461	45,211	16	2	0.883
nomao	1486	34,465	118	2	0.714
letter	6	20,000	16	26	0.041
PhishingWebsites	4534	11,055	30	2	0.557
pendigits	32	10,992	16	10	0.104
jm1	1053	10,885	21	2	0.807
satimage	182	6430	36	6	0.238
phoneme	1489	5404	5	2	0.707
spambase	44	4601	57	2	0.606
sick	38	3772	27	2	0.939
kr-vs-kp	3	3196	36	2	0.522
segment	40,984	2310	16	7	0.143
credit-g	31	1000	20	2	0.700

Note: All datasets were sourced from OpenML under the Creative Commons Attribution (CC BY) license.

Categorical features are passed natively to each library (LightGBM category dtype, CatBoost cat_features, XGBoost enable_categorical), and missing values are retained as Not a Number (NaN) values for native handling by all three libraries. No one-hot encoding or imputation was performed.

3.2. GBDT Libraries and Hyperparameter Tuning

We compared XGBoost 3.2.0, LightGBM 4.6.0, and CatBoost 1.2.10. To control the capacity confound and ensure that differential noise robustness is not an artifact of differential underfitting, we tuned each (dataset, library) pair on clean training data using a 20-iteration random search [57,58], with each configuration evaluated by the macro-averaged F1 score (macro-F1) [59] on a single stratified 75/25 train/validation holdout drawn from a stratified subsample of at most 15,000 training rows. This 75/25 holdout is an inner split taken from inside the outer 60/20/20 training partition of Section 3.4 and is used only to score candidate configurations; it does not replace the outer split, and no result reported in this paper is evaluated on a 75/25 partition. The outer test partition was excluded from the hyperparameter selection. The 15,000-row figure is an upper bound rather than a fixed subsample size; therefore, it does not apply to all datasets: any dataset whose clean training data contains fewer rows is tuned on its training data in full, and only the largest datasets in Table 2 are subsampled. Because the cap is applied identically to all three libraries in a given dataset, it cannot bias the inter-library comparison. The search space covers learning rate ($\log \text{-}\mathrm{uniform} [0.01,0.3]$), tree depth ($\left\{4,\dots ,10\right\}$), minimum samples per leaf ($\left\{1,5,20,50\right\}$), subsample ratio ($\mathrm{uniform} [0.6,1.0]$), column subsample ($\mathrm{uniform} [0.6,1.0]$), and squared Euclidean norm (L2) regularization ($\log \text{-}\mathrm{uniform} [0.1,10]$). The tuned configuration per (dataset, library) was frozen and reused for each noise condition, ensuring that the hyperparameter quality was constant across conditions.

We considered $k$-fold cross-validation for the inner tuning split and rejected it on cost grounds: the tuning stage already comprises 900 fits (15 datasets $\times$ 3 libraries $\times$ 20 candidate configurations), and five-fold cross-validation would have multiplied that by five without changing what the comparison requires, which is not a maximally precise tuned configuration but an identical tuning budget and protocol across libraries. Because the tuned configuration is frozen and reused across all 13 noise conditions, any tuning error enters every condition for a given (dataset, library) pair identically and largely cancels in the degradation contrast, and the variance in the reported results is instead estimated at the outer level over three seeds. Nevertheless, we agree that a single inner split yields a noisier hyperparameter estimate on the smallest datasets, such as credit-g and segment, and we note this in Section 5.3.

Table A1 (Appendix A) lists the representative tuned hyperparameters. The clean-data validation macro-F1 scores were comparable across libraries (mean difference < 0.01), confirming that the libraries started from a matched capacity.

3.3. Noise Injection Protocol

For each main-grid experimental unit, we applied noise only to the training labels, whereas the validation and test labels remained clean. The early stopping mitigation was the sole exception; its validation labels were corrupted using the same symmetric noise setting as its training labels. The test labels remained clean in all experiments. The three noise models were implemented as follows.

In symmetric noise, each training label is independently flipped to a uniformly random different class with probability $\eta \in \left\{0.1,0.2,0.3,0.4\right\}$. The realized flip rate is equal to the target rate by the construction.

In asymmetric (pair-flip) noise, each training label of class $c$ is flipped to class $(c + 1)\mathrm{m}\mathrm{o}\mathrm{d}K$ with probability $\eta$ [26]. The transition matrix was verified to carry a zero off-pair mass. We note an important structural property: for binary datasets ($K=2$), the pair-flip construction reduces to flipping to the only other class and is therefore operationally equivalent to symmetric binary flipping at the same rate. Consequently, the distinction between symmetric and asymmetric noise is informative primarily for the four multiclass datasets in this benchmark study. We report a binary/multiclass breakdown of the asymmetric results in Section 4.2. Future studies should evaluate the class-conditional transition rates for binary tasks (e.g., unequal flip probabilities per class).

In instance-dependent noise, a logistic regression surrogate is fitted to the clean training fold (using ordinal-encoded, median-imputed features, separate from the GBDT input). The flip probability for sample $i$ is proportional to its predicted-margin uncertainty ($1-\mathrm{m}\mathrm{a}\mathrm{x}\_\mathrm{prob}$), rescaled so the realized flip rate equals the target. The flipped labels were set to the second most probable class of the surrogate. This concentrates mislabeling on decision boundary samples, as verified by the requirement that flipped samples have a mean surrogate uncertainty exceeding non-flipped samples by at least 0.05. This 0.05 margin is an acceptance threshold on the noise generator rather than a tuned parameter of the experiment, since the flip probabilities themselves are set by the surrogate’s uncertainty ranking; its value was fixed before the production run and was not varied, and we record the absence of a sensitivity analysis over it as a limitation in Section 5.3.

All noise injectors were unit-tested for correctness: realized rates within one percentage point of the target, exact transition-matrix structure, and concentration of instance-dependent flips on uncertain samples.

3.4. Experimental Protocol

The data were split into stratified 60/20/20 train/validation/test partitions per seed (seeds 42, 43, 44); this is the outer split referred to in Section 3.2, and all reported metrics were computed on the clean test partition. The validation set was used only by the early stopping mitigation; the main arm trained a fixed 500 trees without early stopping to ensure identical stopping rules across the libraries.

The main grid consisted of 15 datasets $\times$ 13 noise conditions (clean + 4 rates $\times$ 3 noise types) $\times$ 3 libraries $\times$ 3 seeds = 1755 fits. Including the mitigation arms (540 fits) and the tuned-versus-default ablation (A1; 234 fits), the full study comprised 1755 + 540 + 234 = 2529 production units (plus 900 short tuning fits) and was completed in approximately 4 h on a single consumer graphics processing unit (GPU) workstation. All unit counts reported in this study were generated programmatically from the experimental ledger.

Two mitigations were evaluated at rates of 0.2 and 0.4 under symmetric noise. M1 (early stopping) uses 50-round patience on validation log-loss with validation labels noisy, representing a realistic scenario. M2 (small-loss reweighting) fits the same library for 100 trees on the noisy training fold, computes per-sample log-loss, down-weights the top $\eta$ fraction of highest-loss samples to weight 0.1, and retrains the full model with sample weights.

The experimental design enforces three fairness rules: identical tuned hyperparameters across noise conditions, identical splits, and noise realizations across libraries (noise is a function of dataset and seed, not of library), and fixed 500 trees in the main arm.

All experiments were performed on a single workstation with Windows 11, an NVIDIA GeForce RTX 3080 (10 GB), and Python 3.10. The five largest datasets used graphics processing unit (GPU) acceleration (XGBoost device = “cuda”, CatBoost task_type = “GPU”); all other fits used a central processing unit (CPU) worker pool (six workers, OMP_NUM_THREADS = 2). The full production run (2529 units, including mitigation and ablation) was completed in approximately 4 h. The experimental framework supports ledger-based resume: each unit writes its result atomically, and interrupted runs resume by re-invoking the same command.

3.5. Metrics

The predictive performance was measured on the clean test set using macro-averaged precision, recall, and macro-F1. For a problem with $K$ classes, let ${\mathrm{TP}}_c$, ${\mathrm{FP}}_c$, and ${\mathrm{FN}}_c$ denote the numbers of true positives, false positives, and false negatives for class $c$, respectively. Per-class precision ${\mathrm{P}}_c$ and recall ${\mathrm{R}}_c$ are then given by Equations (1) and (2):

$${\mathrm{P}}_c={\displaystyle \frac{{\mathrm{TP}}_c}{{\mathrm{TP}}_c+{\mathrm{FP}}_c}}$$

(1)

$${\mathrm{R}}_c={\displaystyle \frac{{\mathrm{TP}}_c}{{\mathrm{TP}}_c+{\mathrm{FN}}_c}}$$

(2)

Their macro averages weight every class equally, irrespective of its prevalence, as in Equations (3) and (4), respectively:

$${\mathrm{P}}_{\mathrm{macro}}={\displaystyle \frac{1}{K}}{\displaystyle {\sum }_{c=1}^K}{\mathrm{P}}_c$$

(3)

$${\mathrm{R}}_{\mathrm{macro}}={\displaystyle \frac{1}{K}}{\displaystyle \sum _{c=1}^K}{\mathrm{R}}_c$$

(4)

Macro-F1 is the unweighted mean of the per-class harmonic means of precision and recall [59], as defined in Equation (5).

$${\mathrm{F}1}_{\mathrm{macro}}={\displaystyle \frac{1}{K}}{\displaystyle \sum _{c=1}^K}{\displaystyle \frac{2{\mathrm{P}}_c{\mathrm{R}}_c}{{\mathrm{P}}_c+{\mathrm{R}}_c}}$$

(5)

Macro-F1 is the headline metric because it aggregates both error types into the single per-unit scalar our rank-based tests require, and because it weights all classes equally and is therefore not dominated by the majority class on datasets whose majority share reaches 0.94. Because noise is injected into the training labels only while the test set stays clean, degradation moves macro-precision and macro-recall in the same direction rather than trading one against the other; therefore, the harmonic mean does not conceal a precision-recall trade-off.

Degradation is the drop in macro-F1 relative to the clean baseline of the same dataset and library, as defined in Equation (6) for dataset $d$, library $l$, noise type $n$, and rate $r$:

$${\mathsf{\Delta }}_{d,l,n,r}={\mathrm{F}1}_{\mathrm{macro}}^{\mathrm{clean}}(d,l)-{\mathrm{F}1}_{\mathrm{macro}}^{\mathrm{noisy}}(d,l,n,r)$$

(6)

The Expected Calibration Error (ECE) is computed using 15-bin confidence-based calibration on the clean test set. The mislabel detection performance is measured by the area under the precision–recall curve (AUPRC) for identifying mislabeled training samples using per-sample training log loss, which is evaluated against the ground-truth flip mask. This is a diagnostic measurement rather than a label-cleaning method because our noise is synthetically injected; the flip mask is available to us as an evaluation label, so the AUPRC quantifies how much mislabel information each library’s training loss carries under conditions more favorable than any deployment. Practical label-cleaning methods, such as Confident Learning [52], operate in the opposite situation, in which no flip mask exists, and the joint distribution of noisy and latent true labels must be estimated from cross-validated out-of-sample predicted probabilities. Therefore, our values should not be read as detection rates achievable in deployment or as a comparison against Confident Learning; they indicate which library loss signal supplies the most mislabel information to a cleaning pipeline of that kind, holding the pipeline fixed. Feature importance stability was measured as the Spearman rank correlation between the gain-based feature importance of each noisy unit and its clean-condition counterpart (same dataset, library, and seed).

3.6. Statistical Testing

Our primary estimand is the degradation ${\mathsf{\Delta }}_{d,l,n,r}$ defined in Equation (6). We apply the Friedman test [60] across 15 datasets at each (noise type, rate) combination, with libraries as treatments and datasets as blocks, on both per-dataset degradation (primary, matching the estimand) and mean-over-seeds noisy macro-F1 (secondary); the two analyses agree on every conclusion reported in this study. When the Friedman test rejects the null ($\alpha =0.05$), we apply the post hoc Nemenyi test to identify which pairs differ significantly, visualized via critical-difference diagrams [19,20]. Wilcoxon signed-rank tests [54] complement the analysis for targeted pairwise comparisons, including mitigation gains and beyond-accuracy metrics. Our design tests for differences; where we report a null result, we report it as the absence of statistically significant differences, together with the observed effect size, not as an established equivalence.

We also imposed a noise-floor criterion: we required that a claimed library gap exceeded twice the pooled seed standard deviation (computed from the clean condition across all seeds) before reporting it as a finding. This criterion ensured that the apparent differences were not attributable to the run-to-run variance.

Three confound checks were mandatory. The capacity confound requires that the most robust library also wins or ties on absolute noisy macro-F1 (not just relative degradation) and that a tuned-vs-default ablation (A1) on two representative datasets preserves this ranking. The noise floor confounds requires that the library gap at 40% noise exceeds $2\times$2 times the pooled seed standard deviation. The dataset-selection confound is addressed by reporting the per-dataset results in full in Figure A1 and Table A1 and Table A2.

This study was not formally preregistered; therefore, we state here which analyses were fixed in the experiment configuration and analysis plan before the production run and which were added after inspecting the results. The confirmatory analyses are the Friedman tests on per-dataset degradation for each noise type and rate with the post hoc Nemenyi procedure, the Wilcoxon tests of each mitigation against no mitigation under symmetric noise, and the three confound criteria of Section 3.6 together with the noise-floor rule above; these carry the claims of this study. The exploratory analyses are the binary versus multiclass breakdown of the asymmetric results (Section 4.2), beyond-accuracy comparisons (Section 4.4), mitigation-versus-mitigation comparisons (Section 4.5), imbalance-by-noise regression (Section 4.6), and mechanistic discussion in Section 5.1. They are hypothesis-generating, labeled as exploratory, and not corrected for multiplicity across the exploratory family; therefore, a claim resting on them requires independent confirmation.

4. Results

This section reports the results of the protocol in Section 3 in the order in which the claims depend on one another. Section 4.1 establishes the central accuracy result, and Section 4.2 subjects it to formal multi-dataset tests, including the null result under instance-dependent noise. Section 4.3 reports the three mandatory confound checks that the ranking must pass to be valid. Section 4.4 and Section 4.5 then ask whether the accuracy ranking transfers to calibration, mislabel detectability, and feature-importance stability, and how much of the noise damage two inexpensive mitigations recover. Section 4.6 tests for an interaction between the class imbalance and noise damage. Throughout, degradation is the primary estimand, and absolute noisy macro-F1 is reported alongside it so that every ranking can be read against both.

4.1. Library Robustness Under Label-Conditional Noise

The noise damage grew steeply and unevenly across the three libraries. All three lose accuracy monotonically as the noise rate increases, but not at the same rate: from rate 0.1 to rate 0.4 under symmetric noise, CatBoost’s mean degradation grows from 0.021 to 0.162, whereas LightGBM’s grows from 0.033 to 0.236 and XGBoost’s from 0.030 to 0.228. Therefore, the libraries are practically interchangeable at low noise and clearly separated at high noise, and the same pattern holds under asymmetric pair-flip noise but not under instance-dependent noise, where all three converge. Table 3 reports the absolute macro-F1 and degradation per library across noise types and rates, aggregated over the 15 datasets, with the four rate rows aligned across the three noise-type blocks so that a given rate can also be compared across the noise types. Figure 1 shows the corresponding degradation curves, in which the shaded bands denote the 95% confidence intervals across the 15 datasets.

Table 3. Macro-F1 and degradation by noise type, rate, and library.

Rate	Noise Type	CatBoost F1	LightGBM F1	XGBoost F1	CatBoost Deg.	LightGBM Deg.	XGBoost Deg.	Friedman p
0.1	Symmetric	0.860 ± 0.032	0.853 ± 0.031	0.858 ± 0.029	0.021	0.033	0.030	0.420
	Asymmetric	0.855 ± 0.032	0.851 ± 0.031	0.856 ± 0.030	0.026	0.036	0.033	0.549
	Instance	0.827 ± 0.034	0.820 ± 0.033	0.831 ± 0.031	0.055	0.066	0.058	0.006
0.2	Symmetric	0.822 ± 0.034	0.803 ± 0.033	0.815 ± 0.028	0.059	0.084	0.073	0.015
	Asymmetric	0.817 ± 0.030	0.797 ± 0.029	0.802 ± 0.028	0.064	0.090	0.086	0.127
	Instance	0.765 ± 0.033	0.751 ± 0.032	0.760 ± 0.031	0.116	0.136	0.129	0.074
0.3	Symmetric	0.779 ± 0.037	0.737 ± 0.034	0.750 ± 0.032	0.102	0.150	0.139	0.004
	Asymmetric	0.748 ± 0.033	0.715 ± 0.027	0.726 ± 0.025	0.133	0.171	0.162	0.091
	Instance	0.689 ± 0.027	0.675 ± 0.025	0.682 ± 0.026	0.192	0.212	0.207	0.127
0.4	Symmetric	0.720 ± 0.041	0.651 ± 0.033	0.660 ± 0.036	0.162	0.236	0.228	0.006
	Asymmetric	0.662 ± 0.029	0.602 ± 0.022	0.616 ± 0.020	0.219	0.285	0.272	<0.001
	Instance	0.606 ± 0.022	0.597 ± 0.020	0.603 ± 0.021	0.275	0.290	0.286	0.420

Note: Values are means across the 15 datasets, reported as macro-F1 ± the half-width of the 95% confidence interval (CI); Deg. is the degradation, macro-F1 clean minus macro-F1 noisy. The best absolute macro-F1 per row is bold. Each rate label applies to the three noise-type rows that follow it, so the three noise types can be compared directly at a common rate. The Friedman p-value tests the null of equal ranks across datasets on absolute noisy macro-F1; the primary degradation-based tests are reported in Section 4.2.

Figure 1. Macro-F1 degradation versus noise rate by library and noise type.

At low noise rates (0.1), no library demonstrates a statistically significant advantage under symmetric or asymmetric noise (Friedman $p=0.42$ and $p=0.55$ on noisy macro-F1, respectively). Under instance-dependent noise at rate 0.1, the Friedman test is significant ($p=0.006$), driven by XGBoost’s slight advantage; however, this advantage does not persist at higher rates where noise damage is substantial. As the noise rate increased, CatBoost was separated from the other libraries under both symmetric and asymmetric noise. At the highest rate (0.4), the Friedman test on per-dataset degradation rejects the null of equal ranks for symmetric noise ($p<0.001$) and asymmetric noise ($p<0.001$), with CatBoost achieving a mean rank of 1.20 in both cases (the secondary analysis on absolute noisy macro-F1 agrees: $p=0.006$ and $p<0.001$, mean rank 1.33).

Under instance-dependent noise, the picture is rate-dependent. At 0.2, the degradation-based Friedman test is significant ($p=0.008$), with CatBoost ranked best (1.47). At 0.4, however, the test fails to reject ($p=0.63$ on degradation; $p=0.42$ on noisy macro-F1). The absolute macro-F1 gap between the best- and worst-performing library at rate 0.4 is only 0.009 (CatBoost 0.606 vs. LightGBM 0.597), which falls below our predefined noise-floor threshold of twice the pooled seed standard deviation (0.018). At the highest instance-dependent noise rate, we found no statistically significant and persistent library ranking, and the observed gaps were small relative to the predefined variability thresholds. We note that this is a failure to detect differences in our design and not a demonstration of equivalence. The takeaway is that library choice matters more as the noise rate rises under label-conditional noise, and barely at all under instance-dependent noise.

4.2. Statistical Significance and Critical Differences

The rankings in Section 4.1 survive formal testing under label-conditional noise and fail under feature-dependent noise, and the point at which they fail is the most consequential result in this study: at a rate of 0.4, CatBoost is separated from LightGBM under both symmetric and asymmetric noise, whereas no pairwise gap is resolvable under instance-dependent noise. Table 4 summarizes these statistical results, and Figure 2 displays the same pairwise ranking relationships as critical-difference diagrams, in which libraries joined by a bar are not significantly different at $\alpha =0.05$.

Table 4. Friedman test results at a noise rate of 0.4.

Noise Type	CatBoost Rank	XGBoost Rank	LightGBM Rank	CatBoost Wins	Friedman p
Symmetric	1.33	2.27	2.40	11/15	0.006
Asymmetric	1.33	1.93	2.73	10/15	<0.001
Instance	1.93	1.80	2.27	7/15	0.420

Note: Mean ranks are computed across the 15 datasets on absolute noisy macro-F1 (lower is better); the Nemenyi critical difference is $\mathrm{CD}=0.86$ ($\alpha =0.05$, $k=3$, $N=15$). The primary degradation-based analysis agrees: $p<0.001$ for symmetric and asymmetric noise with CatBoost mean rank 1.20 in both cases (LightGBM 2.47 and 2.67; XGBoost 2.33 and 2.13), and $p=0.63$ for instance-dependent noise.

Figure 2. Nemenyi critical difference diagrams at a noise rate of 0.4.

Under symmetric noise, the gap between CatBoost (rank 1.33) and LightGBM (rank 2.40) was 1.07, exceeding the Nemenyi critical difference ($\mathrm{CD}=0.86$), establishing statistical significance. The gap between CatBoost and XGBoost (0.93) also exceeded the CD. Under asymmetric noise, the CatBoost-LightGBM gap (1.40) is even larger, whereas the CatBoost-XGBoost gap (0.60) does not exceed the CD. CatBoost and XGBoost cannot be distinguished at this significance level in the presence of asymmetric noise.

Under instance-dependent noise at a rate of 0.4, no pairwise gap exceeded the CD, and the Friedman test itself was non-significant. This null result is a central finding rather than an inconclusive one; it indicates that robustness conclusions are noise-model-dependent and that symmetric/asymmetric noise studies may overstate library differences relative to feature-dependent noise conditions. The disappearance is not merely a loss of statistical power. The largest inter-library gap in that condition, 0.009 macro-F1, was smaller than the 0.018 noise-floor threshold derived from seed variability; therefore, the effect was small in absolute terms as well as unresolved by the test.

Because pair-flip noise is operationally equivalent to symmetric flipping on binary datasets (Section 3.3), we additionally analyzed the asymmetric results separately for the 11 binary and four multi-class datasets used in this study. On the binary subset, asymmetric degradation at rate 0.4 is numerically almost identical to symmetric degradation (e.g., CatBoost 0.192 vs. 0.191), as the construction predicts, and the degradation-based Friedman test remains significant ($p=0.001$; CatBoost mean rank 1.18). On the multiclass subset, where pair-flip noise is structurally distinct, the direction is the same (CatBoost has the lowest mean degradation, rank 1.25), but with only four datasets, the test does not reach significance ($p=0.17$). Therefore, we present the multiclass pair-flip comparison as exploratory (Section 3.6) on two grounds: it was prompted by structural equivalence rather than planned, and it rests on four datasets. The headline asymmetric result should be interpreted as being driven largely by binary label-reversal noise. The takeaway is that the robustness ranking is statistically supported under label-conditional noise and undetectable under feature-dependent noise at the highest rate tested.

4.3. Confound Checks

The ranking of Section 4.1 and Section 4.2 passes all three mandatory confound checks, which is what licenses reporting it as a robustness difference, rather than an artifact of capacity, seed variance, or dataset selection.

The capacity confound check confirmed that CatBoost achieved the highest absolute noisy macro-F1, rather than merely the lowest degradation, in 10 of the 12 noise conditions, including all conditions at moderate and high rates (0.2–0.4) for all three noise types. The two exceptions are the lowest rate (0.1) under asymmetric and instance-dependent noise, where XGBoost is marginally the highest (Table 3). This result makes it implausible that the robustness advantage is an artifact of underfitting, while the tuned-versus-default ablation (A1) on electricity and credit-g preserves the CatBoost robustness ranking and further argues against a hyperparameter artifact.

The noise-floor confound check compares inter-library gaps against the $2\times$ pooled seed standard deviation threshold of 0.018. Under symmetric noise at a rate of 0.4, the absolute macro-F1 gap between CatBoost (0.720) and LightGBM (0.651) was 0.069, which was well above the threshold. Under asymmetric noise, the gap was 0.061, which was also above the threshold value. Under instance-dependent noise, however, the gap is only 0.009, falling below the threshold, and the non-significant test result is accompanied by an observed effect smaller than run-to-run seed variability. This makes a large undetected difference unlikely at a rate of 0.4, although with three seeds and 15 datasets, we cannot formally establish the equivalence.

The dataset-selection confound is addressed through per-dataset results (Table A2, Appendix B; Figure A1), which show that CatBoost is not uniformly superior; on some datasets (e.g., credit-g, kr-vs-kp), the libraries perform comparably. The Friedman test appropriately accounts for heterogeneity using rank-based aggregation. The takeaway is that the ranking survives all three checks, so it reflects the libraries themselves rather than capacity, seed variance, or a favorable choice of datasets.

4.4. Beyond-Accuracy Dimensions

The accuracy ranking does not transfer to the other properties on which a deployment may depend, and no single library wins on all of them. LightGBM produced the most stable feature importances under every condition tested, CatBoost’s per-sample training loss was the most informative mislabel signal under every condition tested, and the best-calibrated library changed with the noise model. Table 5 presents these three dimensions beyond predictive accuracy: probability calibration (ECE), mislabel identification (AUPRC of per-sample loss against the ground-truth flip mask), and feature importance stability (Spearman correlation with clean-condition importance). Consistent with Section 3.6, these comparisons were exploratory.

Table 5. Beyond-accuracy metrics at noise rates of 0.2 and 0.4.

Rate	Noise Type	Metric	CatBoost	LightGBM	XGBoost
0.2	Symmetric	ECE	0.109	0.065	0.077
		Mislabel AUPRC	0.745	0.642	0.683
		Importance Spearman	0.737	0.811	0.704
	Asymmetric	ECE	0.103	0.070	0.076
		Mislabel AUPRC	0.710	0.615	0.652
		Importance Spearman	0.789	0.814	0.719
	Instance	ECE	0.070	0.083	0.077
		Mislabel AUPRC	0.598	0.533	0.563
		Importance Spearman	0.786	0.874	0.762
0.4	Symmetric	ECE	0.196	0.110	0.135
		Mislabel AUPRC	0.721	0.605	0.635
		Importance Spearman	0.579	0.632	0.483
	Asymmetric	ECE	0.145	0.158	0.137
		Mislabel AUPRC	0.643	0.552	0.577
		Importance Spearman	0.684	0.715	0.482
	Instance	ECE	0.163	0.206	0.184
		Mislabel AUPRC	0.610	0.578	0.596
		Importance Spearman	0.758	0.809	0.666

Note: Values are means across the 15 datasets, with the best per row in bold. A lower expected calibration error (ECE), higher area under the precision–recall curve (AUPRC), and higher Spearman correlation are better. Each rate label applies to the nine rows that follow it, and each noise-type label to the three metric rows that follow it.

Three ranking tensions emerge from Table 5. First, regarding calibration versus accuracy, the library with the lowest accuracy degradation under label-conditional noise (CatBoost) is not the best calibrated in those conditions, although the calibration ranking itself depends on the noise model used. LightGBM achieves the lowest ECE under symmetric noise (0.065 vs. CatBoost’s 0.109 at rate 0.2; 0.110 vs. 0.196 at rate 0.4), XGBoost is lowest at high asymmetric noise (0.4), and CatBoost is lowest under instance-dependent noise at both rates. None of these pairwise calibration differences reach significance in dataset-level Wilcoxon tests at rate 0.4 (all $p>0.38$), so we present them as observed mean patterns rather than established rankings. For practitioners making thresholded decisions (e.g., accept/reject based on the predicted probability), calibration quality may be more important than accuracy [61,62,63].

Second, regarding mislabel detection, CatBoost’s per-sample training loss achieves the highest AUPRC for identifying mislabeled samples in every condition tested, and this advantage is statistically significant under symmetric noise at rate 0.4 (Wilcoxon vs. XGBoost, $p=0.012$); under asymmetric and instance-dependent noise, the same ordering holds but does not reach significance ($p=0.11$ and $p=0.28$). Therefore, CatBoost provides a promising loss signal for label-cleaning pipelines under the tested conditions [52], although the nonsignificant comparisons under asymmetric and instance-dependent noise require confirmation.

Third, regarding feature importance stability, LightGBM’s gain-based feature importances exhibited the highest mean stability under noise in every condition (highest Spearman correlation with clean-condition importances), whereas XGBoost’s importances were the least stable, particularly at high noise rates. The pairwise LightGBM-vs-CatBoost differences do not reach significance at rate 0.4 ($p\ge 0.09$ across noise types), so this too is an observed mean pattern, albeit a highly consistent one. For practitioners who rely on feature importance for model interpretation [64], this represents a practical consideration. The takeaway is that no single library is best on every dimension, so the choice should follow the property the deployment actually depends on.

4.5. Mitigation Effectiveness

Two mitigations that cost almost nothing to apply recover a substantial share of the noise damage, and they do not help the three libraries equally: the library that loses the most without mitigation gains the most from it, which compresses but does not eliminate the inter-library gap. At 40% symmetric noise, early stopping recovers 8.5 pp for LightGBM, roughly double the 4.3 points it recovers for CatBoost. Table 6 presents the effects of early stopping (M1) and small-loss reweighting (M2) at rates of 0.2 and 0.4. All results in this subsection are obtained under symmetric noise only; asymmetric and instance-dependent noise are outside the mitigation grid.

Table 6. Mitigation effectiveness under symmetric noise conditions.

Rate	Library	Main	Early Stop	Reweight	Early-Stop Gain	Reweighting Gain
0.2	CatBoost	0.822	0.837	0.848	+0.015	+0.026
0.2	LightGBM	0.803	0.837	0.828	+0.034	+0.026
0.2	XGBoost	0.815	0.838	0.843	+0.022	+0.027
0.4	CatBoost	0.720	0.762	0.765	+0.043	+0.045
0.4	LightGBM	0.651	0.735	0.702	+0.085	+0.052
0.4	XGBoost	0.660	0.712	0.714	+0.052	+0.054

Note: Macro-F1 without mitigation (main), early stopping (M1), and reweighting (M2). The gain is the mitigated macro-F1 minus the main macro-F1. Each rate label applies to the three library rows that follow.

Figure 3 complements Table 6 by showing the effects of no mitigation, early stopping (M1), and reweighting (M2) across the libraries at rates of 0.2 and 0.4.

Figure 3. Mitigation effect on macro-F1 under symmetric noise.

Several patterns emerged from these mitigation results. Reweighting yields statistically significant gains for all three libraries at rate 0.4 (dataset-level Wilcoxon vs. no mitigation: $p<0.001$ for all three), and early stopping yields significant gains for LightGBM ($p=0.005$) and CatBoost ($p=0.02$), with XGBoost marginal ($p=0.055$). In observed means, early stopping benefits LightGBM the most: at 40% noise, it recovers 8.5 pp for LightGBM, roughly double the gain for CatBoost (4.3 pp), consistent with LightGBM overfitting the noisy labels in later boosting rounds [21]. The two mitigations are statistically indistinguishable within every library at 0.4 (Wilcoxon early stopping vs. reweighting, all $p>0.12$), so we do not recommend one over the other on these data. CatBoost gained the least from both mitigations because it lost the least amount of data without mitigation. At a rate of 0.4, unmitigated CatBoost (0.720) underperforms mitigated LightGBM with early stopping (0.735) by 1.5 pp, demonstrating that inexpensive mitigation can partially close the inter-library gap. After mitigation, library differences shrink but do not disappear: CatBoost with reweighting (0.765) still outperforms LightGBM with its best mitigation (early stopping, 0.735) by three percentage points at a rate of 0.4. We emphasize that all mitigation results are under symmetric noise; the mitigation effectiveness under asymmetric and instance-dependent noise was not evaluated. The takeaway is that inexpensive mitigation recovers a substantial share of the noise damage and narrows, without closing, the gap between libraries, under symmetric noise only.

4.6. Imbalance and Noise Interaction

We tested whether class imbalance amplified noise damage by regressing degradation at a rate of 0.4 on the majority class share across the 15 datasets. Prior studies have suggested that imbalanced datasets may be more susceptible to label noise because minority class samples are already underrepresented [11]. The result shows no significant relationship: $\mathrm{slope}=0.002$, $r=0.005$, $p=0.986$. We did not observe evidence of a linear association between the majority-class share and degradation across these 15 datasets; highly imbalanced datasets (sick at 0.94, bank-marketing at 0.88) and balanced datasets (kr-vs-kp at 0.52, PhishingWebsites at 0.56) exhibited comparable noise damage relative to their clean baselines. This analysis had limited power with 15 datasets and should be considered exploratory. This null result should be read against recent work that treats imbalance and label noise as a joint problem: a 2025 systematic review and comparative analysis of imbalanced classification under label noise surveys the resampling and filtering methods designed for the two degradations together [65], and a 2026 benchmark of more than 40 imbalanced-learning algorithms over 57 tabular datasets reports that no method is consistently superior and that effectiveness depends on the data regime [66]. Our 15 datasets span majority shares from 0.52 to 0.94 but were not selected to stress that interaction. The takeaway is that we found no evidence that class imbalance amplifies noise damage, on an analysis that is exploratory and underpowered.

5. Discussion

5.1. Why Might CatBoost Resist Label-Conditional Noise?

One seemingly natural explanation, CatBoost’s ordered boosting, was ruled out by our configuration audit. We verified from the saved run configurations and library defaults that CatBoost was executed in its default plain boosting mode (boosting_type = Plain) in all experiments, on both the CPU and GPU. Therefore, ordered boosting cannot explain the robustness observed here, and our results characterize CatBoost as practitioners run it by default, not its ordered-boosting variant.

The remaining candidate mechanisms remain speculative. CatBoost differs from XGBoost and LightGBM in its use of oblivious (symmetric) decision trees, which force the same split condition across the entire tree level. This acts as a strong structural regularizer that may limit the ability of the ensemble to carve out small regions around the individual mislabeled points. The default leaf-value estimation and regularization settings of CatBoost also differ from those of other libraries. Under label-conditional noise (symmetric and asymmetric), corrupted labels are independent of features; therefore, isolating them requires the exact type of fine-grained, low-support splits that oblivious trees resist. Under instance-dependent noise, corrupted labels cluster near decision boundaries, where all libraries must place splits regardless of tree structure; this could contribute to the reduced separation between libraries observed at the highest instance-dependent noise rates.

We emphasize that these explanations are post hoc and speculative; the present benchmark does not isolate any single mechanism, and differences in regularization defaults, tree construction, and leaf estimation may all contribute to the results. Ablating individual mechanisms (e.g., enabling ordered boosting or forcing non-oblivious trees via CatBoost’s grow_policy) is a natural direction for follow-up work.

5.2. Practitioner Recommendations

No single library is the right default under noise; the correct choice depends on the noise process a deployment expects and the property that the deployment depends on. Table 7 translates our results into actionable guidelines, with the noise processes under which each recommendation was tested and the strength of its supporting evidence given as separate columns, because the scope of the evidence matters as much as its direction. Two scope limits deserve emphasis: every mitigation result in this study comes from symmetric-noise experiments, so the reweighting and early stopping recommendations must not be transferred to asymmetric or instance-dependent noise, and the beyond-accuracy recommendations rest on consistent observed means whose pairwise differences did not reach significance.

Table 7. Practical recommendations.

Scenario	Evidence-Supported Guidance	Noise Types Tested	Strength of Evidence	Evidence
Symmetric noise, accuracy priority	CatBoost; small-loss reweighting adds a further significant gain	Library choice: symmetric and asymmetric; Reweighting: symmetric only	Significant (Friedman and Nemenyi at rate 0.4; Wilcoxon p < 0.001 for reweighting)	Table 2, Table 3 and Table 5; Figure 1
Symmetric noise when using LightGBM	Apply early stopping (largest observed recovery, +8.5 pp at rate 0.4)	Symmetric only	Significant (Wilcoxon p = 0.005)	Table 6; Figure 3
Class-dependent (asymmetric) noise	CatBoost at moderate–high rates	Asymmetric	Significant at rate 0.4	Table 2 and Table 3; Figure 1
Feature-dependent (instance) noise	CatBoost showed significantly lower degradation at rate 0.2; at rate 0.4, expect no library to buy robustness	Instance-dependent	Significant at rate 0.2; no significant ranking at rate 0.4	Table 2 and Table 3
Mitigating asymmetric or instance-dependent noise	Not evaluated; no recommendation is issued	None	No evidence in this study	Section 4.5
Need to find mislabeled points	CatBoost training loss	Symmetric (significant); asymmetric and instance-dependent (same ordering, not significant)	Significant under symmetric noise only; diagnostic upper bound (Section 3.5)	Table 5; Figure A2
Need stable feature importances	LightGBM (most stable in observed means in every condition)	Symmetric, asymmetric, and instance-dependent	Observed means only; pairwise differences not significant	Table 5; Figure A2
Need calibrated probabilities under noise	Depends on noise type: LightGBM (symmetric), XGBoost (high asymmetric), CatBoost (instance-dependent)	Symmetric, asymmetric, and instance-dependent	Observed means only (all p > 0.38)	Table 5; Figure A2

Note: Every recommendation is scoped to the conditions tested; a recommendation must not be transferred to a noise process listed as not evaluated.

The central practical insight is that the “best library under noise” depends on which dimension the practitioner prioritizes and on the noise model: CatBoost retains the most accuracy under the tested label-conditional noise settings, LightGBM has the highest observed mean feature-ranking stability, and calibration patterns vary with the noise model.

5.3. Threats to Validity

Several threats to validity must be acknowledged, beginning with the fact that all noise in this benchmark was synthetically injected. Real-world label noise is more complex because it may be non-stationary, annotator-dependent, or correlated with unmeasured confounders [10,11]. The instance-dependent model is a synthetic proxy for ambiguity-dependent labeling errors because it concentrates noise on difficult samples [31,32]. However, no synthetic model can be considered fully realistic or accurate. Relatedly, 11 of the 15 benchmark datasets are binary, for which the pair-flip asymmetric construction is operationally equivalent to symmetric flipping at the same noise rate (Section 3.3). Therefore, the aggregate asymmetric-noise results should not be interpreted as strong evidence for robustness under a genuinely distinct class-conditional asymmetric-noise process. Such evidence is provided only by the four multiclass datasets, and the corresponding analysis is exploratory because of this limited sample (Section 4.2).

The instance-dependent generator also contains one constant that we fixed rather than studied: the 0.05 minimum separation in mean surrogate uncertainty between flipped and non-flipped samples. Its influence is confined to rejecting generator realizations that are insufficiently boundary-concentrated, since the flip probabilities are set by the surrogate’s uncertainty ranking; however, we acknowledge that its value is arbitrary. A sensitivity analysis over this threshold, and more generally over the strength of the instance dependence, would clarify how sharply the instance-dependent results, including the null result at rate 0.4, depend on the boundary-focused injected noise. We did not perform one.

Hyperparameter tuning was performed using clean training data, with the outer test partition excluded from model selection. The same tuned configurations were reused across the noise conditions to prevent condition-specific tuning from confounding the comparison. However, optimizing hyperparameters exclusively on clean data may favor configurations that perform well before label corruption rather than those that are most robust under noisy labels. Consequently, the absolute degradation, and potentially the relative robustness ranking, may depend on this clean-data tuning choice; noise-aware or condition-specific tuning could yield different results. Tuning used a single stratified 75/25 inner split rather than cross-validation (Section 3.2). The tuning budget and protocol were identical across libraries, which controls differences in tuning effort but does not eliminate this clean-data tuning limitation. The single inner split may also yield a noisier hyperparameter estimate on the smallest datasets, particularly credit-g and segment.

Only three random seeds were used because of computational constraints, which limits the precision of the estimated seed-to-seed variability and therefore reduces confidence in small reported inter-library differences. When reporting a finding, we also compared the observed inter-library gap with twice the pooled seed standard deviation as a descriptive noise-floor check. Where the gap does not exceed that threshold, as under instance-dependent noise at a rate of 0.4, we report the absence of a statistically significant difference rather than claiming equivalence. Additional seeds would provide more precise uncertainty estimates and greater confidence in the stability of the reported differences.

All the experiments were performed on a single RTX 3080. GPU non-determinism in CatBoost and XGBoost introduces small numerical variations [67]; we documented these in the provenance record and verified that the relative rankings were stable across re-runs, in accordance with community recommendations on reproducibility reporting for machine learning research [68].

Although OpenML-CC18 is diverse [55,56], 15 datasets from a single suite are not exhaustive. The results may differ for very high-dimensional data, time-series data, or domains with structured label-noise patterns that are not captured by the three models. Recent tabular-data surveys [7] confirm that dataset diversity is a challenge in benchmarking studies.

The practical stakes of these choices are not confined to the tabular setting studied. Robustness to noisy and imperfect training data is a recurring requirement in high-stakes decision-making domains. Our results speak only to tabular GBDT pipelines and synthetic noise, but they reinforce the general point that a clean-data ranking is not a safe proxy for behavior under imperfect labels.

Finally, the results were specific to XGBoost 3.2.0, LightGBM 4.6.0, and CatBoost 1.2.10. Future library updates may alter this relative robustness.

5.4. Implications for the Noise-Robustness Literature

The disappearance of the robustness ranking at the highest instance-dependent noise rate is the result with the broadest implications, and it deserves to be stated plainly: a library ordering that is statistically significant under symmetric and asymmetric noise, and that survives all three of our confound checks in those conditions, becomes undetectable when the same libraries are exposed to noise whose probability depends on the features of the individual sample. Much of the label-noise literature evaluates methods under symmetric noise only [12,17,18], which our results suggest may overstate the practical importance of the differences in noise robustness. We recommend that future noise-robustness studies routinely include instance-dependent noise as a minimum realism check [30,32,34] and that conclusions be qualified to the specific noise models under which they are obtained. A parallel caution applies to the beyond-accuracy dimensions. Recent work on gradient-boosting explanations identifies a path-dependent mechanism by which sequential residual fitting concentrates attribution on whichever of a group of correlated features is split on first, destabilizing importance rankings for reasons that have nothing to do with label quality, and proposes aggregation-based diagnostics and corrections for it [69]; related work adjusts attribution scores for model complexity and feature interaction with the same stabilizing aim [70]. Our importance–stability comparison holds the feature set fixed across noise conditions, so it cannot separate noise-induced instability from this correlation-induced source, and future work that varies feature correlation alongside the noise rate would be needed to do so.

6. Conclusions

We presented a controlled benchmark that compared XGBoost, LightGBM, and CatBoost under three label-noise models across 15 datasets. CatBoost exhibited measurably lower degradation under symmetric and asymmetric (label-conditional) noise; however, at the highest rate of instance-dependent noise, a synthetic proxy for ambiguity-dependent labeling errors, we found no statistically significant library ranking, and the observed gaps were small relative to the seed variability. Beyond accuracy, library rankings differ across dimensions: LightGBM feature importances are the most stable in observed means, calibration rankings depend on the noise model, and CatBoost’s loss signal is the most effective mislabel detector.

These findings lead to practical recommendations that are scoped to the conditions tested in this study. When label-conditional noise is suspected and accuracy is a priority, CatBoost produces the strongest results in this benchmark. LightGBM had the highest observed mean feature importance stability, although its pairwise differences were not statistically significant, and the calibration patterns depended on the noise model. Under symmetric noise, small-loss reweighting delivered statistically significant gains for all three libraries, and early stopping produced the largest observed recovery for the LightGBM. We did not evaluate mitigations for other noise types, and mitigation guidance should not be extrapolated beyond symmetric noise, without further evidence.

The experimental framework supports the 1755-fit main grid, per-unit artifacts, and automated statistical reporting, and it was released so that the comparison could be reproduced and extended. The two most immediate extensions are the ablation of the individual architectural and regularization mechanisms that Section 5.1 could only implicate indirectly and a mitigation grid covering asymmetric and instance-dependent noise.