DecayBench: A Reference-Free Benchmark for Trustworthy Drift Detection

Abstract

Distribution drift can substantially degrade the performance of deployed machine learning models; for example, accuracy on SST-2 can fall from $88\%$ to $58\%$. Detecting such degradation is fundamentally challenging because deployment provides inputs but not labels, so the detection itself must be reference-free. We introduce DecayBench, the first reference-free, calibrated benchmark for evaluating drift detectors. DecayBench measures detector trustworthiness along five axes (calibrated, valid, timely, no-regret, adaptive), and compares ten existing detectors across ten NLP, vision, and multimodal datasets using paired-bootstrap significance testing. Evaluation on DecayBench shows that no existing detector is uniformly optimal. Motivated by this observation, we propose Alert, a label-free aggregation rule for drift detection. Unlike all competing combiners, it uses a label-free self-configuring selection rule with a no-regret guarantee. Alert has three contributions: (i) a dilution analysis yielding a self-configuring detector selection rule; (ii) a finite-sample conformal guarantee that controls the false-alarm probability on clean data at any prescribed level (e.g., $5\%$) for arbitrary score distributions; and (iii) a no-regret result: when no single detector dominates (constituents of comparable effect size, a condition checkable offline), Alert matches or beats the best constituent, being never significantly worse and sometimes better by a large margin; this holds across NLP, NLI, and vision (ResNet), with the largest gains under multimodal drift, and the proof identifies a dominant single detector (MMD on CLIP) as the only dilution exception. We prove the no-regret property and, across the benchmark, report its empirical counterpart, non-dominance under a paired bootstrap (Alert is never significantly worse than the best constituent), which at some operating points is statistically inconclusive rather than a strict win. Because Alert combines only embedding- and logit-based detector scores, it directly transfers across NLP, vision, and multimodal models. Empirically, Alert strictly improves over single-modality monitoring, increasing AUC by up to 25 points under mixed-modality drift and by approximately 50 points under cross-modal mismatch, where individual modality-specific detectors perform near chance. Alert also matches or outperforms the Fisher, Simes, Bonferroni, and median combiners, performs best under low-severity drift, and matches or surpasses early fusion (Concat-MMD) in both multimodal settings.

1. Introduction

Deployed machine learning models operate in non-stationary environments: as the input stream drifts away from the training distribution, predictive accuracy can decay substantially while model confidence remains unchanged. In our benchmark (Table 7), a sentiment classifier on SST-2 [1] drops from $88\%$ to $58\%$ accuracy under graded drift, and a linguistic acceptability classifier on CoLA [2] drops from $77\%$ to $33\%$, without any signal in the model outputs. Such failures are therefore inherently silent.

To mitigate this risk, deployed systems rely on drift detectors, which monitor an input stream and raise an alarm once the distribution has shifted sufficiently that predictions can no longer be trusted. A correct alarm enables downstream intervention such as deferral, human review, or model updating. The fundamental difficulty is that drift detection is a reference-free statistical problem [3,4]: at deployment time, inputs are observed, but labels are unavailable, so neither detector performance nor drift severity can be directly evaluated.

This creates a second-order problem: one must not only detect distribution shift without labels, but also decide which detector to trust, despite the fact that detector quality itself cannot be observed online. Existing evaluation practice does not resolve this issue [5]. Detectors are typically compared at inconsistent thresholds, against weak or task-specific baselines, without calibrated operating points or rigorous significance testing [6,7,8]. As a result, there is no principled method for selecting drift detectors under deployment constraints.

We address this reference-free detector selection problem along two axes: benchmarking and inference.

First, we introduce DecayBench, a reference-free, calibrated benchmark for drift detection. Each detector is evaluated at a fixed operating point determined using only clean data, yielding a uniform false-positive rate constraint. This calibration corresponds to a conformal threshold and induces a finite-sample, distribution-free guarantee on the false-alarm rate. The benchmark further incorporates paired-bootstrap significance testing and evaluates detectors across ten NLP, vision, and multimodal datasets using ROC-AUC, AUPRC, precision/recall at fixed false-alarm rate, and detection delay.

Beyond standard metrics, DecayBench quantifies trustworthiness along five axes: calibrated false-alarm control (the alarm fires on clean data at most a chosen target rate, e.g., $5\%$), validity of detected drift (i.e., whether alarms correspond to real accuracy degradation), timeliness (it raises the alarm within a bounded delay after drift onset), robustness across shifts (no-regret behaviour), and adaptability to new models without labels.

Second, the benchmark reveals a structural phenomenon: after statistical control, no single detector is reliably optimal across regimes. Different detector families dominate in different settings: confidence-based methods in NLP, kernel-based tests in vision, and distance-based methods in selected shifts; none is uniformly superior across tasks, severities, or modalities.

This motivates treating drift detection as an aggregation problem rather than a single-detector selection problem: the goal is to fuse several detectors, not to crown one.

We therefore introduce Alert (Figure 1), a label-free detector aggregation rule that produces the deployment-time alarm at the end of the detection pipeline. Unlike standard combiners (e.g., Fisher [9], Simes [10], Bonferroni [11], Stouffer [12]), which weight all detectors equally and cannot drop an unhelpful one, Alert adds a label-free self-configuring step that prunes the detectors that would dilute the combination; it still aggregates the surviving detectors rather than reducing to a single one. Because its threshold is conformally calibrated, Alert inherits finite-sample false-alarm control: its false-positive rate on clean data is provably bounded at the chosen target (e.g., $5\%$) for any number of calibration batches. We further prove a conditional no-regret guarantee (Proposition A2): when the constituents have comparable effect sizes, Alert is never worse than the best constituent and can be strictly better under heterogeneous shift; when one detector’s effect size dominates, the average can be diluted below it (Corollary A2), which is exactly what the self-configuring selection targets.

Empirically, when no single detector dominates, Alert matches or exceeds every individual detector (Table 9) and every p-value combiner (Table 5). It is particularly strong under low-severity drift, where detection is most critical (Table 9). In multimodal settings, it matches or surpasses early fusion (Concat-MMD), reaching AUCs of $1.000$ vs. $0.892$ under compound drift (Appendix E) and $1.000$ vs. $0.970$ under cross-modal mismatch (Table 13). It is no-regret wherever no single detector dominates: on NLP, it is never significantly worse than the best single detector and often better at low severity (Table 9); on vision, it transfers unchanged and, on ResNet where the detectors are on par, matches or beats the best single detector, while on CLIP, where MMD dominates, MMD leads at low severity (Table 12); and in multimodal settings fusion strictly wins, significantly beating the best single-modality monitor (paired bootstrap, $p<0.001$) even when each modality alone falls to near chance (Table 13).

Summary of contributions.

• DecayBench: A reference-free, calibrated benchmark for drift detection with conformal false-alarm guarantees (Proposition A1), statistical significance testing (Section 4.4), and cross-modal evaluation (Table 13).
• Alert: A label-free, self-configuring aggregation rule with finite-sample calibration (Proposition A1) and a no-regret guarantee (Proposition A2).
• Trustworthiness measurement: A definition of reference-free drift-detection trustworthiness (Definition 3), made measurable along five axes (calibrated false-alarm control, validity, timeliness, no-regret, and adaptivity) under the label-free precondition (Section 4.4).
• Empirical results: A significance-tested study showing that drift detectors are regime-dependent rather than uniformly optimal (Table 4), with consistent Alert gains across NLP (Table 9), vision (Table 12), and multimodal settings (Table 13), especially under low-severity (Table 9) and cross-modal drift (Table 13).

The remainder of this paper is organised as follows. Section 2 reviews related work on reference-free quality estimation, drift and out-of-distribution detection, and how such detectors are evaluated. Section 3 sets up reference-free drift detection. Section 4 introduces DecayBench: its typed and graded drift streams, its calibrated no-peeking operating point with a conformal false-alarm guarantee, its detection metrics, and its trustworthiness criterion. Section 5 presents Alert, a label-free combiner, with its dilution analysis, conformal inheritance, and self-configuring selection rule. Section 6 presents the empirical study across NLP, vision, and multimodal models: it introduces the deployed models, the datasets, and the ten benchmarked detectors, evaluates the single detectors, Alert, and competing combiners, and discusses the implications (Section 6.6). Section 7 concludes, and Section 8 covers limitations and future work.

2. Related Work

2.1. Drift and OOD Detection

A large body of literature develops detectors for distribution shift, but not their rigorous evaluation. Confidence-based methods derive anomaly scores directly from a classifier’s outputs, including maximum softmax probability [13], ODIN [14], energy [15], and later post hoc variants such as ReAct, ViM, and deep nearest neighbours [16,17,18], as well as Mahalanobis feature-density methods [19]; see Yang et al. [6] for a survey. Distributional approaches instead compare batches using kernel two-sample tests such as MMD [20], classifier two-sample tests [21], and black-box shift detection [5,22]. For NLP, DriftLens [23] compares class-conditional embedding distributions, while reconstruction-based monitors learn generative models of in-distribution data [24], despite known limitations for OOD detection [25]. Text-specific OOD work studies transformer representations [26,27,28] and confidence calibration [29,30]. Our work is complementary: rather than proposing another detector, we evaluate ten label-free detectors drawn from these families (MSP, ODIN, Energy, MMD, BBSD, Mahalanobis, KNN, DriftLens, an industrial cosine baseline, and the reconstruction monitor Recon), all computable from a forward pass of the deployed model, under one calibrated protocol (Section 4).

2.2. Evaluating Drift Detectors

Monitoring non-stationary data streams has long been studied [7,31], yet evaluation remains fragmented. OpenOOD [32] standardises primarily vision OOD detection, while WILDS [33] measures post-shift accuracy rather than label-free detection. The seminal “Failing Loudly” study [5] compares shift detectors but does not use calibrated operating points or significance testing, and conformal test martingales [34] provide sequential change detection rather than a general evaluation benchmark. Benchmarks named for “drift” address different problems entirely, including databases [8,35] and LLM agents [36]. By contrast, DecayBench is, to our knowledge, the first released benchmark for reference-free drift detection combining calibrated fixed-FPR evaluation, significance testing, detection delay, typed and graded drift streams, and multimodal coverage (

Table 1. Benchmark Comparison. Only DecayBench pairs reference-free detection with calibration, significance, delay, and multimodal coverage. DecayBench versus prior benchmarks and evaluation suites; ✓ present, ∼ partial, × absent. Benchmarks named for “drift” (DriftBench, NeurBench, Drift-Bench, driftbench) target databases, agents, or industrial processes, a different task; among the detection-oriented suites, only DecayBench pairs reference-free detection with a calibrated fixed-FPR operating point, significance testing, detection delay, typed and graded drift streams, and multimodal coverage, plus a released harness. Calib. FPR = a no-peeking threshold set on clean data to a target false-positive rate; Typed/graded = multiple drift types at controlled severity.

Benchmark/Suite	Domain	Ref.- Free	Calib. FPR	Signif.	Delay	Typed/ Graded	Multi- Modal	Released
OpenOOD [32]	vision OOD	✓	∼	×	×	∼	×	✓
WILDS [33]	in-the-wild	×	×	×	×	×	∼	✓
Failing-Loudly [5]	shift detect.	✓	×	×	×	∼	×	∼
Conf. martingale [34]	sequential CP	✓	✓	∼	✓	×	×	×
DriftBench [8]	DB workload	×	×	×	×	✓	×	✓
NeurBench [35]	DB components	×	×	×	×	✓	×	✓
Drift-Bench [36]	agent dialogue	×	×	×	×	✓	×	✓
DecayBench (ours)	NLP/vision/MM	✓	✓	✓	✓	✓	✓	✓

).

2.3. Reference-Free Reliability Estimation and Score Combination

Drift detection is closely related to reference-free quality estimation, which predicts model reliability without labels [3,4]. Likewise, combining evidence from multiple label-free statistics has a long history, from Fisher’s and Stouffer’s methods [9,12] to recent conformal and OOD combination schemes [34,37,38]. We build directly on these ideas rather than introducing a new combination rule. Our contribution is a calibrated benchmark demonstrating that, across heterogeneous detector families, no individual detector is consistently best and a simple label-free combiner is empirically no-regret whenever no single detector dominates.

3. Problem Setup

3.1. Distribution Drift

The deployed classifier $f_{\theta }$ acts on inputs x in an input space $\mathcal{X}$, producing for each input a logit vector $\ell \left(x\right)\in {\mathbb{R}}^K$ over the K class labels and a [CLS] embedding $e\left(x\right)\in {\mathbb{R}}^d$ of dimension d. Write $P_0$ for the in-distribution (training) input marginal, the reference against which drift is measured. At deployment, the model sees a stream of input batches $B_1,B_2,\dots$, each a set $B_t=\{x_1,\dots ,x_n\}$ of n inputs drawn from a batch marginal $P_t$.

Definition 1

(Distribution drift). A batch $B_t\sim P_t$ exhibits drift relative to $P_0$ when $P_t\ne P_0$. We model the stream with an unknown onset $t^{★}$: $P_t=P_0$ for $t<t^{★}$ and $P_t\ne P_0$ for $t\ge t^{★}$, possibly with graded severity. The drift is consequential when it raises the deployed classifier’s error against the true label y, ${\mathbb{E}}_{P_t}[⊮(f_{\theta }\left(x\right)\ne y)]>{\mathbb{E}}_{P_0}[⊮(f_{\theta }\left(x\right)\ne y)]$.

Definitions 1 and 2 formalise standard notions from the dataset-shift and reference-free out-of-distribution detection literature reviewed in Section 2 [5,6]; the trustworthiness axes of Definition 3 are, by contrast, introduced in this work. Two instances of Definition 1 bound the detectors we study. Data (input-marginal) drift changes the input distribution directly, $P_t\left(x\right)\ne P_0\left(x\right)$ (new vocabulary, noise, domain change); this is the silent–accuracy–loss regime and our main focus. Representation drift is a meaning-preserving map (such as paraphrase) under which the label posterior $P(y\mid x)$ is essentially unchanged, yet the induced distribution of the embedding $e\left(x\right)$ shifts. Detectors differ in which of these they can see, a distinction the typology probe (Appendix C.1) makes concrete.

3.2. Drift Detection

Definition 2

(Reference-free drift detector). A drift detector is a map g from a batch to a scalar score $g\left(B_t\right)\in \mathbb{R}$, together with a threshold τ, that raises an alert on $B_t$ when $g\left(B_t\right)>\tau$. It is reference-free (label-free) when $g\left(B_t\right)$ depends only on the inputs $\left\{x_i\right\}$ and the deployed model $f_{\theta }$ (their embeddings $e_i$ and logits ${\ell }_i$), using no test labels, and τ is calibrated on clean ($P_0$) data alone.

The detection problem is therefore to choose g and $\tau$ so that the alert fires under consequential drift ($t\ge t^{★}$) but rarely on clean batches, all without deployment labels. The calibration of $\tau$ and the metrics that score g are defined in Section 4; the ten detectors we benchmark and the deployed models are introduced with the experimental setup (Section 6).

4. DecayBench: Evaluation Framework

DecayBench provides a calibrated benchmark for reference-free drift detection. It combines (i) labelled, typed, and graded drift streams; (ii) no-peeking calibration with finite-sample false-alarm guarantees; (iii) standard detection metrics, including detection delay; and (iv) a measurable notion of trustworthy drift detection.

4.1. Typed, Graded Drift Streams

We evaluate detectors on streams whose drift status is known exactly. Starting from held-out clean validation data, we generate drifted batches by applying deterministic character- and word-level perturbations at severity

$$\rho \in \{0.01,0.02,0.03,0.05,0.10,0.15,0.20\},$$

where $\rho$ is the fraction of perturbed tokens. Clean batches receive label 0, and perturbed batches receive label 1. Low $\rho$ produces subtle drift, while larger $\rho$ yields progressively easier detection. Reporting performance across the full severity range avoids conclusions based only on easy drift.

4.2. No-Peeking Calibration

All detectors are evaluated at the same operating point. For each detector, we estimate a threshold $\tau$ using only held-out clean batches, never drifted or future data, by taking the $(1-\alpha )$ quantile of the clean score distribution ($\alpha =5\%$ throughout). This fixes the deployment false-positive rate before evaluation. Throughout, calibrated is used in this conformal sense, controlling the false-alarm (false-positive) rate at the target level $\alpha$, and not in the distinct sense of confidence calibration, the agreement between a model’s predicted class probabilities and its empirical accuracy; only the former is studied here.

This calibration is exactly split conformal, giving a finite-sample, distribution-free guarantee that the clean false-positive rate does not exceed $\alpha$ under exchangeability [34,38]. Because the guarantee depends only on score ordering, it also applies to the combined detector Alert. The proof is given in Appendix F (Proposition A1, Corollary A1).

4.3. Accuracy Metrics

For each detector, and drift severity, over a held-out set of clean (label 0) and drifted (label 1) batches with scores $g\left(B\right)$ (larger means stronger evidence of drift), we report four standard detection–accuracy metrics. ROC-AUC [39] $=\mathrm{Pr}[g\left(B^{+}\right)>g\left(B^{-}\right)]$ is the threshold-free probability that a drifted batch outscores a clean one ($0.5$ for a useless detector); AUPRC [40] summarises the precision–recall curve and is more informative under class imbalance. At the calibrated threshold $\tau$ (Section 4.2), we report precision, recall, and their harmonic mean F1 [41]. Detection delay [42] is the index of the first alerted batch as severity ramps from 0 (lower is faster; undefined if the detector never alerts). All metrics are averaged over three seeds, and reporting all four prevents a single favourable number from standing in for the whole operating curve. Algorithm 1 states the full procedure.

Algorithm 1: DecayBench protocol: calibrated drift detection and evaluation

4.4. Trustworthiness Metrics

Accuracy metrics rank detectors; they do not say whether a detector is trustworthy to deploy. Making “trustworthy” precise and measurable is one of this paper’s contributions: a reference-free drift detector is trustworthy to the degree that it satisfies five testable axes, each answering a concrete deployment question. Prior evaluations test at most one or two of these in isolation; DecayBench makes all five measurable at once.

Definition 3

(Trustworthiness metrics). A drift detector operates label-free, using no test labels at deployment; this is the precondition that defines the reference-free setting. Within it, the detector (and its deployment-time alert) is trustworthy to the degree that it is (i) calibrated, holding the false-alarm rate on clean data at a controlled target; (ii) valid, in that the drift it flags is consequential, associated with real accuracy loss; (iii) timely, alerting within a bounded delay of drift onset; (iv) no-regret, never significantly worse than the individual signals it is built from (proved as Proposition A2; reported empirically as non-dominance under a paired bootstrap); and (v) adaptive, configurable from clean data so it degrades gracefully across tasks and backbones.

Notation. A detector g maps a batch B to a scalar score $g\left(B\right)\in \mathbb{R}$ (larger means stronger evidence of drift) and alerts when $g\left(B\right)$ exceeds the calibrated threshold $\tau$ of Section 4.2; write $A_g\left(B\right)=⊮[g\left(B\right)>\tau ]\in \{0,1\}$ for that alert, with $⊮[·]$ the indicator. Clean batches are drawn $B\sim P_0$ and drifted batches $B\sim P_{\rho }$ at severity $\rho >0$; ${\mathrm{Pr}}_{B\sim P_0}$ is the probability over a clean batch and ${\mathbb{E}}_{P_0},{\mathbb{E}}_{P_{\rho }}$ are expectations under the clean and drifted batch distributions. Let $a\left(B\right)\in [0,1]$ be the deployed classifier’s accuracy on B, $\mathrm{AUC}\left(g\right)\in [0,1]$ its detection ROC-AUC, and $\alpha \in (0,1)$ the target false-positive rate. Each axis is then a testable property, answering a deployment question and established by a specific measurement.

• Calibrated. Can an operator trust and budget the alarm? The clean false-alarm rate is held at the target,

  $${\mathrm{Pr}}_{B\sim P_0}\left[A_g\left(B\right)=1\right]\le \alpha (\alpha =0.05),$$

  fixing $\tau$ on clean data alone (Section 4.2); we ground this in a finite-sample conformal guarantee (Proposition A1) that the combiner inherits (Corollary A1). Calibration plays a dual role, both an axis and the common operating point at which the others are read.

• Valid. Does an alert mean something has actually gone wrong? The flagged drift is consequential: deployed accuracy is strictly decreasing in severity,

  $${\mathbb{E}}_{P_{\rho }}\left[a\right]<{\mathbb{E}}_{P_0}\left[a\right]\mathrm{for} \mathrm{every} \rho >0,$$

  verified by the accuracy–decay study (Section 6.4.2).

• Timely. Does the alert arrive in time to act? Given a severity ramp ${\left(B_t\right)}_{t\ge 1}$ (a stream of batches with drift onset at $t=1$) and a delay budget $D_{\max }$, the detection delay is bounded,

  $$\mathcal{D}\left(g\right)=\mathbb{E}\left[\min \{t\ge 1:A_g\left(B_t\right)=1\}\right]\le D_{\max },$$

  measured in Section 6.4.3.

• No-regret. Is the deployed monitor a dependable default? For M constituent detectors $g_1,\dots ,g_M$, their combiner $g^{★}=Alert$, and a tolerance $\epsilon \ge 0$, the regret $\mathcal{R}$ (the AUC gap to the best constituent) is non-positive up to significance,

  $$\mathcal{R}=\underset{1\le k\le M}{\max }\mathrm{AUC}\left(g_k\right)-\mathrm{AUC}\left(g^{★}\right)\le \epsilon ,$$

  with a paired bootstrap failing to reject $\mathrm{AUC}\left(g^{★}\right)\ge \mathrm{AUC}\left(g_k\right)$ for every k (Table 9; empirically $\epsilon \to 0$).

• Adaptive. Will it still work on the next task or backbone? A label-free selector S maps a deployment j’s clean data $C_j$ to a detector subset $g_{S\left(C_j\right)}$; writing ${\mathrm{AUC}}_j$ for detection ROC-AUC on deployment j, re-selecting is never worse than transferring a family chosen on a different deployment $j^{\prime }$,

  $${\mathrm{AUC}}_j\left(g_{S\left(C_j\right)}\right)\ge {\mathrm{AUC}}_j\left(g_{S\left(C_{j^{\prime }}\right)}\right)(j^{\prime }\ne j),$$

  realised by the label-free contextual selection (Section 5.3).

The axes are not new in isolation: they synthesise established notions, calibrated coverage from conformal prediction [34,38], no-regret from online learning [43], and bounded detection delay from change-point detection [42]. The contribution is to make all five simultaneously measurable and label-free for drift detection.

5. Alert: A Calibrated Label-Free Combiner

Alert is our deployment-time method. It does not introduce a new drift signal; instead, it combines existing detectors into a single calibrated drift alert. The combination rule itself is classical. Our contributions are threefold: (i) showing that the combined detector inherits the finite-sample calibration guarantee of Section 4.2; (ii) establishing conditions under which the combiner is no-regret; and (iii) introducing a label-free rule that automatically selects which detectors to combine. Formal results are given in Appendix F.

5.1. Combining Complementary Detectors

Different detectors capture different manifestations of drift. MMD measures changes in embedding distributions; energy and ODIN measure predictive confidence; Recon measures reconstruction error; and DriftLens measures changes in class-conditional embedding geometry.

Alert first standardises every detector using only the clean calibration batches: writing $s_k=g_k\left(B\right)$ for the k-th detector’s score on batch B,

$$z_k={\displaystyle \frac{s_k-{\mu }_k}{{\sigma }_k}},$$

where $({\mu }_k,{\sigma }_k)$ are the mean and standard deviation of $s_k$ on clean data. The standardised scores (or their equivalent clean-calibrated p-values) are then combined into a single statistic; Alert’s default is their mean (equivalently Stouffer’s Z), and we compare it against the median, Fisher, Simes, and Bonferroni rules in Section 6.4.4. This follows the classical multiple-testing framework of Fisher [9] and Stouffer [12], applied here across heterogeneous detector families.

5.2. Why Combining Helps

When the constituent detectors carry complementary information, averaging their standardised scores reinforces the shared drift signal while reducing independent noise, improving detection over any single detector. When the constituents have comparable effect sizes, this yields a no-regret combiner; on the NLP benchmark, Alert is never significantly worse than its best constituent, and is significantly better where their signals are complementary (Table 9). When one detector’s effect size instead dominates, the average can fall below it (Corollary A2). Empirically, no-regret holds wherever no single detector dominates: NLP (Table 9), NLI and vision (Table 12), and multimodal drift, where the gain is largest; CLIP single-modality drift, where MMD dominates, is the predicted dilution exception. Because the alert threshold is calibrated exactly as in Section 4.2, the combined detector inherits the same finite-sample false-positive-rate guarantee.

The principal failure mode is dilution: including detectors with little or no drift signal can weaken the combined statistic. Appendix F (Propositions A2 and A3, Corollary A2) formalises these results under a location-shift model, deriving the $\sqrt{M}$ signal amplification, its extension to correlated detectors, and the precise condition under which dilution occurs. Section 6.4.4 confirms these predictions empirically.

5.3. Label-Free Detector Selection

The theoretical analysis suggests that combining every available detector is unnecessary and can even be harmful. We therefore make Alert self-configuring by selecting detectors using only clean calibration data.

For each detector, we estimate its self-detectability: the ROC-AUC separating clean calibration batches from lightly perturbed copies generated by the benchmark itself. This procedure requires no deployment labels and never accesses the evaluation stream.

We then combine only the top-M detectors by self-detectability (we report $M\in \{3,5\}$). Intuitively, self-detectability estimates a detector’s effect size under small perturbations, so retaining high-scoring detectors while removing near-zero ones maximises the combiner’s signal-to-noise deflection and avoids the dilution characterised in Appendix F (Proposition A3, Corollary A2). Section 6.4.5 evaluates this selection rule. Algorithm 2 states the full procedure: label-free selection, clean standardisation, mean-z combination, and the conformal threshold it inherits from Section 4.2.    

Algorithm 2: Alert: label-free self-configuring drift alert

6. Experiments

We evaluate the paper’s two contributions: DecayBench, a calibrated benchmark for reference-free drift detection, and Alert, a label-free combiner of existing detectors. Unless otherwise stated, we evaluate ten label-free detectors on SST-2 and CoLA under seven drift severities and three random seeds, using the full validation split in batches of size 64; thresholds are calibrated to a $5\%$ false-positive rate on held-out clean batches only. We further test transfer to natural domain shift, a three-class NLI task (MNLI), and vision and multimodal backbones.

6.1. Datasets and Deployed Models

The primary deployed models are the GLUE [44] tasks SST-2 [1] and CoLA [2], each with a fine-tuned bert-base-uncased classifier [45] monitored by the detectors; the training split defines the in-distribution reference, and the validation split provides the clean stream from which drifted streams are constructed. DecayBench additionally covers natural domain shift (IMDB [46], Yelp [47], and Amazon [48] sentiment), a three-class sentence-pair task (MNLI [49]), and modality transfer to vision (CIFAR-10-C [50,51], STL-10 [52]) and multimodal (CLIP [53] image-caption) backbones, ten datasets in all (

Table 2. Benchmark Datasets. The ten datasets in DecayBench, spanning NLP, vision, and multimodal sources. All are scored under the same calibrated, reference-free protocol.

Dataset	Modality/Domain	Role in DecayBench	Source
SST-2	NLP, sentiment (GLUE)	Primary task; graded synthetic drift	[1]
CoLA	NLP, acceptability (GLUE)	Primary task; graded synthetic drift	[2]
MNLI	NLP, 3-class NLI	Task/label-space transfer	[49]
IMDB	NLP, sentiment	Natural domain shift	[46]
Yelp	NLP, sentiment	Natural domain shift	[47]
Amazon	NLP, sentiment (by year)	Natural/temporal domain shift	[48]
CIFAR-10-C	Vision, image	Modality transfer; corruption drift	[50,51]
STL-10	Vision, image	Modality transfer	[52]
CIFAR-10 (CLIP pairs)	Multimodal, image-caption	Cross-modal fusion drift	[53]
Flickr8k	Multimodal, real pairs	Cross-modal, real-pair check	[54]

), introduced where they are used in the experiments that follow.

6.2. Baseline Detectors

We evaluate ten existing label-free detectors spanning five families (

Table 3. Baseline Detectors. The ten benchmarked detectors. All are prior work and read only the deployed model’s logits or embeddings; only Recon is trained. Score definitions are in Appendix B.

Detector	Family	Reads	Trained	Source
MSP	confidence	softmax	no	[13]
ODIN	confidence	logits	no	[14]
Energy	confidence	logits	no	[15]
MMD	distributional	embeddings	no	[20]
BBSD	distributional	softmax	no	[5]
Mahalanobis	feature-density	embeddings	no	[19]
KNN	feature-density	embeddings	no	[18]
DriftLens	per-class emb.	embeddings	no	[23]
Cosine baseline	industrial	embeddings	no	SageMaker-style
Recon	reconstruction	embeddings	yes	[24]

); each maps a batch $B_t$ to a scalar drift score using only the deployed classifier $f_{\theta }$ and its [CLS] embeddings $e_i\in {\mathbb{R}}^{768}$ (the bert-base-uncased hidden size) or logits ${\ell }_i\in {\mathbb{R}}^K$ (over the K classes), and only Recon [24] is trained (an auxiliary autoencoder). All are established prior work; we claim none as our own. Our contributions are the benchmark (Section 4) and Alert, a label-free combiner that aggregates these signals into one deployment-time alert (Section 5.1). Full score definitions, the Recon autoencoder and its drift-score algorithm, and Recon’s three-signal typology decomposition are in Appendix B; no signal is privileged, and all are scored identically under the protocol of Section 4.

6.3. Label-Free Detector Performance

Table 4. Motivation to Combine Detectors. No single detector wins; trained Recon never beats training-free MMD and energy. Detection ROC-AUC by task and severity $\rho$ (mean over three seeds); bold = best per row. Mahalanobis is anti-correlated under token noise but works on natural shift (Table 11); precision/recall at the calibrated FPR in Table 6, AUPRC in Table A6.

Task	$\mathit{\rho }$	Recon	Cosine	MMD	Mahalanobis	BBSD	Energy	MSP
SST-2	$0.01$	$0.683$	$0.352$	$\mathbf{0.835}$	$0.385$	$0.784$	$0.809$	$0.733$
	$0.02$	$0.872$	$0.352$	$\mathbf{0.923}$	$0.234$	$0.897$	$0.916$	$0.890$
	$0.03$	$0.888$	$0.311$	$\mathbf{0.978}$	$0.183$	$0.963$	$0.963$	$0.934$
	$0.05$	$0.947$	$0.300$	$\mathbf{1.000}$	$0.169$	$\mathbf{1.000}$	$\mathbf{1.000}$	$\mathbf{1.000}$
CoLA	$0.01$	$0.932$	$0.687$	$0.766$	$0.086$	$0.799$	$\mathbf{0.958}$	$0.917$
	$0.02$	$0.997$	$0.766$	$0.935$	$0.031$	$0.961$	$0.992$	$0.984$
	$0.03$	$\mathbf{1.000}$	$0.823$	$0.997$	$0.005$	$\mathbf{1.000}$	$\mathbf{1.000}$	$0.997$
	$0.05$	$\mathbf{1.000}$	$0.932$	$\mathbf{1.000}$	$0.003$	$\mathbf{1.000}$	$\mathbf{1.000}$	$\mathbf{1.000}$

reports detection ROC-AUC across tasks and severities and exposes the benchmark’s central finding: no single detector wins. Training-free MMD and energy are the strongest overall and consistently match or beat the trained reconstruction monitor, while the industrial cosine baseline collapses to near-random; yet, which method leads still flips with task and severity. The full severity curves (Figure A2), the calibrated-operating-point metrics (Table 6), and three further signals (DriftLens, ODIN, KNN) all tell the same story. This instability, not any fixed winner, is what motivates a combiner.

Table 5. Accuracy. Alert (mean-z) matches or beats every label-free combiner on SST-2. ROC-AUC over {energy, MMD, Recon} across four severities; the small spread shows calibration and correlation matter more than the aggregation rule. Paired-bootstrap tests (${10}^4$ resamples) confirm Alert is never significantly worse than any combiner and significantly exceeds Fisher and Bonferroni at low severity ($\rho =0.01$–$0.02$, $p<0.05$), while tying the median rule. Fisher, Simes, Bonferroni are conformal p-value rules [34,38]; bold = best per row.

SST-2	Median	Fisher	GLRT	Bonferroni	Simes	Alert (Mean-z, Ours)
$\rho =0.01$	$\mathbf{0.861}$	$0.806$	$0.826$	$0.731$	$0.793$	$\mathbf{0.861}$
$\rho =0.02$	$0.949$	$0.945$	$0.940$	$0.854$	$0.943$	$\mathbf{0.971}$
$\rho =0.03$	$0.982$	$\mathbf{0.993}$	$\mathbf{0.993}$	$0.923$	$0.978$	$\mathbf{0.993}$
$\rho =0.05$	$\mathbf{1.000}$	$\mathbf{1.000}$	$\mathbf{1.000}$	$0.929$	$\mathbf{1.000}$	$\mathbf{1.000}$

illustrates why Alert works: on SST-2, its mean-z rule matches or beats every classical combiner (Fisher, Simes, Bonferroni, GLRT, median), so the advantage comes from calibration and correlation structure, not a bespoke aggregation. The location-shift model of Proposition A2 predicts the observed gains, with the correlation-corrected form closest to the measured AUC (

Table A7. The location-shift model predicts the observed combiner AUC (Proposition A2). The correlation-corrected form is closest where detectors are correlated (CoLA).

Task	Severity	Observed	Pred (Indep.)	Pred (Corr.)
SST-2	$\rho =0.01$	$0.861$	$0.833$	$0.796$
	$\rho =0.02$	$0.971$	$0.973$	$0.950$
CoLA	$\rho =0.01$	$0.901$	$0.987$	$0.937$
	$\rho =0.02$	$0.982$	$1.000$	$0.999$

); fusion gain is thus governed by redundancy among detectors, not their count.

6.4. Trustworthiness

Beyond detection accuracy, we evaluate every detector and Alert on the five trustworthiness axes of Definition 3, in their defining order: calibrated false-alarm control, validity, timeliness, no-regret, and adaptivity.

6.4.1. Calibrated

Table 6. Trustworthiness: Calibrated. Alert and the training-free leaders stay accurate at the calibrated $5\%$ FPR. Precision/recall/F1 at drift severity $\rho =0.05$ (mean over three seeds); Alert reaches perfect precision and recall at a measured $0.0$ false-positive rate. Bold = best F1 per task.

Task	Detector	Precision	Recall	F1
SST-2	Recon	$0.97$	$0.85$	$0.90$
	MMD	$0.93$	$1.00$	$0.96$
	Energy	$1.00$	$0.95$	$0.97$
	Alert (ours)	$1.00$	$1.00$	$\mathbf{1.00}$
CoLA	Recon	$1.00$	$1.00$	$\mathbf{1.00}$
	MMD	$0.87$	$1.00$	$0.93$
	Energy	$1.00$	$1.00$	$\mathbf{1.00}$
	Alert (ours)	$1.00$	$1.00$	$\mathbf{1.00}$

shows the leading detectors and Alert stay accurate at the calibrated operating point: at the common $5\%$ false-positive target fixed on clean data alone, training-free MMD and energy reach precision and recall at or near $1.0$ on both tasks, and Alert attains perfect precision and recall at a measured $0.0$ false-positive rate, so the no-peeking calibration that gives every detector one controlled threshold (the conformal guarantee of Proposition A1) costs nothing in detection quality.

6.4.2. Valid

Table 7. Trustworthiness: Valid. Injected drift is consequential: accuracy decays steeply with severity. Deployed-classifier accuracy under graded drift (true labels, mean over three seeds; used only for this check).

Task	Clean	$0.01$	$0.02$	$0.03$	$0.05$	$0.10$	$0.15$	$0.20$
SST-2	$0.88$	$0.88$	$0.85$	$0.84$	$0.80$	$0.71$	$0.62$	$0.58$
CoLA	$0.77$	$0.73$	$0.69$	$0.66$	$0.59$	$0.44$	$0.38$	$0.33$

shows the injected drift is consequential: deployed accuracy falls monotonically with severity, from $0.88$ to $0.58$ on SST-2 and from $0.77$ to $0.33$ on CoLA, with comparable decay under natural domain shift (IMDB, Yelp). A benchmark is informative only when its perturbations cause genuine degradation, so every detection score below reflects real accuracy loss rather than cosmetic noise. True labels are used for this check alone, never by any detector.

6.4.3. Timely

Table 8. Trustworthiness: Timely. Alert is among the fastest detectors to raise an alert. Each number is the number of batches to first alert under a severity ramp (mean ± std, three seeds; lower = faster; bold = fastest). ${13}^{†}$ = never fired within the $L=13$ stream.

Detector	SST-2		CoLA
	$\mathit{\rho }=\mathbf{0.01}$	$\mathit{\rho }=\mathbf{0.03}$	$\mathit{\rho }=\mathbf{0.01}$	$\mathit{\rho }=\mathbf{0.03}$
Recon	$2.67\pm 0.94$	$1.67\pm 0.47$	$1.33\pm 0.47$	$1.00\pm 0.00$
MMD	$2.00\pm 0.00$	$1.67\pm 0.47$	$1.00\pm 0.00$	$1.00\pm 0.00$
Energy	$2.00\pm 0.00$	$2.00\pm 0.00$	$1.00\pm 0.00$	$1.00\pm 0.00$
MSP	$2.67\pm 0.94$	$2.00\pm 0.00$	$1.00\pm 0.00$	$1.00\pm 0.00$
Mahalanobis	$5.67\pm 1.25$	$9.00\pm 5.66$	${13}^{†}$	${13}^{†}$
SageMaker (cosine)	${13}^{†}$	${13}^{†}$	${13}^{†}$	${13}^{†}$
Alert (ours)	2.00 ± 0.00	$\mathbf{1.33}\pm \mathbf{0.47}$	$\mathbf{1.00}\pm \mathbf{0.00}$	$\mathbf{1.00}\pm \mathbf{0.00}$

shows that the strongest detectors also alert earliest, and Alert stays in the fastest tier on both tasks, so its accuracy gains cost nothing in timeliness.

6.4.4. No-Regret

Table 9. Trustworthiness: No-regret. Alert consistently matches or exceeds individual detectors. (a) ROC-AUC across severities (SST-2). (b) Paired-bootstrap comparison at the hardest severity ($\rho =0.01$, two batch sizes); positive $\Delta$ favours Alert. Alert is never significantly worse than any constituent, and significantly improves ($p<0.05$) by up to $+0.15$ ROC-AUC over Recon (SST-2) and $+0.10$ over MMD (CoLA). Bold indicates the best result per column.

(a) ROC-AUC versus drift severity (SST-2)
Detector		$\rho =0.01$	$0.02$	$0.03$	$0.05$
Recon		$0.729$	$0.872$	$0.903$	$0.952$
MMD		$0.835$	$0.927$	$0.978$	$\mathbf{1.000}$
Energy		$0.809$	$0.916$	$0.963$	$\mathbf{1.000}$
Alert (ours)		$\mathbf{0.861}$	$\mathbf{0.971}$	$\mathbf{0.993}$	$\mathbf{1.000}$
(b) Paired-bootstrap no-regret test ($\Delta$, p) at $\rho =0.01$
Task	Alert vs.	Batch size 64		Batch size 32
		$\Delta$	p	$\Delta$	p
SST-2	Recon	$+0.132$	$0.116$	$+0.152$	$0.007$
	MMD	$+0.026$	$0.579$	$-0.004$	$0.951$
	Energy	$+0.051$	$0.393$	$+0.019$	$0.733$
CoLA	Recon	$-0.045$	$0.225$	$+0.019$	$0.597$
	MMD	$+0.098$	$0.001$	$+0.099$	$0.000$
	Energy	$-0.057$	$0.094$	$-0.071$	$0.099$

demonstrates the main result: on this on-par detector pool, Alert turns the unstable signals into one calibrated score that is never beaten. It attains the highest ROC-AUC at every SST-2 severity (panel a) and matches the best single detector on CoLA, where a confidence score already saturates. Paired-bootstrap tests (panel b) confirm Alert is never significantly worse than any constituent at the hardest severity $\rho =0.01$, the no-regret guarantee of Proposition A2. Where the paired-bootstrap p-value is large (e.g., $p=0.579$ for MMD and $p=0.393$ for Energy in panel b), the comparison is statistically inconclusive in either direction; we read these cells as non-dominance (Alert is not significantly worse), not as strict wins. Alert is therefore a simple, label-free default that deploys without knowing in advance which detector would have won.

6.4.5. Adaptive

Table 10. Trustworthiness: Adaptive. Label-free self-selection (ctx) recovers the hand-picked fixed family on a matched task, and yields decisive gains under mismatch. Every row is an Alert combiner: ctx selects its detector family from clean data with no labels; fixed hand-picks it. (a) on a matched task, ctx tracks fixed (fixed is best on SST-2 at $\rho =0.02$–$0.03$, ctx best at $\rho =0.01$ and on CoLA); (b) under task/backbone mismatch, Alert re-selection recovers up to $+0.136$ ROC-AUC ($\rho =0.01$). Bold indicates the best result per column.

(a) Self-configured Alert (ctx) vs. fixed-family Alert
Task	Alert variant (selected family)	$\rho =0.01$	$0.02$	$0.03$	$0.05$
SST-2	Alert fixed {Energy, MMD, Recon}	$0.861$	$\mathbf{0.971}$	$\mathbf{0.993}$	$1.000$
	Alert ctx top-3 {BBSD, Recon, MMD}	$0.865$	$0.960$	$0.985$	$1.000$
	Alert ctx top-5	$\mathbf{0.872}$	$0.960$	$0.989$	$1.000$
CoLA	Alert fixed {Energy, MMD, Recon}	$0.901$	$0.982$	$1.000$	$1.000$
	Alert ctx top-3 {MSP, ODIN, Energy}	$\mathbf{0.927}$	$\mathbf{0.990}$	$1.000$	$1.000$
	Alert ctx top-5	$0.911$	$0.987$	$1.000$	$1.000$
(b) Where selection helps most: mismatch
Mismatch setting		Alert fixed		Alert ctx	$\Delta$
CoLA drift, family chosen on SST-2		$0.862$		$0.927$	$+0.065$
SST-2 drift, family chosen on CoLA		$0.798$		$0.865$	$+0.067$
SST-2, roberta-base backbone (BERT family)		$0.608$		$\mathbf{0.744}$	$+\mathbf{0.136}$

shows Alert also configures itself. Choosing detector families by self-detectability on clean data alone recovers the right family per task with no labels or tuning (panel a), and helps most under mismatch: when a fixed family meets the wrong task or a RoBERTa [55] backbone, re-selection recovers up to $+0.136$ ROC-AUC (panel b).

6.5. Generalisation

Alert and the benchmark protocol generalise well beyond the SST-2/CoLA token-noise setting. We test three forms of transfer in turn: natural domain shift, single-modality backbones, and multimodal fusion.

Table 11. Generalisation: Natural-shift transfer. On natural shift, the cosine baseline fails, while confidence scores, MMD, and Alert detect (ROC-AUC; clean SST-2 vs. whole OOD corpora). Acc. is classifier accuracy on each source (in-domain $0.88$). Bold indicates the best result per column.

OOD Set	Acc.	Recon	MMD	Energy	MSP	Mahalanobis	Cosine	Alert (Ours)
IMDB	$0.79$	$0.995$	$1.000$	$1.000$	$1.000$	$0.571$	$0.000$	$\mathbf{1.000}$
Yelp	$0.78$	$1.000$	$1.000$	$1.000$	$1.000$	$0.992$	$0.044$	$\mathbf{1.000}$

begins with natural domain shift: monitoring clean SST-2 against whole out-of-domain corpora (IMDB, Yelp, which drop classifier accuracy to ∼$0.78$), confidence scores, MMD, and Alert all detect the shift at ROC-AUC $1.000$, while the industrial cosine baseline collapses to near-zero ($0.000$ and $0.044$) despite the real accuracy loss.

Table 12. Generalisation: Single-modal transfer. Alert transfers to a three-class NLI task (a) and to vision/CLIP backbones (b) (ROC-AUC; MNLI and CLIP mean over three seeds; ResNet-18 a single checkpoint retrained for this study, clean accuracy $0.94$). (a) MNLI; the top Classifier accuracy row confirms the flagged drift is consequential: accuracy falls toward chance ($0.33$, 3-class) as severity rises. (b) ResNet-18 and zero-shot CLIP (CIFAR-10-C); CLIP energy excepted (uncalibrated logits). On ResNet no single detector dominates (all within $0.85$–$0.91$), and Alert matches or beats the best single detector at every severity (paired bootstrap, never significantly worse); on CLIP MMD dominates at low severity, the dilution regime of Corollary A2 where one strong detector can lead. Bold = best per column.

(a) NLP transfer: three-class NLI (MNLI)
Detector	$\rho =0.01$		$\rho =0.05$		$\rho =0.20$
Classifier acc.	$0.79$		$0.67$		$0.40$
BBSD	$0.890$		$1.000$		$1.000$
ODIN	$0.890$		$1.000$		$1.000$
Energy	$0.867$		$1.000$		$1.000$
MMD	$0.857$		$1.000$		$1.000$
MSP	$0.843$		$1.000$		$1.000$
DriftLens	$0.820$		$1.000$		$1.000$
Recon	$0.563$		$0.913$		$1.000$
Mahalanobis	$0.540$		$0.593$		$0.863$
Cosine	$0.003$		$0.000$		$0.000$
Alert (ours)	$\mathbf{0.890}$		$1.000$		$1.000$
(b) Vision/backbone transfer
Detector	ResNet-18			CLIP
	sev 1	3	5	sev 1	3	5
MMD	$0.912$	$0.990$	$1.000$	$\mathbf{0.986}$	$\mathbf{1.000}$	$1.000$
MSP	$0.865$	$0.953$	$1.000$	$0.907$	$0.997$	$1.000$
ODIN	$0.907$	$0.990$	$1.000$	$0.933$	$\mathbf{1.000}$	$1.000$
Energy	$0.904$	$0.990$	$1.000$	$0.348$	$0.388$	$0.524$
KNN	$0.913$	$0.989$	$1.000$	$0.769$	$0.879$	$0.977$
Mahalanobis	$0.846$	$0.864$	$0.947$	$0.676$	$0.673$	$0.743$
Recon	$0.877$	$0.943$	$0.998$	$0.720$	$0.794$	$0.880$
Alert (ours)	$\mathbf{0.920}$	$\mathbf{0.993}$	$1.000$	$0.933$	$\mathbf{1.000}$	$1.000$

extends the protocol to new single-modality backbones: a three-class NLI task (MNLI, panel a) and vision models (ResNet-18 [56] and zero-shot CLIP, panel b). Strong detectors stay strong, and the cosine baseline still fails. The outcome follows detector dominance, as the theory predicts: on MNLI and ResNet, where no single detector dominates (on ResNet all detectors fall within $0.85$–$0.91$), Alert matches or beats the best single detector ($Alert=0.920$ vs. best single $0.913$ at the lowest ResNet severity, never significantly worse); on CLIP, where MMD dominates ($0.986$ vs. $0.933$ for the next detector), Alert stays competitive but MMD leads at low severity, exactly the dilution regime of Corollary A2.

Table 13. Generalisation: Multimodal transfer. Fusion strictly improves on single-modality monitoring; only the cross-modal member catches mismatch. All values are ROC-AUC (frozen CLIP on CIFAR-10 image-caption pairs). Rows = which modality each detector monitors; columns = which part of the (image, caption) stream drifts: Image/Text corrupts that modality, Mixed corrupts half each, Mismatch pairs each image with a wrong-class caption (both marginals unchanged). Single-modality detectors are blind outside their modality; Alert fuses them by a clean, calibrated mean-z. A marginals-only fusion catches modality corruption but is near chance on mismatch ($0.542$); adding the alignment score recovers it ($1.000$). Paired-bootstrap tests confirm the fusion gain is significant: Alert exceeds the best single-modality detector by $+0.25$ (mixed) and $+0.48$ (mismatch), $p<0.001$, and matches or surpasses early fusion (concat-MMD). Bold = best per column.

Detector (What It Monitors)	Image Drift	Text Drift	Mixed	Mismatch
MMD, image only	$\mathbf{1.000}$	$0.500$	$0.750$	$0.500$
MMD, text only	$0.485$	$\mathbf{1.000}$	$0.743$	$0.522$
concat-MMD, both (early fusion)	$\mathbf{1.000}$	$\mathbf{1.000}$	$\mathbf{1.000}$	$0.970$
alignment, cross-modal	$0.756$	$0.942$	$0.849$	$\mathbf{1.000}$
Alert, marginals only {img, txt}	$\mathbf{1.000}$	$\mathbf{1.000}$	$\mathbf{1.000}$	$0.542$
Alert + alignment (ours)	$\mathbf{1.000}$	$\mathbf{1.000}$	$\mathbf{1.000}$	$\mathbf{1.000}$

closes with multimodal drift, the one setting where fusion strictly wins: under mixed-modality and cross-modal mismatch drift invisible to any single stream, Alert reaches ROC-AUC $1.000$, while single-modality monitors sit near chance. It improves on the best single-modality monitor by up to 25 points (mixed) and roughly 50 points (mismatch), significant under a paired bootstrap ($p<0.001$), and matches or surpasses early fusion (concat-MMD [20], $0.970$ on mismatch). The cross-modal alignment score is essential: a marginals-only Alert catches modality corruption but is near chance on mismatch ($0.542$), and adding alignment to the fused family recovers $1.000$. These results confirm the benchmark’s conclusions are not specific to one dataset or modality. Real-pair, dead-channel, and larger-backbone variants confirm this (Appendix E).

Is the Multimodal Gain the Combiner or the Alignment Feature?

On caption mismatch, the alignment score alone already reaches ROC-AUC $1.000$, so we isolate the source of Alert’s performance (

Table 14. The caption-mismatch gain comes from the alignment member, not the aggregation rule. ROC-AUC on the caption-mismatch task (mean over three seeds), for combiners all given the same family {image-MMD, text-MMD, alignment}. Every combiner that includes the alignment member reaches $1.000$; a fusion without it stays near chance. Bold = includes the cross-modal alignment member.

Combiner/Detector (Features)	Mismatch ROC-AUC
image-MMD alone	$0.500$
text-MMD alone	$0.364$
concat-MMD (early fusion of marginals)	$0.929$
alignment alone	$\mathbf{1.000}$
Alert mean-z, marginals only {img, txt}	$0.417$
Alert mean-z + alignment (ours)	$\mathbf{1.000}$
unstandardised mean + alignment	$\mathbf{1.000}$
logistic regression + alignment (supervised)	$\mathbf{1.000}$

). Given the same family that includes the alignment member, every combiner, our mean-z, a naive unstandardised mean, and a supervised logistic regression, reaches $1.000$, whereas a fusion without the alignment member (marginals only) stays near chance ($0.417$). The gain on this task is therefore attributable to including the cross-modal alignment member, which Alert selects label-free, not to the specific aggregation rule; this is exactly the dilution regime of Corollary A2, where one dominant member makes the combiner reduce to it. The aggregation rule earns its keep in the settings where no single member dominates, the mixed-modality columns of Table 13 and the on-par detector pools of Table 9 and Table 12.

6.6. Discussion

The central finding is not that some detectors beat others, but that detector performance is governed by drift geometry: apparent superiority is often an artefact of the chosen baseline or perturbation, and against strong training-free baselines such as energy and MMD, learned reconstruction monitors are no longer dominant. Detector families respond to different statistical signals, so their assumptions break in different and sometimes opposite ways; for example, token-level corruption can pull representations toward the training centroid and invert Mahalanobis-style scores while the same statistic stays effective under natural shift, which is why rankings vary across tasks and severities. Drift detection is therefore not a model-selection problem with one optimal solution but a context-dependent one, and robustness comes primarily from calibration and combination rather than detector complexity: calibrated confidence scores are strong defaults, distributional tests help when embeddings and covariate shift are expected, and fusing complementary detectors is more reliable than betting on a single best one. Progress should thus be measured less by new detector design and more by understanding when and why detectors fail.

7. Conclusions

We introduced DecayBench, a calibrated benchmark for reference-free drift detection that enforces consistent evaluation under controlled drift severity, calibrated thresholds (a clean false-positive target that admits a finite-sample, distribution-free conformal interpretation), and standardised metrics. Across ten representative label-free detectors, we find that no method is uniformly dominant. We further propose Alert, a label-free combiner that aggregates detector signals into a single calibrated alert. Whenever no single detector dominates, it is no-regret, matching or beating the best constituent across NLP, NLI, and vision and beating it by a large margin under multimodal drift, never significantly worse; a dominant single detector (the dilution corollary, e.g., MMD on CLIP) is the only exception. This condition is checkable offline, and the guarantee follows from the comparable-effect-size regime of our analysis. Together, these contributions establish a reproducible evaluation setting and a simple but effective aggregation mechanism that reveal how much robust signal already exists in standard detectors when they are measured consistently and combined carefully. The result is a clearer path forward for reliable drift detection grounded in calibration, comparability, and compositionality of existing signals.

8. Limitations and Future Work

We study reference-free drift detection, deciding from unlabelled data when to raise an alert, on encoder-based classification models under synthetic and natural shift. We do not address concept drift, where the label changes with no observable input change and which unlabelled inputs cannot reveal. Alert is also only as strong as the detectors it fuses: its no-regret guarantee assumes an appropriate detector family, and its label-free selection assumes the self-perturbation probe resembles the real drift. When that fails, as under a cross-modal mismatch no single-modality probe can mimic, selection can drop the decisive detector and a single strong detector may win.

What to do after an alert (human review, abstention, adaptation, or retraining) is outside our scope, as are multilingual and long-horizon deployment, richer distribution shift, and foundation-model backbones. Because DecayBench fixes one calibrated protocol and metric set, any new detector or deployment setting can be scored the same way and compared head-to-head with the baselines reported here.