The Synthetic Fire Drill: A Reproducible Benchmark for Novel Topic Detection in Text Streams

Abstract

Proactive novel topic detection in text streams lacks standardized evaluation: systems are tested on different data, under different definitions of novelty, with no shared ground truth. The goal of this study is a standardized, reproducible way to evaluate proactive topic detection under identical conditions. We introduce the Synthetic Fire Drill, a benchmark methodology that injects articles about fictitious topics, generated by large language models, into real news streams under controlled semantic distance, volume, and timing. Since the topics are fictitious, every detection outcome is classifiable against complete ground truth, and a validation battery separates topical detection from generation artifacts. We demonstrate the benchmark on a broad general news corpus and a focused science-technology corpus, with two generators, four injection schedules, and 16 scenarios spanning a calibrated distance gradient, evaluating four detection approaches (the micro-cluster stream clusterers DBSTREAM and DenStream, the windowed topic-modeling detector BERTrend, and a cosine novelty detector) under two sentence encoders. The experiment reveals that every stream-clustering detection, across all runs, seeds, and encoders, occurs through absorption into pre-existing clusters, never through the formation of new ones, with the absorbing cluster’s topical relatedness depending on the corpus, not the detector; only the windowed paradigm produces an emergence signal. The approaches prove complementary along mechanism lines, with each covering difficulty regions the others miss. Finally, the benchmark’s ground truth demonstrates that the detection rate alone is a misleading metric, requiring a computable quality dimension like enrichment to evaluate true system performance. The findings’ scope is bounded by two embedding-model families, two corpora from different eras, and 16 scenarios. The synthetic articles, embeddings, ground truth, evaluation protocol, and refresh recipe are publicly released.

1. Introduction

The proactive detection of novel topics in text streams, as opposed to retrospective batch analysis, is a prerequisite for applications that depend on early awareness: media intelligence, horizon scanning, emerging-risk monitoring, and rapid response to novel narratives. The task has attracted sustained research attention, but its evaluation is ad hoc, with each system being tested on a different dataset, against a different definition of novelty, under a different time horizon, and with no shared ground truth. BERTrend, a windowed neural topic-modeling system for emerging-trend detection, based on Bidirectional Encoder Representations from Transformers (BERT), identifies the gap directly: no established metrics for comparing detection performance, no benchmark datasets carrying ground truth, and existing topic-model measures unsuitable for the task [1]. The Topic Detection and Tracking (TDT) program standardized retrospective topic detection in broadcast news [2], but its datasets are decades old, and its task definition differs from proactive detection. In adjacent fields, General Language Understanding Evaluation (GLUE) and Super General Language Understanding Evaluation (SuperGLUE) have established shared benchmarks for language understanding [3,4], and the LFR framework did the same for community detection [5], yet streaming topic detection has none.

To close that gap, this paper proposes the Synthetic Fire Drill, a benchmark that evaluates topic-detection systems the way a fire drill evaluates an alarm: by staging a known event under controlled conditions and measuring what the system does with it. The staged events are LLM-written articles about nonexistent topics, appended to real news streams. Because the topics are fictitious, the injected articles carry known membership, and no article in the host corpus belongs to the fictitious topic, so every detection outcome is classifiable against complete ground truth. Together, generation and injection put the parameters that govern detection difficulty under experimental control: how close the topic sits to existing coverage, how many articles arrive, and when. The benchmark is reproducible, with the synthetic articles, embeddings, run configurations, and evaluation code released as part of the publication.

We demonstrate the benchmark with four detection approaches spanning the mechanism space (two micro-cluster stream clusterers, DBSTREAM and DenStream; a windowed topic-modeling detector, BERTrend; and a per-article cosine novelty detector), two sentence encoders from different training families, two news corpora, two Large Language Model (LLM) generators, four injection schedules, and 16 scenarios along a calibrated semantic-distance gradient. Our evaluation uncovers three dynamics of streaming detection: the stream clusterers detect injected topics by absorbing their articles into pre-existing clusters, never forming new ones, under either encoder, while only the windowed paradigm produces an emergence signal; the approaches complement one another along mechanism lines, with each responding where the others are weak; and the detection rate alone is misleading without a quality dimension. None of these are observable without ground truth, which controlled injection supplies and a live deployment lacks.

Prior work has used synthetic or injected content for evaluation in adjacent settings. The work of Lau et al. is the closest methodological precursor, replacing tweet text with sentences sampled from existing TDT3 documents in a collected Twitter corpus for novel event detection [6], but without controlled generation, difficulty calibration, or a validation battery. SAIBERSOC injects synthetic attacks into security operations centers for controlled evaluation [7] but focuses on analyst workflows instead of detection algorithms and has no semantic difficulty gradient, and Shi et al. evaluate topic models on fully synthetic corpora with planted topic structure [8], without a real corpus or a streaming setting.

To our knowledge, no prior approach combines controlled generation, a calibrated semantic difficulty gradient, and injection into real streams in a released benchmark. To demonstrate the benchmark’s utility, our evaluation addresses six core research questions:

• RQ1: Does the framework produce a functioning benchmark, measuring topical detection instead of generation artifacts?
• RQ2: How does the semantic distance between an injected topic and the host corpus affect detection?
• RQ3: What does the benchmark reveal about how the tested detectors register injected topics?
• RQ4: Do the detection approaches complement each other?
• RQ5: Does injection schedule affect detection?
• RQ6: Do detection outcomes depend on the LLM generator?

This design lets us examine three expectations about current practice: that stream-clustering algorithms may assimilate novel topics into pre-existing clusters instead of isolating them as distinct entities (absorption failure); that the detection rate alone can mask this behavior, since a system can register a detection without topical specificity; and that the proposed enrichment metric makes the difference measurable.

The primary contributions of this study include: the Synthetic Fire Drill, a reproducible and standardized benchmark methodology that provides computable ground truth for evaluating streaming topic-detection systems; the construction of a dual-domain news corpus of 250,678 collected publications (231,413 after cleaning and deduplication), released as embeddings; the empirical identification of systemic absorption failure modes and detector complementarity in stream-clustering environments; and the introduction of the Enrichment metric, a quality dimension that shifts the evaluation focus from detection probability to the specific, actionable isolation of emerging narratives.

2. Materials and Methods

The Topic Detection and Tracking (TDT) program gave the task its name and its first shared evaluation infrastructure [2], while contemporary research has shifted toward using time-aware document embeddings to capture topical dynamics [9]. Beyond these, recent advancements in streaming topic detection have increasingly shifted toward the incremental processing of transformer-based representations. For example, dynamic and online extensions of embedding-based topic models, such as those utilizing continuous optimal transport for document streams, allow for the continuous updating of topic representations without requiring full retraining or a predefined number of topics [10]. Additionally, continual contrastive learning frameworks leveraging cloze-style prompting [11] and real-time multimodal stream detection approaches [12] have been recently proposed to better capture short-term semantic shifts and evolving narratives in news streams. However, while these novel architectures report high topic coherence, evaluating their actual responsiveness to sudden, zero-day narrative injections remains difficult. This is largely due to the lack of standardized, high-resolution streaming benchmarks—a specific methodological gap that the Synthetic Fire Drill explicitly targets. Manually annotated corpora of broadcast news allowed systems to be compared on the same data under the same definitions, and for a decade, TDT was the field’s reference point. The task they define is different from the one contemporary systems face: TDT evaluation is retrospective, operating over closed corpora with batch annotation, while proactive detection must handle open-ended streams with no fixed endpoint. First-story detection was introduced into streaming settings using locality-sensitive hashing for efficient nearest-neighbor comparison on Twitter at scale [13]. The computational requirements changed, the evaluation approach did not: detection was still validated against manually annotated known events, and the events themselves were selected because they were already prominent, leaving the question of what a detector missed structurally unanswerable.

A standardized benchmark holds evaluation conditions fixed, while the systems under test can vary. When tasks, difficulty, and ground truth are shared, differences in reported performance reflect those in method rather than in data. GLUE and SuperGLUE demonstrated this for language understanding, exposing failure patterns across architectures that per-system evaluation had concealed [3,4]. The Lancichinetti–Fortunato–Radicchi (LFR) framework gave community-detection researchers synthetic networks whose planted structure and tunable mixing parameter control detection difficulty [5]. Baes et al. recently extended the pattern to lexical semantic change, using LLM-generated synthetic data to benchmark detection methods where historical evaluation datasets do not exist [14]. Streaming topic detection has not undergone this transition. Detectors are evaluated against ad hoc collections of annotated real events, and because no shared difficulty scale exists, differences in reported performance cannot be separated from differences in evaluation conditions.

The research of Lau et al. [6] is a prior work that constructs topic-detection ground truth by placing known event content into a collected stream, and the differences from the present work reflect structural constraints of their design. Their datasets substituted existing human-written TDT3 sentences for real tweet text, removing the original content in the process. The present work generates new articles through LLM prompting and appends them to the stream, preserving the original corpus while controlling the injected topics’ thematic overlap with existing coverage. In evaluation design, their five datasets each tested a single known event against small backgrounds of 25 to 150 events in 9-day offline windows. The present benchmark tests 16 scenarios across a controlled difficulty gradient. Evaluation runs against full-scale corpora in streaming temporal order. Their design could evaluate whether a detector found a known event; it could not test how detection behavior varies across a difficulty range in a realistic stream.

The pipeline of the research is shown in Figure 1.

The pipeline is designed to supply topics that are verifiably novel, placed at known semantic distances, and injected with controlled volume and timing. Corpus preparation turns the raw article collections into clean, deduplicated, chronologically ordered streams; every later analysis stage operates on their sentence embeddings. Topic cluster analysis maps the topical structure of each corpus with Hierarchical Density-Based Spatial Clustering of Applications with Noise (HDBSCAN) [15,16]. Scenario design uses this map to decide where 16 fictitious topics should sit, from topics adjacent to well-populated clusters to topics in sparse regions of the embedding space; the distance itself is measured directly against corpus articles, so the gradient is calibrated per corpus and not in absolute terms. The gradient is the benchmark’s difficulty axis.

The remaining stages turn the designed scenarios into a controlled experiment. Article generation is where the control enters: two LLM generators produce each scenario’s articles under fixed volume, rotating stylistic templates, and temporal grounding matched to the corpus era. Real documents offer none of these levers, which is why the benchmark generates its articles. Injection scheduling assigns each scenario’s articles to timestamps under one of four arrival patterns, and stream construction merges them into the real stream in arrival order, leading to a stream identical to the real one except for the injected articles. Detection is the most replaceable stage: any streaming detector that consumes the ordered stream and emits cluster assignments or anomaly flags can be evaluated. The demonstration in this paper uses four: two micro-cluster stream clusterers, one windowed topic-modeling detector, and one per-article novelty scorer. Evaluation uses what a live deployment never has: complete ground truth about which articles belong to the injected topic. From it, the benchmark computes detection rate at a fixed threshold, enrichment ratio, permutation significance, and detection latency.

Validation runs as a parallel branch alongside the main flow: the two control experiments and the construction-validation tests feed the detection and evaluation stages directly, each of which is detailed within our validation framework. The pipeline runs per corpus as well. Because the distance measure is corpus-relative, a scenario placed near GN’s topical structure can sit far from ST’s; the benchmark treats these as distinct test conditions, and results are reported per corpus throughout, never pooled.

The benchmark artifact evaluated in this paper comprises the 16 scenarios, 884 generated articles, the four injection schedules, and the ground-truth labels; the demonstration applies it to the four detection approaches of Section 2.5 under two sentence encoders.

2.1. Data Collection

To construct the benchmark, an automated web scraping methodology was deployed. This programmatic approach facilitated the extraction of large-scale, unstructured text data, ensuring a highly representative empirical foundation for evaluating LLM-generated synthetic narratives. The data collection process was structured around two distinct corpora, targeting different thematic domains and temporal windows. The selection of specific web portals for data extraction was guided by three primary criteria: editorial authority, volume of daily publication, and domain-specific linguistic diversity. By sampling from globally recognized outlets, the benchmark ensures that the LLM is evaluated against high-quality, professionally moderated text.

The primary dataset, General News Corpus, serves as a comprehensive baseline of mainstream media discourse. Data was systematically extracted from ten prominent international news portals and financial platforms: The Guardian, Deutsche Welle (DW), The Wall Street Journal (WSJ), ABC News (Australia), CNN (Money/Business), Sky News, Reuters, The Independent (Business), The Economist, and Forbes. The timeframe for this dataset includes a period of one year (1 January 2019–31 December 2019).

Table 1. Overview of media outlets in General News Corpus.

Media	Format	Primary Editorial Focus
ABC News	Public Broadcaster	Authoritative journalism
CNN	Broadcast News Network	Multimedia network
DW	Public Broadcaster	Public broadcasting
Forbes	Business Magazine	Business publication
Reuters	International News Agency	Global wire reporting
Sky News	Broadcast News Network	Commercial broadcasting
The Economist	Weekly News Magazine	Analytical journalism
The Guardian	Daily Newspaper	Comprehensive political commentary
The Independent	Digital Newspaper	Digital-first portal
WSJ	Daily Financial Newspaper	Leading financial daily

presents an overview of media outlets in General News Corpus selected for research.

The Guardian [17] and The Independent [18] have been included for their high editorial standards, comprehensive digital archives, and diverse socio-political commentary, providing a rich dataset of European and global narratives. DW [19] has been selected to capture the European perspective on global events, characterized by objective, public-broadcasting linguistic patterns. The Wall Street Journal [20], The Economist [21] and Forbes [22] are essential for incorporating complex financial, economic, and entrepreneurial terminology into the baseline model. Reuters [23] serves as a primary global news agency; it provides highly factual, neutral, and rapid reporting, serving as the foundational linguistic layer of breaking news. CNN [24] and Sky News [25] have been included for their high-volume, real-time multimedia news cycles, which heavily influence global attention dynamics. ABC News [26] adds geographic and linguistic diversity by providing authoritative coverage from the Asia-Pacific region. The purpose of this dataset is to capture a broad spectrum of general public narratives, socio-political events, and baseline linguistic patterns prior to recent shifts in the digital attention economy.

The secondary dataset—Science/Technology Corpus—focuses on domain-specific discourse, capturing specialized terminology and emerging trends within the technology and engineering sectors. This data was harvested from eight targeted scientific and industry platforms: everything RF, Defense & Security Monitor, Interesting Engineering, MarkTechPost, SciTechDaily, ScienceDaily, New Scientist, and IEEE Spectrum. The timeframe for this dataset includes a period of one year (1 April 2025–31 March 2026).

Table 2. Overview of media outlets in Science/Technology Corpus.

Media	Format	Primary Editorial Focus
Defense & Security Monitor	Industry Publication	Defense and aerospace technology
everything RF	Specialized Industry Portal	Radio frequency and microwaves
IEEE Spectrum	Professional Magazine	Flagship engineering publication
Interesting Engineering	Digital Magazine	General engineering news
MarkTechPost	Tech News Portal	AI and machine learning focus
New Scientist	Science Magazine	Applied sciences and research
ScienceDaily	Research Aggregator	Aggregates university press releases
SciTechDaily	Science News Portal	Curated scientific discoveries

presents an overview of media outlets in Science/Technology Corpus selected for research.

IEEE Spectrum [27] and New Scientist [28] have been recognized as flagship publications in engineering and applied sciences, offering in-depth, rigorously fact-checked features on technological advancements. SciTechDaily [29] and ScienceDaily [30] have been selected for their aggregation of peer-reviewed research and university press releases, providing a structured, highly academic vocabulary. MarkTechPost [31] is crucial for its specific focus on Artificial Intelligence (AI) and Machine Learning (ML) research, representing the cutting edge of current digital narratives. Interesting Engineering [32] bridges the gap between complex engineering breakthroughs and accessible technological news. Defense & Security Monitor [33] has been included to capture the intersection of geopolitics and military technology, a niche but highly impactful narrative domain. The site everything RF [34] represents a highly specialized, niche industry (radio frequency and microwaves), great for testing the model’s ability to detect topics in extreme technical jargon. The purpose of this dataset is to isolate specialized narratives and evaluate the model’s capacity for topic detection within highly technical, fast-evolving informational ecosystems.

The automated extraction frameworks were designed to navigate the dynamic architectures of the selected platforms, bypassing superficial structural changes to consistently harvest the core textual content and relevant metadata. This dual-dataset approach provides both the volume required for deep semantic indexing and the categorical variance necessary to rigorously test topic-detection algorithms across different media segments. The technical details regarding the web scraping are provided in Appendix A.

Our primary focus in preparing the corpus was a rigorous data cleaning and deduplication pipeline to ensure high signal quality for the benchmark. First, all boilerplate content—such as navigation menus, advertisements, and embedded widgets—was stripped, isolating only the main editorial text. Second, because news platforms frequently syndicate the same wire stories, we applied a comprehensive deduplication strategy. We identified and removed duplicate texts that shared substantial overlapping content, alongside exact-match filtering. This robust cleaning process eliminated redundant noise from the stream, resulting in a highly curated final dataset of unique publications, which served as the reliable foundation for the streaming benchmark.

To facilitate a consistent comparison, all articles were embedded using the all-mpnet-base-v2 model [35], which provides a robust 768-dimensional representation suitable for semantic novelty detection. A second encoder, bge-base-en-v1.5 [36], from a different training family at the same dimensionality, provides the benchmark’s representation-sensitivity arm; its role and calibration protocol are described in Section 2.2. Furthermore, we established a 10-NN (k-Nearest Neighbors) self-similarity baseline for each corpus. This baseline acts as an objective, corpus-relative difficulty anchor, quantifying the inherent “density” of the informational space and allowing for a calibrated semantic-distance gradient during the subsequent injection phase. To maintain experimental consistency, we established fixed operational parameters for both corpora, as presented in

Table 3. Experimental corpus parameters and semantic baselines.

Parameter	General News (GN)	Science/Technology (ST)
Sources	10 outlets	8 outlets
Timespan	1 January 2019–31 December 2019	1 April 2025–31 March 2026
Primary embedding model	all-mpnet-base-v2 (768-dim)	all-mpnet-base-v2 (768-dim)
10-NN baseline (all-mpnet-base-v2)	0.745	0.665
Secondary embedding model	bge-base-en-v1.5 (768-dim)	bge-base-en-v1.5 (768-dim)
10-NN baseline (bge-base-en-v1.5)	0.791	0.785

.

The primary embedding-model choice and the 10-NN self-similarity baselines function as the static anchors for our difficulty gradient. Semantic distance is therefore calibrated to the specific topology of each corpus rather than to absolute values.

2.2. Scenario Design and the Semantic-Distance Gradient

Each scenario represents a fictitious topic governed by a strict editorial brief, including a non-existent named entity, a specific domain, and a one-line premise. Its difficulty is its semantic distance from the corpus, and this is measured without any clustering step: a scenario’s position is the mean cosine similarity of its articles to their 10 nearest neighbors among the corpus articles (10-NN similarity), computed separately for each corpus. Higher similarity means nearer; throughout, far means low similarity. The measure is deliberately clustering-free, so that the benchmark’s difficulty axis does not depend on the hyperparameters of any clustering method, least of all the family of methods under evaluation. Two reference points anchor the scale per encoder: the corpus self-similarity baselines, 0.745 for GN and 0.665 for ST under the primary encoder and 0.791 and 0.785 under the second, computed with the same measure over all real articles in each corpus. These distance quantities, including the baselines, the per-scenario k-Nearest Neighbors (kNN) values, and the HDBSCAN topic map, are computed against the full cleaned corpora (211,212 GN and 20,201 ST articles); the GN detection runs stream a contiguous slice of 29,999 articles (21 June to 15 August 2019, 56 days) of the same corpus, sized for tractability and selected by a scored scan for gap-free mid-year coverage; the ST detection runs stream the full corpus. All detection arms consume the identical slice, so the per-corpus results are comparable across detectors by construction. A scenario at the baseline is as close to the corpus as the corpus is to itself. The HDBSCAN map from the previous stage (974 clusters on GN at min_cluster_size 50, 362 on ST at min_cluster_size 15; noise ratios near 30% on both) served as the placement guide: near scenarios were aimed at the vocabulary of well-populated topic clusters, far scenarios at regions the map showed as sparse.

The representation layer established in Section 2.1 carries one primary and one secondary encoder. All distance measurements, the gradient calibration, and the full detection grid run on all-mpnet-base-v2; bge-base-en-v1.5, an encoder from a different training family with the same dimensionality, reruns the distance measurement and the DBSTREAM detection protocol on a six-scenario subset spanning both corpora’s tiers. Detection thresholds are encoder-specific by protocol: a clustering threshold tuned in one embedding space is re-derived in the other from that space’s own pairwise-distance distribution, per corpus, with a pre-stated acceptance check, because similarity scales differ across encoders. Under this protocol, the bge clustering threshold resolved to 0.7868 on GN and 0.7516 on ST against the mpnet-space default of 1.0, and the cosine novelty detector’s thresholds were recalibrated from the second encoder’s own false-positive baseline run. The subset comprises aethelgard, velosphere, srvp7, luma_krill, vangard, and mof_sable, chosen to cover the three distance tiers on both corpora and both detection outcomes under the primary encoder.

Everything downstream rests on the topics being verifiably fictitious, so this property was engineered and not assumed. We brainstormed roughly 80 candidate topics within LLM sessions, using prompts designed to elicit domain diversity without revealing the research goal. Each entity name was then screened for collisions with two search-grounded services, Perplexity and Google Search: names were queried against real companies, organisms, products, and organizations and rated 1 to 5 for uniqueness, where a conflict meant an existing entity with the same or a confusable similar name, particularly in the same domain. Ten candidates failed the screen; eight were eliminated and two renamed. Some conflicts were subtle: Norøvir sat one diacritic away from Norvir, a widely prescribed HIV drug, and was dropped. Domain deduplication then thinned the surviving pool to one candidate per domain.

Placement was calibrated empirically and not trusted from design intuition. For each of the 18 shortlisted candidates, three short test articles (roughly 150 words each, generated manually via the Gemini web interface) were embedded with the primary sentence encoder (all-mpnet-base-v2, truncating at the model’s default of 384 tokens) and scored with the 10-NN measure on both corpora. Several candidates landed elsewhere than intended and were reassigned; two whose positions duplicated coverage already on the gradient were dropped. The final set contains 16 scenarios in three tiers relative to the corpus baselines: six near scenarios that reach or approach at least one corpus’s baseline, five medium, and five far scenarios sitting well below both. Tiers are design labels fixed at calibration time, not threshold rules; the placement they summarize is continuous and two-dimensional. Article volume is a design variable separate from the distance tier: 13 strong scenarios carry 12 articles each, and three small scenarios carry four, five, and six articles, probing detection near the minimum volume. The smallest, calcifex, sits below the detection threshold by construction and probes the benchmark’s floor. It is excluded from the tier groupings and per-scenario statistical tests, which therefore report the medium tier at n = 4 and compare nh = 15 scenarios. Figure 2 illustrates the placement of the scenarios on the semantic-distance plane.

Table 4. The 16 fictitious scenarios spanning the semantic-distance gradient. kNN similarity is the mean 10-nearest-neighbor cosine similarity of the scenario’s articles to the corpus (Claude-generated articles; the Gemini values differ by at most 0.03). Tier reflects the design assignment. Calcifex (4 articles < N = 5, structural cap) is included in the headline detection-rate denominators (n = 16) but excluded from the tier groupings and the per-scenario statistical tests (n = 15).

Scenario	Topic	Tier	n	kNN GN	kNN ST
aethelgard	Caspian oil consortium announces surprise production cut	near	12	0.779	0.547
velosphere	Chinese network-equipment maker hit with export restrictions	near	12	0.728	0.599
cordis_ai	Nonprofit lab releases LLM trained only on licensed data	near	12	0.640	0.676
frb_coronet	Fast radio burst repeating on a perfect 22-min cycle	near	5	0.634	0.727
novalith	Fusion startup sustains net-positive reaction for 12 s	near	12	0.625	0.696
mof_sable	Metal-organic framework captures CO2 directly from air	near	12	0.574	0.732
luma_krill	Bioluminescent krill population collapse in Southern Ocean	medium	12	0.685	0.646
quietmantis	Espionage malware infiltrating energy-grid operators	medium	12	0.656	0.589
meridazole	Compound immobilizes PFAS in contaminated soil	medium	6	0.626	0.655
calcifex	Bio-concrete injected into aging subway tunnels	medium	4	0.556	0.624
srvp7	Perovskite compound stores single photons at room temp	medium	12	0.527	0.651
valenzia	Misattributed Renaissance painting confirmed as masterwork	far	12	0.657	0.532
vangard	Literary prize awarded to anonymous reclusive author	far	12	0.653	0.416
veloria	Mining group assets frozen after reserve misstatement	far	12	0.644	0.479
naledi_okoro	Distance runner breaks national 10,000 m record	far	12	0.633	0.467
veristhorn	Invasive vine threatens European grain yields	far	12	0.588	0.512

Note: In this table, kNN GN and kNN ST denote the k-Nearest Neighbors similarity metrics for the General News and Science/Technology corpora, respectively. LLM stands for Large Language Model, and PFAS refers to per- and polyfluoroalkyl substances.

lists the 16 scenarios with their topics, tiers, article counts, and measured similarities.

The values there, and in every analysis, come from the articles actually generated for the benchmark, not from the design-time test articles. The two differ by up to roughly 0.08 per scenario, but the ordering along the gradient is preserved (Spearman rho = 0.879 on GN and 0.985 on ST between test-article and generated-article placements), so the calibrated design survived generation intact.

2.3. Article Generation

Every scenario was generated independently by each of two LLM generators, Claude (claude-sonnet-4-6) and Gemini (gemini-2.5-flash), and independently for each target corpus, yielding four article sets per scenario. No sampling parameters were overridden; both generators used provider defaults, documented in

Table A2. Generation configuration. Parameters marked “not set” were omitted from the API call; the provider default applied.

Parameter	Claude	Gemini
Model version	claude-sonnet-4-6	gemini-2.5-flash
Max output tokens	2000	Uncapped
Temperature	Not set	Not set
top_p	Not set	Not set
top_k	Not set	Not set

. All articles for a scenario share the same prompt anatomy: a constant journalist persona, the scenario’s fixed brief of 60 to 100 words, a cast of named figures unique to that scenario (early test generations reused the same few invented names across scenarios, a fingerprint the per-scenario casts eliminate), and one of a small set of rotating templates. Each template locks a reporting angle, a format ranging from a wire dispatch to a feature of an instructed length of roughly 150 to 400 words, and an editorial tone; strong scenarios cycle five templates, small scenarios three. We held the brief identical across a scenario’s templates so that differences among its articles are attributable to the controlled framing dimensions and not to differences in factual content. Temporal grounding is the one asymmetry. GN articles are written as if published in 2019, with publication months cycled through the calendar and an instruction to use no post-2019 knowledge; a post hoc term-prevalence check confirmed compliance in 15 of 16 scenarios, the exception being cordis_ai, whose premise inherently requires post-2019 vocabulary. ST articles are written as contemporary reporting.

Each generator produced 442 articles: 171 scenario articles per corpus, plus 100 corpus-neutral noise articles (one per topic from a list of 100 unrelated domains) for the Control B experiment, giving 884 articles in total. The 74 templates and 16 briefs are built deterministically by a committed script whose validator checks, among other things, for cast-name collisions, missing scenario vocabulary, and meta-commentary leakage (phrases revealing the article is generated, such as “synthetic article”). The validator operates at template-authoring time; generation itself applies no per-article content gate, and every application programming interface (API) response was saved and released as returned. No rejection or regeneration of individual articles occurred, so the released article set is the generators’ unfiltered output under the documented prompts. The full templates and worked generation examples are included in the released artifact; the template rotation and recorded model versions are summarized in Appendix B.

2.4. Injection Schedules and Stream Construction

Semantic distance determines where a scenario sits on the difficulty gradient; the injection schedule determines how its articles arrive over time. Arrival is the benchmark’s second experimental axis. Each run injects a single scenario into one corpus stream under one schedule, and both generators’ article sets receive identical timestamps by position, so a generator comparison varies article content and nothing else. Timestamps are assigned by a committed script and are fully deterministic: each scenario-corpus-schedule combination derives its own seed from a sha256 hash of its identifiers, and two consecutive executions produce byte-identical schedule files. When the merged real and synthetic stream is assembled for replay, articles are sorted by timestamp, with ties broken by a per-article random key drawn once from a seeded generator; two runs that differ only in this seed differ only in the arrival order of articles that share an identical timestamp.

The four schedules span the arrival patterns an emerging topic can plausibly take. Burst distributes all of a scenario’s articles evenly across a 4-day window placed at a seeded-random offset within [c_min + 10 days, c_max − 10 days], ensuring a 10-day margin from both corpus boundaries; it is the concentrated-arrival case, the most favorable for cluster-based detection. Steady spaces the articles evenly across the middle 80% of the corpus range, deterministically placed: 44.78 days on GN, one article roughly every 4.1 days for a 12-article scenario, and 283.37 days on ST, roughly every 25.8 days. It models gradual emergence. We matched accelerating to steady deliberately, same window, same mean interval, so that any difference between the two schedules is attributable to the arrival pattern alone; under accelerating, the gaps shrink exponentially, from roughly 7 days at the start to 1.16 days at the end on GN, and from roughly 44 days to 7.36 days on ST. Flash-then-silence (hereafter flash) is the minimum-volume probe: only the first five articles by identifier, front-loaded across a 7-day window at a seeded-random offset, and nothing afterward. Five is exactly the strong-signal threshold defined in our benchmark protocol, so flash operates with zero scatter margin, and one article failing to co-cluster makes the threshold unreachable. Because templates rotate T1–T5 by article index, the flash slice of a strong scenario contains each of the five templates exactly once, preserving full stylistic diversity despite the volume cap. Among the small scenarios, calcifex injects its full four articles under flash and meridazole five of its six. Flash is therefore not a shorter burst: burst injects everything a scenario has, while flash caps the injected signal at the detection threshold itself.

Stream construction merges the timestamped synthetic articles into the real stream and sorts by timestamp. Ties are resolved by a pre-assigned random key drawn from an independent seeded generator, rather than by provenance or insertion order; this tie-breaking seed is the one varied during the robustness and stability validation phase. The result is a stream identical to the real one except for a single scenario’s articles, at most 12 among roughly 30,000 on GN and 20,000 on ST.

2.5. Detection Approaches

Four detection approaches close the pipeline, evaluated as one comparison set under a shared protocol: the same streams, the same ground truth, and the same scenario-level detection criterion. This reflects a design decision: the benchmark evaluates existing detectors and proposes no new architecture, so any member of the set can be substituted with any method that consumes the ordered stream of embeddings and emits cluster assignments or anomaly flags. The set spans the mechanism space deliberately. DBSTREAM and DenStream are micro-cluster stream clusterers from the same algorithmic family with different maintenance semantics; BERTrend detects through windowed topic modeling, a paradigm that batches the stream rather than clustering it point by point; the cosine novelty detector scores each article individually and never groups anything.

DBSTREAM is a density-based stream-clustering algorithm that maintains a population of micro-clusters, assigning each arriving point to the nearest micro-cluster within a clustering threshold or opening a new one, while clusters that receive no points fade over time [37]. We run the River 0.24.2 implementation [38], pinned in the release, at the library-default clustering_threshold = 1.0 and fading_factor = 0.01 under the primary encoder, with the second encoder’s thresholds re-derived per the protocol in Section 2.2; on L2-normalized embeddings, the threshold corresponds to a cosine similarity of approximately 0.5, with the effective cutoff slightly more permissive for multi-member clusters whose centers drift below unit norm under River’s weighted update. Articles are processed sequentially in the seeded stream order established during the stream generation phase. A thin tracking layer records, at stream time, each article’s micro-cluster assignment and the member count of the assigned cluster at that moment; it adds no clustering logic of its own.

One bookkeeping step on the raw library output is needed before its assignments can be counted. The published DBSTREAM algorithm defines no persistent cluster identifiers, but the implementation does, and it recycles them: keys are allocated as one plus the current maximum, so a key freed when the highest-numbered cluster fades is immediately reused by the next new cluster. Grouping articles by raw cluster ID over a full run would therefore conflate distinct clusters that happen to share a recycled key. Attribution is instead incarnation-aware and post hoc: the per-article assignment log is segmented into incarnations, with a new incarnation opened whenever a cluster’s member count fails to strictly increase, and all detection accounting operates on incarnations rather than raw IDs. This bookkeeping is what allows our subsequent analysis to determine, for each detection, whether the scenario’s articles joined an already-populated cluster or formed a fresh one. Under DBSTREAM, a scenario is detected when N or more of its articles are assigned to the same micro-cluster incarnation; every injected article counts toward the criterion, including the article whose arrival opened the incarnation.

DenStream is the second density-based stream clusterer, run from the same pinned River release. It maintains two tiers of micro-clusters, potential and outlier, promotes outlier clusters that gain sufficient weight, and buffers the first 1000 stream articles for initialization before online processing begins. Its neighborhood radius epsilon was selected by a pre-stated criterion on the zero-injection Control A streams alone: the smallest grid value at which clustering is non-degenerate on both corpora, which resolved to 0.7. The remaining five parameters coincide with River 0.24.2’s defaults; only epsilon was tuned. All six are set explicitly in every released configuration rather than inherited from the library. The identity bookkeeping differs from DBSTREAM’s by necessity. DenStream’s cluster identifiers are unstable across its merge and promotion operations, so the tracking layer mints an object-tagged lineage identifier when a micro-cluster is created and carries it through merges and promotions; all detection accounting operates on these lineages. A scenario is detected when N or more of its articles are assigned to the same lineage, the same criterion that the incarnation rule applies to DBSTREAM. One structural consequence of the initialization buffer is that articles injected among the first 1000 of a stream enter the buffer rather than the online phase, which caps the reachable co-occupancy for early-injection schedules.

BERTrend [1] is the one member of the set outside the stream-clustering family: it slices the stream into two-day windows, following its own paper’s granularity, fits a topic model per window on the same precomputed embeddings the other detectors consume, and matches topics across windows into lineages. The core detection path runs without any external language-model call; BERTrend’s optional LLM-based topic labeling is not used. Its native output is a per-topic popularity classification with dynamically adapted thresholds; the benchmark additionally applies its own fixed criterion to BERTrend’s assignments, so a scenario is detected when N or more of its articles share one topic, evaluated at two levels: within a single window, and across a lineage of matched topics. The two levels are reported separately throughout, with the lineage level being primary. Two properties of the paradigm bound what BERTrend can be compared on. Every article in a window is assigned to some topic, so topic membership is forced rather than radius-gated, and synthetic share within a topic is diluted accordingly; the concepts of an absorbing cluster’s enrichment ratio and of a false-positive incarnation count have no analog in a windowed topic model, so BERTrend appears in the comparisons where the concepts translate, detection and the classification of Section 2.6, and not in the absorption map or the false-positive trade-off.

The cosine novelty detector is the set’s non-clustering member: it asks, for each arriving article, whether anything similar has been seen recently. Each incoming article is scored by its maximum cosine similarity to the W = 1000 most recently seen articles (equivalently, the 1-nearest-neighbor similarity within the trailing window) and flagged when that score falls below the corpus-specific 5th-percentile threshold calibrated during the false-positive baseline validation (Control A): 0.3950 on GN and 0.4518 on ST under the primary encoder, and 0.6078 and 0.6734 under the second, each calibrated from its own Control A run. Because the window is defined in articles, not in time, its temporal span is corpus-dependent, roughly 1.9 days on GN at roughly 536 articles per day and roughly 17.5 days on ST at roughly 57 per day. A scenario is detected when N or more of its articles are flagged. All four detectors therefore share a single scenario-level definition of detection and differ only in the per-article mechanism: co-occupancy of one incarnation for DBSTREAM, of one lineage for DenStream, of one topic or topic lineage for BERTrend, and an individual novelty flag for the cosine novelty detector.

Coverage differs by arm and is stated here once. The three clustering arms run the full grid of 16 scenarios by two corpora under the burst schedule; DBSTREAM and the cosine novelty detector additionally run the full schedule and generator study of Section 2.4. The second encoder runs the DBSTREAM protocol and the cosine novelty detector on the six-scenario subset of Section 2.2, twelve cells per detector. Every denominator in the results names the grid it is computed over.

The benchmark defines two fixed signal thresholds. The strong signal, N = 5, is the operating threshold for all headline detection rates; N is additionally swept over 3, 5, 7, and 10 in our parameter sensitivity analysis, so the value is an explicit protocol parameter. Its derivation is a subsample-matched permutation null: drawing k = 12 articles at random from the Control B reference noise set 20,000 times to match the strong-scenario volume, and measuring their maximum co-occupancy within a single incarnation. Five is the smallest count at which chance co-occupancy falls to approximately 5% (P(X ≤ 4) = 94.2% on GN, 94.4% on ST); the probability of 5 or more articles co-occupying one incarnation by chance is 5.8% on GN and 5.6% on ST. The same null machinery, re-parameterized with each scenario’s own article count k rather than the volume-matched k = 12, supplies the per-scenario significance tests reported with the results; the threshold derivation and the significance testing are two uses of the same null model. The null is volume-matched but not schedule-matched: its noise articles enter at uniform-random timestamps across the corpus span, while burst injection concentrates arrivals in four days, so chance co-occupancy under burst is plausibly higher than the scattered null estimates. The exposure is limited: most detected cells reach maximum co-occupancy of 8 to 12, where the scattered null already assigns probabilities below 2 × 10⁻³, and only significance verdicts for cells at the 5-to-6 boundary could shift under a plausibly fatter burst null. The weak signal, N = 2, marks the first time 2 or more of a scenario’s articles co-occupy one incarnation and is reported as an early-warning latency alongside the per-scenario results. Both thresholds are fixed constants of the benchmark, not quantities tuned per run, a deliberate contrast with systems such as BERTrend [1] that adapt their detection thresholds dynamically: a benchmark needs a stable definition of success for results to be comparable across detectors, corpora, and studies. The fixed criterion is applied uniformly to every arm, including BERTrend, whose native dynamic classification is reported beside it rather than in place of it.

2.6. Evaluation Metrics and the Benchmark Protocol

The benchmark consists of the fixed benchmark artifact (versioned v1 in the released repository), the 884 articles, their embeddings, the four injection schedules, and the ground-truth labels, evaluated at the N = 5 threshold and reporting detection rate and enrichment ratio per corpus. Sensitivity analyses (sweeps over N, the cosine window size, and the DBSTREAM clustering threshold) probe the protocol’s neighborhood but do not redefine it. This positions the Synthetic Fire Drill as a benchmark construction methodology together with its first instantiation, following the Lancichinetti–Fortunato–Radicchi (LFR) precedent [5] of a generator methodology that the field treats as a benchmark once a parameter set is fixed and released.

The detection rate is the fraction of scenarios detected at threshold N. The enrichment ratio measures how concentrated a scenario’s articles are within the detecting cluster: the scenario’s density in the cluster (the N detected articles divided by the incarnation’s total membership at that moment) divided by the scenario’s base rate in the stream (its injected count divided by all articles processed). Recall, the fraction of the scenario’s articles injected up to the detection step that were captured in the detecting incarnation, is reported per scenario in Supplementary Materials, Table S1; conventional precision and F1 are uninformative here: 5 synthetic articles in a cluster of 2500 real members gives precision 0.002, and F1 is similarly near zero. We organize detection quality along two axes: significance, derived from our statistical permutation null, and specificity, measured by the enrichment ratio with a predefined baseline floor of 5×. Significance is reported with Benjamini–Hochberg correction applied within each test family, co-clustering tests and correlation tests separately, and corrected q-values are quoted beside the raw statistics. Because the co-occupancy structure is a property of the detector, the permutation null is constructed per detector from that detector’s own zero-injection and noise-injection behavior; Section 2.7 details the construction.

The absolute delay, the time between the earliest point at which detection becomes arithmetically possible and the point at which the detector reaches the threshold, is the primary latency metric; cross-schedule delay comparisons are not reported because schedules differ in injection span.

For the windowed paradigm, each detected cell is additionally classified on two axes, birth timing and topic purity, into emergence and absorption analogs: a topic born with a majority-synthetic membership is an emergence analog, and one whose synthetic share never reaches a majority is an absorption analog, with the majority bound at 0.5 mirroring the benchmark’s own majority logic. The classification is computed at both levels of Section 2.5, single-window and lineage, and reported with the lineage level being primary. Forced topic membership dilutes the synthetic share, which biases this classification toward absorption; the direction of the bias is conservative for the comparison made in the results and is disclosed where the classification is used.

The cosine novelty detector’s per-article score (windowed max-sim) and the benchmark’s difficulty measure (global 10-NN similarity) both derive from cosine similarity to the corpus, raising the question of whether the observed distance-detection gradient is circular. On 5000 real articles per corpus, the two quantities correlate at Pearson r = 0.76 on GN and 0.61 on ST, sharing 37–58% of their variance: related, as expected from a common geometric basis, but not identical.

2.7. Validation Framework

Five tests probe whether measured detections reflect the injected topics rather than artifacts of LLM generation style, synthetic cluster tightness, or chance co-occurrence. Control A runs both corpora through the full detection pipeline with no synthetic injection. A false positive is one micro-cluster incarnation whose lifetime membership reaches N or more articles in the zero-injection stream, counted once per incarnation, regardless of its size or duration; the resulting counts are the baseline against which all detection rates are read. Control A runs per detector and per encoder: each stream clusterer’s false-positive floor and the cosine novelty detector’s threshold are calibrated from its own zero-injection run. For BERTrend, the zero-injection run is a descriptive null rather than a false-positive baseline, because forced topic membership means every window produces topics; its Control A characterizes native base rates and defines no false-positive count.

Control B injects 200 noise articles (the same 100 unrelated topics generated independently by both models), assigned random timestamps spanning the full corpus date range, into each corpus. The test targets the most direct threat to the benchmark’s validity: if LLM generation style rather than topical coherence drove clustering, 200 articles sharing the same generative origin should form cohesive clusters regardless of topic. The formation route exists by construction, since DBSTREAM opens a new micro-cluster whenever an article falls outside every existing cluster’s radius, so a stylistic attractor strong enough to pull the noise articles together has a standalone-cluster pathway available. The 100 noise topics were hand-curated for domain diversity and validated programmatically for count and uniqueness, though no automated deduplication against the 16 scenario topics was applied. The permutation null of Section 2.5 draws from Control B’s labels under the detector being tested: the threshold derivation is defined once, on DBSTREAM’s incarnations, as a fixed constant of the benchmark, while the per-scenario significance tests for each clusterer draw from that clusterer’s own Control B assignments, since co-occupancy under noise is detector-specific.

A logistic regression classifier on linguistic features (average word length, type-token ratio, vocabulary richness) tests whether synthetic articles are stylistically separable from real ones, with accuracy reported under length-controlled conditions to remove a paragraph-count artifact. The intra-cluster variance check measures the mean pairwise similarity of each scenario’s generated articles against the natural-cluster distribution, bounding the consequence of generation-induced tightness: shared briefs, shared casts, and limited template rotation are expected to produce tighter-than-natural clusters.

The variance robustness test degrades synthetic embeddings for four scenarios spanning the gradient with calibrated Gaussian noise to mean pairwise similarity targets of 0.70, 0.60, and 0.50, the last below the natural median, and reruns detection on both corpora. A kNN sanity check verifies that degraded articles remain above a signal-destruction floor of 0.30 in 10-NN similarity, so that the noise diversifies the clusters without destroying their topical signal. The cosine novelty detector needs no equivalent test: reduced sibling similarity would only weaken the mutual nearest-neighbor masking that burst injection creates, so elevated tightness cannot inflate cosine detection rates.

As a reproducibility check, each DBSTREAM and DenStream burst run was repeated under three stream-order tie-breaking seeds (42, 123, 456); the outcome splits are reported with the validation results, and the false-positive floors are single-seed quantities for both clusterers. BERTrend is run once per cell: its pipeline is seeded and reproducible for a fixed input order but not bit-identical on repetition, and its within-window processing is sensitive to article order as a property of the paradigm, so per-seed robustness is reported for the two micro-cluster detectors only.

2.8. Applying the Benchmark to New Detectors and Corpora

Three operations define the benchmark’s adoption interface: evaluating a new detector, porting to a new corpus, and refreshing the scenario set.

Evaluating a new detector on the v1 artifact requires only that the detector consumes a chronologically ordered stream of (article_id, timestamp, embedding) tuples and emits cluster assignments or anomaly flags. The released analysis scripts compute detection at threshold N, enrichment, permutation significance, and latency over the released logs; adapting them to a new detector’s output requires mapping its assignments or flags to the same per-article schema. The integration effort is demonstrated twice in this paper rather than estimated. Integrating DenStream required a tracking wrapper that mints stable lineage identifiers over the library’s unstable cluster IDs and a one-parameter calibration on the zero-injection streams; integrating BERTrend, a detector from a different paradigm, required an adapter that slices the stream into windows, feeds the precomputed embeddings, and maps per-window topic assignments to the same per-article schema. Both are released with their configurations.

Porting the benchmark to a new corpus requires four steps: embedding the corpus with any sentence encoder, running the corpus-relative kNN distance calibration to place existing or new scenarios on the target corpus’s gradient, rerunning Control A on the new corpus to establish its false-positive baseline and cosine threshold, and optionally regenerating articles with corpus-appropriate temporal grounding. Porting across encoders follows the same shape and is likewise demonstrated: the second-encoder arm required re-embedding, per-corpus threshold re-derivation under the pre-stated protocol of Section 2.2, and a fresh Control A calibration, and nothing else in the pipeline changed. The distance measure is corpus-relative by design, so ported results remain per-corpus.

Regeneration also addresses shelf life: the v1 articles will eventually enter training corpora, and a refreshed scenario set restores the fictitiousness guarantee. The prompt templates, uniqueness verification protocol, and distance calibration procedure are all released, so generating a fresh scenario set requires only rerunning the documented generation and calibration scripts.

3. Results

This section reports the composition and yield of the two corpora and then the detection outcomes of four detector arms and two embedding models across the 16 scenarios, organized by research question.

Table 5. Publications in General News Corpus (January 2019–December 2019).

Media	Website	Publications
ABC News	https://www.abc.net.au/	24,025
CNN	https://edition.cnn.com/	3984
DW	http://www.dw.com/	15,278
Forbes	https://www.forbes.com/	3076
Reuters	https://www.reuters.com/	90,760
Sky News	https://news.sky.com/	1008
The Economist	https://www.economist.com/	5895
The Guardian	https://www.theguardian.com/	71,949
The Independent	https://www.independent.co.uk/	2212
WSJ	https://www.wsj.com/	10,098

presents the total number of publications for each of the digital media across Mainstream Discourse within the period from January 2019 to December 2019.

As is evident from the distribution, the corpus exhibits a volume skew typical of digital news ecosystems. Global wire services (Reuters) and high-output daily publishers (The Guardian) account for over 71% of the total dataset. In contrast, publications with strict paywalls, deeper analytical focuses, or lower daily editorial output (e.g., The Economist, WSJ, and Forbes) contribute to smaller, yet linguistically dense, subsets. This structural bias is acknowledged and preserved, as it authentically mirrors the natural volume of mainstream informational exposure in the digital attention economy.

Table 6. Publications in Science/Technology Corpus (April 2025–March 2026).

Media	Website	Publications
Defense & Security Monitor	https://dsm.forecastinternational.com/	203
everything RF	https://www.everythingrf.com/	1455
IEEE Spectrum	https://spectrum.ieee.org/	646
Interesting Engineering	https://interestingengineering.com/	7166
MarkTechPost	https://www.marktechpost.com/	1394
New Scientist	https://www.newscientist.com/	2098
ScienceDaily	https://www.sciencedaily.com/	4836
SciTechDaily	https://scitechdaily.com/	4595

presents the total number of publications for each of the digital media across Specialized Discourse within the period from April 2025 to March 2026.

The distribution within the Science/Technology Corpus highlights the structural differences between broad technical aggregators and highly niche publications. High-volume platforms that aggregate general engineering news and university research (e.g., Interesting Engineering, ScienceDaily, and SciTechDaily) collectively represent nearly 74% of the dataset. Conversely, deeply specialized, industry-specific portals such as IEEE Spectrum and Defense & Security Monitor contribute much smaller volumes but inject dense, highly specialized vocabulary into the vector space. This balance ensures the novel topic-detection model is tested against both accessible scientific journalism and rigorous engineering terminology.

The raw data extraction pipeline demonstrated robust yield, with minimal loss during the deduplication phase, as presented in

Table 7. Data acquisition yields and cleaning statistics.

Metric	General News (GN)	Science/Technology (ST)
Articles loaded (raw)	228,285	22,393
Post-cleaning (deduplication)	211,212	20,201
Final detection stream	29,999 (mid-year slice)	20,201 (full)

.

The resulting operational datasets provide a high-fidelity representation of the respective domains, with the GN corpus utilizing a computationally optimal slice and the ST corpus processed in its full, domain-specific volume.

Table 8. Cross-paradigm detection and absorption summary. Brackets: three-seed range where available. Absorbed column: count classified as absorption (A) or emergence (E) under the benchmark’s rule; cosine has no clustering step. FP floor: Control A false positives at N = 5. BERTrend’s Control A is a descriptive null with no false-positive analog (Section 2.7).

Arm	Encoder	Corpus	Detected (N = 5)	Absorbed	FP Floor	Note
DBSTREAM	mpnet	GN	10/16 [9–12]	10/10	85	3-seed range
DBSTREAM	mpnet	ST	12/16 [11–12]	12/12	16	3-seed range
DBSTREAM	bge	GN	6/6	6/6	88	6-scenario subset
DBSTREAM	bge	ST	4/6	4/4	10	6-scenario subset
DenStream	mpnet	GN	15/16 [15–15]	15/15	20	3-seed range; flat across N
DenStream	mpnet	ST	15/16 [14–15]	15/15	6	3-seed range; flat across N
Cosine	mpnet	GN	0 burst; 2 steady	n/a	—	No clustering
Cosine	mpnet	ST	0 burst; 4 steady	n/a	—	No clustering
Cosine	bge	GN	0 burst; 0 steady	n/a	—	6-scenario subset
Cosine	bge	ST	0 burst; 2 steady	n/a	—	6-scenario subset
BERTrend	mpnet	GN	12/16	7A/5E	—	Lineage N ≥ 5; descriptive null
BERTrend	mpnet	ST	13/16	7A/6E	—	Lineage N ≥ 5; descriptive null

Note: In this table, FP stands for false positive and n/a indicates not applicable. GN and ST denote the General News and Science/Technology corpora, respectively.

summarizes the detection and absorption outcomes across the four detector arms and two embedding models. Each row reports the arm’s detection count out of its eligible cell count, the absorption classification, and the false-positive baseline from its Control A.

3.1. Benchmark Validation (RQ1)

Five validation tests address this; all return outcomes consistent with a valid benchmark.

Control A establishes the false-positive baseline: at the operating threshold N = 5, DBSTREAM produces 85 false positives on GN and 16 on ST, and DenStream produces 20 on GN and 6 on ST. The DenStream floors are flat across all four tested N values (Section 3.7). BERTrend’s Control A is a descriptive null: it records topic counts and native classification base rates under zero injection, with no false-positive analog, because BERTrend’s per-window topic discovery has no operating threshold comparable to N. Control B confirms that detection is topical and not stylistic. After incarnation-aware attribution, the apparent multi-noise clusters collapse (GN from 29 to 2, ST from 12 to 6), and every survivor sits inside a background mega-cluster, mirroring the low-specificity injected scenarios; the 200 noise articles form zero standalone same-topic clusters despite the available formation pathway. Several noise topics have both generators’ articles co-absorbed within the background clusters, an incidental co-absorption into thousand-member real clusters distinct from the standalone clusters a style effect would predict. Across the k = 12 noise subsamples of the permutation null, maximum co-occupancy reaches 5 by chance in only 5.8% (GN) and 5.6% (ST) of draws, while shared topics produce 5 to 12. Detection is driven by topical coherence, not generation style; the absence of cohesive noise clusters indicates the style-versus-topic separation held in practice.

The stylistic classifier separates synthetic from real articles at 94.9% length-controlled accuracy; the raw figure of 99.8% is inflated by a paragraph-count artifact. Detection itself operates on semantic embeddings, not these surface markers, so the classifier result is a transparency check, not a confound. Intra-cluster variance is high: mean pairwise similarity reaches 0.911 on GN and 0.914 on ST, well above the natural-cluster median of approximately 0.55–0.58 (range 0.26–0.95). Because tighter clusters are easier to co-assign, the benchmark operates at the favorable end of the difficulty spectrum for detectors.

The variance robustness test confirms stability: all seven informative single-seed conditions survived at every degradation level; the eighth, quietmantis/GN, was undetected at the baseline and, therefore, excluded. Across three seeds on GN, 9 of 10 multi-seed informative conditions held at the 0.50 target, the single boundary failure falling at the harshest level. The kNN sanity check recorded a minimum 10-NN similarity of 0.39 across all conditions, against the 0.30 signal-destruction floor. The full grid is in Supplementary Materials, Table S5; co-membership margins narrow at the harshest target, with scenario-level erosion detail in the limitations section.

Cross-seed stability provides a reproducibility check, not a sixth validation test. Of the 15 scenarios eligible for detection at N = 5 (calcifex is excluded by its four-article count), GN yields 7 stable detections, 2 stable misses, and 6 seed-sensitive outcomes; ST yields 9, 2, and 4. DenStream is stable across the same three seeds: GN detects 15 of 16 at every seed, and ST detects 15 at seeds 42 and 123 and 14 at seed 456, with st/meridazole as the single cell whose N = 5 verdict varies. meridazole is a 6-article scenario where N = 5 leaves a one-article margin; it detects at N = 3 under every seed and at N = 7 under none. The borderline cell differs from DBSTREAM’s own seed-sensitive cases, so the two detectors’ thin margins do not coincide.

The five tests confirm that detection signals reported below reflect topical co-occurrence rather than generation artifacts and can be interpreted against quantified baselines for false positives, style leakage, and seed variability.

3.2. Semantic Distance and Detection (RQ2)

Table 9. Detection rate and enrichment by distance tier (DBSTREAM, N = 5, seed 42; calcifex excluded). Mean and median enrichment are both reported because small tier sizes (n = 2–6) make the mean sensitive to single outliers: the GN near-tier mean (24.1×) is more than twice the median (10.4×). Failure mode: scatter = no two synthetic articles ever co-clustered; fragmentation = partial co-clustering below N. Dominant failure mode from Supplementary Materials, Table S3.

Corpus	Tier	n	Det.	Rate	Mean Enrich.	Median Enrich.	IQR	Enrich. Range	Failure Mode
GN	Near	6	3	0.50	24.1×	10.4×	9.9–31.5	9.4–52.5×	scatter
GN	Medium	4	2	0.50	7.2×	7.2×	6.0–8.4	4.7–9.7×	fragmentation
GN	Far	5	5	1.00	4.3×	5.2×	2.3–5.3	2.2–6.6×	-(all detected)
ST	Near	6	6	1.00	9.2×	8.0×	6.5–9.7	4.9–17.9×	-(all detected)
ST	Medium	4	3	0.75	11.9×	7.7×	7.1–14.6	6.5–21.5×	fragmentation
ST	Far	5	3	0.60	29.7×	15.7×	11.2–41.3	6.7–66.8×	scatter

Note: In this table, IQR stands for interquartile range. GN and ST denote the General News and Science/Technology corpora, respectively.

summarizes DBSTREAM outcomes at N = 5 (seed 42) grouped by the defined difficulty tiers established in our experimental design.

The GN far tier detects all five scenarios, yet its mean enrichment is 4.3×, below the 5× specificity floor. Every cell in the tier is permutation-significant, so the detections are real; they are not specific to the injected topic.

Overall performance across the four detector arms, alongside mean enrichment by tier, is summarized in Figure 3.

The enrichment ranges expose what the means compress. GN far spans 2.2–6.6× (no cell clears even 7×), despite a perfect detection rate. ST far spans 6.7–66.8×, with valenzia’s 66.8× pulling the mean to 29.7×. ST near detects at 6/6 with enrichment spanning 4.9–17.9× (mean 9.2×). The two corpora show opposite quality-distance relationships: GN detects far scenarios at high rates but low enrichment, while ST detects near scenarios at high rates and moderate-to-high enrichment. The GN near tier (3/6, 9.4–52.5×, mean 24.1×) confirms the inversion: fewer detections, but those that occur are highly enriched. The enrichment bars slope downward across GN tiers and upward across ST, forming the crossed pattern.

Scenario-level distance-detection correlations are consistent with the grouped picture but do not reach significance. Co-clustering strength correlates at rho = +0.34 on GN and rho = +0.19 on ST (Supplementary Materials, Figure S1). The raw binary correlation on GN, the strongest individual result, is rho = +0.49 (uncorrected p = 0.063, bootstrap 95% CI [0.0, 0.81]). None of the DBSTREAM-related correlations survive Benjamini–Hochberg correction within the 11-test correlation family at q < 0.05. At n = 15, these tests have low statistical power, and the wide confidence interval (CI) on the strongest result spans zero to strong positive. The grouped tier comparison is the more informative presentation; the full correlation set, with q-values for both corrected families, appears in Supplementary Materials, Table S7.

The distance gradient is preserved under a second embedding model. When the 16 scenarios are re-embedded with bge-base-en-v1.5, the within-space rank correlations between kNN similarity and the mpnet design-time ordering are rho = 0.893 on GN and 0.968 on ST (both p < 1 × 10⁻⁵), comparable to mpnet’s own 0.879 and 0.985. Cross-encoder scenario ordering is a different statistic, weaker on GN (Spearman 0.53) than ST (0.93), and is never merged with the within-space gradient. The full comparison appears in Supplementary Materials, Table S8.

The 10 undetected cells at seed 42 fall into three failure modes. Two are structural caps: calcifex on both corpora has 4 articles, making detection at N = 5 arithmetically impossible. Four are scatter failures, where no two of the scenario’s articles ever co-occurred in the same micro-cluster incarnation. Four are fragmentation: partial co-clustering (maximum 2–4 articles) that never reached the threshold. The two corpora do not differ significantly in the overall detection rate (McNemar exact p = 0.73); they differ in detection mechanisms, which is the focus of our subsequent analysis. The Wilson 95% confidence intervals on the headline DBSTREAM rates overlap substantially: GN 10/16 [0.39, 0.82], ST 12/16 [0.51, 0.90].

3.3. Detection Mechanism (RQ3)

Across 66 detected cells over three seeds, every detection occurred through absorption into a pre-existing real cluster. No detection arose through new-cluster formation. Synthetic fractions at detection range from 0.0009 to 0.04: the clusters that absorbed the injected articles were 96–99.9% real articles at the moment of detection. Figure 4 maps the absorption pattern on both corpora.

On GN, the pattern is concentrated. Eight of the 10 detections funnel into two background mega-clusters: mc 229 (background news A, 2642 members at its largest detection snapshot) absorbs six scenarios, and mc 184 (background news B, 5806 members at its largest detection snapshot) absorbs two. The medium-sized cluster is the busier absorber. The remaining two land in single-scenario clusters: aethelgard in mc 64 (markets/shares, 1334 members, 9.4×, moderate specificity) and velosphere in mc 247 (238 members, 52.5×, the only high-specificity GN detection). mc 247’s topic tag is US–N.Korea politics, semantically unrelated to the telecom scenario velosphere; on GN, even the highest-specificity detection lands in an unrelated cluster. Within the background cells, enrichment still varies: cordis_ai in mc 229 reaches 10.4×, showing that cluster size and enrichment specificity are orthogonal.

On ST, absorption is more distributed. Four scenarios sitting mid-to-far on ST’s realized gradient share the paleontology cluster mc 297 (1287 members at its largest detection snapshot): luma_krill (ST kNN 0.646), aethelgard (0.547), veristhorn (0.512), and veloria (0.479). aethelgard absorbs into markets/shares on GN and paleontology on ST, illustrating corpus-dependent absorption. Five scenarios reach clusters with thematic overlap: frb_coronet into astronomy, srvp7 into quantum, cordis_ai into AI, valenzia into archaeology (66.8×), and quietmantis into defense/drones (21.5×); the last is the loosest match, with espionage malware landing in a defense-adjacent cluster. The remaining three share two additional absorbers: mof_sable and novalith enter EV/engine tech (mc 460), and velosphere enters the same defense/drones cluster as quietmantis at 6.9×.

Our variance robustness analysis independently corroborates absorption as the primary detection mechanism. Detection that depended on synthetic articles co-binding with each other would have failed under the degraded similarity conditions tested there; absorption into a large pre-existing cluster is insensitive to sibling tightness.

The all-absorption pattern holds under a second embedding model. On the six scenarios tested with bge-base-en-v1.5, all 10 DBSTREAM-detected cells (6 GN, 4 ST) classify as absorption under the same rule, with synthetic fractions between 0.005 and 0.007. Two additional cells are detected by the cosine detector under steady injection only (aethelgard/ST and vangard/ST); cosine detections have no clustering step and are reported in Section 3.4.

DenStream absorbs every detected scenario on both corpora: 30 of 30 at seed 42 and 89 of 89 across all three tie-breaking seeds, with zero new-cluster formations. Its detections are a strict superset of DBSTREAM’s 22 cells: 8 DenStream cells have no DBSTREAM counterpart at any overlap level. Where both detectors find the same scenario (22 cells: 10 GN, 12 ST), the relationship is containment with dilution rather than agreement. The median DenStream lineage is 2.26 times the corresponding DBSTREAM absorber on GN and 2.86 times on ST, and the median share of the DBSTREAM absorber contained inside it is 40.8% on GN and 85.5% on ST (overall Jaccard 0.15, with only 1 of 22 cells at or above 0.5). On ST, the two detectors find largely the same real articles at a coarser granularity; on GN, they absorb substantially different real articles. The disagreements run in one direction on both corpora: GN 11 agree/5 disagree, ST 13 agree/3 disagree, and every disagreement is a DenStream detection with no DBSTREAM counterpart. Supplementary Materials, Table S9 reports the per-cell overlaps.

BERTrend, the one detector outside the micro-cluster family, produces a different pattern. At the lineage level (primary), 14 of the 25 cells detected at N ≥ 5 classify as absorption and 11 as emergence. At the single-window snapshot level, 13 of the 24 detected cells classify as emergence and 11 as absorption. The 25th cell (gn/novalith) is detected at the lineage level but does not reach N = 5 within any single window. On the 24 cells detected at both levels, exactly two flip from emergence to absorption between the snapshot and lineage views: gn/cordis_ai and gn/luma_krill, both on General News. These two cells are the two downward bound-crossings in the purity data, where a scenario’s purity falls from above 0.5 at birth to below 0.5 at lineage close. The level split and the crossings are one finding about two GN cells, not two corroborating results; ST reads 7 absorption and 6 emergence at both levels, identical cell for cell. Under the alternative tie-breaking rule (share-last), two ST cells (st/vangard and st/veloria) also move from emergence to absorption, giving a lineage emergence count between 11 (share-first, adopted) and 9 (share-last). Supplementary Materials, Table S10 reports every cell’s purity at both levels, both strata, direction of change, and tie-split flags.

Beyond detection itself, each cell is scored on two quality measures: significance (permutation p-value) and specificity (enrichment ratio). Of the 30 non-capped DBSTREAM cells, 23 are significant at p < 0.05, and all 23 remain significant after Benjamini–Hochberg correction within the co-clustering family (Supplementary Materials, Figure S3 and Table S7). Significance is computed on maximum co-occupancy for every cell, including undetected ones, so significant cells can outnumber detections: a scenario can co-cluster beyond chance without reaching the N = 5 operating threshold. Four of the 23 fall below the 5× specificity floor: naledi_okoro/GN (2.2×), vangard/GN (2.3×), luma_krill/GN (4.7×), and mof_sable/ST (4.9×). The flagged set is floor-sensitive between 3× and 5×: at a 3× threshold, luma_krill and mof_sable would unflag, leaving only two low-specificity cells. The paper reports the four exact values rather than claiming stability around the chosen threshold.

The DenStream permutation test uses a detector-native null constructed from its own Control B label distribution. On GN, the null saturates: 178 of 195 noise articles (91.3%) land in a single lineage (5 of the 200 fall inside DenStream’s initialization buffer and carry no lineage), so the null’s 95th percentile of co-occupancy reaches 12 of 12, and no GN cell survives at q < 0.05, regardless of its observed count. On ST, 13 of 15 detected cells survive; the two that do not are st/frb_coronet (p = 0.035, q = 0.074, significant before correction) and st/meridazole (p = 0.126). The GN saturation is a structural property of DenStream’s granularity at epsilon = 0.7, not evidence against the detections: the count-based null is uninformative where a single lineage captures most of the noise, and the enrichment ratios continue to discriminate (Section 2.6).

DenStream’s own Control B yields per-corpus p95 enrichment floors: 3.60× on GN and 5.348× on ST. On GN, 3 of 15 detected cells fall below the native floor (compared with 3 below the 5× heuristic); on ST, 11 of 15 fall below it (compared with 11 below the heuristic). The 5× heuristic is conservative on GN (floor 3.60×) and permissive on ST (floor 5.348×), with the two directions reflecting the cluster-size regimes on each corpus. The DBSTREAM-derived floors (GN 1.55×, ST 8.12×) rest on DBSTREAM’s own label distribution and are not compared with DenStream’s native values. Both sets of disagreement cells are named in Supplementary Materials, Table S11.

3.4. Detector Complementarity (RQ4)

The cosine novelty detector scores each article individually against the recent corpus within a sliding window, not by cluster membership. This architecture produces opposite behavior across the distance-schedule space: where DBSTREAM’s distance correlations were non-significant, the cosine novelty detector’s are strongly negative.

Under steady injection, the cosine novelty detector produces the strongest surviving correlation: ST rho = −0.88 (q < 0.001 after Benjamini–Hochberg correction within the 11-test correlation family). GN steady (rho = −0.61, uncorrected p = 0.016) and GN accelerating (rho = −0.56, p = 0.031) do not survive correction (q = 0.060 and q = 0.084 respectively). The gradient on ST is robust; on GN it is suggestive at n = 15 but does not clear the corrected threshold. Far-corpus articles are individually anomalous against the corpus distribution, making them detectable by novelty scoring; near-corpus articles blend in and go unflagged. Figure 5 makes the separation visible: the steady and accelerating points fall along a steep negative slope, while burst points cluster at zero.

At the paper’s operating point (5th-percentile threshold, window size 1000), the cosine novelty detector produces 0/32 detections under burst injection across all 16 scenarios on both corpora. Across the 9 tested parameter combinations (3 percentiles × 3 windows), no setting exceeds 1 of 32 burst detections; 7 of 9 settings detect nothing, and both non-zero settings sit at window 500 (Supplementary Materials, Figure S2). Burst-blindness is architectural: each burst article’s nearest neighbor in the sliding window is a sibling from the same scenario, which lifts its maximum similarity above the flagging threshold. The schedule dependence independently confirms that the distance-detection gradient is not circular, as the established circularity defense requires: identical distance geometry produces 0/32 burst detections and 6/32 steady detections.

Under the second encoder, burst-blindness replicates (0 of 12 detections on both corpora), and the steady-injection results are fragile: of three non-trivial cells, only vangard/ST agrees across encoders.

Table 10. Detector complementarity: Per-scenario detection outcomes across detector arms. DBSTREAM and DenStream under burst injection; cosine novelty detector under steady injection; BERTrend at lineage level; each arm under its most favorable conditions. + detected at N = 5 (seed 42), − missed, n/a structural cap.

Scenario	kNN GN	DB GN	Cos GN	DenS GN	BT GN	kNN ST	DB ST	Cos ST	DenS ST	BT ST
vangard	0.653	+	−	+	+	0.416	−	+	+	+
naledi_okoro	0.633	+	−	+	+	0.467	−	+	+	+
veloria	0.644	+	−	+	+	0.479	+	+	+	+
veristhorn	0.588	+	−	+	+	0.512	+	−	+	+
valenzia	0.657	+	−	+	+	0.532	+	+	+	+
aethelgard	0.779	+	−	+	+	0.547	+	−	+	+
quietmantis	0.656	−	−	+	+	0.589	+	−	+	+
velosphere	0.728	+	−	+	+	0.599	+	−	+	+
calcifex	0.556	n/a	n/a	n/a	n/a	0.624	n/a	n/a	n/a	n/a
luma_krill	0.685	+	−	+	+	0.646	+	−	+	+
srvp7	0.527	−	+	+	+	0.651	+	−	+	+
meridazole	0.626	+	−	+	−	0.655	−	−	+	−
cordis_ai	0.640	+	−	+	+	0.676	+	−	+	+
novalith	0.625	−	+	+	+	0.696	+	−	+	+
frb_coronet	0.634	−	−	+	−	0.727	+	−	+	−
mof_sable	0.574	−	−	+	−	0.732	+	−	+	+

Note: In this table, DB, Cos, DenS, and BT denote the DBSTREAM, Cosine Novelty, DenStream, and BERTrend algorithms, respectively. The suffixes GN and ST indicate the General News and Science/Technology corpora (e.g., DB GN stands for DBSTREAM on General News). kNN ST denotes k-Nearest Neighbors similarity on the Science/Technology corpus, and n/a indicates not applicable.

shows the per-scenario outcomes.

With four arms, the union detects every eligible scenario on both corpora, but DenStream alone accounts for the full 15/15. The three DenStream-only cells (frb_coronet and mof_sable on GN, meridazole on ST) are all detected through absorption into the existing structure, not through the distance-dependent novelty mechanism that the cosine detector provides. The gap in coverage of near-corpus topics under non-burst schedules narrows but does not close.

3.5. Schedule Effects (RQ5)

Burst is the most favorable schedule: articles arriving within a 4-day window maximize the chance of co-clustering, and the full-gradient results presented in our analysis use burst injection for this reason. Steady and accelerating injection produce comparable detection outcomes despite spanning weeks to months, because the detector accumulates articles across the stream. Flash operates at zero scatter tolerance: with 5 articles at the N = 5 threshold, a single article scattered into a different micro-cluster kills detection. In the 12 flash cells of the schedule study (aethelgard, srvp7, and vangard, selected to span the distance gradient; 3 scenarios × 2 corpora × 2 generators), 8 went undetected, including all 6 on GN; the 4 detections (aethelgard and srvp7 on ST, both generators) all occurred at zero delay when all five articles co-clustered (Supplementary Materials, Table S4). This is a minimum-volume finding, not evidence that flash consistently fails; detection succeeds when and only when no article is scattered.

Detection latency is schedule-driven: 17 of 23 detected schedule-study DBSTREAM runs have zero absolute delay, and the non-zero delays are dominated by the injection span under steady and accelerating schedules, not by detector lag. Weak signals (N = 2) typically appear within approximately 0.36 days of injection under burst, confirming that co-clustering begins almost immediately and that the delay to strong-signal detection is a threshold effect, not detector-side lag. Beyond schedule geometry, DBSTREAM detection is sensitive to the clustering_threshold parameter. Across an extended parameter sweep (5 scenarios, 2 corpora, 3 seeds, 3 clustering_threshold values; 22 cells per threshold after reusing the existing Tier-1 baselines), the cliff replicates: 0 of 22 cells detected at ct = 0.8, 15 of 22 at ct = 1.0, and 22 of 22 at ct = 1.2, with a single borderline flip at ct = 1.0 (aethelgard/ST at seed 456). The three-regime shape (silence, sensitive transition, full detection) is stable across seeds and scenarios. The shuffle_seed parameter varies same-timestamp tie-break ordering only, not injection randomization; the one ct = 1.0 flip reflects a genuine borderline case, not a pipeline instability. The full sweep appears in Supplementary Materials, Figure S2.

3.6. Generator Sensitivity (RQ6)

Across the schedule study (3 scenarios × 4 schedules × 2 corpora), Claude and Gemini agree on 23 of 24 detection outcomes (Cohen’s kappa = 0.917); the single discordant cell is aethelgard on GN under accelerating injection. Across the full distance gradient (16 scenarios × 2 corpora under burst injection, seed 42), agreement is 30 of 32 (kappa = 0.855, or 0.83 excluding the structurally capped calcifex; Supplementary Materials, Table S6). Both disagreements fall on meridazole, the smallest strong scenario at 6 articles, and flip in opposite directions: Claude detects on GN where Gemini does not, while Gemini detects on ST where Claude does not. The opposing directions indicate margin-level seed sensitivity on a borderline scenario rather than a systematic generator effect. The benchmark’s findings are not artifacts of a specific LLM generator.

3.7. Detection Sensitivity to N Threshold

The detection threshold N is swept over {3, 5, 7, 10} to test how it affects the balance between sensitivity and false-positive cost. Figure 6 presents the trade-off on both corpora.

At the operating point N = 5, DBSTREAM detects 10/16 on GN and 12/16 on ST against 85 and 16 false positives, respectively (10/15 and 12/15 excluding calcifex, which cannot reach N = 5 by construction). frb_coronet sits at the threshold’s edge for a related structural reason: with exactly five articles, detection requires all five to co-cluster, so its outcomes measure minimum-volume viability as much as semantic placement. Lowering to N = 3 gains one scenario on each corpus (11/16 GN, 13/16 ST), but GN false positives double to 175 and ST rises to 36. Raising to N = 7 or N = 10 reduces false positives (GN: 64, 41; ST: 10, 10) at the cost of detections, with GN dropping to 8/16 at N = 7 and 6/16 at N = 10, while ST holds at 11/16 for both.

DenStream detects more scenarios than DBSTREAM at every tested N on both corpora, and the three-seed ranges are disjoint: DenStream’s minimum across seeds exceeds DBSTREAM’s maximum. At N = 5 on GN, DenStream detects 15 of 16 at every seed (range [15, 15]) against DBSTREAM’s [9, 12]; at N = 5 on ST, DenStream’s range is [14, 15] against DBSTREAM’s [11, 12]. The separation holds at N = 3, 7, and 10 on both corpora (8 of 8 comparisons).

DenStream’s false-positive floor is 20 on GN and 6 on ST. DBSTREAM’s falls from 175 at N = 3 to 41 at N = 10 on GN and from 36 to 10 on ST. Thresholding buys DBSTREAM specificity at a steep cost in detections; it buys DenStream nothing, because its floor is already flat.

4. Discussion

The demonstration shows a consistent split in how detection mechanisms behave across the difficulty space: stream clustering registers an emerging topic as absorption into an existing structure, a behavior that held across two clustering paradigms, three seeds, and two embedding spaces, whereas per-article novelty scoring isolates anomalies but remains sensitive to corpus-relative distance and to injection schedule. The one evaluated paradigm outside the micro-cluster family produces an emergence analog at the single-window level; the lineage view shifts the balance toward absorption (14 of 25 detected cells), driven by two General News cells. These findings show that detection rate alone is insufficient and that evaluation needs a second, quality axis.

4.1. What the Demonstration Reveals About the Benchmark

Comparing the detector arms produced three insights: the absorption census, the rate-quality divergence, and the coverage map across mechanisms. All three show why a controlled benchmark is needed, since each relies on ground truth that a live deployment lacks.

The absorption census is only countable because the injected articles carry known membership; a deployment sees a detection fire and cannot tell a genuinely new cluster from an old one that simply grew, having never labeled the emerging topic in the first place.

Detection rate and detection quality also came apart: the GN far tier detected 5/5 at a sub-floor mean enrichment (4.3×), absorptions into generic background clusters rather than specific ones. Without ground truth, the rate is the only number available, and on its own it misleads, reporting success where known membership shows non-specific absorption.

The arms proved complementary along mechanism lines rather than along a simple division of the difficulty space. The two clustering arms nest: every scenario cell DBSTREAM detects, DenStream detects as well, and the relationship between their absorbers is containment at different scales rather than agreement or disagreement. The cosine novelty detector responds where the clustering arms are weakest, catching spread-out arrival schedules that thin the co-occupancy clustering requires, while remaining blind to bursts. Establishing any of this required running the same scenarios at known distances under matched conditions, which only controlled injection makes possible; against an unlabeled stream, the arms’ hits and misses cannot be lined up at all.

In practical terms, 85 false positives against 10 true detections on GN is not a deployable alarm, and the benchmark does not claim it is. The benchmark measures relative detectability under controlled conditions against a quantified false-positive baseline. Control A exists so that detection rates are interpretable rather than presented in isolation. The counts are the corpus’s noise floor at each operating point, not a precision denominator: the 85 GN incarnations are groups of real articles that reach N = 5 co-occupancy in a stream with nothing injected. Any detector run at this operating point inherits them, and a ratio across the two numbers would mix units, scenario detections against incarnation events. The comparative use is the intended one: a candidate system improves on this baseline if it preserves the detection profile while lowering the floor that Control A computes for it under identical conditions. The demonstration itself contains the case: DenStream’s floor on the same streams is 20 incarnations on General News and 6 on Science/Tech, flat across every threshold tested, alongside detection counts above DBSTREAM’s, which is exactly the comparison Control A was built to support. The supplied ground truth is the distinguishing factor that allows this benchmark to expose behaviors which would remain hidden in operational deployments.

4.2. The Absorption Finding and Its Consequences

The absorption census establishes what stream clustering does when it is asked to find emerging topics. Applied to topic detection, micro-cluster stream clustering does not surface a new topic as a new cluster; it registers the topic’s arrival as intensification inside a cluster that already exists. New-cluster formation was available the whole time. DBSTREAM opens a fresh micro-cluster for any article that falls outside the clustering threshold of every current one, DenStream holds the same standing pathway through its outlier micro-clusters, and the incarnation-aware attribution makes each detection auditable, showing whether it took that pathway. Across 66 DBSTREAM detections over three seeds, none did. Across 89 DenStream detections over the same three seeds, none did. The census repeats in a second embedding space: all 10 cells detected under the bge encoder classify as absorption by the same verbatim rule. The absorption-first tendency is known for this family of algorithms [37,39]; what the benchmark supplies is the exhaustive census under known ground truth, and the census spans two clustering paradigms, two encoders, and three seeds without a single formation event (Supplementary Materials, Figure S4).

The one evaluated paradigm outside the micro-cluster family is also the only one that produces an emergence analog. At the single-window level, 13 of BERTrend’s 24 detected cells are majority-synthetic groupings, and three cells consist of synthetic articles alone. Traced across full lineages, the picture shifts toward the family pattern: 14 absorptions against 11 emergences over the 25 lineage-detected cells, with the two label changes on the comparable cells confined to General News. Formation, where it appears at all in this study, appears at the paradigm boundary, not inside the micro-cluster family.

How specific an absorption is depends on the corpus, not on the detector. aethelgard makes the point cleanly: the same scenario is absorbed into a markets and shares cluster on General News and into a paleontology cluster on Science/Tech, pulled in each case toward whatever sits nearest in that corpus. On General News the nearest thing is usually generic. Two background mega-clusters take eight of the ten detections, and even the single high-specificity General News detection, velosphere, is absorbed into a topically unrelated cluster, which says more about how undifferentiated the corpus is than about how the detector behaves. Science/Tech absorbs more selectively. Five of its twelve detections land in clusters with thematic overlap to the injected scenario, though the fit is loose at the edge, with quietmantis (espionage malware) drawn into a defense and drone cluster. The corpus’s narrower topical structure offers related absorbers that broad news coverage does not.

Hubness is consistent with the split, and the corpora carry the signature. In high-dimensional embedding spaces, a few points accumulate a disproportionate share of nearest-neighbor relationships and draw incoming articles toward them [40]. Measured in the study’s embedding space (k = 10, full deduplicated corpora), the k-occurrence distribution is positively skewed in both corpora, more strongly on General News (skewness 2.01) than on Science/Tech (1.73), and the real members of the absorbing clusters sit modestly above their corpus’s mean hubness, between 1.06 and 1.17 times it pooled by absorber type, with no absorber falling below the mean. The detector’s behavior does not change between the two. The corpus decides whether the cluster waiting to absorb an emerging topic is a thematic neighbor or a generic hub.

The consequence for practitioners who deploy micro-cluster stream clustering as a topic detector is concrete: a firing detector signals intensification within an existing cluster, not a new one, the behavior survives a change of clustering paradigm and a change in the encoder, and the detection rate on its own cannot tell a specific find apart from a generic absorption. Telling them apart needs a quality dimension reported alongside the rate, which we address in the following analysis.

4.3. Complementarity, the Rate-Quality Divergence, and a Two-Axis Framework

Running detectors with different mechanisms turns the benchmark into a diagnostic instrument rather than a ranking exercise. DBSTREAM detects scenarios across the distance gradient by absorbing them into clusters that already exist, with the specificity of the absorption, not the fact of detection, tracking the corpus; the cosine novelty detector flags articles as individually anomalous against the corpus distribution and responds along the schedule dimension where the clustering arms are weakest, detecting spread-out arrivals while remaining blind to bursts. Between the clustering arms, the relationship is containment rather than division of labor: DenStream detects a strict superset of DBSTREAM’s cells, reaching every non-capped scenario on both corpora under the burst schedule, and union coverage across all arms is reported per corpus in Table 10. Single-arm evaluation reports whether a detector fired; the multi-arm evaluation reports where in the difficulty and schedule space each arm is blind. Across all arms at the burst schedule, the only undetected cells are the two calcifex instances, where the structural article cap (four articles) places N = 5 below reach; no scenario that can reach the detection criterion goes undetected by every arm. That the gap is structural rather than mechanism-bound is itself a finding: the benchmark’s difficulty gradient does produce scenarios that challenge individual arms, but at the burst schedule the four arms together cover the full gradient.

The same complementarity shows in the distance gradient, seen from the other side. Neither clustering arm’s distance correlations reached significance, yet the cosine novelty detector’s are strongly negative on the identical scenarios. On Science/Tech, the steady-schedule correlation (rho = −0.88) survives the multiple-comparison correction; the corresponding General News correlations do not and are read as suggestive rather than established (Section 3.2). The difficulty gradient is real, established on Science/Tech and preserved as a within-space ordering under both encoders, and it operates through one detector’s mechanism, where it stayed invisible through the others.

Complementarity is legible only because every arm ran on the same scenarios under identical conditions. A single-detector evaluation would have missed the schedule dependence; a single-schedule evaluation would have missed the distance reversal; a single-paradigm evaluation would have missed the emergence analog at the paradigm boundary. That is the benchmark’s diagnostic contribution, and ad hoc evaluation, without matched scenarios or controlled conditions, cannot deliver it.

Detection rate carries a second, subtler unreliability that the benchmark exposes. Rate can look strong while the detections behind it are generic absorptions, and without ground truth, a reader cannot tell the difference; the fully detected, sub-floor General News far tier of Section 4.1 is the case in point. The deeper problem is directional. On one corpus, detection quality falls as scenarios move farther away; on the other, it rises. The same rate metric tells opposite stories depending on corpus structure, and rate on its own gives no signal that it is doing so. A measure that is unreliable in the direction of its error, and not only in its size, cannot be interpreted alone; a second axis is needed to fix the direction, and that is what specificity supplies alongside the rate. The demonstration also contains the divergence in its sharpest form. DenStream’s three-seed detection range sits above DBSTREAM’s at every threshold tested on both corpora, its false-positive floor sits far below, and every one of its 89 detections across those seeds is an absorption. The rate axis moved substantially between the two arms; the quality-axis behavior did not move at all. A detector that dominates on rate is exactly as absorption-bound as the one it dominates, which is the benchmark’s argument for reporting the two axes together. The comparison describes what this protocol surfaced on these corpora; the benchmark ranks behaviors, not detectors.

We therefore propose reporting detection significance and detection specificity as a candidate standard companion to detection rate for clustering-based detection evaluation. Significance, from the permutation null, asks whether a scenario’s co-occurrence exceeds chance. Specificity, from the enrichment ratio, asks whether the detection is topically concentrated or a generic absorption. For a near-corpus scenario like velosphere, this is the difference between detecting the fictitious topic and detecting the genuine neighboring coverage it was designed to resemble: enrichment separates the synthetic articles pulling together from the articles landing where their neighbors live. The two axes can diverge: naledi_okoro on GN is significant at p < 5 × 10⁻⁵ and non-specific at 2.2× enrichment, passing one test and failing the other, which is why one number cannot stand in for both. The axes are separable in the other direction as well: on General News, the count-based permutation null saturates for DenStream, background co-occupancy alone reaching the detection criterion so often that the significance axis stops discriminating there, and enrichment carries the discrimination the count cannot (Section 3.3). Within this benchmark, the framework has been exercised on both micro-cluster detectors and both encoders, with the specificity floor derived separately per detector rather than assumed shared. Exercised on one benchmark, it remains a candidate companion and not a settled standard. Placing a detection on the two axes requires knowing which cluster absorbed which scenario, which is to say it requires the ground-truth membership the benchmark supplies and ordinary evaluation does not.

4.4. Ecological Validity

Synthetic injection is cleaner than real topic emergence, by design. A real topic breaks across many outlets at once, in conflicting detail and uneven volume, with its early articles being written by people who do not yet agree on what the story is. An injected scenario has none of that noise: fixed brief, controlled arrival, one generator. The benchmark trades this realism for control, and it is the same trade every synthetic benchmark makes. LFR graphs are not real social networks [5]; the TDT topics were curated by hand. The trade is worth making because the benchmark needs known ground truth, controlled difficulty, and reproducibility, and for this task ground truth and realism pull against each other. A realistic stream carries topics that are exactly what a detector is meant to discover, and a topic cannot be labeled before it has been identified, so the realism that would make the stream lifelike is the very thing that denies it the ground truth the benchmark depends on. A missed detection cannot be measured against topics that were never labeled.

This trade-off comes with a cost. Since the injected clusters are tightly bound (roughly 0.91 mean pairwise similarity versus a natural-cluster median near 0.56), the benchmark sits at the easy end of the difficulty range, and detection rates on it are an optimistic estimate of what the same detectors would achieve on messier real topics. That makes the rates an upper bound on what the same detectors would achieve on messier real topics. The obvious worry is that the bound is too loose to mean anything, that the tightness alone manufactures the detections. The variance robustness test was built to check exactly this. Degrading the synthetic embeddings with calibrated noise down to a mean pairwise similarity of 0.50, below the natural median, left detection unchanged across all seven informative single-seed conditions, while corpus proximity was held above the signal-destruction floor throughout. Internal cluster tightness does not drive the reported rates. The most obvious way the upper bound could have been inflated has therefore been tested and ruled out.

What the test does not settle is the other half of the realism gap. It degraded the scenarios’ internal tightness while deliberately holding their distance to the corpus fixed, so it says nothing about corpus-relative dispersion, the possibility that a real emerging topic is internally varied and also spread more widely against the surrounding coverage than an injected scenario is. That second axis remains open. Generation strategies that produce corpus-relative spread, and not just internal variance, are the natural next step for narrowing the gap the benchmark currently trades away.

4.5. Reproducibility Scope and Shelf Life

Reproducibility means two different things for this benchmark, and the distinction matters for API-based generation. The released artifact is reproducible in the strong sense. It fixes the articles, their embeddings, the injection schedules, and the evaluation code, so every experiment reported here reruns deterministically on any machine from the released embeddings and ground truth, both frozen at release. The generation recipe is reproducible only in the weak sense. It calls API models that are non-deterministic and will eventually be deprecated, so regenerating the v1 articles from the same prompts would produce different articles. Every result the paper reports rests on the artifact, which is fixed; none rests on the recipe, which is not fixed. Non-determinism in generation cannot reach a number in this paper. The recipe’s role is to build the next instantiation, not to verify this one.

The recipe does raise one problem the artifact cannot escape: the v1 articles will eventually enter model training corpora, and once they do, the fictitious topics stop being reliably fictitious. Detecting that moment is a provenance question, and the release addresses it with a canary. A contamination-check string ships with the release, out of band from the article text, so it leaves the embeddings untouched, and a model trained on the release can later be probed for it. A positive probe means the v1 articles have entered that model’s training data. The canary does not prevent contamination; it makes contamination visible, which is what gives our refresh protocol something to trigger on. Detection and refresh together do not close the shelf-life problem; they manage it: the release anticipates its own eventual staleness and ships the instrument for noticing when it arrives.

4.6. Position Among Evaluation Approaches

The existing evaluation approaches each supply part of what proactive topic detection needs and leave out a different part. The Topic Detection and Tracking program [2] established the task but on fixed retrospective datasets with no controlled difficulty, and those datasets are decades old. SAIBERSOC [7] injects synthetic attack scenarios assembled from the MITRE ATT&CK framework into security operations, which gives control but targets analyst workflows, not detection algorithms, and carries no semantic difficulty gradient. Shi et al. [8] plant topic structure in fully synthetic corpora, which is reproducible and controlled but replaces the real corpus entirely and evaluates batch topic models instead of streaming detection. The Synthetic Fire Drill is, to our knowledge, the first to combine LLM generation, a calibrated semantic difficulty gradient, multiple injection schedules, and streaming evaluation in a single releasable package. This specific integration of methods represents the main contribution of the paper, as the demonstration required all these elements to operate simultaneously. The absorption census was only countable because ground truth and the difficulty gradient were present together; the complementarity map required the same scenarios run under matched conditions across the detector arms; the two-axis divergence required enrichment computed against known membership. None of the earlier approaches supplies all of the required elements at once: controlled ground truth, a calibrated difficulty gradient, injection into real streams, and matched streaming evaluation. The combination surfaced what the separate pieces could not.

4.7. Limitations

The benchmark rests on a set of design choices, with each carrying a scope boundary. This section maps those boundaries so that the findings of this research can be read with their generalizability constraints visible.

4.7.1. Corpus Representation

Several inherent biases within the collected corpora must be acknowledged. First, there is a pronounced language and geographical bias, as the sources are exclusively English-language platforms predominantly based in North America, the UK, and Australia, meaning non-Anglophone media narratives are underrepresented. Second, the crawler could only capture articles that were still live and accessible during the active collection period. Finally, the selection of specific outlets introduces an editorial skew representing mainstream and established scientific discourse, potentially missing highly localized or alternative narrative formations.

4.7.2. Synthetic Article Properties

Synthetic articles cluster more tightly than natural topics, with a mean pairwise similarity of 0.91 against a natural-cluster median near 0.55 to 0.58. The variance robustness test within our experimental protocol confirmed that this tightness does not drive the reported detection rates, but it also exposed the margin at which GN detection operates. Co-membership counts on GN shrink progressively as tightness is degraded: veristhorn from 12 to 11 to 11 to 6, luma_krill from 12 to 11 to 10 to 9, both trajectories ending at the 0.50 target, below the natural median. velosphere sits at a maximum co-membership of 6 already at the baseline, the thinnest margin in the grid, and holds between 5 and 6 throughout. ST remained steady at 11 to 12 across all conditions. Detection at N = 5 survived every informative single-seed condition, but GN cells near the threshold would flip under still-harsher degradation; across three seeds, the sole failure is quietmantis on GN at the 0.50 target, a fragmentation-borderline scenario that is already seed-sensitive at the baseline. Robustness on GN holds, with shrinking margins. One dimension of the tightness gap remains untested: the degradation perturbed internal similarity while holding each article’s distance to the corpus fixed, leaving corpus-relative dispersion outside the test’s scope.

4.7.3. Detector and Embedding Coverage

The demonstration evaluates four detector arms spanning three paradigms: two micro-cluster stream clusterers (DBSTREAM, DenStream), a windowed topic-model detector (BERTrend), and the cosine novelty detector, with the primary clusterer being additionally run in a second embedding space. Each finding remains conditioned on the arms that produced it: the absorption census covers the micro-cluster family and, at the lineage level, the topic-model paradigm; the clean distance gradient belongs to per-article novelty scoring; and the coverage map is specific to this set of mechanisms. Detector families outside the set, with incremental variants of BERTopic [41] and supervised streaming detectors being among them, remain untested, and the adoption protocol, exercised twice in this study, is the intended route for them. Embedding coverage is partial rather than complete: two encoder families were evaluated, the distance gradient is preserved as a within-space ordering in both, and detection agreement on the shared subset is 9 of 12 cells, with the three disagreements running in both directions, two cells detected only in the second space and one only in the first. Absorption targets, tier boundaries, and the specific numbers remain functions of the representation space, and per-corpus recalibration of the clustering threshold is the porting step the protocol requires. The methodology ports; the specific numbers do not.

4.7.4. Sample Size

Sixteen scenarios, with calcifex excluded for its structural cap, leave n = 15 for every scenario-level statistical test. The sample resolves strong effects cleanly: the cosine steady gradient on ST (rho = −0.88) is significant after multiple-comparison correction (q < 0.001). It cannot resolve the weaker DBSTREAM distance relationships, where the strongest individual result, the raw binary GN correlation of +0.49, has a bootstrap CI of [0.0, 0.81], consistent with anything from no effect to strong positive. The grouped tier comparison is accordingly the more informative level of analysis, but both levels are constrained by the same n. A larger scenario set would sharpen the statistical picture, and the generation methodology supports scaling without a redesign.

4.7.5. Corpus Scope and Temporal Confound

Both evaluation corpora are news article collections. The benchmark methodology applies to any text stream, but the specific findings about absorption patterns, distance gradients, and corpus-dependent detection quality are conditioned on the structure of news coverage. One structural feature of the GN corpus deserves particular attention: approximately 71% of the full corpus comes from two outlets (The Guardian and Reuters), rising to 74.5% within the evaluation slice. The dominant background mega-cluster distribution, where eight of ten GN detections funnel into two generic absorbers, could partly reflect this source concentration rather than a general property of broad news coverage. Measured outlet composition gives the concern only partial support: the two dominant outlets sit at or below their stream base rate in most General News absorbers, so the funneling is not a simple echo of source concentration, though the evidence remains observational, and a reweighted-corpus test has not been run. A more evenly sourced corpus might still distribute absorption across a larger set of clusters. The scientific literature, social media, and regulatory filings have different topical densities, volume profiles, and source-diversity characteristics; the porting protocol accommodates domain transfer, but the behavioral findings would need re-evaluation per domain. Both corpora and all generated articles are English-language; cross-lingual transfer is untested. No additional corpus has yet been run through the porting protocol, so its friction on the corpus side is unmeasured.

The GN corpus is from 2019; the ST corpus is from 2025–2026. Differences in detection behavior between the two corpora could reflect temporal mismatch rather than topical structure alone, and the benchmark cannot separate these factors with two corpora. Temporal mismatch enters in two distinct ways. The first is knowledge leakage: both generators postdate the GN corpus, so generated “2019” articles could carry post-2019 knowledge despite the temporal-grounding instruction. The term-prevalence check found a single exception, cordis_ai, whose premise inherently requires post-2019 vocabulary. The second is an instrument-level confound that no content check can reach: both embedding models postdate the GN corpus and may represent text from the two eras differently in ways that the benchmark cannot distinguish from topical-structure effects. The six-year gap keeps the GN background free of the technology vocabulary that saturates the ST era, sharpening the corpus contrast the benchmark exploits; and for the technology-flavored scenarios, a 2019 stream cannot contain genuine coverage of the injected concepts, which adds a chronological anchor on top of the kNN uniqueness screen. Multi-corpus extension across corpora from comparable time periods would isolate topical structure from both confounds.

4.7.6. Dataset Shelf Life and Responsible Release

The v1 articles will eventually enter LLM training corpora, at which point the fictitious topics stop being reliably fictitious for models trained after release. The canary mechanism and refresh protocol manage this by making contamination detectable and scenario regeneration inexpensive. The artifact contains 884 realistic news articles about entities that do not exist, and synthetic news text can be misused. Three properties of the release limit that risk: every entity was screened to be fictitious, so no real person, organization, or product is implicated; every article ships inside labeled manifests that mark it as machine-generated, with ground-truth membership attached; and the canary string makes downstream training contamination detectable. The release documentation states detector evaluation as the intended use.

5. Conclusions

This paper has introduced the Synthetic Fire Drill, a reproducible benchmark for evaluating proactive novel topic detection in text streams, a task whose evaluation offered no shared ground truth against which detector failures could be counted. The methodology generates fictitious topics through LLM prompting, injects them into real news corpora at controlled difficulty, and evaluates detection against complete ground truth. A demonstration across two corpora, two generators, four injection schedules, four detection approaches from three paradigms, two sentence encoders, and 16 scenarios spanning a calibrated semantic-distance gradient yielded three findings. Every detection by a micro-cluster stream clusterer arose through absorption into pre-existing clusters, with no formation event across 66 DBSTREAM and 89 DenStream detected cells over three seeds and all 10 cells under the second encoder, while the windowed paradigm alone produced an emergence signal alongside absorption. The approaches proved complementary along mechanism lines, with the two clusterers nesting and the cosine novelty detector responding where clustering is weakest. And detection rate required a quality measure alongside it, which the benchmark’s ground truth makes computable as enrichment; the sharpest divergence is DenStream detecting more scenarios than DBSTREAM at every tested threshold, with disjoint seed ranges, while holding a flat false-positive floor. Each of these findings depends on ground truth that only controlled injection can supply.

The current methodology controls scenarios’ internal variance but not their spread relative to the background corpus. Generation strategies that produce corpus-relative spread would narrow this gap. The adoption protocol has been exercised twice within this study, integrating DenStream and BERTrend, and remains open to untested families, including incremental BERTopic and supervised streaming detectors; the generation methodology supports scaling to larger scenario sets without redesign. Extending the evaluation to corpora from comparable time periods would isolate topical-structure effects from the temporal confound that the current two-corpus design cannot resolve. The contribution is the methodology, while the v1 artifact is its first instantiation.