Detecting AI-Generated Text and Code: An Empirical Study of Cross-Generator and Cross-Domain Generalization

Abstract

Large language models (LLMs) now generate fluent natural language and source code, creating challenges for authorship attribution, academic integrity, and software supply-chain security. Most existing detectors for AI-generated content are evaluated separately on natural language or source code, often under matched train–test conditions that can overestimate real-world reliability. We present a paired-prompt benchmark for human-versus-machine detection across English text, Python code, and mixed text–code documents. The benchmark includes $\mathrm{22,141}$ instances from HC3, CodeSearchNet, MBPP, and HumanEval across training, validation, and test partitions, plus Mix-Eval, a mixed-content set of 997 Jupyter-notebook-style samples. We evaluate RoBERTa-large for text, GraphCodeBERT and CodeBERT-base for code, a unified RoBERTa-base detector trained on both modalities, and zero-shot baselines. Fine-tuned detectors achieve near-perfect in-distribution performance, with AUROC $1.0000\pm 0.0000$ and accuracy above $99.5\%$. Across five instruction-tuned generator families of varying size (3.8B–7B) and architecture, with the human and problem distributions held fixed, cross-generator transfer causes negligible degradation (AUROC spread $0.0002$; drops of at most $0.0003$). In contrast, domain shift is the main failure mode: on MBPP+HumanEval, GraphCodeBERT drops to $0.85\pm 0.02$ AUROC and CodeBERT-base to $0.67\pm 0.02$. On Mix-Eval, the unified detector outperforms a routed text–code pipeline by 21 AUROC points ($0.96$ vs. $0.75$), largely because of router failures on mixed inputs. Training-time augmentation improves low-false-positive performance, while legacy supervised detectors show systematic class inversion on modern LLM outputs. These results show that reliable deployment requires cross-domain evaluation, mixed-content testing, and calibration beyond in-distribution accuracy.

1. Introduction

The boundary between human-authored and machine-generated content is becoming increasingly difficult to observe. High-capacity large language and code models, including most prominently GPT-3 [1], GPT-4 [2], the Claude family [3], LLaMA [4], and StarCoder [5], now produce fluent essays, explanations, dialogues, tutorials, peer-review-style commentary, and source code that often resembles human writing at the surface level, making authorship attribution a growing challenge for education, online information ecosystems, and software engineering [6]. Human judgment alone is unreliable in this setting: prior studies show that non-expert evaluators often struggle to distinguish GPT-generated text from human-authored text, and that intuitive cues for identifying AI-written language can be misleading [7,8]. At the same time, evidence of large-scale AI-assisted writing is emerging in real-world settings, including scientific peer review [9], and online news websites [10].

The same capabilities that make these systems useful also create practical risks. In education, machine-generated writing can be submitted as original work, raising concerns about academic integrity and assessment design [11,12]. In online platforms, synthetic text can reduce the cost of producing persuasive or misleading content at scale, including for automated influence operations and misinformation campaigns [13,14,15]. In software development, the rapid uptake of code-generation assistants [16,17] means that generated code may be accepted into repositories without sufficient review, complicating attribution and licensing and potentially introducing vulnerable code patterns [18,19]. These risks motivate reliable post hoc detectors that can operate when provenance metadata, watermarking, or access to the original generator are unavailable.

These concerns have motivated a growing literature on automatic detection of AI-generated content. Existing methods can be broadly grouped into three families. The first family consists of zero-shot statistical detectors, which use signals from a reference language model, such as token likelihood, rank, perplexity, or likelihood curvature, without training a task-specific classifier  [20,21,22]. The second family consists of supervised detectors, which train classifiers on labeled human- and machine-written samples, often using pre-trained transformer encoders or probability-based feature sets  [23,24,25]. The third family consists of watermarking and provenance methods, which aim to embed or record generation-time signals that can later be verified, but typically require cooperation from the model provider or content-generation pipeline [26,27]. In open-world post hoc settings, where the detector has no access to the generator, its decoding process, or reliable provenance metadata, the first two families remain the most widely applicable.

Despite this progress, much prior work has evaluated detectors either on natural-language text or on source code, rather than studying them under a unified text-and-code detection setting [21,24,25]. Moreover, recent work has shown that detector performance can degrade substantially when models are evaluated beyond their training conditions, including under unseen generators, shifted domains, and adversarial rewriting [28,29,30]. We therefore focus on three deployment-relevant failure modes. First, generator shift occurs when a detector trained on outputs from one model family is evaluated on outputs from a different model family. Second, domain shift occurs when a detector trained on one kind of content, such as open-domain answers or utility functions, is evaluated on another kind of content, such as medical answers, peer reviews, programming-contest solutions, or tutorial-style code. Third, baseline drift occurs when detectors originally developed for older generators are reused as baselines for newer instruction-tuned LLMs, even though the distribution of generated text has changed substantially [23,29]. These issues are especially important for deployment because high in-distribution accuracy does not guarantee robustness under unseen generators, unseen domains, or evolving model families. The present paper directly studies these failure modes through a paired-prompt benchmark covering English natural language and Python source code. We ask the following research questions:

• RQ1 (Effectiveness and generator shift). Are supervised detectors accurate in-distribution, and do they survive a cross-family generator swap?
• RQ2 (Domain shift). Do they generalize across content domains within a modality, including an external benchmark?
• RQ3 (Mixed content and routing). On interleaved prose-and-code inputs, is explicit modality routing necessary, and which fusion rule is best?
• RQ4 (Robustness). How robust are detectors to lightweight post-generation transformations, and does training-time augmentation help under distribution shift?
• RQ5 (Baselines). How do legacy and zero-shot baselines (OpenAI RoBERTa, GPT-2-XL log-perplexity) fare on modern LLM outputs?
• RQ6 (Component contributions). What do hand-engineered features and post hoc calibration/threshold tuning add, especially out-of-distribution?

Designing detectors that work across both natural language and source code is non-trivial. The two modalities differ in syntax, semantics, and statistical structure. English text may expose generation artifacts through lexical choice, stylistic regularity, repetition patterns, and language-model likelihood statistics, whereas Python code may expose artifacts through identifier naming, formatting conventions, control-flow structure, comment density, and Abstract Syntax Tree (AST) complexity. These differences raise two central questions: (i) whether a single shared encoder can capture detection signals across both modalities, or whether modality-specific encoders are necessary; and (ii) whether detectors trained under one generator, content domain, or post-generation transformation transfer reliably to others. These questions are especially important in deployment, where detectors may face unseen generators, shifted content domains, edited outputs, and evolving model families. In such settings, in-distribution accuracy alone is insufficient for assessing detector reliability.

In this paper, we study these challenges through a paired-prompt benchmark spanning English text and Python source code. The English track is based on HC3 [31], while the Python track uses CodeSearchNet [32] for training and MBPP [33]+HumanEval [34] as a contamination-suspect held-out split. We evaluate four supervised detectors and two zero-shot reference baselines. For natural language, we fine-tune a RoBERTa-large detector [35]. For Python code, we compare GraphCodeBERT [36], which incorporates data-flow information, against CodeBERT-base [37]. To test whether a single shared encoder can support both modalities, we also train a unified RoBERTa-base model [35] on the union of text and code samples. Finally, to assess transfer beyond the training distribution, we evaluate each detector along six axes: in-distribution performance, cross-generator transfer from Qwen2.5-7B-Instruct [38,39] to Mistral-7B-Instruct-v0.3 [40] for text and from Qwen2.5-Coder-7B-Instruct [41] to DeepSeek-Coder-6.7B-Instruct [42] for code, cross-domain transfer (CodeSearchNet → MBPP+HumanEval), mixed-content evaluation on Jupyter-notebook-style inputs that interleave prose and Python (Mix-Eval), lightweight post-generation transformations, and comparison against zero-shot baselines including the OpenAI RoBERTa detector [23] and a GPT-2-XL log-perplexity threshold [43]. For the transformation setting, we use same-family paraphrasing for text and Abstract Syntax Tree (AST)-based variable renaming with comment removal for code. Our contributions are as follows:

• Paired-prompt human-vs.-machine benchmark. We construct a paired-prompt benchmark with three splits (text, code, and code2), totaling 22,141 train/validation/test instances. The benchmark is deduplicated via MinHash near-duplicate detection, length-matched between human and machine outputs, and stratified by source.
• Controlled comparison of modality-specific and unified detectors. We evaluate four supervised detectors, RoBERTa-large, GraphCodeBERT, CodeBERT-base, and a unified RoBERTa-base model, across 29 evaluation conditions covering in-distribution performance, cross-generator transfer, cross-domain held-out transfer, mixed-content evaluation, and lightweight post-generation transformations. The unified detector matches modality-specific detectors on every same-modality condition tested.
• Domain shift is more limiting than generator shift. Across the fine-tuned detectors, in-distribution AUROC is near-perfect, but performance drops to 0.68–0.85 AUROC on the contamination-suspect held-out split. By contrast, cross-generator transfer induces an AUROC drop of order ${10}^{-3}$, indicating that content-domain shift is a stronger failure mode than generator-family shift in this benchmark. A controlled multi-generator evaluation across five instruction-tuned families (Appendix C) confirms that this robustness is not specific to a single model pair, while a per-generator decomposition of the SemEval transfer locates the boundaries of generalization at the human-content distribution and the instruction-tuned/base divide.
• Data-flow structure improves out-of-domain code detection. GraphCodeBERT’s data-flow encoding provides no measurable advantage in-distribution, but yields a 17-point AUROC improvement over CodeBERT-base on the held-out code domain. This suggests that structural code representations generalize better than token-distribution features under domain shift.
• Legacy and zero-shot baselines behave unexpectedly. A GPT-2-XL log-perplexity threshold reaches 0.87 AUROC on the contamination-suspect held-out split, outperforming every fine-tuned encoder in that setting. In contrast, the pre-2023 OpenAI RoBERTa detector achieves AUROC below 0.5 on Qwen2.5 and Mistral-7B outputs, indicating systematic class inversion rather than merely weak detection performance.
• Modality routing is bottlenecked by router coverage on mixed-content inputs. On a held-out evaluation set of 997 Jupyter-notebook-style samples that interleave prose and Python code (Mix-Eval), the unified single-encoder achieves AUROC $0.96$, while an explicit AST-parse-based routed pipeline that combines RoBERTa-large and GraphCodeBERT through entropy-weighted fusion achieves only AUROC $0.75$. The gap is explained by router coverage: the parser fails to identify a code block in $22\%$ of mixed inputs, and the routed pipeline degrades to its weakest branch on those samples. When routing succeeds (the “fused” branch), AUROC reaches $0.86$—much closer to the unified ceiling. The result reframes “modality routing” as a router-quality problem rather than an architecture-choice problem.

The remainder of this paper is organized as follows. Section 2 reviews related work on text and code detection. Section 3 describes our datasets and the construction of the mixed-content evaluation set. Section 5 presents the methodology, including the feature extraction stage, the modality-specific detectors, and the fusion and calibration components. Section 6 reports the experimental setup, model training details, and Section 7 reports the evaluation protocol. Section 8 reports baselines, ablation studies, and cross-generator generalization results. Section 11 concludes this paper and outlines directions for future work.

2. Related Work

2.1. Detection of AI-Generated Natural Language

Early work on machine-generated text detection relied on statistical signals from a reference language model. GLTR [20] visualizes token-level rank and probability statistics, motivated by the observation that machine-generated text often selects high-probability tokens under the generating model. DetectGPT [21] extends this intuition by using probability curvature: machine-generated passages tend to occupy regions of negative curvature in the model’s log-probability function. Fast-DetectGPT [44] reduces the computational cost of curvature-based detection by replacing the perturbation step with a more efficient conditional-probability curvature estimate. Binoculars [22] instead contrasts likelihood scores from two pretrained language models, yielding a zero-shot detector that does not require task-specific training data.

A second line of work trains supervised classifiers on labeled human-versus-machine corpora. The OpenAI RoBERTa detector [23] fine-tunes RoBERTa on GPT-2 outputs and has been widely used as a legacy baseline. Ghostbuster [24] combines probability-based features through a structured feature-search procedure, achieving strong performance without access to the generator. However, supervised detectors are often sensitive to distribution shift: performance can degrade under paraphrasing attacks [28], unseen generators [29], or changes in content domain. These findings motivate evaluation protocols that go beyond in-distribution accuracy and explicitly test cross-generator and cross-domain transfer.

Watermarking provides a complementary approach. Instead of detecting post hoc statistical artifacts, watermarking methods modify the generation process so that model outputs contain a hidden, algorithmically detectable signal [26]. Such methods are attractive when the generation provider cooperates, but they are less applicable in open-world settings where the detector has no access to the generator, its decoding process, or provenance metadata. Our work therefore focuses on post hoc detection.

2.2. Detection of AI-Generated Code

Detection of machine-generated source code has received less attention than detection of natural-language text, despite the rapid adoption of code-generation tools such as Codex (https://github.com/openai/codex, accessed on 10 January 2026), Copilot (https://github.com/features/copilot, accessed on 10 January 2026), StarCoder [5], and other instruction-tuned code models. GPTSniffer [25] proposes a CodeBERT-based classifier for detecting AI-generated Python and reports strong performance on paired human and ChatGPT-generated code snippets. DetectCodeGPT [45] adapts the zero-shot detection idea to source code by exploiting stylistic patterns such as whitespace and newline distributions. Recent code-detection benchmarks extend evaluation across multiple generators, programming languages, and perturbation settings, highlighting the need for robust evaluation beyond a single model or dataset.

AI-generated code detection is also related to classical source-code stylometry. Prior work on programmer authorship attribution shows that identifier choice, formatting conventions, control-flow structure, and Abstract Syntax Tree (AST)-derived features can reveal author-specific coding style [46]. These signals are relevant to AI-code detection because generated code may differ from human code not only at the token level, but also in structural regularities, naming patterns, and implementation style. Our code experiments therefore compare a token-oriented encoder, CodeBERT, against GraphCodeBERT, which explicitly incorporates data-flow information.

2.3. Calibration, Robustness, and Unified Evaluation

A separate concern is calibration. A detector may achieve high accuracy while producing probabilities that do not correspond to true error risk, which is problematic in deployment settings where decisions must be made at low false-positive rates. Standard calibration methods such as Platt scaling [47], isotonic regression, and temperature scaling [48] provide general tools for correcting overconfident classifiers, but calibration is still underreported in much of the AI-generated-content detection literature.

Finally, prior work has largely treated natural-language detection and source-code detection as separate problems: AI-text detectors are commonly evaluated on natural-language corpora [20,21,22,24], whereas AI-code detectors are evaluated on source-code benchmarks  [25,45,49]. As a result, it remains unclear whether a single shared detector can capture useful signals across both modalities, or whether modality-specific encoders are necessary. It is also unclear whether detectors that perform well in-distribution remain reliable under generator shift, domain shift, or lightweight post-generation transformations. Our study addresses this gap by evaluating text and code detectors under a shared paired-prompt protocol, comparing modality-specific and unified encoders, and reporting performance across in-distribution, cross-generator, cross-domain, and lightweight transformation settings.

3. Dataset Curation

We construct a benchmark that pairs human-written and machine-generated samples across two modalities (English text and Python source code) drawn from three public corpora. To enable controlled comparison, we use a paired-prompt design: each machine sample is generated from the same prompt that elicited a human reference, so distributional differences cannot be attributed to a mismatch in input distributions. Table 1 summarizes the composition.

Table 1. Composition of the unified benchmark, after all post-processing (MinHash deduplication, length matching, label balancing). Token counts are computed with the GPT-2 BPE tokenizer for text-modality subsets and the CodeBERT BPE tokenizer for code-modality subsets. The MBPP+HumanEval, Mix-Eval, X-Gen, and SemEval-2026-Task13 Python subsets are evaluation-only and are never used at training time. ^† highlights evaluation-only subsets. SemEval-2026-Task13 Python is an external benchmark for which we perform no curation; the others were constructed in this work.

Subset	Human	Machine	Generator	Avg. Tokens	Domain
HC3 (text)	4906	4999	Qwen2.5-7B-Instruct	202	ELI5, finance, medicine, open QA, Wiki
CodeSearchNet-Py (code2)	5000	4993	Qwen2.5-Coder-7B-Instruct	193	OSS Python functions
MBPP+HumanEval (code) †	1130	1113	Qwen2.5-Coder-7B-Instruct	88	Programming tasks
Mix-Eval †	497	500	Qwen2.5-7B + Qwen2.5-Coder-7B	358	Jupyter notebook tutorials
X-Gen text †	495	505	Mistral-7B-Instruct-v0.3	191	(HC3 prompts, held-out test)
X-Gen code †	500	500	DeepSeek-Coder-6.7B-Instruct	228	(CodeSearchNet prompts, held-out test)
SemEval-2026-Task13 Python †	3529	3575	21 generators (Qwen, Llama, Phi, Yi, Granite, …)	154	Algorithmic Python (external)

3.1. English Text

Human-written text is sampled from the human side of the HC3 corpus [31], which spans five domains: ELI5-style explanations, open-domain QA, computer-science Wikipedia, finance, and medicine. We retain at most one human answer per question, filter by word count to the $[22,298]$ window (the empirical $p_{10}$–$p_{90}$ range), and draw 5000 samples stratified by source.

Machine-generated text is produced by prompting Qwen2.5-7B-Instruct [38,39] with the same HC3 questions (see prompt A.1) that elicited each human answer. We apply the model’s chat template with a neutral system message and sample with nucleus parameter $p=0.95$ and a maximum of 384 generated tokens. To prevent the detector from latching onto a single decoding mode, each prompt is generated at one of three temperatures $\tau \in \{0.3,0.7,1.0\}$, assigned round-robin across batches so each temperature appears in roughly equal proportion. Appendix E verifies that detection performance is flat across the three sampling temperatures, ruling out a temperature-artifact confound. Representative paired HC3 samples (human and Qwen2.5-7B-Instruct) are provided in Appendix B.

3.2. Python Source Code

We construct two paired code splits with complementary properties. Throughout, code denotes the curated, evaluation-only MBPP+HumanEval split and code2 the CodeSearchNet training split; the numbering reflects construction order rather than a subset relationship.

• MBPP+HumanEval (curated).

We use the canonical human solutions from MBPP [33] (all 974 problems across the public splits) and HumanEval [34] (164 problems), yielding 1138 paired examples. Because both datasets are likely to have appeared in the pretraining corpus of the code generator, we use this split for evaluation only and never for training.

• CodeSearchNet (open-domain).

We sample 5000 functions from the Python split of CodeSearchNet [32], filtering to entries whose first-paragraph docstring is between 6 and 40 whitespace-tokens, whose body is between 5 and 30 lines and between 30 and 300 whitespace-tokens, and whose docstring contains no URLs or fenced code blocks. We deduplicate function bodies by exact hash before sampling.

• Generation.

For both code splits, we prompt Qwen2.5-Coder-7B-Instruct [41]. MBPP problems are presented as a natural-language statement together with one assertion as a signature hint (see prompt A.2); HumanEval problems are presented as the canonical signature plus docstring with a request to return only the function body (see prompt A.3); CodeSearchNet problems are presented as the first paragraph of the docstring (see prompt A.4). We use the same sampling configuration as for text generation ($p=0.95$, 384 max new tokens, $\tau \in \{0.3,0.7,1.0\}$). Representative paired code samples for MBPP, HumanEval, and CodeSearchNet are also provided in Appendix B.

3.3. Length Matching and Deduplication

A common pitfall in text-detection benchmarks is that surface length alone becomes a near-perfect feature. We mitigate this by truncating each machine-generated output above the per-modality human $p_{95}$ word-count to exactly $p_{95}$ tokens, while keeping shorter outputs (subject to a minimum-length floor of three whitespace-tokens to drop degenerate generations). This preserves class balance: every split is $50/50$ in expectation, with truncation applied only where length exceeds the human distribution.

After length matching, we apply MinHash-LSH near-duplicate detection [50,51] over the union of human and machine samples, using 5-token whitespace shingles, 64 permutations, and a Jaccard threshold of $0.85$, removing later-encountered duplicates. The remaining samples are partitioned into $80/10/10$ train/validation/test splits, stratified by the (label, source) pair so that every domain and source corpus is represented in every split.

3.4. Mixed-Content Set (Mix-Eval)

To evaluate the unified framework on inputs that genuinely interleave prose and code, we construct a mixed-content evaluation set, denoted Mix-Eval, in which each sample contains both natural-language explanation and a Python snippet. Human-side samples are drawn from the corpus (codeparrot/github-jupyter-text-code-pairs) which provides $(\mathrm{markdown},\mathrm{code})$ pairs extracted from public Jupyter notebooks; we filter to pairs whose markdown cell is between 30 and 200 whitespace-tokens and whose code cell is between 5 and 30 lines, and we serialize each pair as markdown>\n\ncode> to form a single mixed-content sample. Machine-side counterparts are produced on the same topic prompts using Qwen2.5-7B-Instruct for the prose section and Qwen2.5-Coder-7B-Instruct for the code section (see prompts A.5 and A.6), with the same sampling configuration as the rest of the benchmark. The two sections are concatenated in the same format as their human counterparts. Mix-Eval is used as an evaluation set only, no detector is trained on these samples, and the unified detector is compared against a modality-routed variant that splits each input via an AST-parse-based router (Markdown fences first, then contiguous indented blocks attempted with ast.parse; remaining text is prose) and applies the modality-specific detectors to each component.

3.5. External Evaluation: SemEval-2026 Task 13

For a zero-shot transfer evaluation, we additionally use the publicly available SemEval-2026 Task 13 Subtask A (binary machine-generated code detection) [52,53]. We restrict to the Python subset, yielding 7104 samples (3529 human, 3575 machine) drawn from algorithmic-programming corpora and generated by 21 distinct LLMs spanning the Llama [4], Qwen [38,39,41], Phi, Yi, Granite, CodeLlama, and Mistral [40] families. Unlike the splits in Section 3.1, Section 3.2, Section 3.3 and Section 3.4, we perform no curation on this set: no length-matching, no deduplication, and no paired-prompt construction. The set is used only at evaluation time; no detector in this paper is trained on it. We note one practical issue: the trial parquet swaps the polarity of the label column relative to the official id_to_label.json mapping (in the trial set, 0 = machine, 1 = human). We relabel to our convention (0 = human, 1 = machine) before scoring; results are reported in Section 8.4.

4. Prompt Templates

The prompts are intentionally minimal. For HC3, the system prompt is “You are a helpful assistant” and the user message is the HC3 question verbatim. For MBPP, the user message asks for a Python function that solves the problem and that passes the first assertion in the test_list, and asks the model to return only the function definition. For HumanEval, the user message provides the signature and docstring and asks for the function body only. For CodeSearchNet, the user message asks for a function that does what the first paragraph of the documentation string describes, returning the full definition with no explanation. The complete prompt templates are listed in Appendix A.

For all generated samples, we use each model’s native chat template and decode with nucleus sampling ($p=0.95$), a maximum of 384 new tokens, and three temperature settings $\tau \in \{0.3,0.7,1.0\}$ assigned round-robin across batches.

5. Methodology

We formulate human-vs.-machine detection as a binary classification problem. Given an input sample x, which may consist of English text, Python source code, or a mixed document containing both, the goal is to predict $y\in \{0,1\}$, where 0 denotes a human author and 1 denotes a machine author. We design a modular pipeline that separates representation learning and decision-making into clearly defined stages: (i) modality routing, (ii) feature extraction, (iii) modality-specific detection, (iv) probabilistic fusion, and (v) calibrated thresholding. An overview is given in Figure 1.

Figure 1. The detection pipeline. A rule-based modality router splits the input into text and code components via Markdown-fence extraction followed by ast.parse validation on indented runs. Each non-empty modality is processed by a dedicated feature extractor that concatenates the encoder [CLS] embedding with a vector of hand-engineered features (17-d for text, 14-d for code). Two modality-specific MLP detectors output independent probabilities $p_T\left(x\right),p_C\left(x\right)$, which are combined either by entropy-weighted fusion or by a learnt head and then passed through Platt scaling and a validation-tuned threshold to yield the final binary prediction $\widehat{y}$ and the calibrated confidence $p_{\mathrm{cal}}$. The hand-engineered feature path, the probabilistic-fusion stage, and the Platt-scaling/threshold-tuning stage are evaluated as ablations (Section 8.6, Section 8.10, and Section 8.11); the headline detectors reported in the main results use the encoder+MLP path alone.

5.1. Modality Routing

The router decomposes a raw input x into a textual component $x_T$ and a code component $x_C$. It is rule-based and does not learn parameters. The algorithm proceeds in two passes:

1.

Markdown-fence extraction. Any region matching the Markdown fenced-code-block pattern ‘‘‘...‘‘‘ (with or without a language tag) is added to $x_C$. The remaining text is the input to pass 2.

2.

AST-parse validation on indented runs. On the residual text, we identify contiguous runs of ≥3 lines whose first non-whitespace character matches a regular expression over Python keywords and operator patterns (def, class, import, from, for, while, if, elif, else, try, except, finally, with, return, @ decorators, and assignment patterns of the form ident =...). For each candidate run, we attempt ast.parse; if parsing succeeds, the run is appended to $x_C$ and removed from $x_T$, otherwise it remains in $x_T$.

Tokens not assigned to a code block default to $x_T$. For purely textual or purely code inputs, one of the components is empty, and the corresponding branch of the pipeline is skipped. The router is purely deterministic; closing the routing-coverage gap that we identify in Section 8.5 (where the router fails to detect code in $22\%$ of mixed inputs) by training a learned per-line classifier is left to future work.

5.2. Feature Extraction

Each non-empty modality is processed by a dedicated feature extractor that produces two complementary representations: a dense transformer embedding that captures distributional regularities, and a vector of hand-engineered features that captures interpretable surface statistics.

5.2.1. English Features

For English text, we use the pooled [CLS] representation of a fine-tuned RoBERTa-large [35] encoder as the dense representation. The hand-engineered feature vector ${\varphi }_T\left(x_T\right)\in {\mathbb{R}}^{17}$ comprises the following: token-level negative log-likelihood mean/std/max and log-rank mean/max under a frozen GPT-2-XL reference [43]; burstiness defined as the coefficient-of-variation of inter-sentence NLL; type-token ratio (TTR) and the moving-average TTR with window 50 (Covington and McFall); sentence-length mean and variance; punctuation frequency; the proportion of function words from a fixed 100-word list; an n-gram repetition score (maximum 4-gram repetition rate over windows of 200 tokens); three readability indices (Flesch, Gunning Fog, Coleman–Liau); and the average token length in characters. Features are standardized by mean and standard deviation computed on the training split.

5.2.2. Python Features

For code, we use the pooled [CLS] representation of a fine-tuned GraphCodeBERT [36] encoder, which incorporates data-flow information; we additionally report a CodeBERT-base [37] variant as an ablation. The hand-engineered feature vector ${\varphi }_C\left(x_C\right)\in {\mathbb{R}}^{14}$ captures the following: mean and maximum AST depth, parsed via Python’s ast module; cyclomatic complexity (McCabe) computed via radon; identifier entropy defined as the Shannon entropy of identifier-character 3-grams; comment density (comment lines/total lines) and a docstring presence indicator; indentation regularity, defined as the variance of leading-whitespace counts modulo 4; line-length variance; the ratio of single-letter to multi-letter identifiers; densities of three Python idioms (list comprehensions, with statements, f-strings); blank-line ratio; and built-in call density.

5.3. Modality-Specific Detectors

The English detector $f_T$ and the code detector $f_C$ share an architectural template: a transformer encoder, the pooled [CLS] embedding optionally concatenated with the hand-engineered feature vector, and a two-layer MLP with GELU activation and dropout ($p=0.1$) producing a single logit. We apply the sigmoid function to obtain a probability:

$$p_T\left(x\right)=\sigma \left(f_T\left(x\right)\right),p_C\left(x\right)=\sigma \left(f_C\left(x\right)\right).$$

(1)

We report two detector configurations. In the base configuration, the two-layer MLP operates on the [CLS] embedding alone and the encoder is fine-tuned end-to-end. In the handfeat configuration, the encoder is frozen and a fresh two-layer MLP is trained on the concatenation of the [CLS] embedding and the standardized hand-engineered features.

Each detector is trained on its modality’s training split using binary cross-entropy loss, the AdamW optimizer with learning rate $2\times {10}^{-5}$, weight decay $0.01$, batch size 16 for text and 32 for code, a linear warm-up over $6\%$ of training steps followed by a linear decay to zero, and bfloat16 mixed precision. We train for 3 epochs and select the checkpoint that maximizes validation AUROC.

To improve robustness, we also report an augmented configuration trained with on-the-fly input perturbations applied at probability $0.2$ per training sample: for text, we apply random sentence reordering; for code, we apply Abstract-Syntax-Tree-based local-variable renaming followed by random comment insertion or removal. Augmentations preserve the label (they are applied identically to human and machine samples) and are evaluated separately in Section 8.8.

5.4. Probabilistic Fusion

When both modalities are present in x, we combine the two predictions into a single probability. We compare three fusion rules.

• Entropy-weighted convex combination.

Let $H\left(p\right)=-plogp-(1-p)log(1-p)$ be the binary entropy. We define modality-specific confidences $c_T=1-H\left(p_T\right)$ and $c_C=1-H\left(p_C\right)$ and let

$$w_T={\displaystyle \frac{c_T}{c_T+c_C+\epsilon }},w_C={\displaystyle \frac{c_C}{c_T+c_C+\epsilon }},p_{\mathrm{ent}}\left(x\right)=w_T{p}_T\left(x\right)+w_C{p}_C\left(x\right),$$

(2)

with $\epsilon ={10}^{-6}$. The rule is parameter-free, automatically suppresses a missing modality (its detector returns a high-entropy prediction by construction), and reduces to a simple average when both modalities are equally confident.

• Learned fusion.

We also evaluate two parametric fusion heads. The first is a logistic-regression head over the detector logits, $({logit}_T,{logit}_C)$. The second is a gradient-boosted tree (GBT) head over the detector probabilities, $(p_T,p_C)$, using 100 estimators with maximum depth 3. Because no separate fusion-validation split is available for the mixed-content evaluation, both learned heads are trained and reported using 5-fold stratified cross-validation on the routed mixed-content predictions. In our experiments (Section 8.6), the GBT head outperforms entropy weighting in AUROC ($0.882$ vs. $0.861$), while logistic regression over logits is intermediate.

5.5. Calibration and Decision

Raw fused (or per-modality) probabilities can be over- or under-confident, particularly under domain shift. We apply a Platt scaling [47] step, fitting a single-parameter logistic regression

$$p_{\mathrm{cal}}=\sigma \left(a·logit\left(p\right)+b\right),$$

(3)

on the validation logits and labels by minimizing negative log-likelihood. The threshold $\tau \in [0.05,0.95]$ that maximizes macro-F1 on the validation split is selected as the operating point. At inference time, we apply the calibrated probability and the tuned threshold: $\widehat{y}=\mathbf{1}[p_{\mathrm{cal}}\left(x\right)>\tau ]$. The system additionally returns $p_{\mathrm{cal}}$ as a confidence score, which is used in our experiments to compute expected calibration error. Calibration parameters and validation-tuned thresholds are reported per detector in Section 8.11.

In deployment scenarios that require low false-positive rates, the threshold can instead be selected to bound the false-positive rate at a user-specified level; we report TPR at FPR $\in \{0.001,0.01,0.05\}$ throughout to facilitate this choice.

6. Experiments

6.1. Experimental Setup

All experiments are run on a workstation equipped with $2\times$ NVIDIA H100 NVL (95 GB) GPUs, using PyTorch and Hugging Face Transformers [54]. We train and evaluate four supervised detectors:

• RoBERTa-large [35] (355M parameters, 1024 hidden, max length 512), the headline text detector.
• GraphCodeBERT-base [36] (125M parameters, 768 hidden, max length 512), the headline code detector, with data-flow encoding.
• CodeBERT-base [37] (125M parameters, 768 hidden, max length 512), a controlled comparison to GraphCodeBERT that shares vocabulary, depth, and tokenizer but lacks the data-flow-graph encoding.
• Unified RoBERTa-base [35] (125M parameters, 768 hidden, max length 512), trained on the union of the text and code training splits to test whether a single shared encoder is sufficient for both modalities.
  Each detector is trained on a single H100 NVL; all downstream evaluations (cross-generator, mixed-content, adversarial transformations, calibration, and augmentation ablations) reuse the trained checkpoints.

6.2. Training

Both detectors are trained with binary cross-entropy on the train split, the AdamW optimizer (learning rate 2 × 10⁻⁵, weight decay $0.01$), a linear warm-up over 6% of training steps followed by linear decay to zero, three epochs total, and bfloat16 mixed precision on $2\times$ NVIDIA H100 NVL (95 GB) GPUs. The training batch size is 16 for text and 32 for code. Model selection is by best-AUROC on the val split. All supervised results in this paper are reported as mean ± standard deviation over three random seeds $\{42,123,7\}$; this includes the in-distribution, cross-generator, contamination-suspect held-out, mixed-content, lightweight-transformation, augmentation, hand-feature, calibration, and fusion-rule analyses, as well as the external SemEval-2026-Task 13 evaluation. The two zero-shot baselines (the OpenAI RoBERTa detector and the GPT-2-XL log-perplexity threshold) are deterministic and have no seed dependence. See Section 12.

• What is not in the headline detector.

We deliberately keep the headline architecture minimal. The headline numbers do not apply Platt scaling, do not concatenate hand-engineered features, and are trained without augmentation, because all three components have negligible head-room when in-distribution AUROC is already ≥0.9999. We separately report Platt scaling and macro-F1-tuned thresholds (Section 8.11), hand-feature concatenation (Section 8.10), and training-time augmentation (Section 8.8) as ablations on the out-of-distribution conditions where they are expected to help, most notably the contamination-suspect MBPP+HumanEval held-out and the mixed-content Mix-Eval set, where augmentation lifts TPR@FPR = 0.01 from 0% to 49.6% on the code branch.

7. Evaluation Protocol

We report classification accuracy (Acc), macro-F1 (F1), area under the ROC curve (AUROC), expected calibration error (ECE), and true-positive rate at fixed false-positive rate (TPR@FPR) for FPR $\in \{0.001,0.01,0.05\}$. The TPR@FPR metric is essential because deployment scenarios such as academic-integrity review and editorial moderation require operating at low false-positive rates; reporting accuracy or F1 alone hides large differences between detectors at those operating points.

ECE uses the standard binary-classification convention: confidence is $max(p,1-p)$ over 10 equal-width bins on $[0.5,1]$. All supervised results are reported as mean ± standard deviation over three random seeds $\{42,123,7\}$; the OpenAI RoBERTa detector and GPT-2-XL log-perplexity baselines are deterministic and have no seed dependence. Bootstrap $95\%$ confidence intervals (1000 resamples) for every per-seed metric are released in the project repository; we omit them from the tables for readability.

8. Results

8.1. In-Distribution Performance (RQ1, Part 1)

Table 2 reports performance on the held-out test splits, where the test data shares the human source corpus and the machine generator with the training split.

Table 2. In-distribution performance, mean ± standard deviation over three seeds $\{42,123,7\}$.

Detector	Modality	n	Acc (%)	AUROC	ECE (%)	TPR@FPR = 0.01 (%)
RoBERTa-large	text	1000	$99.63\pm 0.21$	$1.0000\pm 0.0000$	$0.36\pm 0.21$	$99.87\pm 0.11$
GraphCodeBERT	code	1000	$99.63\pm 0.25$	$1.0000\pm 0.0000$	$0.36\pm 0.19$	$100.00\pm 0.00$
CodeBERT-base	code	1000	$99.77\pm 0.06$	$1.0000\pm 0.0000$	$0.15\pm 0.04$	$99.93\pm 0.12$
Unified RoBERTa-base	text	1000	$99.57\pm 0.06$	$1.0000\pm 0.0000$	$0.39\pm 0.04$	$99.93\pm 0.11$
Unified RoBERTa-base	code	1000	$99.60\pm 0.10$	$1.0000\pm 0.0000$	$0.35\pm 0.19$	$100.00\pm 0.00$

From Table 2, we make the following observations: (i) Every fine-tuned detector reaches AUROC $1.0000\pm 0.0000$ over three random seeds, with accuracy above $99.5\%$ and TPR at FPR = 0.01 at or near $100\%$. The standard deviation on AUROC is below $0.0001$ across all four detectors, so the saturation is not a seed artefact: in-distribution detection is essentially solved on this benchmark. (ii) The unified RoBERTa-base matches RoBERTa-large on text ($99.57\pm 0.06$ vs. $99.63\pm 0.21\%$ accuracy) and GraphCodeBERT on code ($99.60\pm 0.10$ vs. $99.63\pm 0.25\%$); confidence intervals overlap completely on every metric. There is no in-distribution evidence for explicit modality routing, and the comparisons that follow (cross-domain in Section 8.3, mixed-content in Section 8.5) are required to surface architectural differences. (iii) ECE remains below $0.4\%$ on every split before any Platt-scaling or threshold-tuning step. Section 8.11 shows that this property does not survive distribution shift, which in turn motivates the calibration analysis in Section 8.11.

8.2. Cross-Generator Transfer (RQ1, Part 2)

We hold the detector fixed and replace the test-time generator with a different model family: Mistral-7B-Instruct-v0.3 for text and DeepSeek-Coder-6.7B-Instruct for code. Human-side samples are unchanged.

From Table 3, three observations follow: (i) Across three seeds, AUROC degrades by at most $0.0003$ when the test-time generator is swapped to a different model family, RoBERTa-large $1.0000\pm 0.0000\to 0.9998\pm 0.0001$ (Qwen-7B → Mistral-7B), GraphCodeBERT $1.0000\pm 0.0000\to 0.9997\pm 0.0003$ (Qwen-Coder-7B → DeepSeek-Coder-6.7B). For CodeBERT-base the drop is statistically zero ($1.0000\pm 0.0000\to 1.0000\pm 0.0000$). The confidence intervals on the matched-condition and cross-generator AUROCs overlap completely, so cross-generator generalization is not the limiting factor on this benchmark. (ii) Both modality-specific encoders (RoBERTa-large, GraphCodeBERT, CodeBERT-base) and the unified RoBERTa-base behave identically under a generator swap. Calibration also transfers: ECE stays at $0.40$–$0.94\%$ across detectors, comparable to the in-distribution values in Table 2. The robustness is therefore a property of the data, not of any specific encoder. (iii) A plausible explanation is that paired-prompt generation produces a “polished tutorial” style, structured numbered lists, def/comment patterns, response openers, that is shared across modern instruction-tuned LLMs (Qwen, Mistral, DeepSeek). The detectors latch onto this style rather than onto generator-specific fingerprints, so swapping generators within the same instruction-tuned regime leaves the discriminative signal intact. We return to this interpretation, and to why the same robustness disappears under content-domain shift (Section 8.3), in Section 10.

Table 3. Cross-generator transfer, mean ± standard deviation over three seeds. Each detector is trained on Qwen-generated machine samples; test machine samples come from a different model family.

Detector (Trained On)	Test Generator	Acc (%)	AUROC	ECE (%)	TPR@FPR = 0.01 (%)
RoBERTa-large (Qwen-7B)	Mistral-7B	$99.03\pm 0.45$	$0.9998\pm 0.0001$	$0.94\pm 0.47$	$99.21\pm 0.52$
Unified RoBERTa-base	Mistral-7B	$99.27\pm 0.15$	$0.9998\pm 0.0001$	$0.69\pm 0.14$	$99.41\pm 0.34$
GraphCodeBERT (Qwen-Coder-7B)	DeepSeek-Coder-6.7B	$99.57\pm 0.29$	$0.9997\pm 0.0003$	$0.40\pm 0.30$	$99.87\pm 0.23$
CodeBERT-base (Qwen-Coder-7B)	DeepSeek-Coder-6.7B	$99.87\pm 0.12$	$1.0000\pm 0.0000$	$0.12\pm 0.03$	$100.00\pm 0.00$
Unified RoBERTa-base	DeepSeek-Coder-6.7B	$99.33\pm 0.29$	$0.9997\pm 0.0002$	$0.65\pm 0.34$	$99.40\pm 0.35$

8.3. Cross-Domain Transfer (RQ2, Part 1)

We evaluate every code detector, trained on CodeSearchNet utility functions, on the held-out MBPP+HumanEval set, which contains algorithmic-puzzle solutions and is contamination-suspect in the generator’s pre-training corpus. The held-out set is never used at training time. Table 4 reports performance on this cross-domain evaluation.

Table 4. Cross-domain transfer to MBPP+HumanEval (n = 228), mean ± std over three seeds. TPR@FPR = $0.01$ is highly seed-sensitive on this split: GraphCodeBERT’s std reaches 15 pp.

Method	Acc (%)	AUROC	ECE (%)	TPR@FPR = 0.01 (%)
GraphCodeBERT	$50.73\pm 1.34$	$\mathbf{0.8496}\pm \mathbf{0.0187}$	$48.54\pm 1.65$	$10.32\pm 14.92$
CodeBERT-base	$51.17\pm 1.41$	$0.6704\pm 0.0208$	$48.67\pm 1.40$	$0.00\pm 0.00$
GPT-2-XL log-PPL (0-shot)	$61.40$	$0.8666$	$15.64$	$0.00$

From Table 4, we make the following observations: (i) Both fine-tuned code detectors collapse on the held-out domain. GraphCodeBERT’s AUROC drops from $1.0000\pm 0.0000$ in-distribution to $0.8496\pm 0.0187$ on MBPP+HumanEval, and CodeBERT-base’s drops from $1.0000\pm 0.0000$ to $0.6704\pm 0.0208$. Default-threshold accuracy collapses to ∼51% for both detectors and ECE exceeds $48\%$. The standard deviation on the held-out AUROC is approximately two orders of magnitude larger than on any in-distribution split, but the magnitude of the collapse (≥15 AUROC points for both detectors) is far larger than the seed-induced variability, so the failure mode is robust rather than fragile. (ii) The $+18$-point AUROC gap between GraphCodeBERT ($0.8496\pm 0.0187$) and CodeBERT-base ($0.6704\pm 0.0208$) is roughly $6\times$ its joint standard error. This advantage is invisible in-distribution, where both encoders sit at AUROC $1.0\pm 0.0$. Structural code representations generalize better than pure-token encodings under this domain shift; we revisit this claim under a second OOD distribution in Section 8.4, where the ranking flips. (iii) GPT-2-XL log-perplexity reaches AUROC $0.8666$, within the seed-noise band of GraphCodeBERT’s $0.8496\pm 0.0187$. The frozen reference LM is therefore statistically equivalent to our best fine-tuned encoder on this distribution shift, not categorically better; the more honest reading is that under domain shift, fine-tuning to a single code distribution does not gain over a held-fixed reference model. (iv) ECE on this split sits between $44.3\%$ (after Platt) and $50.2\%$ (raw) for every supervised detector. The TPR at the strict FPR = 0.01 operating point is also seed-sensitive in the extreme: GraphCodeBERT’s TPR@FPR = 0.01 has mean $10.3\%$ but standard deviation $14.9$ percentage points across seeds, while CodeBERT-base’s is $0.0\pm 0.0\%$. Section 8.8 shows that training-time augmentation recovers usable low-FPR performance ($0\%\to 49.6\%$) where post hoc calibration alone cannot.

8.4. External Benchmark: SemEval-2026 Task 13 (RQ2, Part 2)

We additionally evaluate the code branch on the public trial sets of SemEval-2026 Task 13 [52,53], restricted to the Python subset in every subtask. Our detectors are evaluated zero-shot, they have neither been trained on this corpus nor on its content domain (algorithmic Python sources such as Leetcode-style problems). For Subtasks B and C, we collapse the multi-class labels to binary by treating all non-human classes (LLM family in Subtask B; machine, hybrid, and adversarial in Subtask C) as the positive class. We relabel to our convention (0 = human, 1 = machine) before scoring; trial-parquet label polarity is documented in Section 3.5.

From Table 5, we make the following observations: (i) Across Subtasks A, B, and C, AUROC over three seeds stays in $[0.64,0.87]$, substantially above chance, while accuracy at the default $0.5$ threshold collapses to 27–$67\%$ and ECE explodes to 31–$71\%$. The threshold-and-calibration collapse pattern we documented on MBPP+HumanEval (Section 8.3) reproduces on three independent external evaluations. (ii) On MBPP+HumanEval GraphCodeBERT wins by $17.9$ AUROC points ($0.8496\pm 0.0187$ vs. $0.6704\pm 0.0208$). On every SemEval Python subtask the ranking inverts: CodeBERT-base wins by $+15.2$ points on Subtask A ($0.8693\pm 0.0141$ vs. $0.7172\pm 0.0579$), $+7.8$ on Subtask B ($0.7224\pm 0.0118$ vs. $0.6446\pm 0.0354$), and $+4.9$ on Subtask C ($0.7053\pm 0.0152$ vs. $0.6568\pm 0.0382$). All four gaps are several times their joint seed-std, so the inversion is statistically robust rather than seed-driven. The data-flow advantage of GraphCodeBERT is therefore specific to the MBPP+HumanEval style (short, single-function, algorithmic-puzzle solutions) and does not transfer to the broader algorithmic-Python distribution sampled by SemEval-2026 Task 13. (iii) On Subtask C, GraphCodeBERT flags pure-machine samples at $96.85\pm 1.16\%$, hybrid samples at $93.59\pm 1.58\%$, and adversarial (RLHF-mimicking) samples at $92.92\pm 3.73\%$. The pure-machine-to-adversarial gap is $3.9$ percentage points across three seeds. The threats anticipated to erode detection most aggressively, Subtask C’s hybrid and adversarial classes, erode it least. (iv) The dominant failure mode is over-confident false positives on human algorithmic Python. GraphCodeBERT classifies $83.35\pm 7.72\%$ of Subtask C human samples as machine, and CodeBERT-base classifies $72.75\pm 12.02\%$. Both false-positive rates are highly seed-sensitive ($\sigma =8$–12 percentage points), the largest seed-induced variability in the paper outside the MBPP+HumanEval TPR@FPR = 0.01 point estimate. This is precisely the threshold-collapse mechanism from Section 8.3: short, clean, idiomatic human Python sits inside the detector’s machine class because the detector learned a “polished tutorial” style boundary that does not transfer. Threshold tuning on a domain-matched validation split (Section 8.11) is therefore the operational lever; the discrimination signal needed by the threshold is already present (AUROC $\ge 0.64$ across all SemEval subtasks).

Table 5. Zero-shot transfer to SemEval-2026 Task 13, Python subsets. All numbers are mean ± standard deviation over three random seeds $\{42,123,7\}$. Trial-parquet label polarity for Task A is documented in Section 3.5; we relabel to our convention (0 = human, 1 = machine) before scoring. Tasks B and C are binary-collapsed (any non-human class → machine).

Subtask	Detector	n	Acc (%)	AUROC	ECE (%)	TPR@FPR = 0.01 (%)
A	GraphCodeBERT	7104	$51.29\pm 3.69$	$0.7172\pm 0.0579$	$47.31\pm 5.67$	$5.11\pm 4.57$
A	CodeBERT-base	7104	$66.61\pm 11.59$	$\mathbf{0.8693}\pm \mathbf{0.0141}$	$31.39\pm 11.82$	$\mathbf{15.42}\pm \mathbf{12.45}$
B (binary)	GraphCodeBERT	2785	$27.50\pm 7.44$	$0.6446\pm 0.0354$	$71.02\pm 8.58$	$1.26\pm 0.55$
B (binary)	CodeBERT-base	2785	$36.53\pm 9.65$	$\mathbf{0.7224}\pm \mathbf{0.0118}$	$61.62\pm 9.67$	$1.74\pm 2.07$
C (binary)	GraphCodeBERT	2775	$55.26\pm 3.09$	$0.6568\pm 0.0382$	$43.52\pm 3.90$	$0.98\pm 0.22$
C (binary)	CodeBERT-base	2775	$57.60\pm 3.31$	$\mathbf{0.7053}\pm \mathbf{0.0152}$	$40.87\pm 3.53$	$1.30\pm 1.36$

Table 6 reports the corresponding per-class mean detector probabilities and prediction rates for Subtask C.

Table 6. Per-class mean detector probability $p\left(\mathrm{machine}\right)$ on the Subtask C Python subset, averaged over three seeds. “Pred-as-machine” is the fraction of samples in the class that the detector classifies as machine at the default $0.5$ threshold.

Detector	Class	n	Mean p (Machine)	Pred-as-Machine (%)
GraphCodeBERT	Human	1413	$0.829\pm 0.077$	$83.35\pm 7.72$
	Pure machine	782	$\mathbf{0.966}\pm \mathbf{0.013}$	$96.85\pm 1.16$
	Hybrid	255	$0.931\pm 0.019$	$93.59\pm 1.58$
	Adversarial	325	$0.929\pm 0.032$	$92.92\pm 3.73$
CodeBERT-base	Human	1413	$0.728\pm 0.118$	$72.75\pm 12.02$
	Pure machine	782	$\mathbf{0.912}\pm \mathbf{0.047}$	$91.56\pm 4.88$
	Hybrid	255	$0.894\pm 0.046$	$89.41\pm 4.91$
	Adversarial	325	$0.830\pm 0.093$	$82.87\pm 9.87$

8.5. Mixed-Content Generalization (RQ3, Part 1)

We evaluate every method on Mix-Eval: 997 paired samples (497 human + 500 machine) where each input interleaves a Markdown explanation with a Python snippet. Human-side samples are real Jupyter (markdown, code) cell pairs; machine-side counterparts are generated by Qwen2.5-7B (prose) and Qwen2.5-Coder-7B (code) on the same topic. “Naive” detectors apply the modality-specific encoder to the full sample as plain text. “Routed” applies the AST-parse-based router and combines per-modality predictions via entropy-weighted fusion.

From Table 7, we make the following observations: (i) Every method degrades on Mix-Eval relative to its in-distribution numbers. The best fine-tuned AUROC is $0.94$, down from ∼1.0, and accuracy at the default threshold falls below $70\%$ for every supervised method. (ii) The fused branch (when both modalities are detected) reaches AUROC $0.8612\pm 0.0073$ across seeds, much closer to the unified ceiling. The text-only branch (router missed the code) achieves only $0.6668\pm 0.0589$, and its seed-std of nearly 6 AUROC points is the largest seed-induced variability in the paper. The routed pipeline’s overall ceiling is therefore set both by its weakest branch and by that branch’s seed instability. (iii) The cause is router coverage: the AST-parse heuristic does not detect a code block in 216 of 997 samples ($22\%$). On those samples, the routed pipeline degrades to RoBERTa-large alone, which has never seen training samples that contain Python code (text-only branch AUROC $0.62$). When routing succeeds, the fused branch reaches AUROC $0.86$, much closer to the unified ceiling. (iv) GPT-2-XL log-perplexity has the best calibration on this set (ECE $7.5\%$) despite low AUROC, illustrating that frozen reference language models produce miscalibration-resistant scores even when discrimination is weak. (v) Pre-2023 supervised detectors remain useless on mixed content (OpenAI RoBERTa: AUROC $0.59$, barely above chance).

Table 7. Mixed-content evaluation (Mix-Eval, n = 997). Supervised detectors and the routed pipeline are reported as mean ± standard deviation over three random seeds $\{42,123,7\}$. The GPT-2-XL log-perplexity and OpenAI RoBERTa baselines are deterministic and have no seed dependence.

Method	Acc (%)	F1 (%)	AUROC	ECE (%)	TPR@FPR = 0.01 (%)
Unified RoBERTa-base	$68.30\pm 2.44$	$75.94\pm 1.40$	$\mathbf{0.9553}\pm \mathbf{0.0139}$	$30.79\pm 2.73$	$14.20\pm 23.05$
Naive CodeBERT-base	$60.58\pm 9.20$	$71.96\pm 4.84$	$0.8781\pm 0.0114$	$37.52\pm 10.57$	$1.07\pm 1.68$
Naive GraphCodeBERT	$56.34\pm 7.43$	$69.79\pm 3.69$	$0.7917\pm 0.0230$	$41.05\pm 10.21$	$0.33\pm 0.58$
Naive RoBERTa-large	$62.02\pm 9.81$	$71.80\pm 4.71$	$0.7906\pm 0.0484$	$37.17\pm 10.67$	$0.87\pm 0.50$
Routed (RoBERTa-large + GraphCodeBERT, entropy fusion)	$67.13\pm 7.55$	$74.33\pm 3.92$	$0.7508\pm 0.0562$	$30.21\pm 4.51$	$1.67\pm 1.33$
GPT-2-XL log-PPL (deterministic)	61.18	71.18	0.6667	$\mathbf{7.54}$	1.80
OpenAI RoBERTa (deterministic)	52.76	64.40	0.5869	40.86	2.60

Routed pipeline per branch (3-seed mean ± std): fused (n = 781) AUROC $0.8612\pm 0.0073$; text-only (n = 216) AUROC $0.6668\pm 0.0589$.

Figure 2 plots the ROC curves for these conditions; the held-out and Mix-Eval curves are visibly the only ones that bow away from the top-left corner.

Figure 2. ROC curves across modalities and test conditions. Left: text detector. Middle: code detector across in-domain, cross-generator, domain-held-out, and Mix-Eval splits. Right: augmentation effect on out-of-distribution conditions. The held-out (red dashed) and Mix-Eval (purple) curves are the only conditions where AUROC drops noticeably.

8.6. Fusion-Rule Comparison (RQ3, Part 2)

The Mix-Eval result motivates the question: when both modalities are present, what fusion rule should we use? We compare the parameter-free entropy-weighted rule (Equation (2)) against two learnt fusion heads on the routed Mix-Eval predictions, restricted to the 781 samples for which the router identified both a text and a code component (the fused branch). Because Mix-Eval has no separate fusion-validation split, the learnt heads are trained and evaluated via 5-fold stratified cross-validation.

From Table 8, we make the following observations: (i) A learnt fusion head outperforms parameter-free entropy-weighting on Mix-Eval across all three seeds. The gradient-boosted-trees variant achieves AUROC $0.8817\pm 0.0071$ versus $0.8610\pm 0.0072$ for entropy-weighting, a $+0.021$ AUROC gap that holds individually on every seed. (ii) The naive logistic regression over probabilities achieves higher accuracy but lower AUROC than entropy-weighting, suggesting a non-monotonic calibration in the per-modality probabilities that a tree-based head models better. (iii) Entropy-weighting remains useful in deployment when no fusion-validation data is available, since it requires no fitting. When fusion-validation data is available, the gradient-boosted-trees head gives ∼+10 percentage points of accuracy at substantially lower seed variance (Acc std $0.13$ vs. $6.77$). The learnt head’s robustness to underlying-detector seed variability is, in our experiments, a more compelling argument for it than its raw accuracy gain.

Table 8. Fusion rules on the Mix-Eval fused branch (n = 781), mean ± standard deviation over three random seeds $\{42,123,7\}$. Entropy-weighted is parameter-free; learnt heads are evaluated via 5-fold stratified cross-validation on each seed’s routed predictions.

Fusion Rule	Acc (%)	F1 (%)	AUROC
Entropy-weighted (parameter-free)	$70.85\pm 6.77$	$77.69\pm 3.82$	$0.8610\pm 0.0072$
Logistic regression on $(p_T,p_C)$	$76.40\pm 2.15$	$79.93\pm 0.38$	$0.8056\pm 0.0154$
Logistic regression on $({\mathrm{logit}}_T,{\mathrm{logit}}_C)$	$79.30\pm 1.15$	$80.96\pm 0.12$	$0.8671\pm 0.0066$
Gradient-boosted trees on $(p_T,p_C)$	$\mathbf{81.05}\pm \mathbf{0.13}$	$\mathbf{82.18}\pm \mathbf{0.17}$	$\mathbf{0.8817}\pm \mathbf{0.0071}$

8.7. Robustness to Lightweight Transformations (RQ4, Part 1)

We apply two lightweight attacks to the machine half of the test sets. For text, we use Qwen2.5-7B-Instruct as a same-family paraphraser. For code, we apply AST-based local-variable renaming ($x\to v_1$, $y\to v_2$, …) followed by comment stripping. Human-side samples are unchanged.

From Table 9, we make the following observations: (i) Both attacks leave detection performance essentially unchanged. AUROC, accuracy, and TPR@FPR all match the no-attack in-distribution baseline. (ii) Two caveats are worth declaring up front. First, the text paraphraser is the same model family as the generator, which preserves the stylistic regularities the detector relies on; a cross-family paraphraser (e.g., DIPPER, Pegasus) is the standard stronger attack. Second, AST-rename with comment stripping preserves AST structure and identifier-distribution-conditional features, so structural-feature detectors are insensitive to it by design. (iii) In Section 8.8, we show that on-the-fly training-time augmentation provides additional robustness gains on the out-of-distribution conditions (MBPP+HumanEval held-out and Mix-Eval), which test-time attacks alone do not surface.

Table 9. Robustness to lightweight post-generation transformations (headline detectors, no augmentation training).

Detector	Attack	Acc (%)	AUROC	ECE (%)	TPR@FPR = 0.01
RoBERTa-large	Qwen-7B paraphrase	99.20	0.9999	0.79	99.21
GraphCodeBERT	rename + comment-strip	99.60	1.0000	0.44	100.00
CodeBERT-base	rename + comment-strip	99.80	1.0000	0.15	100.00

8.8. Training-Time Augmentation (RQ4, Part 2)

We retrain each headline detector with on-the-fly input augmentation at probability $p_{\mathrm{aug}}=0.2$. For text, augmentation is random sentence reordering. For code, AST-based local-variable renaming followed by random comment insertion or removal. Augmentation is applied identically to human and machine samples and preserves the label.

From Table 10, we make the following observations: (i) Across three seeds the augmented detectors tie or improve AUROC on every same-modality condition, drop ECE on every such condition (e.g., RoBERTa xgen ECE $0.94\to 0.67\%$; GraphCodeBERT xgen ECE $0.40\to 0.21\%$), and reach TPR@FPR = 0.01 of $100\%$ on every in-distribution and cross-generator condition for both detectors. The single exception is GraphCodeBERT on the MBPP+HumanEval held-out, where the AUROC mean dips slightly ($0.8496\pm 0.0187\to 0.8299\pm 0.0481$) while the TPR@FPR = 0.01 mean roughly doubles (see (ii)); the AUROC confidence intervals overlap, so the effect on ranking is statistically zero. (ii) GraphCodeBERT’s mean TPR@FPR = 0.01 improves from $10.3\pm 14.9\%$ to $24.8\pm 22.6\%$, more than doubling on average. The metric is the most seed-sensitive number in the paper outside the routed text-only branch, standard deviations on both conditions exceed 15 percentage points, so we report it with explicit ± std rather than as a single-point estimate. Augmentation does not repair the calibration collapse on this split (ECE remains $\sim 48\%$ in both conditions); it repairs only the low-FPR usability of the detector under domain shift, and even that gain is large in expectation but variable across seeds. (iii) RoBERTa-aug improves AUROC $0.7906\pm 0.0484\to 0.8124\pm 0.0208$ and TPR@FPR = 0.01 $0.87\pm 0.50\%\to 1.80\pm 1.71\%$, with no material ECE change ($37.17\%\to 41.58\%$, within seed noise). GraphCodeBERT-aug shows the same AUROC improvement ($0.7917\pm 0.0230\to 0.8234\pm 0.0078$, with an order-of-magnitude lower std), a modest ECE reduction ($41.05\%\to 39.06\%$), and a much larger TPR@FPR = 0.01 gain ($0.33\pm 0.58\%\to 4.80\pm 3.17\%$).

Table 10. Augmentation ablation, mean ± standard deviation over three random seeds $\{42,123,7\}$. Encoder, optimizer, and all other hyperparameters are unchanged; only the augmentation policy differs.

Detector	Test Set	AUROC		ECE (%)		TPR@FPR = 0.01 (%)
		no-aug	+aug	no-aug	+aug	no-aug	+aug
RoBERTa-large	in-domain	$1.0000\pm 0.0000$	$1.0000\pm 0.0000$	$0.36\pm 0.21$	$\mathbf{0.33}\pm \mathbf{0.15}$	$99.87\pm 0.11$	$\mathbf{99.93}\pm \mathbf{0.11}$
RoBERTa-large	xgen (Mistral)	$0.9998\pm 0.0001$	$\mathbf{0.9999}\pm \mathbf{0.0001}$	$0.94\pm 0.47$	$\mathbf{0.67}\pm \mathbf{0.20}$	$99.21\pm 0.52$	$\mathbf{99.80}\pm \mathbf{0.34}$
RoBERTa-large	paraphrase attack	$0.9998\pm 0.0002$	$\mathbf{0.9999}\pm \mathbf{0.0000}$	$1.20\pm 0.91$	$\mathbf{0.50}\pm \mathbf{0.17}$	$99.54\pm 0.41$	$\mathbf{99.87}\pm \mathbf{0.23}$
RoBERTa-large	Mix-Eval	$0.7906\pm 0.0484$	$\mathbf{0.8124}\pm \mathbf{0.0208}$	$37.17\pm 10.67$	$41.58\pm 8.74$	$0.87\pm 0.50$	$\mathbf{1.80}\pm \mathbf{1.71}$
GraphCodeBERT	in-domain (CSN)	$1.0000\pm 0.0000$	$1.0000\pm 0.0000$	$0.36\pm 0.19$	$\mathbf{0.26}\pm \mathbf{0.07}$	$100.00\pm 0.00$	$100.00\pm 0.00$
GraphCodeBERT	xgen (DeepSeek-Coder)	$0.9997\pm 0.0003$	$\mathbf{0.9999}\pm \mathbf{0.0000}$	$0.40\pm 0.30$	$\mathbf{0.21}\pm \mathbf{0.06}$	$99.87\pm 0.23$	$\mathbf{100.00}\pm \mathbf{0.00}$
GraphCodeBERT	obfuscation attack	$1.0000\pm 0.0000$	$1.0000\pm 0.0000$	$0.38\pm 0.14$	$\mathbf{0.29}\pm \mathbf{0.06}$	$100.00\pm 0.00$	$100.00\pm 0.00$
GraphCodeBERT	MBPP+HE held-out	$\mathbf{0.8496}\pm \mathbf{0.0187}$	$0.8299\pm 0.0481$	$48.54\pm 1.65$	$\mathbf{48.11}\pm \mathbf{1.67}$	$10.32\pm 14.92$	$\mathbf{24.78}\pm \mathbf{22.60}$
GraphCodeBERT	Mix-Eval	$0.7917\pm 0.0230$	$\mathbf{0.8234}\pm \mathbf{0.0078}$	$41.05\pm 10.21$	$\mathbf{39.06}\pm \mathbf{7.25}$	$0.33\pm 0.58$	$\mathbf{4.80}\pm \mathbf{3.17}$

Overall, Augmentation is therefore a partial substitute for in-domain mixed-content training data, particularly on the code branch, where all three of AUROC, ECE, and TPR@FPR = 0.01 improve. On the text branch, the AUROC and TPR gains hold, while ECE is statistically unchanged.

8.9. Comparison with Baselines (RQ5)

We compare against two zero-shot baselines on text test sets: the OpenAI RoBERTa detector [23] (a pre-2023 supervised detector originally fine-tuned on GPT-2) and a GPT-2-XL log-perplexity threshold [43].

From Table 11, we make the following observations: (i) The OpenAI RoBERTa detector achieves AUROC of $0.298$ on Qwen-7B outputs and $0.069$ on Mistral-7B outputs, both well below the chance level of $0.5$. This pattern indicates systematic class inversion rather than random performance: higher machine-likelihood scores are assigned to human samples than to machine samples, and $1-\mathrm{AUROC}$ would recover most of the discriminative signal. Reporting AUROC alongside accuracy is therefore essential to distinguish inversion (correctable by flipping the predicted labels) from genuine inability to discriminate, which a $34.7\%$ or $15.5\%$ accuracy figure alone would conflate. (ii) The GPT-2-XL log-perplexity baseline reaches AUROC $0.984$ on Qwen outputs and $0.995$ on Mistral outputs, substantially higher than the OpenAI detector but $\sim 0.02$ below the fine-tuned detector. Its default-threshold accuracy is lower ($73.2\%$ on Qwen, $82.2\%$ on Mistral) because a fixed perplexity threshold does not transfer cleanly across the two generator distributions; the AUROC–accuracy gap quantifies the threshold-transfer cost of using a zero-shot signal at deployment. (iii) Section 8.3 re-examines GPT-2-XL log-perplexity under content-domain shift, where it reaches AUROC $0.8666$ on the MBPP+HumanEval held-out and exceeds the AUROC of every supervised detector evaluated on that split. The frozen reference language model is consequently less accurate at the default threshold but more domain-transferable in ranking; the supervised detectors exhibit the opposite trade-off, retaining tight calibration on matched distributions while collapsing under content-domain shift. (iv) The fine-tuned RoBERTa-large detector exceeds both baselines on every reported metric. AUROC improves by approximately $0.7$ over the OpenAI detector and by ∼0.016 over the GPT-2-XL threshold, while default-threshold accuracy reaches $99.63\%$ and $99.03\%$ on the two test sets compared with $34.70\%$/$15.50\%$ and $73.20\%$/$82.20\%$ for the respective baselines.

Table 11. Comparison with zero-shot baselines on text test sets. “Ours” is the headline RoBERTa-large detector fine-tuned on text_train.

Method	Qwen-7B Test		Mistral-7B Xgen
	AUROC	Acc (%)	AUROC	Acc (%)
OpenAI RoBERTa (zero-shot)	0.298	34.70	0.069	15.50
GPT-2-XL log-PPL	0.984	73.20	0.995	82.20
Ours (RoBERTa-large)	0.9999	99.63	0.9998	99.03

8.10. Component-Contribution Ablations: Hand-Engineered Features (RQ6, Part 1)

We compare a head trained on the [CLS] embedding alone (cls_only) against a head trained on $[\mathtt{CLS}\parallel \varphi (x\left)\right]$ (cls+hand), where $\varphi \left(x\right)$ is the standardized hand-engineered feature vector from Section 5.2 (17-d for text, 14-d for code). The encoder is frozen for both configurations; only the two-layer MLP head is trained.

From Table 12, we make the following observations: (i) Text hand features yield a small but consistent improvement ($+0.001$ AUROC across every text test, $+0.01$ on Mix-Eval). The features are anchored on a frozen reference language model (GPT-2-XL perplexity, log-rank, burstiness) plus distribution-invariant style statistics (MATTR, readability indices), which appear to transfer. (ii) Code hand features do not help in distribution and actively hurt on the contamination-suspect MBPP+HumanEval held-out (AUROC $0.86\to 0.80$). The hand-engineered code features (AST depth, comment density, idiom counts) overfit to the CodeSearchNet utility-function distribution and fail to generalize to algorithmic puzzles. (iii) The gap suggests that a more transferable code-feature design, one anchored on a frozen reference code language model rather than on training-domain syntactic counts, is an open problem.

Table 12. Hand-feature ablation, mean ± standard deviation over three random seeds $\{42,123,7\}$. Both rows for each test set share a frozen encoder; only the head input differs.

Modality	Test Set	cls_only AUROC	cls+hand AUROC
Text	in-domain	$0.9993\pm 0.0006$	$\mathbf{0.9996}\pm \mathbf{0.0006}$
Text	xgen (Mistral)	$0.9991\pm 0.0005$	$\mathbf{0.9995}\pm \mathbf{0.0005}$
Text	paraphrase attack	$0.9992\pm 0.0004$	$\mathbf{0.9995}\pm \mathbf{0.0005}$
Text	Mix-Eval	$0.6683\pm 0.1268$	$\mathbf{0.6809}\pm \mathbf{0.1338}$
Code	in-domain (CSN)	$1.0000\pm 0.0000$	$1.0000\pm 0.0000$
Code	MBPP+HE held-out	$\mathbf{0.8461}\pm \mathbf{0.0472}$	$0.8136\pm 0.0160$
Code	xgen (DeepSeek)	$0.9999\pm 0.0001$	$0.9999\pm 0.0000$
Code	obfuscation attack	$1.0000\pm 0.0000$	$1.0000\pm 0.0000$
Code	Mix-Eval	$0.8022\pm 0.0151$	$\mathbf{0.8111}\pm \mathbf{0.0205}$

8.11. Calibration and Threshold Tuning (RQ6, Part 2)

For each detector, we fit a single-parameter Platt scaling step (Equation (3)) on validation logits and labels, and select the threshold $\tau \in [0.05,0.95]$ that maximizes macro-F1 on the validation split. Both fitting steps use only validation data; test data is unseen.

From Table 13, we make the following observations: (i) Platt scaling reduces but does not repair calibration under heavy distribution shift. On the MBPP+HumanEval held-out, Platt scaling lowers GraphCodeBERT’s ECE from $48.54\pm 1.65\%$ to $44.78\pm 0.66\%$—a statistically robust $3.8$-point reduction that is nonetheless small relative to the magnitude of the calibration collapse. ECE on Mix-Eval for the same detector drops from $41.05\pm 10.21\%$ to $26.36\pm 9.31\%$ under Platt scaling; the larger improvement on Mix-Eval reflects a less severe distribution shift relative to MBPP+HumanEval, not a property of the scaling step itself. (ii) Threshold tuning on a domain-matched validation split is a substantially larger lever than Platt scaling when the test distribution shifts. Replacing the default $0.5$ threshold with the macro-F1-maximising $\tau$ improves GraphCodeBERT’s accuracy on Mix-Eval from $56.34\pm 7.43\%$ to $70.61\pm 5.69\%$ and on MBPP+HumanEval from $50.73\pm 1.34\%$ to $54.82\pm 0.44\%$. The intervention requires no retraining and no modification of the encoder; only a single threshold parameter is fitted on validation data. (iii) In-distribution detectors do not benefit from either intervention. ECE is below $0.5\%$ before any post hoc step ($0.36\pm 0.21\%$ for RoBERTa-large on the text test split; $0.36\pm 0.19\%$ for GraphCodeBERT on the code2 test split), and threshold tuning leaves in-distribution accuracy essentially unchanged ($99.63\to 99.63\%$ for RoBERTa-large; $99.63\to 99.83\%$ for GraphCodeBERT). The calibration machinery is consequential only when distribution shift breaks the score distribution; under matched conditions, the raw logits are already well-calibrated and the default threshold is already near-optimal.

Table 13. Effect of Platt scaling and macro-F1 threshold tuning, mean ± standard deviation over three random seeds $\{42,123,7\}$. “raw” columns: raw $\sigma \left(\mathrm{logit}\right)$ at threshold $0.5$; “+Platt” columns: fitted Platt scaling; “tuned acc” uses $p_{\mathrm{cal}}$ at the val-tuned $\tau$.

Detector	Test Set	raw ECE (%)	+Platt ECE (%)	raw acc (%)	tuned acc (%)
RoBERTa-large	text test	$0.36\pm 0.21$	$\mathbf{0.25}\pm \mathbf{0.18}$	$99.63\pm 0.21$	$99.63\pm 0.21$
RoBERTa-large	Mix-Eval	$37.17\pm 10.67$	$\mathbf{36.34}\pm \mathbf{9.49}$	$62.02\pm 9.81$	$61.28\pm 7.48$
GraphCodeBERT	code2 test	$\mathbf{0.36}\pm \mathbf{0.19}$	$0.41\pm 0.26$	$99.63\pm 0.25$	$\mathbf{99.83}\pm \mathbf{0.15}$
GraphCodeBERT	MBPP+HE held-out	$48.54\pm 1.65$	$\mathbf{44.78}\pm \mathbf{0.66}$	$50.73\pm 1.34$	$\mathbf{54.82}\pm \mathbf{0.44}$
GraphCodeBERT	Mix-Eval	$41.05\pm 10.21$	$\mathbf{26.36}\pm \mathbf{9.31}$	$56.34\pm 7.43$	$\mathbf{70.61}\pm \mathbf{5.69}$
Unified RoBERTa-base	text test	$0.36\pm 0.11$	$\mathbf{0.28}\pm \mathbf{0.09}$	$99.58\pm 0.06$	$\mathbf{99.67}\pm \mathbf{0.03}$
Unified RoBERTa-base	Mix-Eval	$30.79\pm 2.73$	$\mathbf{29.11}\pm \mathbf{3.51}$	$68.30\pm 2.44$	$\mathbf{71.18}\pm \mathbf{4.00}$

Figure 3 shows the corresponding reliability diagrams, near-diagonal in-distribution, severely bowed on the contamination-suspect held-out and Mix-Eval splits, and Figure 4 shows the underlying score distributions that explain the collapse.

Figure 3. Reliability diagrams. Bars: per-bin accuracy. Red line: per-bin mean confidence. Dotted: perfect calibration. The detector is well calibrated in-distribution (left, ECE $0.6\%$), badly miscalibrated on the contamination-suspect held-out (centre, ECE $48.4\%$), and on Mix-Eval (right, ECE $29.3\%$).

Figure 4. Score distributions $p\left(\mathrm{machine}\right)$. Human samples are shown in green, machine in red. In-distribution scores cluster tightly at the extremes; on the held-out and Mix-Eval splits, the two distributions overlap heavily near the threshold, explaining the ECE collapse and the threshold-tuning gains in Section 8.11.

9. Qualitative Analysis

What Does the Detector Latch onto?

Table 14 shows a curated subset of representative detector outputs; a larger set spanning every test condition, including true/false positives and negatives on the contamination-suspect held-out and Mix-Eval splits, is provided in Appendix B. Confidently detected machine samples consistently exhibit polished-tutorial markers: explicit Markdown structure (### headings, numbered lists), step-by-step inline comments in code, and an explanatory tone. Confidently identified human samples on HC3 show informal Reddit-style hedging (“plenty of room”, citation links, hyphenated qualifications); on CodeSearchNet, they show idiosyncratic docstring conventions (:param/:returns) and dense business-logic implementations rather than clean toy examples. The most diagnostic failures are the false positives: clean encyclopedic human prose and short idiomatic human Python both look “too neat” and trip the machine class. This is consistent with our broader finding that fine-tuned detectors classify on style polish rather than on generation provenance directly, and explains why training-time augmentation, which forces the encoder to ignore some of the polish markers, is the most effective intervention we have for transfer to out-of-distribution domains (Section 8.8).

Table 14. Representative detector outputs across modalities and test conditions. TP = correctly detected machine-generated; TN = correctly identified human; FP = human misclassified as machine; FN = machine misclassified as human. Each row shows the detector probability $p\left(\mathrm{machine}\right)$. The extended set is provided in Appendix B.

Setting	Outcome	p	Sample (Truncated)
RoBERTa-large/in-domain (HC3)	TP	1.000	### Imports and Exports Imagine you have a toy store. Sometimes, you get toys from other stores to sell in your store. These are like "imports." Other times, you might send some of your toys to other stores. These are like "exports." In the same way, countries buy things from each other (imports)...
RoBERTa-large/in-domain (HC3)	TN	0.000	[The Shuttle was one of the most woefully inefficient launch platforms ever deployed] (URL_0) . It cost around $ 8000/pound to put something in orbit using the shuttle compared to $ 2300/pound for late-60s era rockets . The biggest advantage the shuttle had was that it had plenty of room...
RoBERTa-large/in-domain (HC3)	FP	1.000	After months of rumors, on April 18, 2013, Showtime announced via social media that season eight would be the final season of Dexter.
RoBERTa-large/in-domain (HC3)	FN	0.000	Yes, typically companies can only pay dividends if they are profitable and have sufficient retained earnings or cash reserves to distribute to shareholders. Dividends are usually paid out of the company’s net income and any available retained earnings. If a company is not profitable or does not h...
GraphCodeBERT/in-domain (CodeSearchNet)	TP	1.000	def remove_machine_from_model(machine_id): # Code to remove the machine with the given ID from the model pass # Placeholder for actual implementation
GraphCodeBERT/in-domain (CodeSearchNet)	TN	0.001	def _get_history_id(self, connection, agent_id, instance_id): ”’ Checks own cache for history_id for agent_id and instance_id. If information is missing fetch it from database. If it is not there create the new record. BEWARE: This method runs in a thread...
GraphCodeBERT/in-domain (CodeSearchNet)	FP	0.995	def clear_table(dbconn, table_name): """ Delete all rows from a table :param dbconn: data base connection :param table_name: name of the table :return: """ cur = dbconn.cursor() cur.execute("DELETE FROM ’{name}’".format(name=table_name)) dbconn.commit()

10. Discussion

Across our evaluation axes, a consistent picture emerges: detectors that appear nearly perfect in-distribution can fail in very different ways once the test setting changes. Generator-family changes have little measurable effect in our benchmark, whereas content-domain shifts and mixed-content inputs expose substantial weaknesses. Importantly, these failures are not only failures of discrimination. In several settings, AUROC partially survives while calibration and threshold-based usability collapse. Thus, the central lesson is that AI-generated-content detection should not be evaluated only by in-distribution accuracy or AUROC; it must also be tested under domain shift, mixed-modality inputs, low-false-positive operating points, and calibration-sensitive deployment conditions.

• Domain shift is more limiting than generator shift.

The strongest empirical pattern is that changing the test-time generator family has little effect, while changing the content domain substantially degrades performance. Detectors trained on Qwen-generated samples transfer almost unchanged to Mistral or DeepSeek-Coder outputs, suggesting that modern instruction-tuned models share detectable stylistic regularities under the paired-prompt setup. By contrast, transferring from CodeSearchNet utility functions to MBPP+HumanEval algorithmic problems causes large drops for every fine-tuned code detector. This indicates that the detector is not simply learning a universal “machine-code” signature; it is also learning domain-specific regularities of the training corpus. For benchmark design, this means that generator-held-out evaluation is necessary but not sufficient. A single domain-held-out split can reveal failure modes that cross-generator tests alone would miss.

• The discriminative signal is instruction-tuned surface style.

Three revision-stage analyses converge on this interpretation. First, a format-only classifier restricted to shallow surface features (length, whitespace, punctuation, docstring/markdown markers) already separates the in-distribution classes at AUROC $0.994$ (code)/$0.998$ (text), and an identical surface canonicalization applied to both classes drops the unchanged code detector from $1.0000$ to $0.8151\pm 0.069$ (Appendix D), so roughly a fifth of the in-distribution code margin is carried by surface formatting, while substantial non-surface signal remains. Second, per-generator decomposition of the SemEval transfer (Appendix C) shows the detector transfers across instruction-tuned families (AUROC $0.806\pm 0.047$) but degrades sharply on base/pretrained models ($0.561\pm 0.049$), with model size uncorrelated with detectability; base-model output lacks the instruction-tuned formatting polish. Third, $84\%$ of SemEval’s curated, competition-style human code is flagged as machine at the default threshold, clean human code falls on the machine side of a style-based boundary. Notably, the post-canonicalization AUROC ($0.815$) closely matches the cross-domain held-out AUROC ($0.850$): both estimate the signal that remains once surface cues stop being informative. The detector should therefore be understood as identifying format-and-style-typical instruction-tuned output rather than generation provenance per se, consistent with the qualitative analysis of Section 8.

• Structural code representations matter under domain shift.

The comparison between CodeBERT and GraphCodeBERT shows that architectural differences may be invisible in-distribution but important out of domain. Both encoders perform similarly on matched and cross-generator test sets, where performance is already near ceiling. However, on the held-out MBPP+HumanEval split, GraphCodeBERT substantially outperforms CodeBERT. This suggests that data-flow-aware representations provide signals that transfer better than purely token-distributional cues when the code domain changes. The result also cautions against choosing code detectors based only on in-distribution performance: when AUROC is already saturated, architectural advantages may only appear under more realistic deployment stress tests.

• Adversarial detection is not the bottleneck; threshold is.

On SemEval-2026 Task C Python (Table 6), GraphCodeBERT flags pure-machine samples as machine $95.8\%$ of the time, hybrid samples (partially completed by LLMs) $92.2\%$, and adversarial samples (RLHF-tuned to mimic humans) $88.9\%$. The adversarial threat reduces the flagging rate by only ∼7 percentage points relative to pure machine—much smaller than the $74.6\%$ false-positive rate the same detector exhibits on human algorithmic Python in the same evaluation. Across our entire study the dominant deployment risk is therefore not adversarial evasion but threshold mis-transfer under content-domain shift; the same discrimination signal is present, but the operating point is wrong. This reinforces the Section 8.11 finding that threshold tuning, not architecture, is the practical lever.

• Mixed-content detection is limited by routing coverage.

A natural expectation is that a routed pipeline with separate text and code detectors should outperform a single shared encoder on mixed inputs. Our results complicate this expectation. On Mix-Eval, the unified RoBERTa-base detector outperforms the routed pipeline, largely because the router fails to identify code regions in a non-trivial fraction of mixed documents. When routing succeeds, fusion performs much better; when routing fails, the system effectively falls back to a text-only detector that was not trained on code-containing inputs. This suggests that mixed-content detection is not only a question of detector architecture, but also a router-quality problem. A learned line- or span-level modality router is therefore a natural next step.

• Frozen reference-model scores can be surprisingly robust.

Although fine-tuned detectors dominate zero-shot baselines in most matched settings, the GPT-2-XL log-perplexity baseline remains competitive under the strongest domain shift. On the contamination-suspect held-out split, it outperforms the fine-tuned encoders in AUROC. One interpretation is that fine-tuning improves sensitivity to the training distribution, but also increases dependence on domain-specific artifacts. A frozen reference language model, by contrast, provides a more generic likelihood signal that may transfer better when the test domain changes. This does not make perplexity a stronger detector overall, but it shows that simple reference LM baselines remain important sanity checks for deployment-oriented evaluation. Two properties of the held-out split make the frozen reference particularly competitive there. First, fine-tuning to CodeSearchNet utility functions ties the learned decision boundary to domain-specific artifacts: docstring conventions, identifier styles, and function length that do not transfer to short algorithmic-puzzle solutions, whereas the frozen model’s likelihood signal is independent of any training distribution. Second, short, single-function algorithmic solutions exhibit strong statistical regularities under a reference LM (constrained vocabulary, canonical implementations), which favors likelihood-based ranking in exactly this domain. This reading is consistent with the per-class analysis of Table 6: the fine-tuned detectors’ dominant failure on this distribution is over-confident false positives on clean idiomatic human code, a domain-transfer failure of the learned decision boundary, not of the underlying discriminative signal.

• Calibration and low-FPR usability require separate treatment.

Our results also separate ranking performance from deployable decision quality. In-distribution ECE is below 1% for the fine-tuned detectors, but under domain shift and mixed-content evaluation, ECE rises sharply. In these settings, a detector can retain reasonable AUROC while becoming unreliable at the default threshold. Platt scaling helps in some cases, but threshold tuning on a domain-matched validation set is often more directly useful for improving accuracy and low-FPR behavior. This is especially important for applications such as academic-integrity review, where false positives are costly and a detector must be operated at conservative thresholds.

• Augmentation improves robustness more than post hoc repair.

Among the interventions we evaluate, training-time augmentation provides the clearest improvement under deployment-relevant shifts. Code augmentation through variable renaming and comment insertion/removal substantially improves TPR at low FPR on the held-out code domain while preserving in-distribution performance. Text augmentation produces analogous gains for the language branch. This suggests that robustness is better encouraged during training than repaired after the fact. By forcing the model to ignore superficial perturbations, augmentation appears to encourage reliance on signals that transfer better across domains.

• Hand-engineered features are useful only when they transfer.

The hand-feature results are modality-asymmetric. For text, reference-LM and stylometric features provide small but consistent improvements, suggesting that likelihood, repetition, and readability statistics capture signals that are not fully absorbed by the transformer embedding. For code, however, simple syntactic and AST-count features do not reliably improve out-of-domain performance and can even hurt. This suggests that raw syntactic counts may overfit to the training domain. A more promising direction is to design code features that are anchored in a frozen reference code model, analogous to perplexity and log-rank features for natural language.

• Practical implications.

For practitioners, the main recommendation is to treat in-distribution performance as only a first diagnostic. A detector that performs well on matched data should still be evaluated on at least one content-domain held-out split, at least one generator-held-out split, and at low false-positive operating points. Training-time augmentation should be used by default when robustness to paraphrasing or code rewriting is expected. For mixed text–code inputs, router coverage should be measured explicitly, because routing failures can dominate fusion performance. Finally, calibration and threshold selection should be performed on data that matches the intended deployment domain whenever possible. Together, these steps provide a more realistic estimate of detector reliability than headline in-distribution accuracy alone.

11. Conclusions and Future Work

We have presented a paired-prompt benchmark and an empirical study of text-level AI-content detection across English natural language and Python source code. Modality-specific transformer detectors (RoBERTa-large for text, GraphCodeBERT for code) achieve near-perfect in-distribution performance and generalize almost without loss to a different generator family at test time—contradicting the common worry that supervised detectors overfit narrow model-specific surface cues. The same detectors, however, collapse on a domain-held-out, contamination-suspect set, losing 49 accuracy points and an order of magnitude in calibration error. A widely cited zero-shot baseline (the OpenAI RoBERTa detector) scores below random AUROC on outputs from current LLMs, while a GPT-2-XL log-perplexity threshold remains a strong if blunt baseline. We argue that benchmarks for AI-content detection should explicitly include domain-held-out and contamination-suspect splits, that zero-shot legacy detectors should be reported with AUROC alongside accuracy to avoid misleading “bad detector” interpretations, and that robustness claims should be qualified by the family relationship between attacker and generator.

Future Work

The natural next steps are as follows: (i) multi-family training and evaluation; (ii) stronger attacks, DIPPER cross-family paraphrase, and structural code obfuscation; (iii) Layer-2 ablation that concatenates hand-engineered linguistic and structural features into the MLP head and quantifies their value on out-of-distribution sets; (iv) Platt scaling on a domain-matched validation split, with reliability diagrams reported at deployment-relevant operating points; (v) probabilistic fusion of the text and code detectors for mixed-content inputs such as Jupyter notebooks and technical-tutorial documents; and (vi) replacing the rule-based modality router with a learned per-line or per-token modality classifier, closing the 22% routing-coverage gap identified in Section 8.5 that currently bottlenecks the routed pipeline on mixed-content inputs.

12. Limitations

• Single generator family per modality for the main training runs. Cross-generator robustness was evaluated on a single alternative family per modality (Mistral for text, DeepSeek-Coder for code). A multi-family training distribution and a multi-family evaluation matrix is the natural next step.
• Limited code training distribution. Real-world Python is considerably more diverse than CodeSearchNet utility functions. Notebook code, scripts, tests, and competitive-programming submissions are not represented in training, and the code-set collapse suggests these would all be out-of-distribution.
• No hand-engineered features in the headline numbers. With AUROC already ≥0.9999 in-distribution, there is no head-room for hand features to demonstrate value at the headline operating point. Their natural place is on out-of-distribution and domain-shift evaluations, where the present results suggest they would help; the hand-feature extractors are implemented and ready to be wired into the MLP head as $[\mathrm{CLS}\oplus \mathrm{scaled}\mathrm{hand}\mathrm{features}]$ in follow-up work.
• No fusion across modalities. The architecture is intentionally minimal: two independent modality-specific detectors. Probabilistic fusion of the two detectors on mixed text-and-code inputs (e.g., Jupyter notebooks, Stack-Overflow-style answers) is a natural extension and is discussed in Section 11.
• Boundaries of generator generalization. Generalization is bounded in two ways our controlled setting does not probe. (i) Under human-distribution shift, e.g., the SemEval-2026 transfer, whose human code is curated, competition-style, overall AUROC falls to $0.72$ because human code is itself flagged as machine (Appendix C). (ii) The detector transfers across instruction-tuned generators but degrades sharply on base/pre-trained models (AUROC $0.81$ vs. $0.56$), consistent with keying on instruction-tuned stylistic regularities. Training on multiple generator families and on base-model output is future work.
• Response-format confound on the text side. Human and machine responses are matched on the prompt/question but not on response format. A format-only classifier reaches AUROC $\approx 0.998$ in-distribution, and an identical surface normalization applied to both classes moves the neural detector from $1.000$ to $0.815$ on code (AST canonicalization removing docstrings, comments, and indentation variance) but leaves text essentially unchanged, because text-side cues (length, verbosity, punctuation density) are entangled with content and cannot be reformatted away (Appendix D). Our detector should therefore be understood as detecting format-and-style-typical instruction-tuned output; robustly detecting machine text deliberately format-matched to human conventions remains future work, as the human responses cannot be re-collected under a matched template.
• Python-only code scope. All code experiments use Python exclusively. AST structure, naming conventions, and formatting norms differ substantially across programming languages, and generalization to Java, C++, JavaScript, or other languages is untested.
• Attack strength is a lower bound. The tested transformations (same-family paraphrase; AST variable renaming with comment stripping) constitute a lower bound on adversarial pressure. The robustness results should not be read as resistance to determined evasion such as cross-family rewriting, control-flow flattening, or dead-code insertion, which are listed as future work.
• Standard decoding only. Our machine samples span three decoding temperatures ($0.3/0.7/1.0$), and Appendix E shows detection AUROC is flat across them in-domain, out-of-domain, and on Mix-Eval. All samples are nonetheless produced with standard temperature/nucleus sampling from instruction-tuned models; we do not evaluate adversarial decoding strategies designed specifically to evade detection (e.g., logit steering), which we leave to future work.