Are Scientists Starting to Sound like ChatGPT? Stylistic Diffusion in Biomedical Abstracts in the LLM Era

Abstract

Large language models are part of everyday scholarly writing, but their influence may not always appear as obvious AI authorship. More often, it may appear through gradual changes in academic style. This study examines whether biomedical abstracts show measurable signs of LLM-associated “same-ification” after the public release of ChatGPT. Instead of classifying individual abstracts as AI-written, our study analyses population-level changes across 28,415 PubMed abstracts published between 2018 and 2025. A strict dictionary of LLM-associated stylistic markers was used alongside a control vocabulary, lexical diversity, semantic similarity, interrupted time-series modelling, journal-level heterogeneity analysis, and a study-specific validation framework called SAME-ML. The findings show a clear post-2022 stylistic shift. Mean strict marker use increased from 4.995 to 11.658 markers per 1000 words, representing a 133.4% rise, while the control vocabulary slightly declined. This increase was strongest in 2024 and 2025 and was mainly driven by markers linked to value framing, contribution signalling, connective phrasing, improvement rhetoric, and problem-solution positioning. SAME-ML further showed that the post-ChatGPT period contained a learnable stylistic signal that remained visible after topic-masked and matched-sample validation. Overall, the findings show a corpus-level rhetorical shift and provide a reproducible approach for tracking stylistic change in scholarly communication over time.

1. Introduction

When the world changes, the words people use change with it. Such changes often leave detectable fingerprints in the frequency distributions of written language [1]. Few changes have affected word usage as abruptly as the public release of ChatGPT in late 2022, and few have been so quietly pervasive. Within a single year, keyword-based estimates suggested that more than 60,000 scholarly articles, more than one percent of the literature, already bore the lexical signature of large language models (LLMs) [2], and in biomedicine the figure was far higher: at least 13.5% of 2024 PubMed abstracts showed evidence of LLM processing, an impact exceeding even that of COVID-19 on scientific vocabulary [1]. The intuitive response has been to police this influence at the level of the individual author, asking whether a given text was machine-written or human-written. Yet detection is an unstable foundation. State-of-the-art classifiers can be evaded by simple paraphrasing and, more troublingly, systematically misclassify the prose of non-native English writers as AI-generated, penalising precisely those whom the tools could most assist [3]. This suggests a more interesting possibility than machine-versus-human authorship. If the telltale words of LLMs are diffusing through the literature even where no model touched the keyboard, then the bigger change may not be that some humans are letting machines write for them but that humans are beginning to write like machines. The convergence is occurring not in what science says but in the increasingly uniform rhetoric of how science is presented.

LLMs have moved quickly from experimental natural language processing (NLP) systems into everyday writing infrastructure, supported by the scaling of GPT-3, the public launch of ChatGPT, and the later development of GPT-4 [4,5,6]. Since late 2022, these tools have become visible across scientific work because they can draft, revise, summarise, translate, and polish prose with a level of fluency that makes them attractive to researchers and students [5,7,8]. Academic writing is not only a container for results, it is also a rhetorical system through which authors signal novelty, caution, contribution, and disciplinary belonging [7,8]. Biomedical abstracts are especially important in this debate because they are short, highly searchable, widely indexed, and often used by readers and automated systems to make rapid judgements about a study [1,9].

Existing work shows that generative AI (GenAI) is already influencing science, biomedicine, and scholarly publishing, although its benefits and risks vary across tasks and contexts [7,8,9,10,11]. Editorial and publishing organisations have responded by emphasising transparency, responsibility, and the need to prevent AI tools from being treated as authors because such systems cannot take responsibility for the integrity of a manuscript [11,12,13,14]. However, disclosure policies mainly address declared use, authorship responsibility, and editorial governance rather than the quieter possibility that LLMs are reshaping the style of academic language itself [10,11,12,13,14]. At the same time, AI-text detection remains fragile, with evidence showing that detectors can fail under paraphrasing, vary across tools, and unfairly misclassify non-native English writing as AI-generated [3,15,16,17].

The strongest corpus-level evidence to date comes from studies that track LLM-associated vocabulary rather than individual authorship [1,2]. Kobak et al. showed that PubMed abstracts experienced an abrupt post-ChatGPT increase in style-heavy words and estimated that at least 13.5% of 2024 biomedical abstracts showed evidence of LLM processing [1]. Gray similarly used indicative keywords to estimate the prevalence of LLM-assisted writing across scholarly literature, showing that such signals can be measured at scale even when individual disclosure is unavailable [2]. These studies are valuable, but they leave a further question unresolved: whether the observed vocabulary changes are only traces of possible LLM use or whether they reflect a broader diffusion of rhetorical habits across academic writing [1,2].

Our study addresses that gap by reframing post-ChatGPT language change as LLM-associated same-ification, understood as a corpus-level movement toward repeated rhetorical and stylistic patterns [1,2,18]. Same-ification is important because LLM influence may appear not only through fully generated text but also through repeated patterns of polishing, connective phrasing, value framing, contribution signalling, and improvement-oriented rhetoric [1,7,8].

Accordingly, the primary aim of this study was to determine whether PubMed biomedical abstracts published after the public release of ChatGPT show a measurable population-level increase in LLM-associated stylistic markers. We compared the pre-ChatGPT (2018–2022) and post-ChatGPT (2023–2025) periods, tested whether changes exceeded those in a biomedical control vocabulary, and evaluated the robustness of the signal using distributional, temporal, journal-level, topic-controlled, matched-sample, and explainable machine-learning analyses.

To address this aim, the study examined the following three research questions:

• RQ1. Did strict LLM-associated stylistic markers increase in PubMed biomedical abstracts after the public release of ChatGPT?
• RQ2. Is any observed increase stronger than changes in general biomedical and academic control vocabulary, suggesting a stylistic rather than purely topical shift?
• RQ3. Does the same-ification signal remain visible across distributional, temporal, topic-controlled, matched-sample, and explainable machine-learning validation analyses?

By focusing on corpus-level rhetorical convergence, this study reframes how LLM influence in scholarly writing can be examined. Existing work has mainly treated LLM-associated vocabulary as a signal for estimating possible AI use, but this paper examines whether such vocabulary reflects a broader diffusion of academic style. This study advances this argument through a reproducible measurement pipeline that combines strict stylistic markers, control vocabulary, interrupted time-series modelling, lexical diversity, semantic similarity, and journal-level heterogeneity analysis. It also extends corpus-based evidence with SAME-ML, a style-aware, matched, and explainable machine-learning validation framework that tests whether the observed signal is learnable, robust to topic composition, and interpretable at the feature level.

2. Related Work

2.1. Generative AI as Scholarly Writing Infrastructure

LLMs are part of scholarly writing because they can support drafting, rewriting, summarisation, translation, coding assistance, and stylistic polishing [4,5,6,7,8]. GPT-3 demonstrated that large-scale language models could perform many tasks through prompting, while ChatGPT made this capability widely accessible through a conversational interface [4,5]. GPT-4 extended this trajectory by showing stronger performance across professional and academic benchmarks, which increased interest in using LLMs across research workflows [6,8]. In biomedicine, reviews show that LLMs are being explored for information retrieval, question answering, summarisation, medical education, clinical documentation, and knowledge extraction, although reliability and validation remain major concerns [9,10].

2.2. Publishing Policy, Transparency, and Accountability

The rapid adoption of GenAI has pushed journals and publishers to develop policies around disclosure, responsibility, and authorship [11,12,13,14]. Ganjavi et al. found substantial variation in publisher and journal instructions for GenAI use, which suggests that authors face inconsistent expectations across venues [11]. The Committee on Publication Ethics, the World Association of Medical Editors, and Nature Portfolio all emphasise that AI tools cannot be credited as authors because they cannot approve a final manuscript, accept accountability, or respond to questions about research integrity [12,13,14]. These policies are necessary, but they do not fully address how repeated AI-assisted editing may gradually influence the rhetorical norms of published writing [7,11,12,13,14].

2.3. Detection, Fairness, and the Limits of Authorship Classification

A major strand of early LLM-writing research focused on detection, using probability curvature, classifiers, watermarking, perplexity, or lexical signals to distinguish machine-generated from human-written text [3,15,16,17]. DetectGPT showed that probability curvature could support zero-shot detection, but later work showed that detection can become unreliable when outputs are paraphrased or deliberately modified [15,17]. Weber-Wulff et al. tested multiple AI-generated-text detectors and reported that available tools were not consistently reliable across realistic conditions [16]. Liang et al. showed that GPT detectors can misclassify non-native English writing as AI-generated, which creates a serious fairness risk for multilingual academic communities [3]. These limitations support a shift away from policing individual authorship and toward measuring population-level changes in academic language [1,2,3,15,16,17].

2.4. From Excess Vocabulary to LLM-Associated Same-Ification

Corpus-based studies show that LLM influence can be measured through vocabulary patterns that change after the public release of ChatGPT [1,2]. Kobak et al. analysed more than 15 million PubMed abstracts and found abrupt increases in style words after the emergence of ChatGPT-like tools, with words such as delve and underscore becoming unusually frequent [1]. Gray used indicative keywords to estimate LLM-assisted writing in scholarly literature more broadly, showing that the issue extends beyond isolated examples or obvious misuse [2]. However, excess-vocabulary studies mainly estimate possible LLM use, while the present study asks what kind of academic style is spreading and whether that style remains visible after topic, length, control vocabulary, and venue-related variations are considered [1,2].

2.5. Corpus Measurement and Explainable Validation

Measuring same-ification requires methods that can separate temporal change, stylistic density, lexical diversity, topic composition, and model interpretability [19,20,21,22,23,24]. Interrupted time-series analysis is useful because it tests whether a post-event pattern differs from an earlier baseline through level or slope changes [19]. Lexical diversity measures such as the Measure of Textual Lexical Diversity (MTLD) and text-analysis tools such as quanteda support reproducible document-level measurement beyond simple word counts [20,21]. Sentence embeddings and topic models help reduce the risk that apparent stylistic change is actually a change in biomedical subject matter [22,23]. SHAP-based explanation provides an interpretable way to examine which features push a machine-learning model toward a post-ChatGPT classification [24]. SAME-ML brings these strands together by using machine learning as a validation layer for testing whether the same-ification signal is learnable, topic-robust, matched-sample consistent, and explainable [3,15,16,17,18,19,20,21,22,23,24].

3. Methods

3.1. Methodological Positioning and Novelty

This study investigates whether public biomedical academic writing has begun to show measurable signs of LLM-associated stylistic diffusion after the public release of ChatGPT. Existing studies have mostly focused on detecting whether academic texts were generated or processed by large language models, often by identifying “excess vocabulary” or estimating possible LLM usage in published writing. For example, existing excess-vocabulary work on PubMed abstracts showed that many words increased abruptly after the emergence of ChatGPT-like LLMs, with the strongest post-2023 changes appearing in stylistic rather than purely content-related vocabulary [1].

The present study takes a different direction. It does not attempt to classify individual abstracts as AI-written or human-written. Instead, it asks whether biomedical academic writing, as a public corpus, is showing signs of same-ification, understood here as a population-level movement toward repeated lexical and rhetorical patterns associated with LLM-assisted writing. This distinction is important because the influence of LLMs may not only appear through fully generated text. It may also appear more subtly, through the gradual normalisation of particular stylistic habits, such as more frequent use of polished, evaluative, connective, and emphasis-building expressions.

The novelty of our study, therefore, lies in reframing post-ChatGPT vocabulary change as a broader phenomenon of stylistic diffusion. Rather than treating LLM-associated vocabulary only as a detection signal, this study uses it as evidence of possible language change in academic writing. In doing so, the study contributes a reproducible method for measuring how public academic language may be shifting in the LLM era.

3.2. Study Design

The study adopted a retrospective computational corpus design. PubMed abstracts published between 2018 and 2025 were used to examine changes in biomedical academic writing before and after the public release of ChatGPT in November 2022. The period from 2018 to 2022 was treated as the pre-ChatGPT baseline, while 2023 to 2025 was treated as the post-ChatGPT period. This coding is used consistently throughout the analyses: pre-ChatGPT observations satisfy $y_i$ ≤ 2022, and post-ChatGPT observations satisfy $y_i$ ≥ 2023.

This design was chosen because ChatGPT’s public release provides a defined historical boundary for examining population-level language change. The design follows the logic of interrupted time-series research, where a clearly identifiable event is used to evaluate whether an outcome departs from its earlier baseline through a level or slope change [19]. It is used as a historically meaningful exposure point after which LLM-based writing support became widely accessible through authoring, editing, and revision workflows. The unit of analysis was the individual abstract, with document-level metadata, including publication year, journal name, title, language information, word count, marker frequencies, control vocabulary frequencies, lexical diversity, and similarity-related features. The methodological workflow is summarised in Figure 1.

Figure 1. Methodological workflow for corpus construction, text processing, comparative analysis, and output generation.

3.3. Data Source and Analytical Dataset Construction

The dataset was constructed from PubMed-indexed biomedical abstracts. PubMed was selected because it provides a large, structured, publicly accessible, and time-stamped corpus of biomedical academic writing. Records were retrieved using the National Center for Biotechnology Information’s Entrez Programming Utilities, which provide programmatic access to PubMed and related NCBI databases [25]. For each year from 2018 to 2025, PubMed records were retrieved using publication-date filters. The extracted fields included PubMed ID, publication year, journal title, article title, language metadata, publication type, and abstract text. The abstract was selected as the main textual unit because abstracts are highly visible, relatively standardised, and available across a wide range of biomedical journals. This makes them suitable for comparing language patterns over time.

Records were excluded if they did not contain an abstract, were not in English, had fewer than 80 words after preprocessing, or appeared to represent non-standard publication formats. Titles containing terms such as “erratum”, “corrigendum”, “correction”, “retraction”, “withdrawn”, or “expression of concern” were removed to reduce contamination from correction notices and editorial records. After filtering, the final analytic corpus contained 28,415 abstracts, with 20,236 in the pre-ChatGPT period and 8179 in the post-ChatGPT period.

The corpus should be read as a structured analytic sample rather than as a complete census of PubMed. PubMed retrieval was capped at 5000 records per publication year, and records were retained only when the XML extraction procedure returned an eligible English abstract. The 2023 sample was smaller than the surrounding years because the extraction step yielded fewer usable abstracts for that year. For this reason, the manuscript reports 2023 explicitly, interprets it as a transitional low-yield year, and places greater evidential weight on the sustained 2024–2025 pattern rather than on a single-year discontinuity.

3.4. Text Preprocessing

All text was processed using a reproducible script-based workflow. Abstracts were converted to lowercase, non-alphabetic characters were removed, repeated whitespace was standardised, and each abstract was tokenised into word units. Word counts were calculated for every abstract and used to normalise all marker frequencies. Normalisation was necessary because abstracts differ in length. Without normalisation, longer abstracts would be more likely to contain stylistic markers simply because they contain more words. Therefore, the primary outcome was expressed as the number of LLM-associated stylistic markers per 1000 abstract words. Word-level summaries were also reported as occurrences per million words to support interpretation of individual marker trajectories.

3.5. LLM-Associated Stylistic Marker Dictionary

The main analytical measure was a strict dictionary of LLM-associated stylistic markers. These markers were defined as words that operate primarily as rhetorical, evaluative, connective, or emphasis-building language rather than as biomedical content terms. Examples include words such as delve, intricate, meticulous, pivotal, underscore, comprehensive, crucial, enhance, notably, nuanced, seamless, and transformative. The candidate marker pool was derived from the publicly released excess-vocabulary list accompanying Kobak et al. [1]. Rather than adopting the full list directly, this study refined it into a strict style-focused dictionary by retaining words associated with rhetorical, evaluative, connective, and emphasis-building language while excluding biomedical content terms and generic academic vocabulary. This was an important methodological step because broad excess-vocabulary lists may include content-heavy or generic academic words. Such terms may reflect topic shifts, journal composition, or changes in biomedical publishing rather than stylistic diffusion. The measurement logic for applying this dictionary is formalised in Section 3.7.

To improve construct validity, the strict marker set excluded generic and content-adjacent words, such as “this”, “these”, “based”, “research”, “including”, “clinical”, “treatment”, “patient”, “disease”, “analysis”, “model”, “data”, and “results”. The retained markers were selected because they more clearly reflected style, framing, emphasis, or academic polish. This strict-style version was used as the main analysis. Broader vocabulary patterns were treated only as exploratory or sensitivity evidence. The dictionary was, therefore, designed as a high-specificity measure of rhetorical style rather than a high-recall measure of all possible LLM-associated vocabulary. No marker is unique to ChatGPT; inference, therefore, rests on aggregate frequency changes across the corpus. Hence, our study examines whether the collective frequency of a restricted stylistic marker set changed after 2022 in a way that is stronger than ordinary vocabulary drift.

3.6. Control Vocabulary

A control vocabulary was included to test whether the observed pattern was specific to LLM-associated stylistic markers or simply reflected a general change in biomedical academic language. The control vocabulary contained common biomedical and academic terms, such as “patient”, “clinical”, “treatment”, “disease”, “study”, “data”, “analysis”, “method”, “results”, “risk”, and “association”. For each abstract, both the stylistic marker count and the control-word count were calculated and normalised per 1000 words. This allowed the study to compare two trends, one representing LLM-associated stylistic markers and one representing general biomedical academic vocabulary. If the stylistic-marker rate increased sharply after 2022 while the control vocabulary did not follow the same trajectory, this would strengthen the interpretation that the change reflects stylistic diffusion rather than general vocabulary growth.

3.7. Measuring LLM-Associated Same-Ification

To make the measurement procedure transparent and reproducible, same-ification was operationalised as a document-level and year-level scoring process. This process quantified whether the analytical dataset showed increased use of LLM-associated stylistic markers after the public release of ChatGPT. The algorithm, therefore, treats each abstract as an observation and estimates the density of stylistic markers relative to the document length and to a control vocabulary. To clarify how the different analytical components contribute to the measurement of LLM-associated same-ification, the study used a multi-level evidence framework, shown in Figure 2. The framework integrates the following three complementary evidence lanes: document-level evidence based on marker rate, lexical diversity, and semantic similarity; marker-level evidence based on stylistic marker growth; and journal-level evidence based on venue-specific diffusion intensity and heterogeneity.

Figure 2. Multi-level framework for detecting LLM-associated same-ification.

Let D denote the analytical dataset of PubMed abstracts, where each document, d_i, has a publication year, y_i, a token sequence, T_i, and a word count, N_i = |T_i|. Let L_s denote the strict LLM-associated stylistic marker dictionary, and L_c denote the control vocabulary. For each abstract, the stylistic marker count, M_i, and control vocabulary count, C_i, were calculated as follows:

$$M_i=\sum _{w\in T_i}1\left(w\in L_s\right)$$

$$C_i=\sum _{w\in T_i}1\left(w\in L_c\right)$$

where 1(·) is an indicator function that equals 1 when the token appears in the relevant dictionary and 0 otherwise. To control for variation in abstract length, both counts were normalised per 1000 words, as follows:

$$R_i^s=\left({\displaystyle \frac{M_i}{N_i}}\right)\times 1000 R_i^c=\left({\displaystyle \frac{C_i}{N_i}}\right)\times 1000$$

where $R_i^s$ is the LLM-associated stylistic-marker rate, and $R_i^c$ is the control-vocabulary rate for abstract i. Each abstract was then assigned to either the pre-ChatGPT period or the post-ChatGPT period using the publication year, as follows:

$$P_i=\left\{\begin{array}{cc}0,& y_i\le 2022\\ 1,& y_i\ge 2023\end{array}\right.$$

At the word level, each marker, w, was evaluated by comparing its pre-ChatGPT and post-ChatGPT frequencies per million words. For each period, p, the frequency of marker w was calculated as follows:

$$F_{w,p}=\left({\displaystyle \frac{\mathrm{count}\left(w,p\right)}{{\sum }_{i\in p}{N}_i}}\right)\times {10}^6$$

The absolute post-ChatGPT change and relative post/pre ratio were then calculated as follows:

$${\Delta }_w=F_{w,\mathrm{post}}-F_{w,\mathrm{pre}}$$

$${\rho }_w={\displaystyle \frac{F_{w,\mathrm{post}}+\mathsf{ϵ}}{F_{w,\mathrm{pre}}+\mathsf{ϵ}}}$$

where ε = 0.01 was added to avoid division by zero for markers that were absent or extremely rare in the pre-ChatGPT period. These word-level measures were used to identify which stylistic markers contributed most strongly to the observed post-2022 shift.

An exploratory same-ification score was also calculated to combine multiple indicators of convergence. Let Z(·) denote standardisation using the analytical dataset’s mean and standard deviation. The exploratory score for abstract i was defined as follows:

$$S_i=Z\left(R_i^s\right)+Z\left(Q_i\right)-Z\left(L{D}_i\right)$$

where $R_i^s$ is the stylistic-marker rate, $Q_i$ is the semantic-similarity feature, and $L{D}_i$ is the lexical-diversity score. Higher values of $S_i$ indicate stronger evidence of same-ification only in the narrow operational sense of higher stylistic marker density, greater document-level similarity, and lower lexical diversity. The composite index was not treated as the main outcome because its value depends on how similarity and lexical diversity are operationalised. The primary inference, therefore, rests on the strict marker trajectory and its contrast with the control vocabulary; $Q_i$ and $L{D}_i$ are used as supplementary diagnostics for RQ3. Algorithm 1 summarises the complete analytical procedure used to measure LLM-associated stylistic diffusion and same-ification.

Algorithm 1. LLM-Associated Stylistic Diffusion and Same-ification Measurement
Input: PubMed abstract dataset, D; strict stylistic marker dictionary, Ls; control vocabulary, Lc; ChatGPT boundary year, Y0 = 2022; minimum abstract length, Nmin = 80. Output: Document-level marker table M; yearly trend table Ty; marker-level change table W; exploratory same-ification score table S.
1	Initialise M ← ∅
2	for each abstract di in D do
3	Extract PMID, publication year yi, journal name, and abstract text
4	texti ← LOWERCASE(di.abstract)
5	texti ← REMOVE_NON_ALPHABETIC_CHARACTERS(texti)
6	Ti ← TOKENISE(texti)
7	Ni ← COUNT(Ti)
8	if Ni < Nmin then
9	Exclude di from further analysis
10	continue
11	end if
12	marker_counti ← COUNT(w ∈ Ti such that w ∈ Ls)
13	control_counti ← COUNT(w ∈ Ti such that w ∈ Lc)
14	marker_ratei ← (marker_counti/Ni) × 1000
15	control_ratei ← (control_counti/Ni) × 1000
16	if yi > Y0 then
17	periodi ← Post-ChatGPT
18	else
19	periodi ← Pre-ChatGPT
20	end if
21	lexical_diversityi ← MTLDi
22	for each direction d ∈ {→, ←} do
23	Initialise a new token segment
24	for each token position k in the current segment do
25	TTRid(k) ← COUNT_UNIQUE_TYPES(current segment after k tokens)/k
26	if TTRid(k) ≤ τ, where τ = 0.72, then
27	Record one complete factor
28	Begin a new token segmen
29	end if
30	end for
31	Fid ← fid + (1 − TTRi,finald)/(1 − τ)
32	end for
33	MTLDi ← ½[(Ni/Fi→) + (Ni/Fi←)]
34	semantic_similarityi ← Qγi
35	for each publication year y do
36	Aγ ← RANDOM_SAMPLE(abstracts published in year y,
37	size = min(250, nγ), seed = 42)
38	for each abstract j ∈ Aγ do
39	ej ← SBERT(textj, model = “all-MiniLM-L6-v2”)
40	end for
41	Qγ ← 2/[mγ(mγ − 1)] × Σa<β, a,b ∈ Aγ [(eaTeβ)/(‖ea‖2‖eβ‖2)]
42	end for
43	semantic_similarityi ← Qγi
44	Append to M: {pmidi, yi, journali, Ni, marker_counti, control_counti, marker_ratei, control_ratei, lexical_diversityi, semantic_similarityi, periodi}
45	end for
46	Initialise Ty ← ∅
47	for each publication year y represented in M do
48	marker_meany ← MEAN(marker_ratei where yi = y)
49	control_meany ← MEAN(control_ratei where yi = y)
50	marker_CIy ← COMPUTE_95CI(marker_ratei where yi = y)
51	control_CIy ← COMPUTE_95CI(control_ratei where yi = y)
52	Append to Ty: {y, marker_meany, control_meany, marker_CIy, control_CIy}
53	end for
54	Initialise W ← ∅
55	for each stylistic marker w in Ls do
56	pre_ratew ← FREQUENCY_PER_MILLION(w in Pre-ChatGPT abstracts)
57	post_ratew ← FREQUENCY_PER_MILLION(w in Post-ChatGPT abstracts)
58	absolute_changew ← post_ratew − pre_ratew
59	relative_ratiow ← (post_ratew + 0.01)/(pre_ratew + 0.01)
60	Append to W: {w, pre_ratew, post_ratew, absolute_changew, relative_ratiow}
61	end for
62	Initialise S ← ∅
63	for each retained abstract di in M do
64	z_markeri ← ZSCORE(marker_ratei across all retained abstracts)
65	z_semantici ← ZSCORE(semantic_similarityi across all retained abstracts)
66	z_diversityi ← ZSCORE(lexical_diversityi across all retained abstracts)
67	sameificationi ← z_markeri + z_semantici − z_diversityi
68	Append to S: {pmidi, sameificationi, z_markeri, z_semantici, z_diversityi}
69	end for
70	return M, Ty, W, S

Note. The same-ification score is used as an exploratory composite indicator, not as a direct detector of AI authorship. Higher values indicate abstracts with a stronger LLM-associated stylistic marker concentration, greater semantic similarity to the wider retained corpus, and lower lexical diversity.

3.8. Interpretable SAME-ML Validation Framework

This study proposes Style-Aware, Matched, and Explainable Machine Learning (SAME-ML) as a study-specific validation framework. SAME-ML is not a new classification algorithm. It is an integrated analytical pipeline designed to test whether a corpus-level stylistic shift is learnable, robust to alternative explanations, and interpretable. “Style-aware” refers to the use of document-level lexical, rhetorical, marker, control-vocabulary, and length features rather than individual AI-authorship labels. “Matched” refers to the comparison of post-ChatGPT abstracts with comparable pre-ChatGPT abstracts within text-derived biomedical topics. “Explainable” refers to the use of feature importance and SHAP values to identify the features contributing to period classification. The framework combines style-only classification, topic-adjusted validation, matched-sample comparison, and marker-ablation analysis.

The target was the publication period, coded as pre-ChatGPT for abstracts published from 2018 to 2022 and post-ChatGPT for abstracts published from 2023 to 2025. The predictors were derived from the same document-level feature table used in the computational corpus analysis. The style-feature set included the operational marker rate per 1000 words, control-vocabulary rate per 1000 words, marker-control ratio, marker-minus-control contrast, MTLD lexical diversity, log-transformed word count, and rhetorical-group indicators. Year-level semantic similarity and the composite same-ification index were excluded because they could encode temporal information directly. The rhetorical indicators represented value and possibility framing, scope and connective framing, evidence-and-claim signalling, improvement and integration rhetoric, problem-solution orientation, and overall rhetorical density. The strict-style corpus results remained the primary descriptive evidence. SAME-ML was used only as a validation layer to test whether the broad stylistic shift was learnable after topic, length, control-vocabulary, and matching adjustments.

The composite same-ification index was retained as an exploratory descriptive measure, as presented below in the Section 4, because it summarises the temporal patterns of marker density, semantic similarity, and lexical diversity. However, it was excluded from the SAME-ML predictors, matching variables, and matched outcomes because its year-level semantic-similarity component could encode publication period and thereby compromise the independence of the validation analyses.

The first validation experiment (E1) used style-only supervised learning. LASSO (glmnet version 5.0) logistic regression, random forest, and XGBoost (version 3.2.1.1.) classifiers were trained using only style and same-ification features, with no full-text topical vocabulary supplied to the models. This experiment asked whether the post-ChatGPT period could be ranked or separated using rhetorical features alone. The analysis, therefore, treated supervised learning as a diagnostic validation tool rather than as a detector of AI-generated writing.

The second experiment (E2) introduced topic-masked validation. Abstracts were converted into TF-IDF representations and reduced using latent semantic analysis before being grouped into ten topic clusters. Model performance was then compared under the following three conditions: a topic-only text model, a style-plus-topic model, and a topic-residualised style model. In the residualised condition, each style feature was adjusted for topic cluster and log-transformed word count before classification. This design was used to test whether the stylistic signal persisted after reducing the possibility that the post-ChatGPT pattern was mainly a consequence of changing biomedical subject matter.

The third experiment (E3) used a matched pre/post comparison. Post-ChatGPT abstracts were matched one-to-one, without replacement, with comparable pre-ChatGPT abstracts using nearest-neighbour propensity-score matching. The propensity model included log-transformed word count, control-vocabulary rate, lexical diversity, and journal grouping, while matching was constrained to occur within the same text-derived biomedical topic. Covariate balance was assessed using absolute standardised mean differences before and after matching. The matched pairs were then compared in terms of stylistic marker density, marker-control ratio, overall rhetorical density, and rhetorical-function rates. Year-level semantic similarity and the composite same-ification index were excluded from the matching design. This experiment was interpreted as a design-based robustness check rather than as a causal estimate of ChatGPT exposure.

The fourth experiment (E4) used marker-ablation analysis. Text-based XGBoost models were trained under the following four conditions: full abstract text, abstract text with stylistic markers removed, stylistic markers only, and control vocabulary only. This experiment tested where the period signal lived in the text. If performance dropped after removing stylistic markers, this would suggest that the LLM-associated marker layer contributed directly to the post-ChatGPT signal. If performance remained strong, it would suggest that the period signal was distributed more broadly across the lexical and rhetorical surface of the abstracts.

Model performance was evaluated on held-out test data using ROC AUC, PR AUC, accuracy, sensitivity, specificity, and F1 score. Because the post-ChatGPT class was smaller than the pre-ChatGPT class, ROC AUC and PR AUC were treated as the main ranking-based indicators, while threshold-dependent measures were interpreted cautiously. Complete or near-complete separation in any held-out experiment was interpreted as evidence of separability within this corpus and feature design, not as evidence of deployment-ready classification or individual AI authorship. Model interpretability was added through XGBoost gain-based feature importance and tree SHAP values [24]. These interpretability outputs helped identify whether the period signal was driven mainly by biomedical content, general vocabulary, semantic similarity, or rhetorical same-ification features.

4. Results

The results are organised around the three research questions rather than around separate analytical layers. RQ1 is answered using aggregate, yearly, distributional, and interrupted time-series evidence. RQ2 is answered by comparing strict stylistic markers with the control vocabulary and by examining the rhetorical functions of the fastest-growing markers. RQ3 is answered through secondary same-ification indicators, journal-level heterogeneity, topic-masked modelling, matched-sample comparison, marker-ablation, and interpretable SAME-ML analyses. This structure keeps the primary claim tied to marker/control evidence while treating semantic similarity, composite scores, and machine-learning outputs as validation rather than as standalone proof of LLM authorship.

4.1. RQ1: Post-ChatGPT Increase in Strict Stylistic Markers

RQ1 asked whether strict LLM-associated stylistic markers increased in PubMed biomedical abstracts after the public release of ChatGPT. The aggregate evidence answers this question affirmatively. Figure 3 shows that the strict marker rate was relatively stable before the ChatGPT boundary, remaining close to five markers per 1000 abstract words between 2018 and 2022. After the boundary, the trajectory changed substantially: The mean strict marker rate increased from 5.965 in 2023 to 10.595 in 2024 and 13.263 in 2025. This increase is interpreted as a population-level shift in rhetorical style, not as evidence that any individual abstract was written by an LLM.

Figure 3. Yearly trend in strict LLM-associated stylistic-marker rates compared with the control vocabulary. The dashed vertical line marks the ChatGPT boundary, and the ribbon shows the 95% confidence interval.

This divergence is central to the interpretation of the results. If both the strict marker vocabulary and the control vocabulary had increased in parallel, the post-2022 pattern could be explained as a broad change in biomedical abstract language, abstract length, or publication composition. Instead, the sharp growth is concentrated in the style-only marker set. The comparison, therefore, strengthens the claim that the post-2022 shift is stylistic rather than simply a by-product of general vocabulary growth.

The indexed comparison in Figure 4 makes the imbalance clearer. When 2022 is treated as the baseline, LLM-associated stylistic markers rise to more than 250% of their 2022 value by 2025. The control vocabulary remains near the baseline and falls below it in 2025. Lexical diversity also increases but at a much slower rate than the stylistic marker index. This pattern suggests that post-ChatGPT abstracts are not simply becoming less varied or more lexically impoverished. Rather, they appear to be accumulating a recognisable stylistic layer while retaining, and in some respects increasing, lexical variety.

Figure 4. Relative post-2022 change in strict stylistic markers, control vocabulary, and lexical diversity. Values are indexed to 2022 = 100 to enable comparison across differently scaled measures.

Table 1 shows that the strongest between-period difference occurred in strict stylistic marker use, with a large standardised effect (g = 0.80). Lexical diversity also increased moderately (g = 0.54), while the increase in abstract length was small (g = 0.20). Although the control-vocabulary difference was statistically significant, its effect size was negligible (g = −0.04). This contrast supports the interpretation that the post-2022 shift was concentrated in stylistic markers rather than representing uniform growth across general biomedical vocabulary.

Table 1. Descriptive and inferential comparison of pre- and post-ChatGPT abstracts.

Outcome	Pre-ChatGPT Mean (SD)	Post-ChatGPT Mean (SD)	Mean Difference [95% CI]	Hedges’ (g)	BH-Adjusted (p)
Strict stylistic-marker rate	4.995 (6.375)	11.658 (11.785)	6.663 [6.393, 6.933]	0.803	<0.001
Control-vocabulary rate	31.152 (21.242)	30.216 (20.946)	−0.936 [−1.476, −0.395]	−0.044	<0.001
Lexical diversity (MTLD)	69.964 (24.322)	85.170 (35.325)	15.205 [14.370, 16.041]	0.544	<0.001
Abstract word count	210.563 (63.517)	223.445 (64.545)	12.882 [11.232, 14.532]	0.202	<0.001

Note. Stylistic-marker and control-vocabulary rates are reported per 1000 abstract words. MTLD refers to the Measure of Textual Lexical Diversity. Values are presented as the mean (SD), and mean differences represent post-ChatGPT minus pre-ChatGPT values. Comparisons were conducted using two-sided Welch t-tests, with 95% confidence intervals and Hedges’ g effect sizes. p-Values were adjusted across the four outcomes using the Benjamini–Hochberg procedure.

The yearly estimates in Table 2 show that the shift was not a gradual continuation of the pre-ChatGPT baseline. Between 2018 and 2022, the strict marker rate fluctuated within a narrow band, from 4.575 to 5.383 per 1000 words. The substantial increase appears after the boundary, especially in 2024 and 2025. The 2025 mean of 13.263 is approximately 2.54 times the 2022 value. This time pattern is consistent with diffusion rather than immediate replacement: ChatGPT’s release marks the historical boundary, but the larger linguistic movement becomes more visible after a period of adoption, normalisation, and possible integration into authoring, editing, and revision workflows. The stronger evidence for RQ1, therefore, comes from the stable pre-2022 baseline and the sustained 2024–2025 increase, not from the 2023 point alone.

Table 2. Yearly strict marker and control-vocabulary rates, 2018–2025.

Year	No. Abstracts	Mean Strict Marker Rate	95% CI: Strict Marker Rate	Mean Control Rate	95% CI: Control Rate	Mean MTLD
2018	4.214	5.383	5.185–5.582	30.806	30.162–31.451	69.837
2019	4.051	4.843	4.652–5.033	30.870	30.217–31.524	68.170
2020	3.866	4.575	4.385–4.765	31.977	31.304–32.649	69.505
2021	4.122	4.914	4.719–5.109	31.270	30.623–31.918	70.903
2022	3.983	5.230	5.024–5.435	30.879	30.225–31.533	71.398
2023	3.778	5.965	5.350–6.580	34.664	32.709–36.619	72.206
2024	3.552	10.595	10.205–10.986	30.806	30.103–31.508	76.542
2025	4.127	13.263	12.902–13.625	29.169	28.549–29.789	94.166

Note. Confidence intervals are 95% intervals around the yearly mean.

The distributional analysis extends the aggregate findings by showing that the post-ChatGPT shift is visible across the spread of abstract-level marker rates. As shown in Figure 5, the post-ChatGPT distribution has a higher centre and a longer upper tail, indicating that the post-2022 period contains more abstracts with comparatively high concentrations of strict stylistic markers. Strict stylistic-marker rates were substantially higher in the post-ChatGPT period (M = 11.658, SD = 11.785) than in the pre-ChatGPT period (M = 4.995, SD = 6.375). The mean difference was 6.663 markers per 1000 words, 95% CI [6.393, 6.933], Welch’s t(10,169) = 48.35, p < 0.001, Hedges’ g = 0.80, indicating a large standardised difference.

Figure 5. Distributional shift in strict LLM-associated stylistic marker use before and after the ChatGPT boundary. Violin plots show the distribution, boxplots show the median and interquartile range, and white-centred points show group means.

The shape of the post-ChatGPT distribution is important because it points to uneven diffusion. Some post-ChatGPT abstracts remain low in stylistic marker density, while others show much higher saturation. This makes the finding more realistic than a claim of universal change. The evidence suggests that LLM-associated phrasing is spreading unevenly across biomedical writing, with particular documents, venues, or subfields adopting the style more strongly than others.

The interrupted time-series model tested whether the post-2022 pattern differed from the pre-existing trend. Table 3 summarises the magnitude of the shift, while Table 4 estimates level and slope changes. The strict marker rate increased by 133.4%, whereas the control-vocabulary rate decreased slightly. The pre-ChatGPT time trend was small and not statistically significant (estimate = 0.082, p = 0.454), indicating no clear year-on-year increase in marker density before the boundary. The post-ChatGPT level shift was positive and statistically significant (estimate = 3.689, p = 0.005), and the post-ChatGPT slope change was stronger (estimate = 5.064, p < 0.001). These results directly support RQ1 by showing both a higher post-boundary level and a steeper post-boundary trajectory.

Table 3. Descriptive post-ChatGPT shift in strict stylistic marker use and comparator measures.

Measure	Pre-ChatGPT	Post-ChatGPT	Absolute Change	Relative Change
Mean strict marker rate	4.995	11.658	+6.663	+133.4%
Median strict marker rate	3.846	8.584	+4.738	+123.2%
Mean control-vocabulary rate	31.152	30.216	−0.936	−3.0%
Median control-vocabulary rate	27.027	25.641	−1.386	−5.1%
Mean MTLD	69.964	85.170	+15.206	+21.7%
Mean word count	210.563	223.445	+12.882	+6.1%

Table 4. Segmented regression model predicting strict LLM-associated stylistic-marker rate per 1000 words.

Predictor	Estimate	Robust SE	t	p
Intercept	5.941	0.592	10.040	<0.001
Time	0.082	0.110	0.750	0.454
Post-ChatGPT level shift	3.689	1.311	2.810	0.005
Post-ChatGPT slope change	5.064	0.515	9.830	<0.001
Control rate per 1000 words	0.021	0.007	3.160	0.002
Word count	−0.026	0.002	−11.440	<0.001

Note. Robust standard errors are reported. The outcome is the strict LLM-associated stylistic-marker rate per 1000 words.

The model, therefore, supports an acceleration interpretation. The post-2022 period was not only higher than the pre-ChatGPT baseline, but the rate of increase also became steeper. This distinction matters for this paper’s contribution. A level shift alone would suggest a one-off vocabulary jump after the release of ChatGPT. A slope change suggests a diffusion process, where stylistic markers become increasingly normalised over time. The positive association with the control vocabulary is small, while word count has a negative coefficient, indicating that the modelled post-ChatGPT increase cannot be reduced to longer abstracts or a broad increase in common biomedical vocabulary.

4.2. RQ2: Specificity of the Shift Relative to Control Vocabulary and Rhetorical Function

RQ2 asked whether the increase was stronger than changes in general biomedical and academic control vocabulary. The marker-level results support that interpretation by showing that the largest contributors to the post-ChatGPT rise are not primarily biomedical content terms. Figure 6 and Table 5 show that the largest absolute increases occurred for markers such as “potential”, “across”, “while”, “strategies”, “challenges”, “particularly”, “demonstrated”, “enhance”, “comprehensive”, and “exhibited”. These words operate as framing, connective, evidential, or emphasis-building language. They, therefore, help explain why the aggregate pattern is better read as a stylistic shift than as a change in biomedical subject matter.

Figure 6. Strict stylistic markers with the largest post-ChatGPT absolute increases. The x-axis and horizontal line length represent the absolute increase in occurrences per million words, calculated as the post-ChatGPT rate minus the pre-ChatGPT rate. Circle size and colour both represent the post/pre frequency ratio: larger and darker-purple circles indicate stronger proportional growth from the pre-ChatGPT baseline, whereas smaller and yellow circles indicate lower proportional growth.

Table 5. Strict stylistic markers with the largest post-ChatGPT increases.

Marker	Pre Rate	Post Rate	Absolute Increase	Post/Pre Ratio
potential	636.711	1184.641	547.930	1.861
across	452.949	897.372	444.423	1.981
while	785.973	1222.943	436.971	1.556
strategies	312.371	636.368	323.998	2.037
challenges	191.037	437.195	246.159	2.288
particularly	196.200	434.460	238.260	2.214
demonstrated	373.390	610.104	236.714	1.634
enhance	85.896	321.194	235.298	3.739
comprehensive	132.599	355.666	223.067	2.682
exhibited	107.722	323.382	215.660	3.002
promising	113.824	327.760	213.936	2.879
insights	61.019	271.948	210.929	4.456
additionally	125.089	316.816	191.727	2.533
crucial	82.376	263.193	180.817	3.195
enhancing	45.295	195.343	150.048	4.312
notably	25.581	140.625	115.044	5.495
integrating	16.428	118.738	102.309	7.224
underscores	4.459	86.454	81.995	19.347
underscore	14.316	91.926	77.610	6.417
enhances	14.551	89.190	74.639	6.126

Note. Rates are reported as occurrences per million words. Absolute increase is calculated as post-ChatGPT rate minus pre-ChatGPT rate. The post/pre ratio captures proportional growth relative to the pre-ChatGPT baseline. The table ranks descriptive marker contributions to the aggregate effect; it is not used to claim marker-level statistical significance without multiple-comparison correction.

The largest absolute growth is not limited to one rhetorical family. “Potential”, “promising”, “crucial”, and “comprehensive” intensify evaluative or possibility-oriented framing. “Across”, “while”, “particularly”, and “additionally” help link claims and extend argumentative scope. “Demonstrated”, “exhibited”, “insights”, “underscore”, and “underscores” signal contribution and evidential force. “Enhance”, “enhancing”, “enhances”, and “integrating” frame research as improvement-oriented and integrative. “Strategies” and “challenges” create a problem-solution narrative. The pattern, therefore, supports RQ2 because the post-ChatGPT increase is concentrated in how abstracts frame significance, contribution, and coherence, not merely in what biomedical topics they describe.

Figure 7 separates absolute prominence from proportional growth. Some markers, such as “potential”, “across”, and “while”, are frequent in the post-ChatGPT period and contribute substantially to the aggregate increase. Other markers have lower post-ChatGPT frequency but stronger post/pre ratios, suggesting rapid emergence from a smaller baseline. Reading both dimensions together avoids overemphasising already common words while overlooking lower-frequency markers that may be more sensitive indicators of stylistic change.

Figure 7. Relationship between post-ChatGPT marker frequency and post/pre ratio among strict stylistic markers. Labels highlight selected high-growth markers.

The yearly trajectories in Figure 8 show that individual markers do not all follow the same path. Some, such as “potential” and “across”, rise steadily after the boundary. Others, such as “enhance”, “challenges”, and “strategies”, show sharper increases in 2024 and 2025. This supports a diffusion account because the markers move in a broadly convergent post-2022 direction while entering that trajectory at different speeds. The pattern is, therefore, more consistent with gradual stylistic uptake than with a single mechanical vocabulary substitution.

Figure 8. Yearly trajectories for selected high-increase strict stylistic markers. The dashed line marks the ChatGPT boundary.

Table 6 reframes the word-level results as rhetorical functions. The strongest grouped contributions come from value and possibility framing and from scope and connective framing. Evidence and claim-signalling markers also increase, indicating a stronger tendency for abstracts to guide the reader toward significance and contribution. This is where the stylistic nature of the finding becomes most visible. The language that grows after 2022 is not mainly terminology about diseases, interventions, or methods. It is language that shapes how research is positioned, polished, and rhetorically presented.

Table 6. Functional grouping of the fastest-growing strict LLM-associated stylistic markers.

Rhetorical Function	Representative Markers	Combined Absolute Increase	Highest Post/Pre Ratio	Reading of the Pattern
Value and possibility framing	potential, comprehensive, promising, crucial, notably, innovative, notable, pivotal, broader	1499.6	6.38	The growth of evaluative and possibility-oriented language gives abstracts a more opportunity-facing tone.
Scope and connective framing	across, while, particularly, additionally	1311.4	2.53	The increased use of connective markers suggests smoother, more expansive claim-building.
Evidence-and-claim signalling	demonstrated, exhibited, insights, underscores, underscore, demonstrating, emphasizing, explores, exhibits	1045.2	19.35	These markers direct attention to contribution, evidential force, and significance.
Improvement and integration rhetoric	enhance, enhancing, integrating, enhances, incorporating, integrates	676.2	9.42	These terms frame research as constructive, integrative, and capability-building.
Problem-solution orientation	strategies, challenges	570.2	2.29	The growth in these markers positions abstracts around challenges, strategies, and response narratives.

Note. Combined absolute increases are based on selected high-growth strict markers. The grouping is interpretive and describes the dominant rhetorical functions represented by the fastest-growing markers.

The secondary indicators qualify the interpretation of RQ2. Figure 9 shows that lexical diversity increased after 2022, especially in 2025. This result is important because it rules against a crude version of same-ification in which abstracts simply become lexically poorer or semantically interchangeable. The post-ChatGPT period can contain broader vocabulary while also carrying a more concentrated layer of stylistic markers.

Figure 9. Secondary same-ification indicators, including lexical diversity, semantic similarity, and the exploratory composite same-ification index.

Semantic similarity and the composite same-ification index show a less stable pattern. Both appear to peak around 2023 and then decline in later years. This weakens any claim that biomedical abstracts became semantically identical after ChatGPT. The more defensible interpretation is narrower and more researchable: a shared rhetorical texture appears to have intensified across otherwise diverse biomedical abstracts. For this reason, the composite same-ification score is treated as supplementary evidence, while the strict marker trajectory and the marker-control contrast remain the primary empirical basis for answering RQ2.

4.3. RQ3: Robustness Across Venue, Topic, Matching, and SAME-ML Validation

RQ3 asked whether the same-ification signal remains visible across distributional, temporal, topic-controlled, matched-sample, and explainable machine-learning validation analyses. The venue-level analysis is the first robustness check. Figure 10 suggests that post-minus-pre stylistic marker shifts are not evenly distributed across publication venues. This heterogeneity is consistent with the possibility that diffusion is shaped by venue-specific writing norms, disciplinary communities, editorial practices, and article composition.

Figure 10. Exploratory journal-level heterogeneity in post-minus-pre stylistic-marker rates. The results should be interpreted cautiously because journal composition and baseline availability can influence apparent changes.

These venue-level results should be interpreted cautiously. Several journals in the exploratory table have limited or absent pre-ChatGPT observations, which can inflate apparent post-minus-pre differences. The journal analysis, therefore, supports heterogeneity rather than journal attribution. It should not be read as evidence that specific journals adopted LLM-assisted writing more heavily. A stronger extension would estimate journal or publisher random effects with post-period interactions and apply leave-one-journal or leave-one-publisher sensitivity checks. Because those models require a more balanced venue panel than the present analytic sample provides, the current manuscript treats Figure 10 as exploratory evidence for uneven diffusion.

The SAME-ML experiment extends the corpus-level evidence by testing whether the post-ChatGPT period can be distinguished using document-level stylistic features. Year-level aggregate measures were excluded so that classification relied exclusively on features measured at the individual-document level. The models were not intended to infer AI authorship for individual abstracts. Instead, they tested whether the post-2022 period contains a recurring stylistic signal that can be recovered using different machine-learning procedures.

The style-only experiment provided the first validation signal for RQ3. As shown in Figure 11 and Table 7, all three models achieved moderate above-chance discrimination on the held-out data. The sROC AUC was 0.724 for the LASSO logistic regression, 0.712 for random forest, and 0.723 for XGBoost, while the PR AUC ranged from 0.567 to 0.580. At the prespecified classification threshold of 0.50, the accuracy ranged from 0.758 to 0.762 and specificity from 0.928 to 0.944, whereas sensitivity remained comparatively low, ranging from 0.312 to 0.337. Thus, the models ranked post-ChatGPT abstracts better than chance but missed many positive-class observations at the default threshold.

Figure 11. Held-out ROC curves for the style-only models using document-level stylistic and rhetorical features. All three models demonstrated moderate above-chance discrimination between pre- and post-ChatGPT abstracts. Year-level semantic similarity and the composite same-ification index were excluded from the predictor set.

Table 7. Combined SAME-ML held-out model performance across style-only, topic-masked, and marker-ablation experiments.

Experiment	Model	ROC AUC	PR AUC	Accuracy	Sensitivity	Specificity	F1
E1 Style-only	LASSO logistic regression	0.724	0.580	0.762	0.312	0.944	0.431
E1 Style-only	Random forest	0.712	0.567	0.758	0.337	0.928	0.445
E1 Style-only	XGBoost	0.723	0.580	0.762	0.317	0.942	0.434
E2 Topic-adjusted	Topic-only text model	0.762	0.584	0.756	0.319	0.933	0.430
E2 Topic-adjusted	Style + topic model	0.799	0.670	0.788	0.408	0.941	0.525
E2 Topic-adjusted	Topic-residualised style model	0.744	0.599	0.767	0.336	0.941	0.454
E4 Marker ablation	Full text	0.812	0.704	0.798	0.400	0.959	0.533
E4 Marker ablation	Style markers removed	0.803	0.681	0.792	0.380	0.958	0.513
E4 Marker ablation	Style markers only	0.703	0.563	0.755	0.296	0.941	0.411
E4 Marker ablation	Control vocabulary only	0.625	0.395	0.712	0.091	0.963	0.154

Note. ROC AUC and PR AUC are ranking-based metrics. Accuracy, sensitivity, specificity, and F1 depend on the decision threshold and should be interpreted cautiously because the post-ChatGPT class is smaller than the pre-ChatGPT class. Values of 1.000 indicate complete separation in this held-out corpus under the stated feature design; they are not interpreted as external generalisation or as individual AI-authorship detection.

The feature-importance results in Figure 12 indicate that the period-related signal was distributed across several dimensions of abstract style. Overall rhetorical density was the most influential predictor in the XGBoost model, followed by MTLD lexical diversity, stylistic-marker rate, and log-transformed abstract length. Evidence-and-claim framing and improvement-and-integration language also contributed to prediction. The ranking, therefore, suggests that the model learned a combination of rhetorical organisation, lexical diversity, marker density, and document length rather than relying on a single feature or a year-level temporal proxy.

Figure 12. XGBoost gain-based importance for the style-only model. Overall rhetorical density contributed the greatest predictive gain, followed by MTLD lexical diversity, stylistic-marker rate, and log-transformed abstract length. Importance represents relative predictive contribution and does not indicate causal influence.

The SHAP analysis in Figure 13 provides a complementary document-level interpretation. Abstract length, overall rhetorical density, lexical diversity, and stylistic-marker rate produced the largest absolute SHAP contributions. The distribution of contributions across multiple features is consistent with a multidimensional stylistic shift. SHAP values indicate how particular feature values influenced the fitted model’s predictions; they do not establish that these features caused the post-2022 change.

Figure 13. SHAP summary plot for the style-only XGBoost model. Features are ordered by mean absolute SHAP value. Positive SHAP values move predictions toward the post-ChatGPT class, whereas negative values move predictions toward the pre-ChatGPT class. Colour represents the relative feature value within the held-out sample.

The topic-masked experiment addresses an important concern: whether the post-2022 increase was caused by a change in biomedical topics rather than by a change in style. Figure 14 shows that the increase in stylistic-marker intensity was not confined to a single biomedical topic. Mean marker rates were higher in the post-ChatGPT period across all ten text-derived topic groups. For example, the mean rate increased from 46.1 to 70.7 in the expression/cell topic, from 54.6 to 76.1 in the compounds/molecular/human topic, from 62.7 to 78.9 in the systematic-review/meta-analysis topic, and from 44.7 to 58.2 in the artery/aortic/endovascular topic. Although the magnitude of the increase varied across topics, its broad direction supports the interpretation that the observed stylistic shift extends across multiple areas of biomedical research.

Figure 14. Topic-masked stylistic intensity across ten topic clusters. Each cell reports the mean operational stylistic-marker rate and sample size for pre-ChatGPT and post-ChatGPT abstracts within the topic cluster.

Figure 15 compares the topic-only, combined style-and-topic, and topic-residualised models. The combined style-and-topic model achieved the strongest held-out performance, with an ROC AUC of 0.799 and a PR AUC of 0.670. The topic-only text model achieved an ROC AUC of 0.762 and a PR AUC of 0.584, indicating that subject composition itself contained some period-related information. After the stylistic variables were residualised for topic and document-level covariates, the resulting model retained moderate discrimination, with an ROC AUC of 0.744 and a PR AUC of 0.599. Thus, topic contributed to classification, but it did not fully account for the stylistic signal.

Figure 15. Topic-masked validation of the SAME-ML signal. The combined style-and-topic model achieved the strongest performance. The topic-only and topic-residualised models both retained moderate above-chance discrimination, indicating that topic composition explains part, but not all, of the observed period-related signal.

The matched comparison provides a more conservative robustness check by comparing post-ChatGPT abstracts with more similar pre-ChatGPT abstracts. Figure 16 presents the covariate balance before and after matching. Before matching, the largest imbalance occurred in the MTLD lexical diversity, followed by log-transformed word count, whereas the control-vocabulary rate showed relatively little imbalance. Matching substantially reduced the absolute standardised mean differences for all three displayed continuous covariates, bringing each below the conventional 0.10 balance threshold. Matching was also constrained by topic group and incorporated journal grouping in the propensity-score specification. The matched analysis should, nevertheless, be interpreted as a design-based robustness check rather than as a causal estimate of ChatGPT exposure.

Figure 16. Absolute standardised mean differences before and after propensity-score matching. Matching substantially improved balance in MTLD lexical diversity and log-transformed word count while retaining the already small imbalance in control-vocabulary rate. All displayed post-matching differences were below 0.10.

The matched comparison produced a rightward shift in the post-ChatGPT distribution of stylistic marker density (Figure 17), indicating that the period difference remained visible among abstracts with comparable measured characteristics. Figure 18 shows that the largest matched differences occurred in overall rhetorical density and stylistic-marker rate. Evidence-and-claim framing also showed a substantial positive difference, while value-and-possibility language, improvement-and-integration language, scope-and-connective language, and problem-and-solution framing showed smaller positive effects. In contrast, the matched difference in the marker-control ratio was close to zero. The results, therefore, indicate broader increases in rhetorical and marker density rather than a large change in the relative marker-to-control-vocabulary ratio.

Figure 17. Distribution of stylistic marker density in the matched pre- and post-ChatGPT samples. The post-period distribution is shifted toward higher marker density after balancing the measured matching variables.

Figure 18. Matched post-minus-pre stylistic differences. Positive values indicate higher values in matched post-ChatGPT abstracts.

In the marker-ablation experiment, the full-text model achieved an ROC AUC of 0.812 and a PR AUC of 0.704 (Figure 19). Removing the prespecified stylistic-marker vocabulary produced only a modest decline to an ROC AUC of 0.803 and a PR AUC of 0.681. A model restricted to the stylistic markers achieved an ROC AUC of 0.703 and a PR AUC of 0.563, whereas the control-vocabulary-only model performed more weaker, with an ROC AUC of 0.625 and a PR AUC of 0.395. The small ROC AUC reduction of 0.009 after marker removal indicates that the prespecified marker vocabulary captures part of the period-related signal but that substantial information also remains in the wider text.

Figure 19. Marker-ablation performance across four text conditions. Removal of the prespecified stylistic markers produced a small reduction in performance, indicating that the broader textual signal is not limited to the marker dictionary.

The feature rankings in Figure 20 help identify the vocabulary associated with each ablation condition. In the full-text model, terms such as “findings”, “enhance”, “across”, “insights”, “potential”, and “comprehensive” ranked among the most influential features. After removal of the prespecified markers, terms including “findings”, “key”, “remains”, “including”, “offering”, and “purpose” continued to contribute to classification. The marker-only model emphasised terms such as “potential”, “significant”, “across”, “enhance”, “strategies”, and “comprehensive”, whereas the control-vocabulary model relied more heavily on conventional biomedical terms such as “results”, “models”, “patients”, “study”, “clinical”, and “outcomes”. These rankings indicate predictive contribution rather than the direction of association with either period.

Figure 20. Top predictive terms by XGBoost gain under each marker-ablation condition. The comparison separates general textual-period effects from stylistic-marker effects.

Figure 21 summarises the four main validation results using their experiment-specific effect measures. The style-only model achieved a held-out ROC AUC of 0.723, while the topic-residualised model achieved an ROC AUC of 0.744. In the matched analysis, the standardised post-minus-pre difference in the stylistic-marker rate was 0.533. Removing the prespecified stylistic-marker vocabulary reduced the full-text ROC AUC by only 0.010. Taken together, these results provide converging evidence of a moderate and broadly distributed post-ChatGPT stylistic shift: the signal is learnable from document-level style, remains visible after topic adjustment and matching, and is not explained exclusively by the prespecified marker vocabulary.

Figure 21. Summary of the principal SAME-ML validation results. The panels report experiment-specific metrics: style-only held-out ROC AUC, topic-residualised held-out ROC AUC, the matched standardised difference in marker rate, and the change in ROC AUC after marker removal. Because the panels use different metrics, their numerical magnitudes should not be interpreted as directly comparable effect sizes.

4.4. Synthesis of Findings

RQ1 was answered affirmatively: strict LLM-associated stylistic markers increased after the ChatGPT boundary, with the clearest growth occurring in 2024 and 2025 rather than as a one-year discontinuity in 2023. The mean marker rate rose from 4.995 per 1000 words in the pre-ChatGPT period to 11.658 in the post-ChatGPT period, a 133.4% increase. Distributional and time-series evidence show that the change is visible beyond the mean and that the post-boundary trajectory is steeper than the pre-boundary baseline.

RQ2 was also answered affirmatively, with qualification. The control vocabulary did not show the same sustained increase, and the fastest-growing words were concentrated in rhetorical functions rather than biomedical content. The shift is, therefore, best interpreted as stylistic specificity: Post-ChatGPT abstracts more frequently use language that frames value, connects claims, signals evidence, and presents research as integrative or consequential. The concurrent increase in MTLD shows that this is not lexical impoverishment. It is a narrower form of rhetorical convergence within otherwise diverse biomedical writing.

RQ3 received a cautiously affirmative answer. The signal remains visible across several validation layers, as follows: distributional comparison, interrupted time-series modelling, topic-masked analysis, matched-sample comparison, marker-ablation, and interpretable SAME-ML. The strongest claim is not that the models can detect AI-authored abstracts. The stronger and more defensible claim is that the post-ChatGPT period contains a learnable and interpretable rhetorical signal that survives several controls for topic, length, control vocabulary, and sample imbalance.

Based on our findings, the three RQs point to a specific form of same-ification. The corpus does not become semantically identical, and the evidence does not support individual authorship attribution. Instead, biomedical abstracts appear to be accumulating a shared layer of polished, contribution-signalling language. Same-ification in this study, therefore, means convergence in rhetorical presentation: diverse studies increasingly rely on similar linguistic moves to package novelty, relevance, and significance.

The evidence also defines the boundary of the claim. The 2023 sample is smaller than the surrounding years, the journal-level analysis is exploratory, marker-level rankings are descriptive unless corrected for multiple comparisons, and SAME-ML requires external replication before it can be read as generalisable classification. These limits do not remove the observed post-2022 signal, but they narrow the interpretation to corpus-level stylistic diffusion rather than causal attribution to ChatGPT or definitive detection of LLM use.

5. Discussion

This study frames post-ChatGPT change in biomedical abstracts as corpus-level rhetorical diffusion. In relation to RQ1, strict LLM-associated stylistic markers increased sharply after the ChatGPT boundary, while the pre-2022 baseline remained comparatively stable. Figure 3 and Figure 4 and Table 1 and Table 2 show that the increase became most visible in 2024 and 2025, supporting a diffusion interpretation rather than a one-time reaction to the release of ChatGPT. This matters because diffusion implies normalisation through writing, editing, reviewing, and institutional authoring practices.

RQ2 asked whether the shift was stylistic rather than merely a general change in biomedical vocabulary. The answer is cautiously affirmative. The control vocabulary did not show the same sustained post-2022 growth, and the segmented regression in Table 4 showed both a significant level shift and a stronger post-ChatGPT slope change. Figure 6, Figure 7 and Figure 8 and Table 5 and Table 6 show that the strongest markers operate as value framing, connective framing, evidence signalling, improvement rhetoric, and problem-solution language. This extends excess-vocabulary work by moving from prevalence estimation toward the rhetorical functions through which LLM-associated style becomes visible in published abstracts.

The findings also complicate a simple meaning of same-ification. Figure 9 shows that lexical diversity increased after 2022, while semantic similarity and the composite same-ification index were less stable. This means that same-ification should not be understood as semantic sameness or lexical impoverishment. A more precise interpretation is rhetorical convergence: abstracts may continue to cover diverse biomedical topics and use varied vocabulary while increasingly drawing on a shared layer of polished, contribution-signalling expressions. This distinction is important because it avoids the exaggerated claim that science is becoming textually identical. The stronger claim is that academic style may be becoming more recognisably LLM-like in how it packages importance and contribution.

RQ3 concerned whether the post-ChatGPT shift was sufficiently systematic to be recovered through machine-learning validation. The SAME-ML experiments provide converging, but appropriately moderate, evidence of period-related stylistic separability. Style-only models achieved above-chance held-out discrimination, and performance improved when topic information was combined with stylistic features. Importantly, the topic-residualised model remained predictive, indicating that topic composition did not fully explain the stylistic signal. The matched analysis also identified positive differences in rhetorical density and stylistic-marker rate after balancing the measured covariates. Finally, marker ablation produced only a small reduction in full-text performance, showing that the shift extends beyond the prespecified marker vocabulary. These findings support a corpus-level change in biomedical abstract style, but they do not establish individual AI authorship or external diagnostic validity.

This study contributes to the literature by shifting the focus from AI-authorship detection to language change. Detection studies remain valuable, but they are vulnerable to fairness, robustness, and interpretive concerns, particularly for multilingual writers [3,15,16]. The present approach does not ask whether an individual author used ChatGPT. It asks whether a public scholarly corpus shows measurable signs of stylistic diffusion. This provides a more responsible and publishable framing for studying GenAI in academic writing, especially in biomedical contexts where authorship, accountability, and trust are closely tied to publication ethics.

The practical implication is that journals and editors may need to look beyond disclosure policies and plagiarism-style detection. If LLM use gradually reshapes the rhetorical norms of abstracts, then editorial attention should also consider the quality, specificity, and evidential modesty of academic prose. A polished abstract is not necessarily a better abstract if rhetorical intensity masks uncertainty, exaggerates contribution, or makes studies sound more similar than they are. Researchers can use the proposed framework to monitor stylistic change over time, compare disciplines, and evaluate whether LLM-assisted writing is improving clarity or producing a narrower style of scholarly expression.

The analysis is based on PubMed abstracts, not full articles, peer reviews, grant applications, or author drafts. The ChatGPT boundary is historically meaningful, but it cannot prove that every post-2022 change was caused by ChatGPT. The 2023 sample was smaller than other years and was interpreted cautiously. The marker dictionary is intentionally strict, but no word is unique to LLM use, and marker-level rankings should not be treated as inferential tests without multiple-comparison correction. Journal-level heterogeneity may reflect editorial pipelines, venue composition, author populations, or publication-type changes rather than LLM uptake alone. The topic controls and SAME-ML analyses reduce, but do not eliminate, these threats to interpretation. Future work should test the framework across larger and more balanced PubMed samples, disciplines, languages, full-text corpora, and author-disclosed LLM-use settings.

6. Conclusions

The findings provide a qualified but clear answer to the title question. Biomedical abstracts are beginning to show a more ChatGPT-like rhetorical profile, not through identical content but through a shared style of value framing, connective flow, evidence signalling, improvement rhetoric, and problem-solution language. LLM-associated same-ification is therefore best understood as rhetorical convergence: the field remains diverse in what it studies but becomes more similar in how it packages significance.