LLM-Linked Chatbot Platforms for Seeded Clinical Trial Randomization Workflows: A Benchmarking Study of Reproducibility, Allocation Integrity, and Operational Traceability

Abstract

Randomization sequence generation is essential in randomized controlled trials, but access to trial-management systems or statistical support may be limited in some settings. This in silico technical benchmarking study evaluated whether four LLM-linked chatbot interfaces can faithfully execute pre-specified deterministic Python code to generate a randomized sequence under fixed-seed conditions. In Experiment 1, four investigators performed 1200 fixed-seed Python runs across two sample-size scenarios (n = 30 and n = 50), benchmarked against seeded Excel/VBA and R Console workflows. In Experiment 2, the same investigators performed 320 NL-only runs without code submission or seed specification. A supplementary permuted block benchmark (n = 60; blocks of six) added 640 runs across both prompting conditions. Fixed-seed code execution achieved 100% exact reproducibility, allocation integrity, format compliance, and operational completion across all platforms. NL-only prompting preserved allocation integrity, format compliance, and operational completion (100%) but yielded 0% exact reproducibility in both simple and permuted block randomization. These findings support only a constrained interpretation: chatbot-mediated reproducibility depends on executable code, fixed-seed specification, preserved documentation, and human verification. These interfaces should not replace dedicated randomization software or validated trial-management systems.

1. Introduction

Randomized clinical trials (RCTs) are the most credible study design for evaluating the effectiveness of health interventions [1,2]. The defining element of the RCT is random allocation, in which participants are assigned to study groups by chance rather than by investigator choice. Randomization serves three principal functions: it mitigates selection bias by preventing predictability of treatment assignments, it promotes baseline similarity between groups with respect to both known and unknown confounders, and it provides a valid statistical basis for evaluating treatment effects [1,3,4], supporting confident attribution of outcome differences to the intervention [5].

Trial credibility is fundamentally compromised when randomization procedures are poorly implemented or inadequately documented, or when they are paired with insufficient allocation concealment [6]. Two distinct but equally essential elements define proper randomization: generating an unpredictable allocation sequence and implementing that sequence while concealing it from those who enroll participants [6]. Without proper allocation concealment, randomization collapses entirely. Empirical investigations have demonstrated that trials with inadequate or unclear concealment yielded treatment effect estimates up to 40% larger than those in adequately concealed trials, providing direct evidence that poor concealment allows selection bias to infiltrate results [6]. Smaller trials, in particular, may show chance baseline imbalances [5], motivating the use of restricted methods such as permuted blocks [5], and regulatory guidance addresses adjustment for prespecified covariates [7,8].

The CONSORT guidelines underscore that randomization procedures must be clearly documented and transparently reported to maintain scientific integrity [9]. The CONSORT 2010 Statement requires authors to report the method used to generate the random allocation sequence, specify the randomization type and any restrictions, and document the mechanism used to implement and conceal the sequence [2]. The recently updated CONSORT 2025 Statement reaffirms these requirements while addressing contemporary methodological challenges, including the mandate that authors document alignment between the randomization procedures reported and those documented in the trial protocol, statistical analysis plan, and trial registry [10,11]. These provisions establish randomization not only as the defining methodological element of RCTs but also as a process that must itself be documented, transparent, and subject to independent verification.

In practice, randomization schedules are generated using a range of readily available tools, including web-based random-number services (e.g., random.org), spreadsheet software (e.g., Microsoft Excel and R Console), dedicated statistical packages, and trial-management systems. The ICH E9 guideline on statistical principles for clinical trials requires that randomization schedules be reproducible when needed and that details facilitating predictability be filed securely rather than disclosed in the protocol [4]. Methodological guidance further emphasizes the importance of selecting a randomization procedure appropriate to the specific trial context, considering factors such as sample size, blinding level, number of study centers, and recruitment period, and of documenting implementation details in the trial protocol [1]. Simple randomization is methodologically valid and easy to implement, but it may produce chance imbalances, particularly in smaller trials or during sequential recruitment. For this reason, restricted randomization methods, especially permuted block designs, with or without stratification, are commonly used in clinical trial practice to preserve allocation balance at predefined intervals while maintaining unpredictability [1,3]. This distinction is relevant when evaluating chatbot-mediated randomization workflows because a tool that performs correctly under simple randomization may not necessarily preserve reproducibility, allocation integrity, or structural constraints under more restricted allocation designs. It should be emphasized that chance baseline imbalance is invoked here only as motivation for why restricted designs such as permuted blocks matter in trial practice; baseline covariate balance was not an evaluated outcome of the present study, which assessed reproducibility, allocation integrity, format compliance, operational completion, and traceability of the generated sequence rather than the statistical balance of any resulting allocation.

The rapid development of large language models (LLMs) has enabled numerous applications across medicine and dentistry, including tools that facilitate patient care, enhance clinical decision-making, and streamline research workflows [12]. LLM-linked chatbot platforms are widely accessible and easy to use on mobile devices and can be guided by natural-language prompts, potentially positioning them as convenient ad hoc tools for generating allocation lists, particularly in resource-limited settings. However, this convenience raises significant methodological concerns.

Under standard inference conditions, LLM outputs are inherently non-deterministic, and reproducibility through fixed random seeds is not uniformly supported across platforms or consumer-facing interfaces, directly undermining the requirement that randomization schedules be reproducible when needed, as specified by ICH E9 [4], a concern now also reflected in FDA draft guidance on AI-enabled medical software, which explicitly recommends evaluating repeatability and reproducibility in AI systems [7,13]. A recently proposed statistical framework formally quantified this stochastic behavior, demonstrating that LLMs generate variable outputs even when the input prompt, model architecture, and generation parameters remain identical and that this variability lacks standardized metrics for characterization [13].

Furthermore, LLM accuracy and consistency vary substantially across models and tasks, with performance that is not always stable across repeated runs [12]. Critically, repeatability and reproducibility have been shown not to correlate with diagnostic accuracy across models, underscoring that a correct output is not equivalent to a consistent one and that accuracy alone is insufficient to characterize LLM reliability [13]. Despite the growing use of generative AI tools in research settings, no study has systematically evaluated whether chatbot-generated randomization outputs meet the basic operational requirements expected of allocation workflows in clinical trials.

The aim of this in silico benchmarking study was therefore to evaluate the execution fidelity of four commercially available LLM-linked chatbot platforms (ChatGPT, Claude, Copilot, and Gemini) when tasked with executing a deterministic, fixed-seed Python script for the randomization process. This study specifically distinguishes code-level determinism from model-level stochasticity, assessing whether public chatbot interfaces introduce execution corruption rather than testing the models’ natural-language generation capabilities. By subjecting each platform to 200 primary benchmark runs across two sample-size scenarios, performed independently by four investigators, the study sought to determine whether chatbot-based randomization workflows can function as operationally stable code-execution environments under the tested conditions. A complementary NL-only benchmark and a supplementary permuted block randomization benchmark were also conducted to determine whether the observed reproducibility pattern depended on the prompting paradigm and whether it extended beyond simple randomization to a restricted allocation design. The evaluated application scenario and the two prompting pathways are summarized in Figure 1. Consistent with this focus, the present work is best characterized as a technical benchmarking and verification study of chatbot-mediated algorithm execution, rather than as a clinical randomization study or as the development of a new randomization algorithm.

2. Materials and Methods

2.1. Study Design and Reporting Framework

This was an in silico benchmarking study designed to evaluate whether large language model (LLM)-linked chatbot platforms can function as code-execution environments for seeded simple-allocation workflows relevant to clinical trial planning. The study focused on three prespecified methodological domains: (1) allocation integrity; (2) seed-based reproducibility; and (3) operational traceability of the generated randomization outputs.

Reporting of the study was guided by a composite framework integrating three complementary reporting guidelines. The GAMER Statement (Reporting guideline for the use of Generative Artificial intelligence tools in MEdical Research) was used as the primary reporting standard for transparent disclosure of generative AI tool use across all study phases, including specification of tool versions, prompting techniques, role of GAI in the study, content verification procedures, and impact on conclusions [14]. The CHART (Chatbot Assessment Reporting Tool)–Collaborative provided supplementary guidance specific to chatbot assessment studies, including standardized reporting of prompt engineering protocols, chat session configurations, query strategies, and performance evaluation methods [15]. The REFINE checklist (Reporting checklist for FoundatIon and large laNguagE models) supplied additional technical reporting items encompassing model specification, stochasticity control parameters, dataset integrity considerations, and implementation-level documentation [16]. A completed GAMER checklist is provided in Supplementary Table S1.

No human participants, biological samples, protected health information, or patient-derived datasets were used. All randomization runs were performed using synthetic sequential identifiers generated specifically for testing purposes. The study, therefore, constituted a bench/in silico methodological evaluation exempt from institutional review board approval.

2.2. Evaluated Platforms and Comparator

2.2.1. LLM-Linked Chatbot Platforms

Four commercially available LLM-linked chatbot platforms were evaluated: ChatGPT (OpenAI, San Francisco, CA, USA), Claude (Anthropic, San Francisco, CA, USA), Copilot (Microsoft Corporation, Redmond, WA, USA), and Gemini (Google DeepMind, London, UK) (Table S2 presents all model versions and identifiers active across platforms during the testing period). All platforms were accessed via their publicly available graphical user interfaces (web-based chat interfaces) during the testing period. No application programming interface (API) access was used.

Because commercial chatbot providers may update underlying models without disclosing persistent public version identifiers or training/knowledge cutoff dates, the following were preserved for each run to support operational input–output traceability and future replication: platform name, provider, exact date of testing, prompt condition used, interaction style (single-turn), session memory policy (new session per run; no persistent memory retained across sessions), and the complete output record. The specific model versions and identifiers active during the testing period are reported in Supplementary Table S2. All prompts were in English.

2.2.2. Non-LLM Comparator

The comparator platform was Microsoft Excel (Microsoft Corporation, Redmond, WA, USA), using a Visual Basic for Applications (VBA)-based, seeded-randomization macro that served as a deterministic, non-LLM control workflow. Additionally, the same seeded Python allocation script was executed in the R Console environment (R version 4.5.3; R Foundation for Statistical Computing, Vienna, Austria) as an independent deterministic verification environment. These non-LLM workflows were selected because fixed-seed execution produces deterministic outputs under identical conditions, thereby providing a reference standard for assessing seed-based reproducibility in the chatbot platforms.

2.3. Investigators and Execution Structure

Four investigators (C.F.M., L.E.J., R.d.S.P., and G.G.A.) participated in the study and executed the benchmark protocol across all six platforms. The investigators were not blinded to platform identity because the study objectives included operational benchmarking, reproducibility testing, and workflow traceability rather than user-independent outcome adjudication. Each investigator was instructed to preserve the exact text of the standardized prompt, seed value, sample-size scenario, and the verbatim saved output for each run.

2.4. Standardized Benchmark Prompt and Prompt Engineering Protocol

A standardized prompt was developed through an iterative prompt engineering process. The development involved the study investigators (domain experts in clinical trial methodology) systematically testing prompt variants, with the objective of producing a clear, unambiguous instruction set that would reliably elicit a complete randomization table. No automated prompt optimization frameworks were used. The final prompt was version-controlled and locked prior to formal benchmarking.

Pilot prompt-development observations. During the pilot prompt-development phase, the investigators also tested natural-language-only randomization requests without explicit Python execution and without fixed-seed specification. These preliminary runs did not yield reproducible outputs across repeated sessions and platforms. For this reason, the formal benchmark was deliberately restricted to an explicit seeded Python-execution workflow, and the final standardized prompt was version-controlled and locked prior to benchmarking.

The standardized prompt requested simple randomization by complete random permutation of identifiers followed by sequential allocation to study groups. In its primary form, the prompt instructed the platform to: (a) execute a Python script performing simple randomization by complete random permutation; (b) allocate identifiers into two groups in a 1:1 ratio; (c) use a fixed random seed (20260307); (d) ensure that each identifier appeared exactly once with no duplication, omission, or addition; (e) return the result as a two-column table (ID and Group); (f) report final counts per group; and (g) automatically generate identifiers 001 to n when identifiers were not manually supplied. The exact wording of the standardized prompt is presented in Figure 2.

Excel/VBA and R Console Comparator Codes

The VBA macro used for the Excel comparator workflow and the R Console setup and Python code executed via the reticulate package are reproduced in Supplementary S1. In brief, the Excel macro automatically generated sequential identifiers (001 to n), initialized a fixed seed (20260307), applied a Fisher–Yates shuffle to create a complete random permutation of identifiers, and then assigned groups sequentially according to the requested 1:1 allocation structure. Output was written directly to the active worksheet with explicit reporting of the seed value and final group counts. For the R Console comparator, Python was executed within the R environment through the reticulate package, which was installed from a CRAN HTTPS mirror located in Pennsylvania, USA (USA PA 1). After configuring the Python bridge via “repl_python()”, the same randomization logic was run using the Python “random” module with the identical fixed seed (20260307), producing the allocation table with group counts and seed confirmation. The reticulate bridge ensures that the Python pseudorandom number generator produces the same sequence as when executed natively, thereby providing an additional non-LLM reference comparator independent of both the chatbot platforms and the Excel/VBA environment.

2.5. Stochasticity Control and Generation Parameters

The chatbot-based prompt was designed to enforce deterministic execution by specifying a fixed random seed (20260307) within the Python code block. Because the prompt was embedded as executable code rather than a natural-language instruction, the expected behavior was deterministic output given faithful code execution. No additional generation parameters (e.g., temperature, top-p, top-k) were accessible or modifiable through the graphical user interfaces of the tested platforms.

For each platform, each investigator executed each prompt in a new, independent chat session (single-turn interaction) to ensure that no prior conversational context influenced the model’s output. Session memory was reset between runs. No persistent memory features were enabled across sessions.

2.6. Primary Benchmark Scenarios

The primary benchmark assessed reproducibility under fixed-seed conditions in two prespecified sample-size scenarios: n = 30 and n = 50. For each platform and each primary scenario, the standardized prompt was executed 100 times, yielding repeated seeded runs under identical conditions. Thus, each platform underwent 200 primary benchmark runs (100 runs for n = 30 and 100 runs for n = 50). Across the four LLM platforms, the Excel/VBA and R Console comparators, this resulted in 1200 primary benchmark runs encompassing 48,000 individual participant allocations (

Table 1. Overview of the benchmarking design used to evaluate allocation integrity, seed-based reproducibility, and operational input–output traceability of chatbot-mediated randomization sequence-generation workflows versus seeded Excel/VBA and R Console control workflows.

Platform	Primary Scenarios	Runs/ Scenario	Primary Runs	Primary Allocations	Exploratory Runs (Text)	Exploratory Runs (Voice)	Total Runs
Experiment 1—Fixed-Seed Python Code Execution
ChatGPT	n = 30	100	200	8000	12	40	252
	n = 50
Claude	n = 30	100	200	8000	12	40	252
	n = 50
Copilot	n = 30	100	200	8000	12	40	252
	n = 50
Gemini	n = 30	100	200	8000	12	40	252
	n = 50
Excel/VBA	n = 30	100	200	8000	12	—	212
	n = 50
R Console	n = 30	100	200	8000	12	—	212
	n = 50
Experiment 1 Subtotal			1200	48,000	72	160	1432
Experiment 2—Natural-Language-Only (NL-Only) Prompting
ChatGPT	n = 30	20	80	2400	—	—	80
Claude	n = 30	20	80	2400	—	—	80
Copilot	n = 30	20	80	2400	—	—	80
Gemini	n = 30	20	80	2400	—	—	80
Experiment 2 Subtotal			320	9600	—	—	320
Supplementary Benchmark—Permuted Block Randomization (n = 60; blocks of 6)
ChatGPT	Python/Seed	20	80	4800	—	—	80
ChatGPT	NL-Only	20	80	4800	—	—	80
Claude	Python/Seed	20	80	4800	—	—	80
Claude	NL-Only	20	80	4800	—	—	80
Copilot	Python/Seed	20	80	4800	—	—	80
Copilot	NL-Only	20	80	4800	—	—	80
Gemini	Python/Seed	20	80	4800	—	—	80
Gemini	NL-Only	20	80	4800	—	—	80
Permuted Block Subtotal			640	38,400	—	—	640
Grand Total			2160	96,000	72	160	2392

Notes. Experiment 1: fixed-seed Python code execution across four LLM-linked chatbot platforms (ChatGPT, Claude, Copilot, Gemini), Excel/VBA, and R Console; two sample-size scenarios (n = 30 and n = 50); 100 runs per scenario per platform. Text-based exploratory perturbation runs: 10-identifier synthetic samples under three modified conditions (group number, allocation structure, and seed); four investigators × six platforms × 3 conditions = 72 runs. Voice-initiated exploratory runs: 10 runs per investigator per LLM platform; 40 runs per platform; 160 runs total (Excel/VBA and R Console excluded). Experiment 2: NL-only prompting without code submission or fixed-seed specification; 20 runs per investigator per platform; 80 runs per platform; 320 runs total (Excel/VBA and R Console excluded); primary allocations based on n = 30 per run. Permuted Block Benchmark: supplementary evaluation using n = 60 participants in blocks of six; two prompting conditions (Python/fixed-seed and NL-only); 20 runs per investigator per platform per condition; 80 runs per platform per condition; 640 total runs (Excel/VBA and R Console excluded). LLM, large language model; VBA, Visual Basic for Applications; NL, natural language.

).

2.7. Exploratory Perturbation Scenarios

In addition to the primary benchmark, exploratory perturbation tests examined whether the workflow remained operational under prompt modifications delivered via text input. These were performed using a 10-identifier synthetic sample and included three modified conditions: (1) modified number of groups (e.g., three or four arms); (2) modified group-allocation structure (2:1 and 3:1 ratios); and (3) modified seed (20260310). Each of the four investigators applied these three conditions across all six platforms, yielding 12 exploratory runs per platform and 72 in total.

2.8. Voice-Initiated Exploratory Runs

To evaluate whether LLM-linked chatbot platforms could also execute randomization workflows through voice-based interaction, an additional exploratory phase was conducted using each platform’s voice input functionality. Because dictating raw Python code is impractical, the standardized code-based prompt was translated into a structured, natural-language spoken command. The baseline dictated instruction was: “Execute, in Python, a randomization with 30 participants. Use a fixed seed of 20260307. Generate IDs from 001 to 030. Randomly shuffle these IDs using the seed. Then, divide the shuffled list into two equal groups: assign the first 15 participants to the “Control” group and the remaining 15 participants to the “Test” group. Present the result in a two-column table with columns “ID” and “Group”, clearly indicating which group each ID belongs to. Also, report the final counts for each group and confirm the seed used.”

Using this baseline instruction as a starting point, each investigator issued spoken commands to modify key parameters, including sample size (n), allocation ratio, seed value, and number of groups, and to execute the modified randomization accordingly. Each of the four investigators performed 10 voice-initiated runs on each of the four LLM-linked chatbot platforms, yielding 40 per platform and 160 in total. The Excel/VBA and R Console comparators were excluded from this phase, as they do not support voice-based interaction.

Because voice interfaces may not automatically display the underlying Python code used for execution, the study protocol required an explicit follow-up text request within the same session, such as “Show me the final Python code you used.” The complete outputs, including the subsequently retrieved code, were preserved as part of the operational traceability record. The overall benchmarking design, encompassing the primary benchmark, text-based perturbation, and voice-initiated exploratory runs, is summarized in Table 1.

2.9. Natural-Language-Only Benchmark

2.9.1. Rationale and Design

A complementary benchmark (Experiment 2) was conducted to evaluate platform behavior under natural-language-only (NL-only) prompting conditions, without submitting a Python code block and without specifying a fixed random seed. This condition represents the prompting strategy most likely to be employed by researchers without programming expertise and directly operationalizes the model-level stochasticity that was intentionally controlled in the primary benchmark (described in Section 2.4, Section 2.5, Section 2.6, Section 2.7 and Section 2.8). The pilot phase that preceded Experiment 1 (Section 2.4) indicated that NL-only prompting did not yield reproducible outputs; Experiment 2 was designed to formalize this observation with the same methodological rigor applied to Experiment 1, thereby providing a direct empirical contrast between the two prompting paradigms and resolving the ceiling effect inherent in the 100% reproducibility result of Experiment 1.

2.9.2. Standardized NL-Only Prompt

The following standardized prompt was used verbatim across all Experiment 2 runs: “Generate a randomization list for 30 participants. Assign each participant a sequential ID from 001 to 030. Randomly shuffle these IDs. Then, divide the shuffled list into two equal groups: assign the first 15 participants to the ‘Control’ group and the remaining 15 participants to the ‘Test’ group. Present the result in a two-column table with columns ‘ID’ and ‘Group’, clearly indicating which group each ID belongs to. Also, report the final counts for each group.” Critically, no Python code block was submitted, no random seed was specified or implied, and no instruction to use a deterministic algorithm was included. These three absences are the defining features of the NL-only condition and distinguish Experiment 2 from Experiment 1. The prompt was version-controlled and locked prior to data collection.

2.9.3. Run Structure and Investigators

All investigators (C.F.M., L.E.J., R.d.S.P., and G.G.A.) independently executed the NL-only benchmark. Each investigator performed 20 independent runs per platform across all four LLM-linked chatbot platforms (ChatGPT, Claude, Copilot, and Gemini), yielding 80 runs per platform and 320 runs total across Experiment 2. Each run was conducted in a new, independent chat session with no persistent memory and no carry-over context from previous runs, consistent with the session policy applied in Experiment 1. All outputs were saved verbatim. The Excel/VBA and R Console comparators were excluded from Experiment 2, as this condition tests model-level stochasticity rather than code-execution fidelity.

2.9.4. Outcomes and Duplicate Sequence Detection

The same four prespecified outcomes used in Experiment 1 were assessed in Experiment 2: (1) exact reproducibility, defined as whether the full allocation sequence, including complete ID order and group assignment, was identical across repeated independent runs on the same platform, with the first run performed by C.F.M. on each platform serving as the platform-specific reference sequence; (2) allocation integrity, defined as the presence of all 30 IDs exactly once, with no duplicated, missing, or extraneous identifiers and a correct 15:15 group split; (3) format compliance, defined as output conforming to the requested two-column table structure with explicit reporting of final group counts; and (4) operational completion, defined as successful generation of a usable randomization list without refusal, truncation, or logic breakdown. All outputs were saved verbatim and independently verified by two investigators. In cases of discrepancy, manual investigator verification served as the definitive source.

2.10. Permuted Block Randomization Benchmark

A supplementary benchmark was conducted to evaluate whether chatbot platforms could implement permuted block randomization under the same two prompting conditions evaluated in Section 2.4, Section 2.5, Section 2.6, Section 2.7, Section 2.8 and Section 2.9: (1) explicit Python code execution with a fixed seed, and (2) natural-language-only (NL-only) prompting without code submission or seed specification. This evaluation was designed to extend the code-execution versus NL-only contrast to a more complex allocation algorithm that is more structured than simple randomization, while still representing a simplified fixed-block, two-arm, equal-allocation scenario. This benchmark was therefore a restricted technical validation scenario, one step more structured than simple randomization, and was not intended to represent the full range of randomization procedures used in contemporary trials; variable block sizes, stratification, multicenter randomization, unequal allocation ratios, and minimization were not evaluated.

The permuted block prompt requested allocation of 60 sequentially numbered participants (1 to 60) into consecutive blocks of six. For each block, the allocation vector contained three assignments to Group A and three assignments to Group B, and the order of assignments within each block was randomly permuted. For the Python/fixed-seed condition, the standardized prompt instructed the platform to execute a Python script using the fixed seed 20260307 (the same seed used in the primary benchmark): participants numbered 1 to 60 were organized into ten consecutive blocks of six; within each block, a balanced allocation vector containing three Group A and three Group B assignments was randomly permuted using the fixed seed; and the permuted allocation vector was then applied to the participant identifiers in that block. This ensured that each block contributed exactly three participants to each group and that the final 30:30 allocation ratio was maintained. For the NL-only condition, an identical allocation task was specified in natural language without any Python code block and without seed specification. The exact code for the Python/fixed-seed condition is presented in Figure 3; the verbatim text of both prompts is reproduced below.

2.10.1. Python/Fixed-Seed Prompt

“Execute the following code in Python. I have 60 participants numbered from 1 to 60. Use the fixed seed 20260307. Divide the participants into consecutive blocks of 6. For each block, create a balanced allocation vector with three assignments to Group A and three assignments to Group B. Randomly permute the order of assignments within each block using the fixed seed. Apply the permuted allocation vector to the participant IDs in that block. At the end, show the final list of each participant and their assigned group.”

2.10.2. NL-Only Prompt

“I have 60 participants numbered from 1 to 60. Please generate a permuted block randomization list using blocks of 6. In each block, assign exactly three participants to Group A and three participants to Group B, with the order of assignments randomly permuted within the block. At the end, show the final list of each participant and their assigned group.”

Each of the four investigators independently executed both prompting conditions across all four LLM-linked chatbot platforms (ChatGPT, Claude, Copilot, and Gemini), performing 20 runs per investigator per platform per condition: 20 Python/fixed-seed runs and 20 NL-only runs. This yielded 80 runs per platform per condition and 320 total runs per condition, for 640 runs in this supplementary benchmark. The Excel/VBA and R Console comparators were not evaluated in this phase. The same four prespecified outcomes from Experiments 1 and 2 were assessed: exact reproducibility, allocation integrity (60 unique identifiers, 30:30 final split), format compliance, and operational completion.

2.11. Run Execution, Output Handling, and Traceability Record

For each run, the following were recorded: (a) platform used; (b) investigator identity; (c) sample-size scenario; (d) seed value; (e) prompt condition (verbatim); (f) the complete, unedited output as returned by the chatbot or generated by the Excel/VBA or R Console workflow; and (g) any deviation from the requested format or allocation logic.

For chatbot platforms, outputs were preserved exactly as returned by the interface (i.e., no post-generation editing or filtering was applied). For voice-initiated runs, both the voice-generated output and the subsequently requested Python code were saved. For the Excel/VBA and R Console comparators, the generated allocation sheets were saved immediately after macro execution. Operational input–output traceability was defined as the ability to preserve and later inspect the input condition, including prompt/code and seed, together with the resulting allocation output.

2.12. Outcomes

2.12.1. Primary Outcome

The primary outcome was exact seed-based reproducibility, defined as full identity of the final allocation table across repeated runs performed on the same platform, with the same sample size, the same prompt logic, and the same fixed seed. Exact reproducibility required complete concordance of: (a) identifier order after permutation; (b) group assignment for each identifier; (c) final group counts; and (d) reported seed value (when displayed).

2.12.2. Secondary Outcomes

Secondary outcomes included: (1) allocation integrity, defined as the presence of each identifier exactly once with no duplicated, missing, or unexpected identifiers; (2) format compliance, defined as output conforming to the requested tabular structure with ID and Group columns, correct group labels, and explicit final counts; (3) operational completion, defined as successful generation of a usable randomization table without logic breakdown or refusal; and (4) operational input–output traceability, defined as the extent to which each run could be fully traced through preserved prompt/code conditions, seed value, and saved output artifact. This narrow construct denotes only the ability to preserve and later inspect the input condition together with its resulting output; it is explicitly distinct from random sequence generation versus allocation concealment, and it does not encompass concealment of the sequence from enrolling personnel, secure storage, role-based access control, investigator masking, validated electronic records, or regulatory-grade audit trails (e.g., 21 CFR Part 11), none of which were evaluated. For the permuted block benchmark, block-structure conformity was additionally assessed, defined as the presence of exactly six participants per block and exactly three allocations to each group within every block.

2.13. Content Verification and Human Oversight

All outputs generated by the LLM-linked chatbot platforms were independently verified by two investigators for each run. Verification encompassed: (a) completeness of the identifier list (no duplications, omissions, or additions); (b) correctness of group assignments relative to the expected allocation ratio; (c) concordance of the reported seed value with the input seed; and (d) structural conformity of the output table. Discrepancies were flagged and classified. The comparator outputs for Excel/VBA and R Console were verified using an identical checklist. No GAI tool was used to verify the outputs.

2.14. Statistical Analysis

The unit of analysis was the individual run. For each platform and scenario, primary and secondary outcomes were summarized as counts and percentages. Exact seed-based reproducibility was reported as the proportion of repeated runs yielding a fully identical output under identical conditions.

Because the main benchmark question concerned deterministic reproducibility rather than stochastic variation across newly generated randomizations, repeated seeded runs were expected to yield identical outputs if the workflow functioned deterministically. Accordingly, descriptive reporting was prioritized. Where variability in outcome proportions existed between platforms or between the LLM-based workflows and the Excel/VBA and R Console comparator, categorical comparisons were assessed using Fisher’s exact test or the chi-square test, as appropriate. A two-sided p < 0.05 was considered statistically significant. If all platforms demonstrated complete reproducibility (zero failures), inferential comparison would not be informative, and results would be presented descriptively only.

Because every fixed-seed condition yielded zero observed failures, the resulting proportions should be interpreted descriptively and with appropriate statistical caution. A 100% observed success rate does not imply that the true failure probability is zero. To quantify this uncertainty, we calculated exact binomial confidence intervals and applied the rule-of-three approximation for zero observed events. For the primary fixed-seed benchmark, the approximate upper 95% confidence bound for the true failure probability was 3/1200, or 0.25%. At the platform level, with 200 fixed-seed runs per platform, the approximate upper 95% bound was 3/200, or 1.5%. For the fixed-seed permuted block benchmark, the corresponding upper bound was 3/320, or 0.94%. When all fixed-seed runs were pooled (n = 1520), the approximate upper 95% bound was 3/1520, or 0.20%. These estimates indicate that the absence of observed failures should be interpreted as evidence of low observed failure under the tested conditions, rather than as proof that execution failures cannot occur.

For the natural-language-only benchmark, exact reproducibility was observed in 0/320 runs. This result should be interpreted differently from the fixed-seed condition: here, the event of interest is exact sequence reproduction, not execution failure. Using the rule-of-three approximation, the upper 95% confidence bound for the probability of exact sequence reproduction under the tested no-code, no-seed condition was approximately 3/320, or 0.94%. Therefore, the NL-only findings support the conclusion that exact reproducibility was not observed under the evaluated conditions, but they do not prove that exact reproduction is impossible in all natural-language-only workflows.

Although a single successful run under fixed-seed conditions suffices to demonstrate deterministic reproducibility, the repeated-run design was adopted to assess operational resilience across multiple investigators, sessions, and time points, thereby monitoring for potential platform-level instabilities such as silent model updates, server-side variability, or session-handling inconsistencies that could compromise execution fidelity.

2.15. Data Privacy and Confidentiality

No patient data, protected health information, or identifiable data of any kind were entered into any chatbot platform during this study. All identifier lists were synthetic sequential numbers generated solely for benchmarking purposes. Prompts contained no sensitive or confidential information. Accordingly, no data privacy safeguards beyond standard institutional information security practices were required.

3. Results

3.1. Overall Execution and Completeness

All four investigators completed the full benchmark protocol without deviation. In the primary benchmark (Experiment 1), a total of 1200 runs were executed across the four LLM-linked chatbot platforms (ChatGPT, Claude, Copilot, and Gemini) and the two non-LLM comparators (Excel/VBA and R Console), encompassing 48,000 individual participant allocations under the two prespecified fixed-seed scenarios (n = 30 and n = 50). An additional 72 text-based exploratory perturbation runs and 160 voice-initiated exploratory runs were performed, yielding 1432 Experiment 1 runs in total. In Experiment 2, a total of 320 independent NL-only runs were executed across the four LLM-linked chatbot platforms (80 runs per platform; 20 per investigator per platform). In the supplementary permuted block benchmark, an additional 640 runs were executed across the same four platforms under two prompting conditions (320 Python/fixed-seed runs and 320 NL-only runs; 80 per platform per condition). Across all three benchmarks, the cumulative run total was 2392, encompassing 96,000 individual participant allocations (Table 1). No run was excluded from analysis across any benchmark or condition.

During the pilot phase preceding formal benchmarking, a series of natural-language-only randomization requests (approximately 20 runs per platform; 80 runs total) without explicit Python execution or fixed-seed specification did not produce reproducible outputs across repeated sessions or platforms. This pilot observation directly informed the design of Experiment 2, which formally benchmarks NL-only prompting under the same outcome criteria applied in Experiment 1; results are reported in Section 3.7. As a reference, the allocation list generated by the seeded Python code (seed = 20260307) for 30 participants assigned to two groups (Control vs. Test) at a 1:1 ratio is shown in Supplementary Figure S1.

3.2. Exact Seed-Based Reproducibility

Exact seed-based reproducibility was 100% across all six evaluated platforms in the primary benchmark (

Table 2. Primary benchmark results across evaluated platforms.

Platform	Primary Runs	Exact Reproducibility n (%)	Allocation Integrity n (%)	Format Compliance n (%)	Operational Completion n (%)
ChatGPT	200	200 (100.0)	200 (100.0)	200 (100.0)	200 (100.0)
Claude	200	200 (100.0)	200 (100.0)	200 (100.0)	200 (100.0)
Copilot	200	200 (100.0)	200 (100.0)	200 (100.0)	200 (100.0)
Gemini	200	200 (100.0)	200 (100.0)	200 (100.0)	200 (100.0)
Excel/VBA	200	200 (100.0)	200 (100.0)	200 (100.0)	200 (100.0)
R Console	200	200 (100.0)	200 (100.0)	200 (100.0)	200 (100.0)
Total	1200	1200 (100.0)	1200 (100.0)	1200 (100.0)	1200 (100.0)

Notes. Primary benchmark results across the four LLM-linked chatbot platforms and the Excel/VBA and R Console comparator workflows. Each platform was tested under two sample-size scenarios (n = 30 and n = 50), with 100 repeated runs per scenario. Under identical code, identical sample-size scenario, and fixed-seed conditions, all platforms yielded exact output concordance across repeated runs, with no failures in allocation integrity, format compliance, or operational completion. LLM, large language model; VBA, Visual Basic for Applications.

). For ChatGPT, Claude, Copilot, Gemini, the Excel/VBA comparator, and the R Console comparator, repeated execution under identical conditions yielded fully identical allocation outputs in every run, with complete concordance in identifier order after permutation, group assignment for each identifier, final group counts, and displayed seed value. No inter-investigator discrepancies were observed across any platform or scenario. Reproducibility was therefore 100/100 for each scenario within each platform, 200/200 for each platform overall, and 1200/1200 across the primary benchmark dataset. Because no failures were observed, these proportions are bounded above rather than exactly zero in failure probability: applying the rule of three to the zero observed failures yields an approximate upper 95% confidence limit on the per-run failure probability of 0.25% across the full primary dataset (n = 1200) and 1.5% per platform (n = 200), consistent with the exact binomial limits reported in Section 2.14.

3.3. Allocation Integrity, Format Compliance, and Operational Completion

Allocation integrity was preserved in 100% of primary benchmark runs (1200/1200). No duplicated, missing, or extraneous identifiers were observed in any output. Group allocation counts matched the prespecified 1:1 allocation structure in all runs. All returned tables conformed to the requested two-column output format (ID and Group) with explicit reporting of final group counts. Operational completion was likewise 100% (1200/1200): no platform refusals, truncated outputs, logic breakdowns, or code execution failures occurred across any tested platform or investigator.

3.4. Operational Traceability

Operational traceability was maintained across all 2392 runs, including the primary benchmark, perturbation scenarios, voice-initiated exploratory runs, NL-only benchmark, and supplementary permuted block benchmark. For each execution, the following were preserved as part of a structured traceability record: platform identity, investigator identity, sample-size scenario, seed value, verbatim prompt condition, and the complete unedited output. This documentation enabled direct, post hoc verification of the correspondence between the input condition and the resulting allocation table for every chatbot, Excel/VBA, and R Console run.

3.5. Text-Based Exploratory Perturbation Scenarios

In the text-based exploratory perturbation phase (72 runs total; 12 per platform), all platforms remained fully operational under three modified conditions applied to a 10-identifier synthetic sample: (1) modified number of groups (three- and four-arm allocations); (2) modified allocation structure (2:1 and 3:1 ratios); and (3) a modified seed value (20260310). No deviations from the requested grouping logic or altered seed specification were identified. Outputs remained internally consistent with the modified input conditions across all investigators and platforms, with correct group counts, no identifier duplication or omission, and appropriate formatting maintained throughout.

3.6. Results of the Voice-Initiated Exploratory Runs

In the voice-initiated exploratory phase (160 runs total; 40 per chatbot platform; 10 per investigator per platform), all four LLM-linked platforms successfully interpreted spoken commands (detailed in Section 2.8) and the subsequent verbal instructions to modify randomization parameters, including changes to sample size (n), allocation ratio, seed value, and number of groups. Each platform correctly applied the requested modifications and generated complete, internally consistent randomization outputs across all investigators. Allocation integrity and format compliance were preserved across all voice-initiated runs (160/160).

Notably, none of the four platforms spontaneously displayed the Python code used to execute the randomization when the prompt was delivered by voice. In all 160 runs, the underlying code was retrievable only after the investigator submitted an explicit follow-up text request within the same chat session. Once requested, all platforms provided the complete Python code, which could then be verified against the output for operational input–output traceability. This additional step represents a workflow-specific limitation of voice-based interaction, as the code execution pathway is less transparent than in text-based prompting, where the code is submitted directly by the user.

3.7. Natural-Language-Only Benchmark Results

3.7.1. Overall Execution and Completeness in NL-Only Runs

The investigators completed the full Experiment 2 protocol without deviation. A total of 320 independent NL-only runs were executed across the four LLM-linked chatbot platforms (80 runs per platform; 20 per investigator per platform). No run was excluded from analysis.

3.7.2. Allocation Integrity, Format Compliance, and Operational Completion in NL-Only Runs

Allocation integrity was 100% across all 320 Experiment 2 runs (320/320). Every output contained all 30 sequential identifiers (001–030) exactly once, with no duplicates, missing, or extraneous identifiers, and exactly 15 participants assigned to each group in every run across all platforms and all investigators. Format compliance was 100% (320/320): all outputs conformed to the requested two-column table structure with explicit group counts. Operational completion was 100% (320/320): no platform refusals, truncated outputs, or logic breakdowns occurred under any condition.

3.7.3. Exact Reproducibility

Exact reproducibility under NL-only prompting was 0% across all four platforms (0/320 runs). No independent session on any platform reproduced the prespecified reference allocation sequence. This pattern was consistent among all four investigators and across all 80 runs conducted on each platform. This result should be read as a direct warning: because no fixed seed or executable code was provided, the platforms could not regenerate an identical sequence, so natural-language-only prompting did not produce reproducible or operationally traceable randomization lists and is unsuitable whenever exact reproducibility, independent verification, or trial documentation is required. Reproducibility was assessed only as exact, sequence-level identity against the reference run; finer-grained characterization of the non-reproducible NL-only outputs (for example, cross-platform sequence similarity, duplicate-pattern frequency, allocation entropy, or run-to-run edit distance) was not computed in the present study and is identified as a direction for future work in Section 4.7.

3.8. Results of the Permuted Block Randomization Benchmark

A total of 640 independent runs were executed across the permuted block supplementary benchmark (320 Python/fixed-seed runs and 320 NL-only runs; 80 per platform per condition; 20 per investigator per platform per condition). No run was excluded from analysis.

3.8.1. Python/Fixed-Seed Condition

Under the Python/fixed-seed prompting condition, exact reproducibility was 100% across all four platforms (320/320 runs). Every platform produced an identical permuted block allocation table in every run: the shuffled identifier order within each block, group assignments, and final counts (30 participants per group) were fully concordant across all runs, investigators, and platforms. Allocation integrity was 100% (320/320): all 60 identifiers appeared exactly once, with no duplications, omissions, or extraneous entries, and each block yielded exactly three assignments to Group A and three to Group B. Format compliance and operational completion were likewise 100% (320/320). These findings directly replicate the pattern observed in Experiment 1 (Section 3.2), confirming that deterministic seeded code execution preserves exact reproducibility under a permuted block algorithm at the same level observed for simple randomization.

3.8.2. NL-Only Condition

Under the NL-only prompting condition, exact reproducibility was 0% across all four platforms (0/320 runs). No independent session on any platform reproduced the reference permuted block allocation sequence. This pattern was consistent across all four investigators and all 80 runs per platform. Allocation integrity remained 100% (320/320): every output contained all 60 identifiers exactly once with a correct 30:30 final split. Format compliance and operational completion were likewise 100% (320/320). These results directly replicate the pattern observed in Experiment 2 (Section 3.7), confirming that structural correctness is preserved under NL-only prompting while reproducibility is entirely absent, irrespective of the complexity of the underlying allocation algorithm.

4. Discussion

4.1. Principal Findings

The present study evaluated the execution fidelity of four commercially available LLM-linked chatbot platforms (ChatGPT, Claude, Copilot, and Gemini) when used as code-execution environments for seeded simple-randomization workflows, using a seeded Excel/VBA macro and R Console as deterministic, non-LLM control comparators. Across 1200 primary benchmark runs encompassing 48,000 individual participant allocations, all six platforms demonstrated perfect performance on every prespecified primary and secondary outcome. Exact seed-based reproducibility, allocation integrity, format compliance, and operational completion were each 100% in both sample-size scenarios (n = 30 and n = 50) and across all four investigators. These findings held across text-based perturbation scenarios involving modified group structures, 2:1 and 3:1 allocation ratios, and altered seed values and were replicated in voice-initiated exploratory runs. No inter-platform or inter-scenario variability was observed, formal comparative statistical analysis was not informative, and results were reported descriptively as prespecified.

The dual-experiment design of this study produces a knowledge contribution that extends beyond a simple code-execution check. Prior to this investigation, no empirical evidence existed to guide researchers on whether code-based or natural-language prompting should be used when a chatbot platform is considered for generating randomization sequences. The present findings establish that this choice is not a matter of preference or convenience, but a determinant of whether the reproducibility requirement of ICH E9 can be met at all. That operational boundary, documented here across four platforms and four investigators, is the study’s primary contribution, and it could not have been established without the direct empirical contrast between the two prompting paradigms.

It should be made explicit that the methodological contribution of this study is not the demonstration that fixed-seed pseudorandom code is deterministic, which follows directly from the pseudorandom number generator. Rather, it is the empirical verification that consumer-facing LLM-linked interfaces preserved that determinism without execution corruption across repeated runs, investigators, and sessions and the parallel demonstration that this property disappeared under NL-only prompting. The study is therefore positioned as a technical benchmarking and verification study rather than as a new randomization algorithm: it defines an operational reliability boundary for chatbot-mediated sequence generation, in which reproducibility is preserved when the chatbot serves as a code-execution wrapper for a fixed-seed algorithm, but not when sequence generation is delegated to unconstrained natural-language prompting.

4.2. Deterministic Code Execution as the Operative Mechanism

The central interpretive finding is that the reproducibility observed here is best understood as a property of deterministic code execution rather than as a general property of LLM outputs. Foundation models and LLMs are characteristically sensitive to prompt design, access configuration, and session handling, dependencies that the GAMER Statement explicitly requires authors to report through tool specifications, prompting techniques, verification procedures, and documentation of access modes [14]. REFINE additionally addresses the technical dependencies specific to these systems, including model pretraining and inference-time adaptation strategies, generation parameters such as temperature and top-p sampling, output stochasticity and reproducibility constraints, and governance and auditability mechanisms [16]. Both frameworks were developed precisely because LLM outputs cannot be assumed to be stable across repeated runs under standard inference conditions, and the REFINE checklist makes reproducibility assessment a mandatory reporting domain for any FM or LLM study in medical research [16].

In the present study, however, the allocation sequence was determined by a Python script that used an explicit, fixed seed and was submitted directly to each platform. Under these conditions, the chatbot interface primarily served as a vehicle for code submission, execution, and output rendering. The deterministic element was therefore the seeded algorithm rather than the model’s generative inference process, a distinction that has direct implications for how chatbot-based allocation workflows should be classified, reported, and replicated in trial documentation. The Excel/VBA and R Console comparator supports the same conclusion: it implements seeded deterministic logic in a conventional software environment, and its equivalent reproducibility across repeated runs confirms that seeded execution, rather than model inference, accounts for the consistency observed in both tools. While restricted randomization methods such as stratified designs introduce additional structural constraints, their implementation in code-execution environments is governed by the same deterministic principles evaluated here. The present study’s permuted block benchmark confirms that this deterministic property extends beyond simple randomization: execution fidelity was preserved across 640 runs under a block-constrained shuffle, with 100% reproducibility under fixed-seed code execution across all four platforms, directly replicating the Experiment 1 pattern under a more structured, block-constrained algorithm. Whether this holds for stratified randomization or covariate-adaptive procedures such as minimization remains an important direction for future investigation. Accordingly, future evaluations of chatbot-assisted allocation workflows should clearly distinguish between model-level reproducibility, which is subject to stochastic inference and generation parameters, and code-level reproducibility, which is governed by algorithmic determinism under fixed seed conditions [4,16]. Shyr et al. (2025) [13] operationalized this distinction formally, demonstrating that semantic and internal repeatability, measuring consistency in output meaning and token-level probability distributions, respectively, varied across models, prompts, and datasets and crucially did not correlate with diagnostic accuracy. This empirical dissociation between correctness and consistency directly supports the methodological position taken here: that a chatbot producing a correct allocation list on a given run does not, by itself, establish that the platform will do so reliably and reproducibly under varied operational conditions.

This interpretation is reinforced by the pilot phase that preceded formal benchmarking. When the task was requested through natural-language-only prompting without explicit Python execution and without fixed-seed specification, outputs were not reproducible across repeated sessions. In other words, reproducibility did not persist outside the deterministic code + seed pathway. This contrast supports the central methodological conclusion of the present study: the observed stability reflects seeded code execution preserved within chatbot interfaces rather than unconstrained chatbot randomization.

A potential objection to the present findings is that deterministic output from seeded Python code is mathematically guaranteed by the pseudorandom number generator and therefore provides no information beyond what could be obtained by running the same script locally. This characterization, however, does not account for the architectural complexity of LLM-linked chatbot platforms, which are not transparent Python interpreters. These platforms route user input through natural-language parsing layers, content-moderation pipelines, session-state management systems, and model inference processes before passing any identified code to an execution environment. Each of these layers could, in principle, alter, truncate, reformat, or substitute the submitted code prior to execution. The fact that none of these layers introduced execution corruption across 1200 runs on four platforms, four investigators, and two sample-size scenarios is a property of these specific systems under the tested conditions and not a foregone conclusion simply because a fixed seed was specified. The distinction is consequential for trial documentation: an investigator citing a chatbot-generated randomization sequence must be able to assert, not merely assume, that the platform preserved the integrity of the execution. The present study provides that assertion for four of the most widely used consumer-facing platforms at the time of testing.

This platform-reliability framing is also consistent with emerging regulatory expectations: the FDA’s draft guidance on AI-enabled device software explicitly recommends evaluating repeatability and reproducibility as components of performance validation, recognizing that variability in AI outputs, even under identical conditions, is a distinct concern from accuracy [7,13].

4.3. The Contrast Between Code-Execution and NL-Only Conditions as the Central Finding

The most important finding of this manuscript is not the 100% reproducibility observed in Experiment 1, nor the 0% reproducibility observed in Experiment 2, but the absolute contrast between them. Under fixed-seed Python code execution, all four platforms functioned as deterministic code-execution environments, a finding attributable to the seeded algorithm rather than to the model’s generative behavior, as explicitly framed throughout this manuscript. Under natural-language-only prompting, without code submission or seed specification, the same four platforms produced entirely non-reproducible allocation sequences across 320 independent runs, with zero exact matches detected. This contrast was perfectly consistent across all four platforms and all four investigators.

This finding directly operationalizes the distinction between code-level reproducibility and model-level stochasticity that has been called for in the methodological literature. Shyr et al. (2025) [13] demonstrated formally that LLM outputs are variable even under identical input conditions and that this variability lacks standardized metrics for characterization. Experiment 2 provides direct empirical evidence of this stochasticity in the specific context of clinical trial randomization: without a fixed seed and explicit code submission, chatbot platforms cannot serve as reproducible allocation tools under the conditions tested here. Under the conditions evaluated in the present study, unconstrained NL-only chatbot prompting was therefore not appropriate for trial randomization.

The finding that allocation integrity remained 100% under NL-only prompting is equally important for correct interpretation. The platforms reliably produced complete, balanced, correctly formatted randomization tables in every run. The failure mode under NL-only prompting is therefore not structural invalidity, but non-reproducibility: any given list may be technically correct, yet it cannot be reconstructed, independently checked, or verified against a pre-registered sequence. A randomization list that cannot be independently reproduced from documented inputs violates the ICH E9 requirement that randomization procedures be reproducible when needed [4] and the CONSORT 2025 requirement for transparent, verifiable sequence generation [9,10,11]. Structural correctness without reproducibility is insufficient for legitimate trial use.

The practical implication is specific and narrow: investigators working in low-resource settings, such as postgraduate dental or medical research programs, trial sites in low- and middle-income countries, or early-phase exploratory studies, who wish to use a chatbot platform for sequence generation have, prior to this study, had no empirical basis for choosing between code-based and NL-only approaches. The present findings provide that basis. The recommendation is not that chatbots should be used for clinical trial randomization, but rather that if they are used, and available evidence suggests this is already occurring informally, the code-based pathway with a fixed seed is the only approach that satisfies the reproducibility requirement of ICH E9. The NL-only pathway, regardless of how structurally valid its outputs may appear, does not.

Taken together, the findings from Experiment 1 and Experiment 2 define a clear operational boundary: fixed-seed code execution provides a reproducible and manually traceable pathway for allocation generation, whereas NL-only prompting yields non-reproducible outputs and is therefore unsuitable for trial contexts requiring a documented and verifiable randomization sequence.

4.4. Allocation Integrity, Trial Validity, and Traceability

The finding of 100% allocation integrity, each identifier appearing exactly once, with no duplicates, missing, or extraneous entries, is methodologically important because valid randomization begins with a correct allocation sequence. Randomized controlled trials are regarded as the reference standard for causal effectiveness research because randomization reduces bias and, particularly in sufficiently large studies, balances both observed and unobserved participant characteristics between groups, thereby strengthening causal interpretation of between-group differences [17]. The contemporary trial-methods literature emphasizes that true random allocation, together with allocation concealment and, where feasible, blinding, safeguards internal validity and protects against systematic error [18]. Against that background, duplication, omission, or contamination of identifiers would represent a substantive failure of the allocation mechanism rather than a minor formatting defect. No such failures occurred in any benchmark or exploratory run, supporting the conclusion that, within the use case tested here, chatbot-mediated code execution implemented simple randomization faithfully at the level of the final allocation list.

That interpretation should nevertheless be bound to simple randomization itself. Braga et al. (2025) [18] note that simple randomization is easy to implement and methodologically valid, but in smaller studies, it may still produce chance imbalances that, although they do not invalidate the statistical test, may affect the credibility of the treatment effect. Block and stratified randomization are used to address specific design problems, particularly when sample size is small or prognostic balance is a priority [1,3,18]. The present findings, therefore, establish the correctness of the implementation in the tested use case, not the superiority of simple randomization over restricted or covariate-adaptive methods.

A second major contribution of this study is the demonstration that reproducibility and traceability can be preserved in a chatbot-mediated workflow when the allocation logic, seed value, and complete raw outputs are systematically archived. The ICH E9 guideline on statistical principles for clinical trials requires that randomization schedules be reproducible when needed and that details sufficient to reconstruct the schedule be securely filed rather than disclosed in documentation accessible to investigators during recruitment [4]. CONSORT 2025 strengthens expectations for transparent trial reporting by requiring explicit identification of who generated the allocation sequence, the method used, the type of randomization, the allocation concealment mechanism, and whether enrolling personnel had access to the sequence; it further adds items on protocol access, data sharing, and code sharing [9,10,11]. GAMER requires reporting of tool specifications, prompts, unedited outputs, verification procedures, data privacy safeguards, and the influence of AI tool use on study conclusions [14]. REFINE adds dedicated items on model access; prompt engineering; generation parameters; output stochasticity and reproducibility constraints; version variation; sharing of code, data, and model artefacts; and governance, auditability, and oversight [16]. Within this combined framework, preservation of the verbatim prompt or code condition, fixed seed, and full unedited output for every run enabled exact post hoc verification and reconstruction of the generated schedule. This study, therefore, demonstrates operational input–output traceability of the tested workflow, not a validated electronic audit trail or regulatory-grade record system.

It must be explicitly stated, however, that the operational input–output traceability demonstrated in this study refers strictly to manual traceability of the prompt/code, seed, and unedited output. It does not constitute, nor replace, the electronic audit trails required for regulatory compliance by agencies such as the FDA or MHRA (e.g., 21 CFR Part 11). Furthermore, any fully reproducible seeded workflow is inherently predictable if the seed and script are exposed. The revelation of the prompt or seed compromises allocation concealment entirely; therefore, the seed, code, and final list must remain strictly under the control of an independent party not involved in participant recruitment.

A related practical consideration concerns allocation concealment. While the present study establishes that chatbot interfaces preserved correct and reproducible outputs when executing the pre-specified fixed-seed code under the tested conditions, the conversational nature of these interfaces introduces a risk of inadvertent exposure. Chat histories may remain accessible within the platform, and the allocation sequence may be visible to individuals who should not have access to it during participant enrollment. To mitigate this risk, investigators using chatbot-generated allocation lists should adopt post-generation safeguards, such as exporting the output to a password-protected file, deleting or archiving the chat session immediately after list retrieval, and restricting access to the allocation schedule to a designated unblinded statistician or independent party, consistent with established allocation concealment principles [6].

The exploratory voice runs nevertheless exposed a workflow-specific vulnerability. All platforms generated correct outputs after voice-delivered instructions, but the underlying code was not displayed automatically and required an additional text prompt within the same session. This does not negate traceability of the workflow record because the code remains retrievable and verifiable against the output upon request; however, it introduces an avoidable procedural dependency. In the context of contemporary reporting standards that emphasize transparent documentation, reproducibility, and traceability, code retrieval and archival should therefore be treated as mandatory components of any future voice-mediated allocation workflow [9,10,11,14,16].

The supplementary permuted block benchmark extends the central finding of this study to a restricted randomization algorithm that is one step more structured than simple randomization. It should be interpreted as a restricted technical validation scenario rather than as a representative model of contemporary clinical trial randomization: it examined only a fixed block size, two arms, equal allocation, and no stratification. Although permuted block designs are widely used in registered clinical trials because they enforce allocation balance at fixed intervals and mitigate temporal recruitment bias [1,3], the present benchmark establishes only that the observed contrast persists under one restricted method that is more structured than simple randomization; it does not demonstrate performance across the broader set of randomization procedures used in contemporary trials, such as variable block sizes, stratification, multicenter workflows, unequal allocation ratios, or minimization. Within these bounds, the permuted block results demonstrate that the same binary pattern, 100% reproducibility under Python/fixed-seed execution, 0% reproducibility under NL-only prompting, holds without exception across all four platforms, all four investigators, and 640 runs under an algorithm that introduces structural complexity absent from simple randomization.

This replication carries two distinct implications. First, it establishes that the deterministic property of fixed-seed code execution is algorithm-agnostic within the code-execution pathway: the chatbot interface preserves execution fidelity regardless of whether the allocation logic is a complete random permutation or a block-constrained shuffle. The operative mechanism remains the seeded pseudorandom number generator, not the structural complexity of the allocation rule, and the platforms evaluated here did not introduce corruption under either design. Second, the NL-only failure mode is equally algorithm-agnostic: NL-only prompting cannot produce reproducible permuted block sequences any more than it can produce reproducible simple-randomization sequences because the absence of a fixed seed means that no allocation algorithm, however correctly implemented, can yield a verifiable, pre-registerable sequence. The structural correctness of every NL-only output (100% allocation integrity across 320 runs) confirms that the platforms understood and correctly implemented the permuted block instruction; the failure is exclusively one of reproducibility, not of algorithmic comprehension.

Taken together, the primary benchmark (Experiments 1 and 2) and the permuted block supplementary benchmark establish a consistent empirical pattern across two fundamentally different allocation algorithms, six platforms, and over 1900 total runs: within chatbot code-execution environments, seeded determinism is preserved; outside that deterministic scaffold, reproducibility disappears entirely. The practical recommendation that follows is correspondingly algorithm-independent: if LLM-linked chatbot platforms are to be used in any allocation context, whether simple or block-based, the fixed-seed explicit-code-execution pathway is the only prompting strategy that satisfies the ICH E9 reproducibility requirement and enables the traceability record demanded by CONSORT 2025 [9], GAMER [14], and REFINE [16].

4.5. Operational Flexibility and Practical Relevance

The exploratory perturbation analyses showed that the workflow extended without failure to modified allocation structures, including, for example, three-arm and four-arm designs and 2:1 and 3:1 allocation ratios. This is practically relevant because a sequence-generation workflow limited to two-arm, 1:1 allocation would have only narrow utility. Within the bounds of simple randomization, the present results suggest that the same seeded code-execution paradigm can be adapted via straightforward parameter modifications without compromising allocation integrity or reproducibility.

This flexibility may have practical value in lower-infrastructure environments, although it should not be interpreted as a substitute for validated randomization systems. At the same time, accessibility should not be conflated with methodological sufficiency. Even a technically correct allocation list requires appropriate implementation, secure concealment from those enrolling participants, and rigorous protocol governance to support valid inference. Empirical evidence demonstrates that trials with inadequate or unclear allocation concealment yield treatment effect estimates inflated by up to 40% compared with adequately concealed trials, establishing concealment as equally critical to the validity of the RCT as the randomization sequence itself [6]. Valid trial conduct, therefore, depends not only on generating a correct allocation list but on how that list is subsequently implemented, concealed, archived, and governed [6,17,18].

To make the positioning of chatbot-mediated workflows explicit relative to the tools used in clinical practice,

Table 3. Qualitative positioning of chatbot-mediated fixed-seed randomization relative to non-LLM deterministic controls and to dedicated randomization tools. Entries summarize documented or study-observed capabilities and were not obtained from a head-to-head empirical comparison; the present study did not test REDCap, randomizeR, Sealed Envelope, or any trial-management system. randomizeR is a sequence-generation and assessment package and is not itself a concealment, access-control, or trial-management layer. “Yes” and “No” denote the presence or absence of the capability as the tool is used in the indicated role; “Not provided” indicates the capability is outside that function; “Partial” indicates partial coverage; and “Not applicable” indicates the condition cannot arise (e.g., a natural-language interface is absent). A “No” does not imply the capability is impossible, only that it was absent or untested within the present scope. TMS, trial-management system; LLM, large language model.

Capability	Chatbot Fixed-Seed Code Execution (This Study)	Non-LLM Deterministic Controls (Excel/VBA, R Console)	Dedicated Randomization Package (e.g., Randomizer)	Validated Randomization Service/TMS (e.g., REDCap, Sealed Envelope)
Deterministic, exactly reproducible sequence under a fixed seed	Yes	Yes	Yes	Yes
Reproducible output under natural-language-only prompting a	No (0/320)	Not applicable	Not applicable	Not applicable
Operational input–output traceability (prompt/code + seed + output)	Yes (manual archiving)	Yes (manual archiving)	Yes	Yes (built-in)
Allocation concealment from enrolling personnel	Not provided	Not provided	Not provided	Yes (built-in)
Role-based access control and secure storage	Not provided	Not provided	Not provided	Yes
Validated electronic records/regulatory-grade audit trail (e.g., 21 CFR Part 11)	No	No	No	Yes (validated deployments)
Integration with participant enrollment and trial conduct	No	No	No	Yes
Formal software validation or vendor qualification	No	No	Partial (open-source, peer-reviewed)	Yes
Empirically evaluated in the present benchmark	Yes	Yes (as controls)	No	No

^a Not applicable for Excel/VBA, R Console, randomizeR, REDCap, and Sealed Envelope because the comparison concerns unconstrained conversational prompting without submitted code or fixed-seed specification, a condition that is specific to chatbot interfaces and does not arise in these non-conversational tools.

summarizes at a feature level how fixed-seed chatbot code execution compares with the non-LLM deterministic controls evaluated here and with dedicated randomization tools that were not tested: a sequence-generation and assessment package (randomizeR) and validated randomization services or trial-management systems (e.g., REDCap and Sealed Envelope). The comparison is qualitative and based on documented capabilities rather than on a head-to-head experiment, consistent with the scope of this study, and is intended only to delineate where chatbot-mediated execution is and is not a substitute. The table shows that the chatbot fixed-seed pathway achieved deterministic reproducibility and basic input–output traceability under the tested fixed-seed conditions, while differing categorically from dedicated tools in allocation concealment, role-based access control, validated electronic records, and integration with trial conduct—precisely the functions that distinguish a randomization utility from a regulated trial-management system. This directly addresses the concern that the present comparator set (Excel/VBA and R Console) does not encompass dedicated randomization software, by clarifying that those non-LLM tools were included as deterministic execution controls rather than as proposed substitutes for validated systems.

4.6. Reporting Implications

From a reporting perspective, this study lies at the intersection of randomized-trial methodology and generative AI evaluation, and no single guideline fully addresses both domains. CONSORT 2025 is the appropriate framework for transparent reporting of the trial-methods components, particularly sequence generation, allocation concealment, implementation roles, protocol access, and data and code sharing [9,10,11]. GAMER contributes a universal framework for reporting the use of generative AI tools across medical research, covering tool specifications, prompting techniques, verification procedures, data privacy, and influence on conclusions [14]. REFINE adds the technical detail required for FM and LLM studies, particularly model specification, prompt design, stochasticity control, output reproducibility constraints, version variation, and implementation governance [16].

A specific implication of the present study is that future evaluations of chatbot-assisted randomization should report reproducibility at two distinct levels: model-level reproducibility, which is subject to stochastic inference and generation parameters, and code-level reproducibility, which is governed by algorithmic determinism under fixed seed conditions. The FDA draft guidance on AI-enabled device software formalizes this requirement from a regulatory standpoint, recommending precision studies that evaluate repeatability (agreement under identical conditions) and reproducibility (agreement under different, prespecified conditions) as part of performance validation documentation [7]. Shyr et al. (2025) [13] provide a statistical framework for operationalizing exactly these metrics in LLM contexts, using pairwise cosine similarity of semantic embeddings for semantic consistency and average token-level entropy for internal stability, offering a methodological template that future chatbot-assisted allocation studies could adapt.

This distinction follows directly from the logic of both REFINE and GAMER, which treat model identity, prompting, access conditions, output handling, and reproducibility constraints as separate reporting domains rather than as a single undifferentiated performance domain [14,16]. In the present study, the key reproducibility claim concerns the latter: deterministic code-level reproducibility under fixed-seed execution. Reporting this distinction precisely is essential for accurate interpretation by trial investigators, methodologists, and regulators who may seek to evaluate or adopt chatbot-mediated allocation workflows in practice.

4.7. Limitations

Some limitations should be considered when interpreting these findings. The present study evaluated simple randomization by complete random permutation as its primary algorithm; the supplementary permuted block benchmark extended this to a restricted design that is one step more structured than simple randomization but still limited in scope. The permuted block benchmark was intentionally simplified, with a fixed block size, two arms, equal allocation, and no stratification, and should not be interpreted as broadly representative of contemporary trial randomization practice. Within these bounds, it confirmed algorithm-agnostic determinism under fixed-seed code execution. However, variable block sizes, stratified randomization, multicenter randomization, unequal allocation ratios, and covariate-adaptive procedures such as minimization were not evaluated and remain outside the scope of this study; whether chatbot-mediated code execution performs reliably under these designs is a priority for future investigation.

First, all testing was conducted through the publicly available graphical user interfaces (GUIs) of the chatbot platforms rather than through application programming interfaces (APIs). GUI-based access is subject to interface changes, platform updates, and potential session instabilities that may affect long-term reproducibility in ways not captured during the testing period.

Second, the comparator set, while including a deterministic Excel/VBA macro and R Console verification, did not encompass dedicated randomization software packages (e.g., REDCap, randomizeR in R, or Sealed Envelope) or regulatory-grade trial management systems. It should also be noted that even Excel-based randomization is increasingly discouraged in professional trial settings in favor of dedicated statistical software, such as SAS and the R statistical computing environment, or validated web-based systems [3]. The study was designed to evaluate operational equivalence in accessible, low-infrastructure settings rather than to benchmark against the full landscape of established randomization tools. To make this boundary explicit in the absence of a head-to-head experiment, Table 3 provides a qualitative, feature-level comparison of chatbot-mediated fixed-seed execution against these dedicated tools; a direct empirical comparison with at least one dedicated randomization tool or trial-management system remains a necessary step before any practical clinical-trial use could be considered.

Third, because all platforms achieved 100% success across all outcomes in Experiment 1 and the permuted block benchmark, the study could not characterize failure modes, error rates, or the boundaries of reliable performance under fixed-seed code execution; ceiling effects preclude estimation of true failure probabilities. Future studies could use deliberately corrupted prompts or intentionally unstable code versions to characterize failure modes and establish error rates more precisely.

Fourth, the reproducibility demonstrated here relies entirely on strict adherence to the locked, standardized prompt as provided. In real-world applications, there is a practical risk of prompt drift or execution-path drift, whereby minor variations in a user’s natural-language instructions may lead the platform to alter the underlying script or invoke different software libraries. Researchers adopting this workflow must therefore use the provided prompt template exactly as specified; any deviation from it, or reliance on free-text prompting, could compromise deterministic execution and invalidate the reproducibility guarantee established here.

Fifth, although Experiment 2 prospectively benchmarked NL-only prompting, it evaluated only one locked prompt, one allocation scenario, and consumer-facing GUI access. These findings establish non-reproducibility under the tested NL-only condition but do not exhaust every possible prompt formulation, interface configuration, or future platform implementation. The scope of the NL-only failure mode should therefore be interpreted as specific to the conditions tested rather than as a universal characterization of NL-prompted randomization. Relatedly, reproducibility was operationalized as exact sequence-level identity against a reference run. The NL-only outputs that did not reproduce the reference sequence were not further characterized using more granular sequence-level metrics, such as within- and cross-platform sequence similarity, duplicate-pattern frequency, positional allocation entropy, or run-to-run edit distance. Such analyses could quantify the degree of divergence among independently generated NL-only sequences and help identify whether latent structure, such as recurring ordering patterns or reduced entropy, is present. However, these analyses were outside the prespecified binary outcome framework of the present study and would not alter the central finding that exact reproducibility was absent under NL-only prompting.

Finally, the rapidly evolving nature of LLM platforms means that model updates, changes to code-execution environments, or modifications to platform policies may alter performance characteristics over time. The findings reported here reflect platform behavior at the time of testing and underscore the need for periodic revalidation before any chatbot-mediated allocation workflow is adopted or continued in a research setting.

5. Conclusions

The present study demonstrates that all four LLM-linked chatbot platforms evaluated (ChatGPT, Claude, Copilot, and Gemini) can generate randomization lists that are exactly reproducible under fixed-seed code conditions, allocation-integrity-preserving, format-compliant, operationally complete, and operationally traceable. These findings are best understood through the mechanistic lens of code execution rather than natural-language generation: the reproducibility achieved reflects the deterministic behavior of seeded Python execution within chatbot interpreter environments, not the generative properties of the underlying language models.

These results apply to simple randomization by complete random permutation and, as confirmed by the supplementary permuted block benchmark, to block-based restricted designs; validation of stratified and covariate-adaptive methods within chatbot code-execution environments remains an important direction for future work. Based on current evidence, these platforms must not replace validated trial-management systems or dedicated statistical software in regulated clinical trials. Their use should, at this stage, be strictly limited to prototyping, educational demonstrations, teaching methods, or exploratory workflows in low-infrastructure settings, ensuring that strict allocation concealment and governance measures are upheld. Whether advances in platform governance, regulatory validation, and algorithm coverage may position chatbot-mediated workflows as viable complements to established randomization tools in future trial contexts remains an open and important question. Even in these limited settings, natural-language-only prompting must be avoided for randomization: across 320 NL-only runs, it produced structurally valid but entirely non-reproducible allocation lists (0% exact reproducibility), so only the fixed-seed, explicit-code pathway, with the prompt and seed preserved and controlled by a party independent of recruitment, satisfies the reproducibility and traceability that any legitimate use requires.