A Modular Evaluation of AI-Assisted Clinical Documentation

Abstract

Clinical documentation in Electronic Health Records (EHRs) remains a substantial source of administrative burden for clinicians. In this study, we evaluate a modular AI-assisted clinical documentation pipeline using two complementary approaches: (1) a controlled benchmark based on multilingual synthetic clinical dialogues, and (2) an observational analysis of real-world usage traces from routine deployments. The benchmark enables systematic comparison of ASR–LLM configurations under fully controlled conditions, using metrics for transcription accuracy (Word Error Rate and Medical WER), report-generation quality, and modeled processing cost. Within this benchmark setting, Voxtral showed the strongest ASR performance among the evaluated models, while GPT-4o and Gemini 1.5 Pro showed the strongest report-generation performance under the automated evaluation used in this study. The real-world trace analysis should be interpreted as descriptive evidence of operational use, not as prospective clinical validation or as a direct evaluation of any single benchmarked configuration. Taken together, the results support the use of this pipeline as a human-supervised draft-generation tool that still requires clinician review, local workflow evaluation, and prospective clinical validation before broader deployment.

1. Introduction

Clinical documentation in Electronic Health Records (EHRs) has become a central component of modern healthcare, improving data accessibility, standardization, and continuity of care. However, this shift has also introduced a substantial administrative burden for clinicians, often associated with increased documentation time, reduced patient interaction, and higher rates of professional burnout [1,2,3]. As a result, there is growing interest in technologies that can assist or partially automate clinical documentation workflows.

Recent advances in Automatic Speech Recognition (ASR) and Large Language Models (LLMs) have enabled a new class of AI-assisted documentation systems, commonly referred to as AI scribes [4,5,6]. These systems aim to capture clinician–patient conversations, transcribe them, and transform them into structured clinical notes. Early evidence suggests that such systems can support parts of the documentation process, but their effectiveness remains highly dependent on transcription accuracy, robustness to real-world variability, and the quality of generated reports [3,5,7].

Despite rapid progress, important challenges remain. ASR systems must handle specialized medical terminology, diverse accents, and noisy clinical environments, while transcription errors may propagate into downstream summaries [8,9,10]. LLMs, although effective at structuring text, may introduce omissions, distortions, or fabricated content, raising concerns about reliability and safety [4,7,11]. For these reasons, current AI-assisted documentation systems are generally positioned as human-in-the-loop tools, where clinicians retain responsibility for verification and final approval rather than delegating decision-making to autonomous systems [12,13,14]. This perspective is consistent with emerging human-centered clinical AI literature, which emphasizes the role of AI as a supporting component within expert-driven workflows rather than a replacement for human judgment [11,12].

Recent work has already introduced datasets, benchmarks, and evaluation frameworks for ambient clinical documentation and clinical note generation, including ACI-BENCH, automatic note-generation evaluation work, and ambient digital scribing evaluation frameworks [15,16,17]. However, the evidence remains heterogeneous, and technical benchmarking, component-level ASR–LLM comparison, multilingual evaluation, and deployment-grounded usage analysis are often reported separately [18,19,20]. In this context, the present study should be read as an integrated evaluation study rather than as a new general evaluation framework.

The contribution of this paper is to bring together two complementary evidence streams. First, we compare several ASR and LLM configurations under controlled conditions using multilingual synthetic clinical dialogues with preserved ground truth. Second, we analyze finalized-export traces from routine deployments as descriptive evidence of operational use. These two parts answer different questions: the benchmark supports technical comparison under known conditions, whereas the deployment traces show observed use in routine settings. The traces do not measure adoption rates, editing effort, clinical safety, time savings, or patient outcomes. Therefore, the goal is not to claim full clinical validation, but to inform cautious pilot-stage deployment under clinician supervision.

This framing leads to the following research questions:

1.

RQ1. How do contemporary ASR–LLM configurations differ in transcription accuracy, medical terminology preservation, and downstream report-generation performance in a controlled multilingual benchmark?

2.

RQ2. What patterns of operational use are observed in finalized-export traces from routine, clinician-supervised deployments?

3.

RQ3. What practical evaluation considerations emerge when controlled benchmarking and deployment-grounded usage traces are interpreted as complementary, but distinct, forms of evidence?

To answer these questions, we pursue three linked objectives: (1) compare ASR–LLM configurations in a controlled multilingual synthetic benchmark with preserved ground truth; (2) describe operational-use patterns from finalized-export traces in clinician-supervised deployments; and (3) identify practical interpretation limits when benchmark and deployment evidence are considered together. These objectives are intended to support cautious pilot-stage assessment, not to establish adoption rates, clinical effectiveness, workflow impact, or autonomous deployment readiness. We also frame AI-assisted documentation as a data-generation process, where reliable downstream information depends on the quality, structure, and governance of the clinical data produced at the point of care.

2. System Architecture

The AI scribe is a modular web application designed to automate the creation of structured clinical reports from doctor–patient conversations. Its architecture emphasizes flexibility, allowing for the benchmarking of different ASR and LLM models to find optimal configurations for various clinical use cases, languages, and specialties. The system workflow is coordinated by a backend orchestration layer that connects its core functional modules, as illustrated in Figure 1.

2.1. Conversion of Audio to Text

The AI scribe’s workflow begins when a healthcare professional interacts with the web-based user interface, which allows for both direct audio recording of doctor–patient conversations and the upload of pre-existing audio files in various formats. Upon submission, the audio is securely transmitted to the backend server, where it undergoes a comprehensive pre-processing pipeline. This pipeline includes validation of audio format and quality, normalization of volume levels, and segmentation into shorter, more manageable chunks using advanced silence detection algorithms. This segmentation is not only necessary to adhere to the time limits of transcription modules, but also, as highlighted by several studies, to improve processing efficiency and the accuracy of downstream speech recognition [21,22].

Once pre-processed, each audio chunk is routed to the ASR module. The AI scribe supports a modular ASR backend, enabling the selection between top-performing models such as OpenAI’s Whisper and GPT-4o-transcribe. The choice of ASR model can be configured per experiment or user preference, facilitating comparative analysis and optimization for different clinical scenarios, languages (including English, Spanish, French, and others), and audio qualities. The ASR module transcribes the spoken dialogue into raw text, which is then subjected to a series of post-processing steps. These include correction of common transcription errors, removal of non-speech artifacts (such as URLs or background noise), and optional speaker diarization if supported by the ASR model. These steps are important because they improve the readability and overall quality of the transcript before the subsequent LLM processing phase.

2.2. Generation of the Clinical Report

The cleaned transcript is then forwarded to the LLM module. This module is also highly configurable, supporting integration with leading LLMs such as Google’s Gemini and OpenAI’s GPT. The LLM receives a dynamically constructed prompt that includes the full transcript, the target language, the relevant medical specialty, and explicit instructions to structure the output as a formal clinical report. The prompt can be tailored to enforce specific report formats (e.g., SOAP: Subjective, Objective, Assessment, and Plan), a capability increasingly explored and validated for structured clinical note generation by LLMs [23,24]. It can also include or exclude certain sections and adapt to local clinical documentation standards. The LLM processes the input and generates a comprehensive, well-structured clinical report, ensuring the use of precise medical terminology and adherence to the requested format.

After generation, the report is presented to the user in the web interface, where the healthcare professional can review, edit, and approve the content. This human oversight process is important in medical applications to support accuracy and safety [12]. The system provides tools for highlighting potential uncertainties or inconsistencies (e.g., flagged medication dosages or ambiguous terms), and supports exporting the final report in markdown, plain text, or PDF formats. Throughout the workflow, all core ASR and LLM processing is performed via secure cloud APIs, supporting scalability and access to current model versions. Importantly, the AI scribe is designed with privacy by default: no patient-identifiable information is required or stored, and all data processing is intended to comply with relevant data protection regulations, an important consideration in healthcare [25]. The system also supports multilingual and specialty-specific workflows, including evaluation with cross-specialty mismatch controls, and can be extended to integrate with EHR systems or other clinical IT infrastructure.

The AI scribe is distinguished by several design principles aimed at ensuring robust evaluation and adaptability. A core feature is its multi-model flexibility, which supports various ASR and LLM backends, thereby enabling rigorous benchmarking to identify the most accurate and cost-effective model configurations. This adaptability extends to different clinical contexts through dynamic prompts tailored for various medical specialties and languages, including English and Spanish.

3. Related Work

The automation of clinical documentation represents a significant area of research and development within health informatics, driven by the need to reduce clinician burden and improve efficiency [2,26,27]. This section reviews the evolution of documentation practices, the core technologies enabling automation, the current landscape of tools, evaluation methodologies, and the associated challenges.

3.1. Evolution of Clinical Documentation and AI Scribes

The evolution of clinical documentation reflects a continuous effort to improve record-keeping accuracy, accessibility, and efficiency in healthcare. Initially, clinical notes were primarily handwritten, a practice that, while personal and immediate, was prone to issues of legibility, completeness, and standardization [1]. The introduction of structured formats represented an early attempt to bring order and consistency to medical records, often utilizing pre-defined templates and checklists to ensure key information was captured. However, these structured approaches could sometimes feel rigid and fail to fully capture the nuances of complex patient encounters.

The digital revolution brought about the era of Electronic Health Records (EHRs), with widespread adoption being significantly accelerated by governmental initiatives such as the HITECH Act in the United States (https://blogs.opentext.com/history-heath-information-management-now/ (accessed on 26 June 2026)). EHRs promised numerous advantages, including enhanced data accessibility across different healthcare providers, improved data management, reduced errors associated with paper-based records, and the potential for data-driven insights. Despite these benefits, the implementation of EHRs often introduced unforeseen challenges. Clinicians frequently reported a substantial increase in the time and complexity associated with documentation, due to requirements for detailed data entry, navigation through complex interfaces, and adherence to standardized coding and billing practices [2,3]. This increased administrative burden has been widely recognized as a significant contributor to clinician burnout and reduced job satisfaction, potentially impacting the quality of patient care.

In response to these challenges, interest has grown in AI-powered medical scribes, which use ASR and LLMs to convert clinician–patient conversations into draft structured clinical notes [28]. These systems are intended to reduce documentation workload and return clinician attention to direct patient care, although their practical value depends on transcription accuracy, report quality, and integration into clinical workflows.

3.2. Core Technologies: ASR and LLMs

Automated clinical documentation tools primarily rely on two core technologies: Automatic Speech Recognition to convert speech to text, and Large Language Models to structure that text into clinically relevant notes [4,6].

ASR models, such as OpenAI’s Whisper [9], Google Cloud Speech-to-Text [29], Amazon Transcribe, and Microsoft Azure Speech, extract audio from the doctor–patient conversation and convert it into a sequence of words. The performance of ASR systems is often quantified using the Word Error Rate (WER), defined as:

$$WER={\displaystyle \frac{S+D+I}{N}}$$

where S is the number of substitutions, D is the number of deletions, I is the number of insertions, and N is the total number of words in the reference transcription. While general ASR accuracy can be high, achieving low WER in the medical field presents unique challenges. These include specialized terminology, variations in accents and dialects, and noisy environments with overlapping speech [8,9,30]. Errors in transcribing medical terms are a critical concern, and specialized models are often employed to improve accuracy in these contexts [10,31,32].

The challenge of robust signal processing under variable acquisition conditions is not unique to clinical speech recognition. In other engineering domains, including structural health monitoring and non-destructive testing, recent studies have investigated methods for improving signal robustness under environmental variation through approaches such as long-term temperature compensation in ultrasonic guided waves [33] and machine learning-based multi-feature integration [34]. Although these methods are not directly transferable to clinical ASR, they reinforce a broader methodological principle: robust feature extraction, calibration, and multimodal or multi-feature analysis are often necessary when signal quality is affected by changing operational conditions. Similar considerations are relevant in healthcare environments, where accents, background noise, recording devices, and workflow variability can influence transcription performance.

The extracted text is then processed by LLMs such as GPT-4 [35], Gemini 1.5 Pro [36], and specialized medical models such as Med-Gemini [37] and ClinicalBERT [38]. These models transform transcripts into structured clinical notes, often in formats such as SOAP (Subjective, Objective, Assessment, and Plan) [6]. The quality of the generated notes can be evaluated using metrics like ROUGE (Recall-Oriented Understudy for Gisting Evaluation) scores, which measure the overlap of n-grams between the generated text and a reference summary. For example, ROUGE-1 focuses on unigram overlap:

$$ROUGE-1={\displaystyle \frac{\mathrm{Number}\mathrm{of}\mathrm{overlapping}\mathrm{unigrams}}{\mathrm{Total}\mathrm{number}\mathrm{of}\mathrm{unigrams}\mathrm{in}\mathrm{the}\mathrm{reference}\mathrm{summary}}}$$

LLMs excel at understanding context and summarization but are prone to “hallucinations,” generating false or inaccurate information, necessitating validation and often clinician review [4,5]. Techniques like prompt engineering and task-specific fine-tuning are often used to improve reliability in medical documentation settings [6]. Research also explores using LLMs to correct ASR errors [10,39], potentially by leveraging the LLM’s contextual understanding to refine the ASR output.

3.3. Existing Systems and Landscape

Numerous commercial AI scribe solutions have emerged, each employing distinct techniques. Nuance DAX from Microsoft leverages ambient clinical intelligence to analyze the natural flow of conversations for detailed note production. Abridge (https://www.abridge.com/ accessed on 26 June 2026) focuses on efficiently summarizing the most critical elements of these interactions, while DeepScribe uses proprietary AI models trained on medical dialogues to enhance accuracy. Other systems like Suki AI offer voice-enabled, hands-free interaction [40], and Augmedix often integrates a human-in-the-loop approach for quality assurance [40]. Nabla emphasizes real-time assistance and seamless integration with existing EHR systems for immediate documentation support [41].

Despite these diverse approaches, several key features are common. EHR integration, particularly with widely adopted systems like Epic and Cerner, is crucial for embedding notes directly into patient records [3,40]. Real-time processing and support for various clinical note formats, such as SOAP notes, ensure compatibility with existing workflows. Recognizing the global nature of healthcare, some systems are also expanding their multilingual capabilities to cater to diverse patient populations [42]. The underlying techniques employed by these AI scribe solutions exhibit notable variations. The choice of Automatic Speech Recognition (ASR) models can differ, with each model possessing unique strengths and weaknesses in handling the complexities of medical language, diverse accents, and noisy clinical environments. Similarly, the Large Language Models (LLMs) utilized for structuring the transcribed text into clinical notes can range from general-purpose models to those specifically fine-tuned for medical text processing. Finally, the degree of human oversight varies significantly, with some systems relying solely on AI processing while others incorporate human review to enhance accuracy and reliability [40]. These differences in techniques and features highlight the ongoing innovation and diverse strategies within the rapidly evolving field of AI-powered clinical documentation.

3.4. Challenges and Research Gaps

The landscape of automated clinical documentation, while promising, faces substantial technical and ethical hurdles. From a technical perspective, the main unresolved issues for evaluation are ASR robustness to language, audio-quality, and specialty-specific terminology variation, together with the risk that LLM-generated notes may omit, distort, or hallucinate clinically relevant information [4,5,42,43,44].

Beyond these technical considerations, ethical challenges demand careful attention. These encompass ensuring stringent data privacy and compliance with regulations like HIPAA, addressing potential biases within AI models stemming from training data that may not adequately represent diverse linguistic or patient populations, defining accountability for system errors, and obtaining informed patient consent [11,45].

Recent literature has introduced important benchmarking frameworks for AI scribes, such as ACI-BENCH [15] and the SCRIBE evaluation framework [17], which combine automated metrics, human evaluation, and simulation testing. Similarly, datasets such as MedSynth [46] and MMedBench [47] have significantly advanced the use of synthetic and multilingual data for medical LLM evaluation. However, many existing studies still evaluate either isolated components, rely on monolingual datasets, or provide limited evidence linking controlled benchmark behavior with routine deployment. Building upon these foundations, our study complements existing frameworks by combining a highly modular ASR–LLM benchmark across English and Spanish with real-world deployment telemetry. A recent systematic review of AI scribes also reinforces the need for such complementary evidence: only eight heterogeneous studies were identified, most from recent and largely U.S.-centric settings, and outcome heterogeneity prevented stronger synthesis [18].

Table 1. Positioning of the Present Study Relative to Prior Evaluation Work.

Prior Work	What It Contributes	How Our Study Is Positioned
ACI-BENCH [15]	Public benchmark corpus for automatic visit-note generation from clinical dialogue.	We are not the first benchmark. We add modular ASR–LLM comparison, English–Spanish synthetic audio, and deployment-grounded usage traces.
Ben Abacha et al. [16]	Analysis of evaluation methods and metrics for automatic medical note generation.	We do not claim that automated metrics solve evaluation. We use them as technical indicators within one applied benchmark.
SCRIBE framework [17]	Broad ambient digital scribing evaluation framework combining simulation, computational metrics, human review, and LLM-based evaluation.	We do not present a new general framework. We provide an applied integrated evaluation of one modular documentation pipeline.
Digital-scribe and ambient-AI studies [3,7,48]	Measure documentation time, note quality, editing, or clinician-reported outcomes.	Our telemetry does not measure these outcomes. It only describes finalized-export usage patterns.
Systematic reviews [18,19,20]	Show that the field is promising but heterogeneous, with variable methods and outcomes.	Our study contributes one structured evaluation, but it cannot close the evidence gap alone.

summarizes this positioning relative to prior evaluation studies and clarifies how the present study complements, rather than replaces, existing benchmarks and evaluation frameworks.

The present study should therefore be interpreted as an applied, early-stage evaluation that combines controlled benchmarking and deployment-grounded operational observations. It complements existing benchmarks and evaluation frameworks rather than replacing them.

4. Experiments

This section outlines the experimental methodology used to evaluate the performance of Automatic Speech Recognition (ASR) and Large Language Models (LLMs) in generating structured clinical reports from medical dialogues. The experimental process involved generating synthetic medical dialogues, converting them to audio, transcribing them using different ASR models, and generating clinical reports using different LLMs. The results were evaluated using predefined metrics, and performance and cost data were recorded across multiple iterations. This benchmark setup allowed us to compare technical performance and modeled processing costs under controlled conditions, while leaving real-world workflow integration and clinical effectiveness for future prospective evaluation.

4.1. Dataset Generation

To rigorously evaluate the performance of our automated clinical documentation system, we generated a set of synthetic medical dialogues across a diverse range of specialties, including dentistry, ophthalmology, general practice, cardiology, and dermatology. The use of synthetic data is an emerging approach to address data scarcity in specialized domains [49,50]. In the context of this hybrid study, the controlled benchmark is deliberately synthetic for early-stage assessment, not merely a fallback for unavailable real data. This enables controlled comparison across ASR–LLM configurations, consistent scenario coverage across specialties and languages, and the use of shareable benchmark materials without exposing patient-identifiable information. The complete list of medical specialties, their associated conditions, and clinical contexts is detailed in Appendix A.1 (

Table A1. Medical specialties with associated medical conditions and clinical contexts. The dataset covers multiple medical domains with realistic clinical scenarios and synthetic doctor–patient interactions for each specialty.

Medical Specialty	Medical Condition	Clinical Context
Cardiology	Chest pain and shortness of breath	Heart health evaluation
Dentistry	Tooth sensitivity and bleeding gums	Regular dental checkup and cleaning
Dermatology	Unusual mole and skin rash	Skin cancer screening
ENT (Otolaryngology)	Chronic sinusitis and hearing loss	ENT evaluation
General Practice	Persistent cough and fever	Follow-up visit for respiratory symptoms
Neurology	Recurring migraines and dizziness	Neurological consultation
Ophthalmology	Gradual vision loss and eye strain	Annual eye examination
Orthopedics	Chronic knee pain after sports injury	Post-injury assessment
Pediatrics	Child with recurring ear infections	Pediatric checkup
Psychiatry	Anxiety and sleep disorders	Mental health consultation

). An illustrative example of a generated doctor–patient discussion is also provided in Appendix A.2. As illustrated in Figure 2, these dialogues were created using Google’s Gemini-2.0-flash model and were designed to mirror realistic doctor–patient interactions. Each simulated encounter followed a structured format, encompassing essential elements such as initial greetings, patient identification, detailed symptom description, the process of clinical examination, a thorough discussion of the diagnosis, proposed treatment recommendations, and follow-up instructions.

A key advantage of generating these synthetic dialogues is that we retain the original, verbatim text of the entire simulated discussion. This text serves as the ground truth against which the accuracy of the ASR component can be measured precisely. It also allows controlled evaluation of whether clinically salient entities and decisions are preserved through the full documentation pipeline. By comparing the original dialogue, the ASR-transcribed text, and the final LLM-generated report, we can quantitatively assess information flow and potential loss or distortion at each stage of the automated documentation pipeline. For these reasons, a synthetic evaluation component is well suited to controlled benchmarking and component-level comparison, even though it does not by itself establish prospective clinical validity.

This synthetic benchmark serves as the controlled benchmark within the study’s bench-to-field evidence framework. The controlled benchmark uses synthetic dialogues because private real-world consultations rarely provide shareable, perfectly aligned ground-truth transcripts and reference notes needed to calculate exact ASR error rates and downstream LLM-report discrepancies at the component level. In contrast, the deployment-grounded evidence stream uses routine deployment traces to test whether the same pipeline is operationally feasible under heterogeneous, messy environments in which microphones, room acoustics, conversational structure, specialty mix, and user behavior are not experimentally controlled. The two evidence streams therefore answer different but complementary questions: technical accuracy under known ground truth, and observed feasibility under real-world workflow conditions.

4.1.1. Dialogue Generation

Our synthetic dialogue generation process leveraged Google’s Gemini-2.0-flash model, with parameters carefully tuned to achieve a balance between creative natural language generation and consistent adherence to the simulated clinical scenarios (temperature = 0.7, top_p = 0.95, top_k = 40). To thoroughly evaluate the multilingual capabilities of our system and its effectiveness in processing local medical terminology, each dialogue was generated in English and Spanish. The generation of these dialogues adhered to a structured JSON schema designed to enforce several key characteristics: clear delineation of speaker roles (doctor and patient), the inclusion of emotional context markers (neutral, concerned, reassuring, professional, anxious, relieved) to add realism to the interactions, the accurate incorporation of language-specific medical terminology, and a natural flow of conversation that mimics real-world clinical discussions. This structured approach ensured a degree of consistency and allowed for targeted analysis of the system’s performance across different linguistic and emotional contexts.

4.1.2. Audio Generation

The textual dialogues generated in the previous phase were then converted into audio using ElevenLabs’ sophisticated multilingual text-to-speech service. We implemented a detailed voice configuration system to further enhance the realism of the simulated interactions. For English dialogues, a professional male voice (identified by the ElevenLabs voice ID ThT5KcBeYPX3keUQqHPh) was consistently used for the doctor, while a neutral female voice (21m00Tcm4TlvDq8ikWAM) was used for the patient. Similarly, for Spanish dialogues, we selected professional male (LlZr3QuzbW4WrPjgATHG) and neutral female (gD1IexrzCvsXPHUuT0s3) voices, specifically chosen for their authentic accents, to ensure linguistic fidelity.

To further emulate natural conversation dynamics in the generated audio, we incorporated several key elements: consistent 500-millisecond silence intervals between speaker turns to provide realistic pauses, emotion-appropriate voice modulation based on the emotional context markers embedded in the dialogues, and language-specific prosody and intonation to ensure the audio sounded natural.

4.1.3. Dataset Characteristics

The descriptive statistics in

Table 2. Descriptive statistics of the synthetic medical dialogues and audio files.

Metric	Mean	Std. Dev.
Audio duration (s)	226.28	38.13
Words per discussion	573.88	99.76
Turns per discussion	22.62	3.50
Doctor words	421.56	69.72
Patient words	152.31	38.87
Doctor turns	14.38	2.32
Patient turns	8.25	1.60
Proportion of doctor words (%)	73.46	–
Proportion of patient words (%)	26.54	–
Proportion of doctor turns (%)	63.54	–
Proportion of patient turns (%)	36.46	–

provide an overview of the characteristics of the synthetic medical dialogues and their corresponding audio files used in our experiments. The average audio duration is 226.28 s (approximately 3.8 min), with a standard deviation of 38.13 s, indicating moderate variability in dialogue length. These files contain approximately four minutes of near-continuous synthesized speech rather than natural consultations with long silences, pauses, interruptions, or background activity. Although this creates a dense benchmark signal, it is shorter than many real-world consultations, particularly in specialties such as psychiatry or general practice, where encounters may exceed 15–30 min. Longer consultations, silence patterns, background noise, and overlapping speech can pose additional challenges for ASR systems and downstream summarization, and noise has been shown to affect the quality of AI-generated clinical summaries [44]. These conditions were not directly studied in the present benchmark. Each discussion contains, on average, 573.88 words and 22.62 conversational turns, reflecting the controlled complexity and depth of the simulated clinical encounters.

A notable observation is the distribution of dialogue between the doctor and patient roles. Doctors contribute the majority of the content, accounting for 73.46% of the total words and 63.54% of the turns, while patients contribute 26.54% of the words and 36.46% of the turns. This asymmetry is consistent with real-world clinical consultations, where the physician typically leads the conversation, provides explanations, and documents findings, while the patient responds to questions and describes symptoms.

The standard deviations for both word counts and turn counts are relatively low compared to their means, suggesting that the dataset is reasonably balanced and does not contain extreme outliers. Overall, these statistics indicate that the generated dataset captures a diverse range of simulated medical dialogues and is suitable for evaluating the performance of ASR and LLM models in this benchmark.

4.2. Experimental Setup

The experimental setup was designed to compare a selection of ASR and LLM models that represent different paradigms: top-performing general-purpose models, a commercial service, and a specialized domain-specific model. A key aspect of this setup was to establish a lower bound on performance by evaluating the models’ behavior when presented with semantically incongruent input, which serves as a benchmark for stress-testing robustness.

4.2.1. ASR Model Comparison

We conducted a comparative evaluation of four prominent ASR models: OpenAI’s Whisper [9] and a specialized version of GPT-4 optimized for transcription tasks (gpt-4o-transcribe) were chosen as representatives of top-performing general-purpose ASR technology. Google Cloud Speech-to-Text [29] was included as a benchmark for a high-quality, widely used commercial service. Finally, we evaluated Voxtral [51], an open-source speech understanding model developed by Mistral AI. Released in July 2025 under the Apache 2.0 license, Voxtral is built upon the Mistral Small 3.1 backbone and features a 32k token context window. We specifically utilized the Voxtral Mini (3B parameters) variant, which is optimized for low-latency transcription and cost-efficiency, to assess the performance of a resource-efficient model designed for production-scale applications. To establish a lower-bound reference condition, we implemented a cross-specialty mismatch control. In what follows, this term refers to the concrete experimental setup, whereas “degraded-input negative control” refers to its analytical role. These labels are used here for clarity and are not intended as universal terminology. In this condition, Whisper transcribed each audio file under a deliberately mismatched specialty context, producing degraded transcripts that simulate strong domain mismatch. This control was intended as a stress test of the pipeline under semantically incongruent input rather than as a clinically realistic operating mode.

4.2.2. LLM Post-Processing

For the task of clinical report generation, we compared the performance of three advanced LLMs. GPT-4o [52] and Gemini 1.5 Pro [36] were selected as leading, general-purpose foundation models to set a high benchmark for performance. In contrast, MedGemma [53] was included as a specialized, smaller-scale (4B parameters) model designed specifically for medical applications. These models were evaluated both on standard ASR transcripts and on the degraded transcripts produced by the cross-specialty mismatch control. When the results tables refer to the cross-specialty mismatch control, this denotes the lower-bound condition built from cross-specialty Whisper transcripts, used as a reference condition for downstream report generation rather than as an additional stand-alone foundation model. This allowed us to assess how the pipeline behaves when the input transcript lacks semantic coherence with the target specialty.

4.2.3. Prompting and Structured Evaluation

The report-generation prompts were designed to keep the task clinically realistic while minimizing opportunities for hallucination. Across English and Spanish, the core instructions were the same: generate a concise physician-style report in the source language, preserve the supplied section headings, follow SOAP-like organization implicitly, include medications and diagnoses only when explicitly mentioned, and avoid adding information not present in the transcript. The Spanish variant additionally required all requested sections to be present and instructed the model to mark missing content explicitly when the dialogue did not support a section.

The automated evaluation prompt likewise used a fixed structured format. Gemini-2.0-flash returned JSON scores for factual accuracy, information coverage, clinical relevance, hallucination score, and terminology accuracy. This design follows the broader LLM-as-a-judge paradigm, in which strong language models can provide scalable approximate judgments for open-ended outputs, while also inheriting known evaluator biases such as position, verbosity, self-enhancement, and self-preference effects [54,55,56]. This allowed the benchmark to compare text quality, semantic preservation, and safety-oriented behavior within the same reproducible evaluation loop while keeping the scoring interface identical across ASR–LLM combinations.

A methodological limitation of this design is that the Gemini family was used in multiple roles across the study: synthetic dialogue generation, one of the evaluated LLM families, and the automated clinical evaluation framework described in Appendix A.4. Recent research suggests that this type of intra-family overlap may influence LLM-as-a-judge evaluations, because evaluators can favor generations that share similar styles, alignments, or post-training distributions [55,56]. We therefore interpret the resulting scores as an applied benchmark within a specific evaluation setup, not as a model-agnostic final ranking. In particular, the absolute Gemini 1.5 Pro scores should be read with caution, whereas cross-model trends, safety-oriented failure patterns, and the comparison between standard and degraded-input conditions remain useful for identifying pipeline behavior under a fixed scoring protocol.

4.2.4. Multilingual and Length Variations

To further assess the robustness and generalizability of the models, we extended our evaluation to include medical dialogues in English and Spanish reflecting the multilingual context of our research. Critically, the cross-specialty mismatch control ensures that we evaluated the lower bound of performance for both languages when the ASR models were presented with semantically mismatched audio. This comprehensive approach allowed us to understand the fundamental limitations of the ASR and LLM pipelines when the initial audio transcription is significantly compromised due to domain mismatch.

5. Results

We conducted a systematic evaluation of Automatic Speech Recognition (ASR) and Large Language Model (LLM) combinations for medical report generation across 230 experiments spanning 10 medical specialties in English and Spanish.

Table 3. Number of reports generated by each ASR–LLM combination.

		LLM Model
		Gemini 1.5 Pro	GPT-4o	MedGemma	Cross-Specialty Mismatch Control
ASR Model	GPT-4o Transcribe	20	20	20	0
	Voxtral	20	20	20	0
	Whisper	20	20	20	0
	GCP	10	10	10	0
	Cross-specialty	0	0	0	20
	mismatch control

presents the experimental design matrix, showing balanced coverage across model combinations with particular emphasis on contemporary ASR–LLM pairings used in current clinical documentation research and practice.

This section presents the results of the controlled benchmark conducted under standardized experimental conditions. These analyses evaluate the relative performance of the investigated ASR–LLM configurations using synthetic multilingual clinical dialogues and should be interpreted as benchmark evidence rather than direct evidence of real-world clinical deployment.

The experimental design reflects the practical constraints and opportunities in clinical settings, where advanced ASR models are typically paired with sophisticated LLM systems, while the cross-specialty mismatch control functions as a degraded-input negative control. The reduced sample size for GCP combinations reflects the restriction to English language evaluation for this commercial solution. Here, the control-only pairing denotes the lower-bound condition derived from cross-specialty Whisper transcripts and is included as a reference condition rather than as an additional standalone foundation model.

5.1. Speech Recognition Results

Our analysis of ASR model performance centers on two critical metrics that capture different aspects of transcription quality in medical contexts. The Word Error Rate (WER) provides the standard measure of transcription accuracy, calculated as $\mathrm{WER}=\frac{S+D+I}{N}$ where S, D, and I represent substitutions, deletions, and insertions, respectively, normalized by the total number of words N in the reference transcription. Additionally, we employed Medical WER (M-WER), a domain-specific variant designed to penalize errors in clinically relevant terminology. M-WER is calculated using a weighted Levenshtein distance alignment, where medical entities identified via the SNOMED CT ontology are assigned a penalty weight of $\lambda =3.0$ for substitutions, deletions, or insertions, compared to $\lambda =1.0$ for general vocabulary. This weighting scheme reflects the disproportionate patient safety risk associated with inaccuracies in drug names, anatomical terms, and clinical procedures.

Figure 3 presents the comparative performance evaluation across all ASR models and shows material differences within this benchmark.

Voxtral [51] achieved the lowest mean WER in this benchmark (0.099 ± 0.207), corresponding to an approximately 80% relative reduction compared with Whisper (0.496 ± 0.307). A similar pattern was observed relative to GPT-4o audio transcription (0.463 ± 0.318 WER), for which Voxtral showed an approximately 78% relative reduction. In this controlled benchmark setting, these results indicate that Voxtral was the strongest ASR model among the evaluated options.

The Medical WER results provide complementary evidence within the same benchmark, with Voxtral scoring 0.311 ± 0.406 compared with Whisper’s 0.409 ± 0.350. Because this metric upweights medically salient terminology errors, the lower value suggests more favorable handling of domain-specific vocabulary under the study conditions.

Statistical significance testing using the Wilcoxon signed-rank test detected a difference between Voxtral and Whisper ($p<0.001$; exact values in Appendix A.6). The associated Wilcoxon effect size was large ($r=0.877$ for the primary WER contrast), suggesting that the magnitude of the difference is unlikely to be trivial within this dataset, although direct workflow impact was not prospectively measured in this benchmark.

5.2. Language Model Results for Report Generation

The evaluation of LLM models for medical report generation encompasses multiple dimensions that capture both technical NLP performance and clinical utility. Our assessment employed ROUGE-L scores, which measure text similarity using longest common subsequences and are particularly effective for evaluating generated medical reports due to their ability to capture structural similarities in clinical documentation. We complemented this with BERT F1 scores using ModernBERT [57] embeddings, which provide semantic similarity measurements that go beyond lexical overlap to capture contextual understanding.

Figure 4 presents the performance comparison across all LLM approaches and shows measurable differences across the evaluated models.

GPT-4o and Gemini 1.5 Pro showed very similar performance profiles, both achieving ROUGE-L scores of 0.841 ± 0.106. Within the evaluated benchmark setting, both models outperformed the cross-specialty mismatch control (0.278 ± 0.056) and the domain-adapted MedGemma model (0.397 ± 0.291). This convergence suggests that, within the constraints of our evaluation setup, the top-performing large language models may be approaching similar performance levels for this task.

The BERT F1 scores of 0.958 ± 0.035 for both GPT-4o and Gemini 1.5 Pro indicate high semantic similarity under the automated evaluation procedure. In this setting, these scores suggest that the models generally preserved the main medical content while transforming conversational speech into structured clinical documentation.

The clinical assessment dimensions include factual accuracy and clinical relevance, both evaluated using Gemini-2.0-flash as an automated assessor following established methodologies from recent literature [58,59]. Factual accuracy measures medical fact preservation on a 0–1 scale by comparing generated reports against reference transcriptions for factual consistency, while clinical relevance evaluates the utility of generated reports for actual clinical decision-making processes. We employed Gemini-2.0-flash for these assessments due to its demonstrated reliability in medical evaluation tasks [36,37], its consistency across large-scale evaluations, and its ability to provide objective, reproducible assessments without the variability and resource constraints associated with human expert evaluation. This automated approach has been validated in numerous studies as providing comparable accuracy to expert assessment while enabling scalable evaluation across diverse medical specialties [27,60].

A crucial aspect of our evaluation, particularly for deployment-oriented benchmarking, is the assessment of clinical quality using Gemini-2.0-flash as the evaluator, as shown in Figure 5. Both GPT-4o and Gemini 1.5 Pro achieved mean clinical relevance scores of 0.944 ± 0.022 within this automated evaluation setup. These scores indicate strong performance within the evaluation setup, but they do not by themselves establish readiness for unsupervised clinical use. Combined with factual accuracy scores of 0.960 ± 0.072 for GPT-4o and 0.958 ± 0.066 for Gemini 1.5 Pro, the results support continued evaluation of these systems as clinician-facing draft-generation tools when embedded within appropriate review and quality-assurance workflows.

The performance of MedGemma provides a useful counterpoint to expectations about domain-specific adaptation. Despite being specifically adapted for medical applications, MedGemma showed higher variability (0.714 ± 0.395 clinical relevance) and lower overall performance than the general-purpose models in this benchmark. This finding suggests that, under the prompting and evaluation conditions used here, larger general-purpose models may currently perform better than this smaller domain-specific alternative for complex medical report generation tasks.

This result highlights an important trade-off. We deliberately selected a very small model (4 billion parameters) that can be run locally on a personal laptop. Local deployment may offer privacy advantages because it can reduce the need to send sensitive information to external cloud providers. However, our findings suggest that while small models are relevant for privacy-sensitive applications, they may not yet match the performance of larger alternatives on all complex medical documentation tasks. This underscores the ongoing challenge of balancing privacy requirements with model performance in the development of clinical AI tools.

5.3. Safety-Oriented Quality Metrics

Because clinically useful draft notes must avoid fabricated content while preserving specialized vocabulary, we also evaluated hallucination score and terminology accuracy across the LLM configurations, as summarized in

Table 4. Safety-oriented quality metrics by LLM model. Higher values indicate fewer hallucinations and more accurate use of medical terminology. The highest values are highlighted in bold.

LLM Model	Hallucination Score	Terminology Accuracy
GPT-4o	0.980 ± 0.076	0.994 ± 0.024
Gemini 1.5 Pro	0.972 ± 0.069	0.993 ± 0.022
MedGemma	0.968 ± 0.068	0.981 ± 0.051
Cross-specialty mismatch control	0.010 ± 0.030	0.276 ± 0.360

. These metrics complement the broader text-quality measures by focusing on two issues that are especially relevant for deployment-oriented benchmarking: whether the generated report stays anchored to the source dialogue and whether medical terms are reproduced appropriately.

The three evaluated LLM pipelines all achieved mean hallucination scores above 0.96 in this benchmark, whereas the lower-bound cross-specialty mismatch control scored 0.010, indicating that downstream report quality depends strongly on upstream transcript quality. GPT-4o and Gemini 1.5 Pro also achieved terminology-accuracy scores above 0.99, with MedGemma remaining high but somewhat lower (0.981 ± 0.051). These results suggest that once the transcript quality is adequate, the stronger LLM configurations generally preserve medical content without adding large amounts of unsupported information under the study conditions.

Pairwise comparisons were directionally consistent with these descriptive differences. For hallucination score, within the evaluated benchmark setting, GPT-4o outperformed MedGemma after Bonferroni correction (exact $p=0.0039$; adjusted $p=0.0117$), whereas the other hallucination comparisons were not significant after correction. For terminology accuracy, none of the pairwise comparisons among the modern LLM models remained significant after multiplicity adjustment. Together, these findings indicate that the main safety-oriented separation in this benchmark is between the modern LLM pipelines and the degraded-input negative control condition, rather than between GPT-4o and Gemini 1.5 Pro.

5.4. Economic Results Under Study Assumptions

The economic dimension of medical AI system deployment is an important factor for real-world feasibility alongside technical performance. Our cost analysis examines modeled cost components associated with different ASR-LLM combinations under the study assumptions.

The pricing structure used in our analysis reflects the published API prices from the respective providers at the time of evaluation (August 2025), as detailed in

Table 5. ASR model pricing as implemented in the experimental code. The highest values are highlighted in bold.

Model	Role	Rate (USD)
Google Cloud STT	ASR	0.078/min
Whisper	ASR	0.006/min
GPT-4o Transcribe	ASR	0.006/min
Voxtral Mini	ASR	0.004/min

and 

Table 6. LLM token pricing used in cost calculations. Input and output prices are for 1 million tokens.

Model	Usage	Input Price ($)	Output Price ($)
GPT-4o	Post-process	2.5	10.0
Gemini 2.0 Flash	Post-process	0.10	0.40
Gemini 1.5 Pro	Evaluation	0.075	0.30

. A notable exception is MedGemma, which with its 4 billion parameters was deployed locally without external API charges, representing a different economic model that trades upfront infrastructure investment for ongoing usage fees. This analysis enables healthcare administrators and clinical informaticists to make informed decisions about system deployment based on both performance requirements and budget constraints.

Our cost analysis encompasses three primary components: ASR transcription prices, LLM inference token prices, and the resulting total AI-processing cost per generated consultation under the study assumptions. Additionally, we calculated efficiency ratios that normalize performance metrics by total cost, providing an internal comparative indicator for contrasting cost-performance trade-offs across different system configurations.

To provide an internal comparison that emphasizes clinical relevance, our efficiency ratio calculation uses a composite performance metric that combines ASR accuracy on medical terminology with LLM semantic understanding. In this study, the efficiency ratio is used as an internal comparative indicator for benchmarking configurations under a common costing assumption. The efficiency ratio is defined as:

$$\mathrm{Efficiency}\mathrm{Ratio}={\displaystyle \frac{\mathrm{Clinical}\mathrm{Performance}\mathrm{Score}}{\mathrm{Total}\mathrm{Cost}}}={\displaystyle \frac{\left(\right(1-\mathrm{Medical}\mathrm{WER})+\mathrm{BERT}\mathrm{F}1)/2}{\mathrm{Total}\mathrm{Cost}}}$$

The efficiency ratio assigns equal weights to Medical WER and BERT F1 for simplicity, though we acknowledge that this choice is study-specific. The Medical WER penalty weight ($\lambda =3.0$) was selected to reflect the higher clinical risk of errors in medical terminology (e.g., drug names, dosages) compared to general vocabulary, but alternative weighting schemes could be explored in future work and adapted to the objectives.

This study-specific approach prioritizes Medical WER over standard WER for ASR evaluation, recognizing that errors in medical terminology (drug names, anatomical terms, clinical procedures) carry disproportionate consequences for patient care. The ASR performance score is calculated as the complement of Medical WER ($1-\mathrm{M-WER}$), representing the proportion of medical terminology correctly transcribed. Combined with ModernBERT F1 scores that capture semantic meaning preservation in LLM-generated reports, this composite metric serves as an internal comparative indicator of benchmark performance per dollar invested, enabling side-by-side comparison of ASR–LLM combinations under the study assumptions.

Figure 6 illustrates the economic analysis across different ASR–LLM combinations, highlighting both absolute costs and efficiency-ratio values for internal comparison across deployment options.

The cost breakdown analysis indicates that transcription-related costs typically dominate the modeled expense structure for most combinations. This suggests that improvements in ASR efficiency may have disproportionate effects on total system economics. GCP’s transcription costs exceed $0.31 per generated consultation under these assumptions, making it economically challenging for large-scale deployment in cost-sensitive healthcare environments. In contrast, Voxtral’s pricing in this benchmark is associated with lower modeled AI-processing cost while maintaining strong technical performance.

Voxtral-based combinations showed favorable cost-effectiveness in this benchmark, with an estimated AI-processing cost of $0.0169 per generated consultation under the study pricing assumptions while maintaining high performance on medical terminology preservation. Under this internal comparative indicator, Voxtral-based combinations yielded efficiency-ratio values of 49.7, compared with 26.6 for ASR (GPT-4o) + LLM (Gemini 1.5 Pro) combinations ($0.0330) and 25.9 for ASR (Whisper) + LLM (Gemini 1.5 Pro) combinations ($0.0330). These differences suggest that the modeled pricing and benchmark accuracy profile of Voxtral may translate into lower AI-processing expenditure, not lower overall consultation cost. Because these estimates depend on vendor pricing, they should be interpreted as pricing-contingent. Under a uniform $\pm 20\%$ variation applied to the August 2025 API prices, the relative efficiency-ratio ranking does not change.

Processing time analysis reveals additional trade-offs that affect both economic and operational considerations. MedGemma combinations achieve the fastest processing times (7–8 s) but with weaker quality metrics across multiple dimensions. Voxtral combinations require moderate processing time (31.7 s) while maintaining stronger quality metrics in this benchmark, suggesting a potentially favorable balance for workflows where both turnaround time and accuracy matter. This processing-time profile suggests that Voxtral-based systems may be compatible with routine clinical workflows, an interpretation that can be read alongside the deployment-grounded evidence presented in later sections.

5.5. Performance Visualizations and Trade-Offs

Figure 7 summarizes ASR model performance across the evaluated dimensions, while Figure 8 complements this analysis with LLM performance comparisons.

The heatmap visualization shows performance patterns across multiple metrics rather than only individual scores. For ASR models, Voxtral performed consistently well across both transcription accuracy measures and downstream quality indicators in this benchmark, suggesting that improved transcription quality may benefit later stages of the documentation pipeline.

The LLM performance heatmap shows similar performance profiles for GPT-4o and Gemini 1.5 Pro across the evaluated medical quality dimensions, while distinguishing these models from both the cross-specialty mismatch control and MedGemma alternatives. This visualization is consistent with the interpretation that, under this benchmark, general-purpose models performed better than the domain-specific alternative for this report-generation task.

5.6. Cross-Language and Medical Specialty Results

Performance across different languages and medical specialties is an important consideration for healthcare systems that serve diverse patient populations. Our evaluation across English and Spanish consultations provides benchmark-level insight into the cross-linguistic stability of different model combinations, while analysis across 10 medical specialties examines performance across the sampled clinical domains.

Cross-language analysis reveals broadly consistent performance patterns across English and Spanish consultations. Voxtral maintained its performance advantage in both languages, achieving WER improvements of 79% for English and 81% for Spanish compared with Whisper in this benchmark. This cross-linguistic consistency is encouraging, but it should be interpreted as benchmark evidence rather than as proof of robust real-world generalization. The slight performance difference between Spanish and English may reflect differences in phonetic complexity or training data characteristics, but the present design does not allow strong inference about the cause.

Medical specialty analysis across cardiology, dermatology, neurology, oncology, and six additional specialties showed consistent performance improvements with Voxtral-based combinations within this benchmark, without clear evidence of specialty-specific degradation. This pattern suggests that the transcription improvements were not limited to a single vocabulary set or conversation type in the sampled synthetic dialogues. However, the present results do not eliminate the need for specialty-specific evaluation in real deployment settings.

The sample distribution of consultations per specialty per language supports the planned benchmark comparisons while maintaining practical constraints on evaluation scope. This balanced design improves coverage of the sampled conditions, but it does not by itself justify confident generalization to routine clinical environments.

5.7. Statistical Validation and Pilot-Stage Feasibility

The statistical foundation of our evaluation supports cautious interpretation of the benchmark results. With 230 total experiments distributed across balanced model configurations ($N=20$ per condition for primary comparisons, $N=10$ for GCP due to language constraints), the study has substantial sample coverage for the planned comparisons. We employed the non-parametric Wilcoxon signed-rank test for paired comparisons of WER and metric scores across identical audio samples, ensuring robustness against non-normal error distributions. To control for family-wise error rates in multiple pairwise comparisons, we applied the Bonferroni correction (${\alpha }_{adj}=0.05/\mathrm{m}$).

Wilcoxon signed-rank tests [61] indicate statistically significant differences between Voxtral and competing ASR solutions ($W=0.0$, exact p-values and Bonferroni-adjusted comparisons in Appendix A.6), with large Wilcoxon effect sizes ($r=0.877$ for the two $n=20$ contrasts and $r=0.886$ for the $n=10$ GCP contrast). These effect sizes suggest that the observed differences are unlikely to be trivial within this benchmark, although direct workflow effects were not measured prospectively.

Taken together, these benchmark results support cautious follow-up in deployment-oriented evaluation but do not establish real-world clinical effectiveness.

The modeled cost profile suggests that Voxtral-based solutions may be economically plausible enough to justify further pilot-stage evaluation. At an estimated $0.0169 in AI-processing cost per generated consultation in this benchmark, the technology appears relatively inexpensive on a per-consultation basis under the study assumptions. However, this is not evidence of reduced physician administrative burden, improved documentation quality in practice, or cost-effectiveness after implementation costs and workflow constraints are considered.

6. Discussion

This evaluation of ASR–LLM combinations for automated clinical documentation provides evidence on the current performance of these systems under the study conditions. Our analysis across 230 experiments spanning 10 medical specialties in English and Spanish suggests potential implementation relevance while also highlighting important constraints for deployment in healthcare settings.

Taken together, the findings provide direct but bounded answers to the three research questions. For RQ1, the controlled benchmark shows measurable differences among ASR–LLM configurations, with Voxtral providing the strongest transcription performance and GPT-4o/Gemini 1.5 Pro providing the strongest report-generation performance under the study conditions. For RQ2, the deployment-grounded traces do not validate any single benchmarked configuration in isolation, but they show that the system was used routinely in real clinical environments and that observed usage was concentrated in documentation-heavy specialties. For RQ3, the combined evidence highlights practical evaluation considerations for future work, including the need to interpret synthetic benchmarks alongside operational data, clinician oversight, workflow impact, editing burden, and prospective clinical validation.

6.1. Key Findings and Clinical Implications

Our evaluation suggests that modern ASR–LLM combinations can achieve performance levels that may be relevant for supervised draft-generation use. In this benchmark, Voxtral achieved the lowest mean speech-recognition error, with a WER of 0.099 ± 0.207 compared with Whisper’s 0.496 ± 0.307. A similar pattern was observed for Medical WER, where Voxtral scored 0.311 ± 0.406 compared with Whisper’s 0.409 ± 0.350. These findings suggest more favorable handling of medically salient terminology under the study conditions, but they should not be interpreted as proof of equivalent performance in prospective clinical settings.

For language model performance, GPT-4o and Gemini 1.5 Pro showed very similar profiles, both achieving ROUGE-L scores of 0.841 ± 0.106 and clinical relevance scores of 0.944 ± 0.022 within our automated evaluation setup. This pattern suggests that the top-performing large language models may be converging for this task, at least within the scope of our benchmark and scoring procedure.

Our Efficiency-Ratio (ER) comparison suggests that Voxtral-based combinations may offer a favorable trade-off between modeled AI-processing cost and benchmark performance in this setting. Voxtral combinations achieve estimated AI-processing costs of 0.0169$ per generated consultation under the study assumptions, with ER values of 49.7 compared with 26.6 for GPT-4o–Gemini 1.5 Pro combinations (0.033$ under the same assumptions). This comparison should be interpreted cautiously because the ratio is intended as an internal benchmarking aid and because economic conclusions remain sensitive to vendor pricing, service availability, and local deployment constraints.

The relatively consistent performance across medical specialties is relevant for future implementation planning. Our analysis across cardiology, dermatology, psychiatry, and other domains showed limited variation in the core performance metrics used here, suggesting that performance was reasonably stable across the sampled specialties within this benchmark. Even so, this interpretation remains bounded by the synthetic-data design and should be validated prospectively before stronger generalization claims are made.

6.2. Multilingual Capabilities and Global Healthcare Implications

The performance observed across English and Spanish suggests potential relevance for multilingual healthcare settings. Voxtral maintained its relative advantage in both languages, achieving WER reductions of 79% for English and 81% for Spanish compared with Whisper in this benchmark. This cross-linguistic consistency is encouraging, but it should be interpreted as evidence from synthetic multilingual dialogues rather than as confirmation of broad real-world generalization.

The multilingual evaluation also provides some support for the usefulness of the synthetic dataset methodology. The consistency of performance patterns across languages suggests that the generated medical dialogues capture at least part of the linguistic and clinical structure relevant to this task, providing a practical basis for future benchmarking work. Studying both English and Spanish allowed us to evaluate the system in the dominant operational languages across the markets examined here, providing a practically relevant multilingual benchmark. Extension to additional languages remains feasible, but further external validation would still be needed before strong generalization claims are made.

6.3. Clinical Safety and Quality Assurance

The hallucination detection and terminology accuracy results summarized in Section 5.3 provide additional information relevant to safety-oriented evaluation. All modern LLM models achieved hallucination scores exceeding 0.96, with GPT-4o and Gemini 1.5 Pro reaching terminology-accuracy scores above 0.99 in the automated framework. These results suggest high internal performance on synthetic data, but they should be interpreted as benchmark indicators rather than as direct evidence of safety in clinical deployment.

Our evaluation included a degraded-input negative control, implemented here as a cross-specialty mismatch control in which semantically incongruent audio was fed to the models to simulate a worst-case scenario with major ASR errors. The markedly lower hallucination score of this control condition (0.010 vs. >0.96 for the modern LLM pipelines) suggests that downstream report quality depends strongly on upstream transcription quality. For implementation, this implies that ASR choice should be treated as an important safety-related design decision, although the direct relationship to patient outcomes was not measured here.

The Gemini-2.0-flash evaluation setup assessed factual accuracy, information coverage, clinical relevance, hallucination detection, and terminology accuracy, providing a practical automated proxy for comparing system outputs at scale. However, this setup introduces a dependency on an external API-based evaluator and its current model behavior, so the resulting scores should be interpreted as heuristic quality indicators rather than as substitutes for clinician review or independent expert adjudication [37].

6.4. Economic and Scalability Considerations

Our benchmark-level cost analysis suggests potential economic advantages for some AI-powered documentation configurations, with Voxtral-based combinations showing favorable modeled cost-effectiveness within the benchmark. With estimated AI-processing costs ranging from 0.0095$ to 0.383$ per generated consultation (mean: 0.062$), these systems appear economically plausible under the study assumptions. Assuming a conservative estimate of 15 min saved per consultation and an average physician hourly rate of 50$, the break-even cost threshold would be approximately 12.50$ per consultation—much higher than the modeled AI system costs reported here. However, our cost analysis focuses on AI-processing costs per consultation and does not account for additional deployment expenses, such as integration with EHR systems, data governance, cybersecurity, local infrastructure, clinician review time, procurement, training, maintenance, monitoring, or liability-related costs. Therefore, the economic conclusions should be interpreted as model-specific and benchmark-level, not as a comprehensive cost-effectiveness assessment for real-world deployment. This narrower benchmark-level view is consistent with broader arguments that the economic value of healthcare AI should be assessed not only in terms of implementation cost but also through longer-term organizational and system-level effects [62].

Voxtral-based combinations show the most favorable economic profile in this benchmark, with an estimated AI-processing cost of 0.0169$ per generated consultation under the study assumptions, compared to 0.033$ for GPT-4o–Gemini 1.5 Pro combinations. The cost breakdown analysis shows that transcription-related costs typically dominate the total modeled expense structure for most combinations, with GCP’s transcription-related cost exceeding 0.31$ per generated consultation under the study assumptions, which may be economically challenging for larger-scale deployment in cost-sensitive healthcare environments.

The emergence of more efficient medical-domain models like MedGemma, while currently showing performance limitations compared to general-purpose models within this benchmark, represents an interesting economic trade-off. MedGemma’s local deployment eliminates ongoing API charges but requires infrastructure investment, achieving faster processing times (7–8 s) at the cost of reduced quality metrics across multiple dimensions.

For large healthcare systems processing thousands of consultations daily, these cost characteristics indicate possible economic headroom under the study assumptions. For example, a 500-consultation-per-day healthcare system would incur an estimated 8.45$ daily in AI-processing cost using the lowest-cost Voxtral combinations in this benchmark. This is a modeled cost illustration, not evidence of realized time savings, return on investment, or operational benefit in practice.

6.5. Methodological Considerations for Medical NLP

Our study uses a modular ASR–LLM architecture that facilitates systematic evaluation of individual system components. This approach enables component-level analysis of performance, cost, and safety-related metrics that would be harder to isolate in monolithic systems. Furthermore, our evaluation setup, using Gemini-2.0-flash as an automated assessor, combines traditional metrics (e.g., WER, ROUGE) with domain-specific measures (e.g., Medical WER, clinical quality scores), providing a broader view of technical performance under the study conditions.

To address the practical issue of data privacy and access in healthcare research, we developed a synthetic data generation methodology using Gemini-2.0-flash. In this hybrid study, the controlled benchmark relies on synthetic dialogues for a specific scientific purpose: it enables controlled, reproducible, component-level comparison while preserving exact ground truth for the full dialogue. This is especially valuable in clinical documentation research, where access to real consultation audio is often heavily restricted and where aligned reference text is needed to measure ASR fidelity, hallucination, and downstream report-generation quality. The consistency of the benchmark patterns across diverse medical scenarios and two languages (English and Spanish) suggests that synthetic data can serve as a useful, privacy-preserving foundation for initial system development and benchmarking under controlled conditions. At the same time, because both the synthetic dialogue generation and parts of the automated evaluation depend on Gemini-based services, the study remains exposed to external model behavior, API availability, and vendor updates. To facilitate further research and reproducibility, the dataset, including both text and audio files, is available upon request.

Just as importantly, the manuscript does not rely on the synthetic benchmark alone. It uses a complementary bench-to-field evidence framework: the controlled benchmark provides measurement of technical accuracy under known ground truth, while the deployment-grounded evidence provides operational evidence from routine use. Across a 120-day study period, the deployment-grounded evidence captured 3107 finalized clinical traces from 301 active users, corresponding to 29,535 min (~492 h) of authentic doctor–patient interaction across LATAM and Europe [5,14]. Although these data do not establish prospective clinical outcomes, they do show that the system was used in real clinical environments at a scale that would be difficult to capture in a small prospective pilot.

The comparable performance of general-purpose LLMs (GPT-4o and Gemini 1.5 Pro) relative to the domain-adapted alternative (MedGemma) complicates conventional assumptions about the necessity of medical domain adaptation for clinical NLP, while the present results suggest that scale and training diversity may currently offer an advantage for complex medical report-generation tasks, smaller domain-specific models remain important, especially for local deployment and privacy-sensitive settings. The performance variability observed in some MedGemma-generated reports may be addressable through further fine-tuning, such as adjusting temperature or modifying the structured output format. Accordingly, the current performance gap should be interpreted as a benchmark finding, not as evidence that smaller privacy-preserving models are unlikely to become competitive.

The evaluation also yields several observations that may inform clinical implementation. Within this benchmark setting, Voxtral and GPT-4o Transcribe outperformed Whisper, suggesting that more recent ASR systems may offer measurable gains in medical documentation tasks. The cost-effectiveness analysis indicates that some of these technical differences may also matter economically, with Voxtral-based combinations showing a 1.9× improvement in cost-effectiveness over selected alternatives under the assumptions used here. Even so, these findings should be interpreted with caution because both performance and cost may change as models, APIs, and local deployment constraints evolve.

7. Real-World Context and Implications

Unlike the controlled benchmark presented in Section 5, this section examines deployment-grounded operational observations derived from routine clinician-supervised use, providing descriptive evidence of operational use and implementation patterns rather than direct evaluation of the benchmarked ASR–LLM configurations or measures of clinical effectiveness.

This section situates the paper as a whole within its broader clinical and implementation context. Within the study’s bench-to-field evidence framework, the controlled benchmark defines technical performance under a known ground truth, whereas the deployment-grounded evidence shows how the same system was used in routine practice under uncontrolled conditions. A central motivation of this study was to examine whether technical benchmark patterns and operational implementation patterns provide consistent or complementary insights. Together, these complementary perspectives provide a broader understanding of how technical performance translates into real-world implementation while identifying priorities for future prospective clinical evaluation. Finalized exports can therefore be read as a bounded behavioral proxy for perceived usefulness: when clinicians export and finalize an AI-generated draft, they have judged it useful enough to refine and incorporate into routine documentation. That does not establish clinical correctness, safety, or efficiency, but it does add operational evidence beyond the synthetic benchmark and avoids some of the recall limitations of self-reported workflow surveys [5,13,14,63]. The subsections that follow move from clinical motivation and evidence gaps to analytical framing, the study’s specific contribution, observed usage patterns, and an overall interpretation of what these findings do and do not support.

7.1. Clinical Relevance and Motivation

Clinical documentation burden remains a significant contributor to clinician workload and burnout, and ambient-AI documentation tools are increasingly proposed as a potential response [3,5,13]. However, empirical evidence remains mixed, while some studies report perceived efficiency gains, systematic reviews continue to highlight limited and heterogeneous evidence regarding effects on documentation quality, workflow efficiency, burnout, and system-level outcomes [13,14,18].

Recent multisite evidence further tempers expectations. A study published in JAMA on 1 April 2026, reported modest reductions in EHR time (13.4 min) and documentation time (16.0 min) per 8 scheduled patient hours, small productivity gains, and no significant change in after-hours “pajama time” across five U.S. health systems [48]. Taken together, these findings suggest that the central question is not whether AI scribes are promising in principle, but how they should be evaluated and integrated cautiously in practice.

7.2. Evidence Gaps in Current Literature

One reason this question remains open is that existing evidence is often fragmented. Many studies emphasize controlled technical performance or self-reported usability, while fewer examine how these systems are used over time in routine clinical environments. This distinction is important because technical performance does not necessarily translate into sustained workflow value.

For example, a Dutch digital-scribe study (22 participants, 430 mock-consultation summaries) found that editing AI-generated drafts was modestly faster than fully manual summarization, but unedited automatic notes had substantially lower documentation quality and less diverse language despite higher word counts [7]. This suggests that any productivity gain depends on clinician editing, quality control, and workflow integration rather than autonomous note finalization.

7.3. Analytical Framework

This section connects controlled benchmarking with observed routine use. Rather than treating technical evaluation and field behavior as interchangeable, we consider them as complementary evidence streams. The controlled benchmark assesses ASR and LLM performance under controlled conditions with preserved ground truth. The deployment-grounded evidence examines how the system was used in routine deployment through retrospective log-mining when real-world consultations are noisy, heterogeneous, and not amenable to gold-standard transcript alignment.

The real-world traces were generated using a production ASR–LLM pipeline that evolved during the 120-day observation period, while the benchmark evaluates specific fixed configurations (e.g., GPT-4o, Gemini 1.5 Pro, MedGemma), the deployed system may include configuration updates, prompt refinements, or model substitutions over time. Therefore, the real-world evidence should be interpreted as complementary operational evidence rather than a direct evaluation of any single benchmarked configuration. Methodologically, digital trace analysis provides a non-interruptive alternative to recall-based survey instruments by relying on timestamped interaction data generated in situ rather than post hoc self-report [5,14]. This perspective linking benchmark results to field use is particularly relevant for systems intended to operate under human oversight, where technical performance and workflow integration jointly determine practical utility.

7.4. Contribution of the Present Study

Building upon recent ambient clinical documentation evaluation studies [15,17], this study is positioned as an early-stage evaluation rather than a purely methodological advance, while individual metrics such as Medical WER and the use of synthetic data generation have been explored previously [46], our primary contribution lies in a bench-to-field evidence framework: a controlled, multi-specialty synthetic benchmark is paired with real-world, human-in-the-loop deployment trace analysis to support research tracking, implementation insight, and interpretation of system use beyond laboratory conditions. These two evidence streams answer different questions. The controlled benchmark evaluates technical performance under standardized conditions, whereas the deployment traces describe operational use in routine clinical settings. Accordingly, they should be interpreted as complementary rather than interchangeable forms of evidence.

Under this framing, the primary contribution of the paper is the explicit combination of two complementary evidence streams within one study. The controlled benchmark is a modular ASR–LLM evaluation based on multilingual synthetic clinical dialogues, which makes exact measurement of WER, Medical WER, semantic preservation, and safety-oriented LLM errors possible. The deployment-grounded evidence stream is an analysis of 3107 finalized traces from LATAM and Europe, which adds complementary operational evidence that the system was used under routine, uncontrolled conditions.

Accordingly, the study should not be read as asking one evidence stream to replace the other. The synthetic benchmark addresses technical validity where private real-world data cannot provide exact ground truth, whereas the deployment-grounded evidence addresses ecological validity by showing sustained use under routine conditions. Together they provide a more informative hybrid evidence base than either a synthetic benchmark or an observational usage study alone, while still stopping short of claiming prospective clinical outcome validation [5,13,14,18,48].

7.5. Observed Usage Patterns During the 120-Day Deployment Period (November 2025–February 2026)

To complement the experimental benchmarks, we conducted a retrospective clinical log-mining analysis of routine system use. Because it relies on digital trace data captured during actual deployments rather than self-reported surveys, this analysis reflects observed physician behavior with minimal workflow disruption. It therefore provides evidence of operational use rather than prospective clinical outcomes [5,14]. It should also be read as product-level observational evidence, not as a direct comparison of real-world performance for a single fixed ASR–LLM configuration over the full study window.

Although the deployment analysis primarily focused on operational usage patterns rather than direct measurement of documentation quality, the broader motivation for this work is the generation of trustworthy clinical data. The value of AI-assisted documentation extends beyond note generation itself, as the resulting information may subsequently support clinical workflows, analytics, interoperability initiatives, quality-improvement activities, and future AI-enabled healthcare applications. Consequently, the quality, consistency, and trustworthiness of generated clinical information remain important considerations when evaluating AI-assisted documentation systems.

This perspective motivated our emphasis on transcription accuracy, medical terminology preservation, hallucination rates, and report quality within the benchmark component of the study, while modern AI systems can generate highly fluent documentation, fluency alone should not be interpreted as correctness. Large language models may occasionally produce outputs that appear plausible yet contain inaccuracies, reinforcing the importance of human review and appropriate governance mechanisms within clinical workflows.

The unit of analysis was the clinical trace, defined as a finalized and exported report. Between November 2025 and February 2026, we recorded 3107 traces across 120 days, corresponding to 29,535 min (~492 h) of authentic doctor–patient interaction. These traces were generated by 301 active users across Latin America (LATAM) and Europe. LATAM accounted for 2314 reports (23,190 min) and Europe for 793 reports (6345 min). Mean consultation duration was 10.02 min in LATAM and 8.00 min in Europe; in LATAM, the 75th percentile was 11.56 min, consistent with use during full consultations rather than only short tests. Usage distributions were also observed to be skewed across users in certain specialties, suggesting concentration effects typical of early-stage deployment [5,14].

These data should be interpreted as observational evidence of operational use, not as direct evidence of clinical effectiveness. They do not establish prospective clinical outcomes or exposure-adjusted adoption rates. Finalized exports are a proxy for perceived usefulness in routine practice, but they do not directly measure documentation quality, patient safety, time savings, or user satisfaction [5,13,14].

Across the observation period, the deployment logs showed continued operational use among active deployment sites, with 3107 finalized reports and 29,535 min of doctor–patient interaction processed. In the operational usage analysis, LATAM generated almost three times as many reports as Europe (2314 vs. 793), while the interpretation of broader uptake emphasized activation and habit formation rather than acquisition alone.

While this operational use should not be interpreted as evidence of adoption rates or clinical effectiveness, it provides complementary evidence that users incorporated the system into routine documentation workflows. We interpret these observations cautiously as indicators of perceived utility among active users rather than definitive measures of impact.

Importantly, clinician review remained part of the workflow throughout the observation period. The system was designed as a clinician-supervised documentation support tool rather than an autonomous documentation solution. This human-in-the-loop approach was considered essential given that highly capable AI systems may occasionally generate plausible but inaccurate outputs.

Table 7. Global in-field usage summary during the 120-day study period (November 2025–February 2026).

Region	Active Users (≥1 Report)	Reports Generated	Total Minutes Processed	Mean Minutes Per Report
LATAM	205	2314	23,190	10.02
Europe	96	793	6345	8.00
Total	301	3107	29,535	—

summarizes the global in-field usage patterns,

Table 8. Specialty-level usage highlights in LATAM.

Specialty	Active Users	Reports Generated	Total Minutes Processed
Psychology	14	120	3345
Traumatology & Orthopedics	15	161	2458
Psychiatry	7	70	1693
Cardiology (Electrophysiology) *	1	311	2482

* Single-user concentration effect; interpret as high-intensity individual use rather than broader specialty-level usage.

highlights the main specialty-level usage patterns observed in LATAM, and Figure 9 provides a visual summary of specialty-level usage intensity in LATAM.

7.5.1. Specialty-Level Usage Patterns

Usage was not evenly distributed across specialties, but concentrated in areas with distinct documentation demands. In LATAM, the highest activity was observed in Psychology (3345 min; 120 reports; 14 users), Traumatology & Orthopedics (2458 min; 161 reports; 15 users), and Psychiatry (1693 min; 70 reports; 7 users). These patterns are consistent with two possible task–technology fits: narrative synthesis settings (e.g., Psychology and Psychiatry), where long-form dialogue benefits from structured summarization, and motor-load-constrained settings (e.g., Traumatology & Orthopedics), where hands-free documentation may reduce workflow friction. These interpretations remain tentative because exposure, rollout strategy, and specialty-level user distribution were not controlled for [5,14].

7.5.2. Concentration Effects and High-Intensity Usage

A notable concentration effect appeared in Cardiology (Electrophysiology), where a single user generated 311 reports, or approximately 10% of total LATAM report volume. This is best interpreted as high-intensity individual use, suggesting that the system can become deeply integrated into a specific clinician’s workflow. However, it should not be treated as representative of broader specialty-level usage patterns without additional exposure and user-distribution data [5,14].

7.5.3. Sensitivity Analysis and Observational Bias

To assess potential selection and concentration effects, additional analyses were conducted at the user level. Usage distributions were found to be skewed, with certain specialties influenced by high-intensity users. For example, excluding the highest-contributing user in Cardiology reduced Cardiology’s observed share of LATAM exports from 13.4% to 0%, indicating strong dependence on a single high-intensity user and highlighting the potential for concentration effects in early-stage deployment. These analyses suggest that some specialty-level patterns are partially influenced by concentration effects. However, because exposure-adjusted denominators (e.g., total eligible consultations or clinicians with access) were not available, these findings should be interpreted as descriptive usage patterns rather than adoption rates.

7.5.4. Interpretation of Finalized Exports as a Behavioral Signal

Although we did not quantify editing effort (e.g., edit distance between draft and final report), the 3107 finalized exports still provide a behavioral signal of continued use. A finalized report can therefore be read as a pragmatic indicator of perceived utility: the draft was useful enough to refine and integrate into documentation. However, this should not be interpreted as evidence of clinical correctness or completeness, since all outputs remained subject to physician verification and editing [5,14,63].

7.6. Summary Interpretation and Human-in-the-Loop Framing

Overall, the log-mining analysis adds an important but bounded source of evidence. It shows sustained operational use under routine conditions, with activity concentrated in specific specialties and, in some cases, among high-frequency users. It does not, by itself, demonstrate clinical quality, safety, time savings, burnout reduction, or adoption rates. The findings should therefore be read as descriptive evidence of observed usage patterns rather than definitive evidence of clinical effectiveness or workflow efficiency [13,14].

Recent work has emphasized the importance of human-centered or “human-allied” AI systems, in which the clinician remains the primary decision-maker and AI functions as a supporting component rather than an autonomous agent. In this context, our findings are most appropriately interpreted within a human-in-the-loop framework. The persistent need to verify generated reports, the variability observed across model configurations, and the observational nature of the deployment evidence all support viewing the present pipeline as a clinician-facing documentation tool rather than an autonomous system. This interpretation is consistent with positioning the system as a draft-generation tool integrated into existing workflows under human oversight [7,11,12,13,14].

More broadly, the findings highlight that trustworthy AI-assisted documentation depends not only on model performance but also on the quality of the data being generated, stored, and made available for later use. As healthcare systems increasingly rely on structured digital information for interoperability, analytics, decision support, organizational learning, and health-system governance, the generation of reliable, reviewable, and well-governed clinical data becomes increasingly important. Future research should therefore evaluate AI-assisted documentation systems not only as note-generation tools, but also as components of a broader healthcare data infrastructure. Prior work on converting clinical reports into FHIR bundles shows that this downstream structuring task is feasible with LLMs but remains sensitive to prompt design, iterative correction, hallucination control, and resource-mapping accuracy, reinforcing the need to treat documentation quality and data normalization as connected problems [27,64,65,66]. This perspective is consistent with the need to connect AI-generated documentation to high-quality data foundations, including interoperable representations such as FHIR, clinically meaningful terminologies such as SNOMED CT, robust storage and collection practices, and governance processes that support trustworthy downstream analysis. Accordingly, we do not interpret operational use as evidence of clinical impact on its own; rather, we view strong data quality and infrastructure as prerequisites for producing reliable information from which higher-level AI-enabled clinical, operational, or organizational value can later be assessed.

8. Future Work and Limitations

This study should be interpreted as an early, pilot-stage evaluation rather than as a definitive implementation study. It combines a controlled benchmark with observational usage traces, but it does not establish prospective clinical effectiveness, validated workflow benefit, or readiness for autonomous deployment.

Rather than proposing a validated implementation framework or positioning a “playbook” as a central contribution, we limit the practical implications of the present study to three cautious points. First, any adoption effort should begin with controlled pilots in clearly defined local settings before broader rollout is considered. Second, clinical verification should remain mandatory, with clinicians retaining responsibility for reviewing, correcting, and approving generated documentation. Third, deployment should be adapted to specialty-specific documentation needs and to the realities of local workflow integration, since operational value is likely to depend on context rather than on a single uniform model of use. If any thresholds, staging logic, or rollout phases are inferred from these findings, they should be interpreted only as indicative, local, and not universally validated [12,62].

8.1. Future Research Directions

The future-work section should follow the same logic as the rest of the discussion: brief, specific, and aligned with what this study can actually support.

8.1.1. Technical Robustness and Real-World Validation

Future work should test the pipeline under more realistic clinical conditions, including diverse speakers, accents, recording devices, background noise, and longer consultations. This would clarify how acoustic variability affects both ASR performance and downstream report quality. The next step should also be prospective validation in routine care, with direct measures such as physician satisfaction, documentation quality, workflow fit, and unintended effects on clinical practice.

8.1.2. Advanced Model Development

Model development should focus on smaller and more specialized clinical models that can be fine-tuned for documentation tasks while remaining feasible for local, privacy-sensitive deployment. Our results suggest that domain-specific models still lag behind the strongest general-purpose systems within this benchmark setting, but they remain promising because they may offer better controllability, lower cost, and easier secure deployment. Another clear direction is multimodal documentation support that combines speech with structured EHR fields, physiological data, or relevant visual context.

8.1.3. Voice as a Clinical Signal Beyond Transcription

In the present study, speech is used mainly as a source of text for documentation. This is appropriate for the goal of the paper. However, speech also contains acoustic information that may have clinical relevance. Previous work has discussed vocal biomarkers for diagnosis, risk prediction, and remote monitoring [67], and recent reviews show that speech-signal analysis may support clinical decision-making in neurological and mental-health contexts [68]. In depression research, Vilenchik et al. showed that the way speech data are collected can affect the performance and generalization of voice-based detection models [69]. This is relevant because AI-scribe systems already capture clinical conversations. Future systems could therefore evolve from documentation tools into multimodal clinical assistants that support documentation, screening, and longitudinal monitoring, but only under explicit consent, appropriate governance, and rigorous clinical validation. This is only a future direction. The present study does not analyze vocal biomarkers and does not support diagnostic claims from voice.

8.1.4. Multilingual and Cross-Cultural Expansion

Future research should extend the benchmark to more languages, specialties, and healthcare settings. This is important because documentation structure, consultation style, and clinically relevant information vary across specialties and across health systems. Cross-cultural adaptation should therefore include local terminology, documentation norms, and mixed-language or code-switching scenarios, especially in multilingual regions.

8.2. Limitations and Methodological Considerations

This subsection summarizes the main constraints on both the benchmark and the deployment-based evidence, and therefore on the claims this study can support.

8.2.1. Dataset, Evaluation, and Generalizability Limits

The benchmark has three main limits. First, the synthetic dialogues support controlled and reproducible comparison, but they cannot fully capture the complexity of real-world clinical consultations, including spontaneous speech, interruptions, overlapping speakers, diverse accents, emotional variation, incomplete histories, complex comorbidity, difficult communication, ambient noise, and device-related recording variation [44,70,71]. Because these features are likely under-represented in TTS-generated dialogues, the benchmark may overestimate ASR and report-generation performance in actual clinical environments. Moreover, no direct comparison between synthetic and real-world dialogues was conducted to quantify this gap. The benchmark should therefore be treated as a reproducible test bed for component-level comparison, not as a substitute for prospective validation in real care; the large effect sizes reported here (e.g., $r\approx 0.88$) are specific to the controlled benchmark conditions and should not be assumed to generalize to noisy routine settings without further validation.

Second, automated evaluation cannot fully replace expert clinical review. Although Gemini-2.0-flash provides a standardized way to compare many model combinations, clinicians may still detect subtle errors, omissions, or contextual problems that automated metrics miss. The use of Gemini models for synthetic dialogue generation, one evaluated report-generation model, and automated scoring also introduces a potential self-preference or intra-family bias. Prior LLM-as-a-judge work supports automated judging as a scalable approximation to human evaluation but also documents systematic judge biases, and more recent studies show that models may recognize and favor their own or same-family generations [54,55,56]. Consequently, the favorable performance of Gemini 1.5 Pro may be slightly inflated by the Gemini-2.0-flash evaluator, even though the evaluated generator and judge were not identical models. Blinded physician review and, ideally, adjudication by evaluators from independent model families should therefore complement automated assessment in future work [72].

Third, the benchmark is limited in both linguistic and temporal generalizability. Results in English and Spanish may not transfer to other languages, dialects, or healthcare systems [73]. In addition, the API-based experiments were run from 5 August to 15 August 2025, so model behavior, pricing, and service quality may change over time [74]. Additionally, the reliance on cloud-based models (e.g., GPT-4o, Gemini 1.5 Pro) introduces reproducibility challenges, as their underlying versions and behaviors may evolve without public transparency, limiting the long-term durability of benchmark rankings. The benchmark should therefore be interpreted as a time-bounded comparative snapshot rather than a fixed statement of future performance.

8.2.2. Operational, Data, and Implementation Constraints

The observational results in Section 7 should likewise be read as complementary evidence of operational use, not as exposure-adjusted measures of adoption. We did not have denominators such as the number of eligible clinicians, eligible consultations, or potential opportunities for use. We also lacked direct measures of editing effort, recurrence depth, and workflow impact. Furthermore, the benchmark’s reliance on clean, TTS-generated audio means that the observed performance—including the large effect sizes—may not reflect real-world conditions, where factors such as background noise, poor microphone quality, or overlapping speech could degrade ASR and LLM performance. For that reason, specialty-level differences are better interpreted as observed usage patterns than as direct measures of penetration or preference [13,14].

Operational logs also show only part of system value. They can reveal repeated use and specialty-level heterogeneity, but they cannot explain why clinicians returned, which features mattered most, or whether repeated use improved documentation quality. Some specialty patterns were also shaped by high-intensity users, rollout effects, training, or local workflow constraints rather than product performance alone.

The present study did not evaluate editing frequency, correction rates, note acceptance rates, clinician satisfaction, workflow outcomes, or patient outcomes. As a result, operational use cannot be interpreted as direct evidence of documentation quality improvements, efficiency gains, or clinical effectiveness. Future studies should incorporate these measures to better understand the relationship between operational use and real-world value, including whether clinicians accepted AI-generated notes with minimal correction or substantially revised them before finalization.

The real-world analysis is also vulnerable to selection and measurement bias. Participating sites had already chosen to trial AI-assisted documentation, so they may over-represent digitally mature settings with stronger implementation support. Specialty labels may also be incomplete or inconsistently assigned across deployments. In addition, because the study evaluates the authors’ own product, the implementation findings require extra caution even though the benchmark comparisons were standardized across multiple ASR and LLM configurations.

Finally, neither the benchmark nor the telemetry establishes clinical effectiveness. The real-world usage analysis provides a complementary observational evidence stream, but it does not directly measure routine-care documentation accuracy, time savings, note completeness, after-hours charting, clinician workload, user satisfaction, or patient outcomes. These outcomes would require a dedicated user-centered and prospective evaluation, such as clinical trials, structured physician surveys, or in-depth workflow studies. Future research should also examine potential unintended consequences of prolonged reliance on AI-assisted documentation systems, including changes in documentation practices, training, clinical reasoning, and possible deskilling effects.

9. Conclusions

This study provides an early-stage evaluation of AI-assisted clinical documentation systems through the combination of controlled benchmarking and deployment-grounded operational observations. The benchmark results identified meaningful differences across ASR and LLM configurations, demonstrating important trade-offs between transcription accuracy, terminology preservation, report quality, hallucination rates, and processing costs. The operational traces provide complementary descriptive evidence that the system was incorporated into routine documentation workflows across multiple healthcare settings, particularly within documentation-intensive specialties.

These findings support continued supervised pilot evaluation of AI-assisted documentation technologies. However, they should not be interpreted as evidence of clinical safety, clinical effectiveness, documentation time savings, adoption rates, workflow impact, or readiness for autonomous deployment. The present study provides benchmark and operational evidence rather than clinical validation.

Accordingly, AI-assisted documentation systems should currently be understood as clinician-facing draft-generation tools that require human review, appropriate governance, workflow integration, and prospective clinical evaluation. Future research should extend beyond technical performance toward broader assessment of interoperability, clinician burden, workflow outcomes, patient experience, economic value, and health-system impact.