Deep Learning in Medical Speech to Text: Methods and Challenges

Abstract

Automated clinical documentation based on clinician-patient conversations is an emerging application of deep learning, driven by advances in medical speech recognition and natural language processing. Despite technological progress, real-world adoption remains limited. This review analyzes deep learning–based medical speech-to-text systems, focusing on methodologies, evaluation strategies, and barriers to clinical implementation. A systematic review of 31 studies was conducted, covering automatic speech recognition, clinical dialogue processing, and large language model-based documentation pipelines. Speech recognition accuracy varies considerably in noisy, multi-speaker, and spontaneous clinical environments. Downstream tasks such as entity extraction and summarization are highly sensitive to transcription errors and constrained by limited real-world datasets. Most systems lack external clinical validation and are tested in controlled settings. Key challenges include speaker diarization, domain adaptation, privacy protection, and the need for standardized evaluation frameworks. Although LLMs demonstrate strong potential, concerns remain regarding hallucinations and factual reliability, necessitating improved robustness and clinician oversight.

1. Introduction

Clinical documentation refers to the systematic process of recording clinical observations, interventions, and services related to patient care, encompassing all patient-related records that document operational activities performed by healthcare professionals. Its principal functions include facilitating the communication of relevant clinical information, supporting regulatory and legal compliance, and providing structured data for clinical decision support systems [1]. Unsurprisingly, capturing clinical documentation consumes a substantial portion of a physician’s workday, a burden strongly associated with professional burnout and reduced quality of care [2]. This administrative overhead has been recognized as a structural challenge, prompting efforts to streamline workflows through more efficient digital infrastructures [3].

Recent breakthroughs in automatic speech recognition (ASR) and large language models (LLMs) have enabled the development of “digital scribe” systems that automate the creation of medical notes directly from clinician-patient conversations. These frameworks aim to transform ambient audio into structured documentation, such as SOAP notes, to preserve clinical accuracy while reducing cognitive load (Figure 1). Hence, digital scribes are perceived as vital tools for supporting clinical reasoning and operational efficiency [4,5,6].

Figure 1. End-to-end conceptual flow of an automated clinical documentation pipeline. The architecture illustrates the transition from raw acoustic signals to ASR, phased information extraction, and LLM-based drafting. Importantly, the architecture incorporates the “Medium Automation” paradigm, which requires explicit human verification before EHR integration to mitigate clinical risks.

Despite evident progress, creating reliable end-to-end pipelines remains challenging. Spontaneous clinical dialogues often involve multiple speakers, overlapping speech, and substantial environmental noise [7]. These factors negatively affect the robustness of automatic speech recognition (ASR) and downstream clinical tasks [8,9]. Furthermore, empirical evidence suggests a substantial “information sparsity problem,” with less than 20% of clinical conversation content typically contributing to the final medical summary [10]. Progress is also hindered by the limited availability of real-world datasets due to privacy regulations and HIPAA-related restrictions. In turn, many studies to rely on synthetic or role-played clinical encounters [11,12,13].

However, our knowledge of the current state-of-the-art solutions that leverage both ASR and LLMs appears to be fragmented and insufficiently synthesized across the existing literature. To address this gap, this systematic literature review synthesizes 31 studies covering medical ASR, clinical dialogue processing, and LLM-based documentation. The objective is to identify current methodologies, evaluate system performance, and highlight the remaining gaps that must be addressed for the safe deployment of automated tools in clinical practice. By consolidating diverse evidence, this review outlines the current technological landscape and the challenges that digital scribes must overcome to become trustworthy components of modern clinical workflows.

The remainder of this paper is organized as follows. Section 2 describes the PRISMA-based systematic review methodology, complemented by a targeted manual search for recent architectures. Section 3 presents the findings concerning deep learning applications in medical Speech-to-Text systems, including challenges related to ASR error propagation, Large Language Models hallucinations, privacy regulations, and data heterogeneity. Section 4 discusses the broader implications of the findings, focusing on barriers to clinical deployment, governance concerns, and future research directions, such as federated learning, de-tokenization mechanisms, multimodal Vision–Language Models, and knowledge distillation approaches. Section 5 concludes the paper and outlines the role of LLMs in advancing clinical documentation systems.

2. Research Methodology

This study follows a structured systematic literature review (SLR) to identify current methods, challenges, and research gaps in automating clinical documentation from clinician-patient audio recordings. The review focuses on technologies spanning medical automatic speech recognition (ASR), clinical dialogue processing, and large language model (LLM)-based information extraction.

2.1. Research Questions

The SLR is guided by four research questions (RQs):

RQ1:

What techniques are used for medical speech-to-text in clinical environments?

RQ2:

How do current systems extract structured information from clinical dialogues?

RQ3:

What challenges exist in applying LLMs to medical documentation?

RQ4:

What gaps remain in automating structured medical note generation?

2.2. Search Strategy

A systematic literature search was conducted across five major bibliographic databases: PubMed, IEEE Xplore, Scopus, Springer, and Web of Science. The search was limited to English-language publications released between 2020 and 2026 and focused on studies related to automatic speech recognition, clinical dialogue processing, electronic health records, and large language models. All searches were completed on 6 February 2026.

The detailed step-by-step selection process, including the exact number of records retrieved from each database, the database-specific filtering criteria, and the deduplication process, is visually illustrated in the PRISMA flowchart (see Figure 2). Notably, as shown in the diagram, the open-access database (PubMed) yielded the vast majority of the initial results compared to commercial databases. This highlights the crucial role of open-access repositories in rapidly sharing knowledge at the intersection of clinical medicine and artificial intelligence, ensuring high accessibility and reproducibility of research.

Figure 2. PRISMA flow diagram detailing the literature search, screening, and selection process.

2.3. Search Query Definition

To identify relevant literature, the search strategy utilized specific combinations of the following keywords and terms found within the title, abstract, or keywords of the articles:

• Speech & Transcription:“speech recognition”, “speech-to-text”, ASR, “audio transcription”, “speech processing”.
• Clinical Context: “clinical conversation”, “medical dialogue”, “doctor patient”, consultation, encounter.
• Documentation: “clinical documentation”, “progress note”, “electronic health record”, EHR.
• AI & NLP: “large language model”, LLM, “clinical NLP”, “information extraction”, “named entity recognition”.

2.4. Inclusion and Exclusion Criteria

Studies were included if they met the following inclusion criteria (IC):

IC1:

Published between 2020 and 2026.

IC2:

Written in English. Importantly, this restriction applied exclusively to the language of the manuscript; studies analyzing non-English, native, or multilingual doctor-patient conversations were fully eligible for inclusion.

IC3:

Classified as peer-reviewed journal/conference papers or preprints.

IC4:

Belonging to the fields of computer science, medicine, health professions, or engineering.

IC5:

Addressing clinical documentation, medical ASR, clinical dialogue processing, or LLM-based information extraction.

In the next stage, studies were evaluated based on the following exclusion criteria (EC):

EC1:

Lacked speech recognition or clinical NLP components.

EC2:

Did not relate to clinical workflows or medical note generation.

EC3:

Provided no methodological or empirical contribution.

EC4:

Were purely opinion-based with no technical or clinical relevance.

Following the application of the inclusion and exclusion criteria, the remaining studies underwent a multi-stage screening process, which is summarized in the flowchart (Figure 2). This process included title screening, abstract screening, and full-text review. As a result of this sequential filtering, a final set of 28 studies was retained for qualitative synthesis and analysis.

The authors conducted manual screening and eligibility assessments through title, abstract, and full-text reviews. Any disagreements about study inclusion were resolved through discussion and consensus. Due to the methodological heterogeneity of the included studies, no formal quantitative quality scoring or meta-analysis was performed.

2.5. Snowballing and Supplementary Search

Following the database search, a substantial number of studies meeting the predefined inclusion and exclusion criteria were identified. As the initial search strategy yielded sufficient and comprehensive coverage of the primary literature, no formal backward or forward snowballing procedures were deemed necessary. However, to ensure the review remains fully contextualized within the most cutting-edge advancements, a supplementary manual search was conducted during the peer-review process. This purposive sampling identified recent comprehensive reviews and pioneering frameworks concerning next-generation AI architectures and privacy-preserving approaches (specifically Vision–Language Models, Knowledge Distillation, and Federated Learning). These additional articles were incorporated explicitly to enrich the discussion on future technological trajectories and secure AI deployment, without altering the primary dataset of 28 studies evaluated in the systematic review.

2.6. Data Extraction

To ensure consistency across the reviewed studies and to directly address the varying performance of automated systems, a structured data extraction process was applied. Information from each study was systematically organized into four analytical categories:

• Authors & Year:The authors and publication year of the study.
• Data Source: The specific datasets utilized, including whether the data consisted of real-world clinical audio, simulated/mock transcripts, online medical forums, or private hospital records.
• Technological Focus: The primary computational components, such as the specific LLM pipeline, ASR system, summarization models (e.g., GPT-4 vs. BioBART), or prompting strategies.
• Evaluation & Validation: The metrics used to assess the models, encompassing quantitative scores (e.g., ROUGE, BLEU, BERTScore) and clinical/human validation instruments (e.g., PDQI-9, SME preference, error counts).

3. Results

The 31 included studies reveal an evolving landscape of technologies and approaches aimed at automating clinical documentation from clinician-patient dialogues. To provide a structured and comprehensive overview, the studies have been grouped by their primary technological focus. Table 1, Table 2, Table 3 and Table 4 present the consolidated data extraction covering Clinical Note Generation, Information Extraction & ASR, Evaluation & Usability, and Methodological Innovations, respectively.

Table 1. Overview of included studies for Clinical Note and Report Generation.

Authors & Year	Data Source	Technological Focus	Evaluation & Validation
Leong et al. (2024) [14]	ACI-BENCH (Simulated)	MediNotes: LLaMA3-8B + RAG vs. GPT-4o.	ROUGE, BERTScore, Clinical review
Neupane et al. (2024) [15]	Simulated dialogues, FigShare, MTS-Dialog	CLINICSUM: Retrieval filtering + LLaMA-3.	ROUGE, BERTScore, SME preference
Haniff et al. (2025) [16]	Mock GP Videos	GPT-4 vs. BioBART, Flan-T5, BART.	Modified Q-note score
Yim et al. (2023) [17]	ACI-BENCH (Simulated)	Division-based generation; BART, LED, GPT-4.	ROUGE, MEDCON
Kernberg et al. (2024) [18]	Simulated ambulatory	Zero-shot ChatGPT-4 (SOAP generation).	PDQI-9, Error counts
Sezgin et al. (2024) [19]	ED referral calls	AWS Transcribe + BART-Large-CNN.	ROUGE, LINK/CORRECT
Ma et al. (2025) [20]	Real hospital EMR	MedStructGen: Two-stage LLM pipeline.	BLEU, BERTScore, Structural compliance
Lu et al. (2023) [21]	IMCS-MRG, MC	Factual Aware Two-Stage Model (REINFORCE).	ROUGE, Terminology Matching Rate (TMR)
Kumar et al. (2024) [22]	IMCS21 dataset	BERT-CRF Entity Extraction + BART + Adapters.	ROUGE, Algorithm Complexity
Abanoub et al. (2023) [23]	HealthCareMagic	Two-layer transformer pipeline (BART + T5).	ROUGE
Duan et al. (2023) [24]	IMCS21, IMCS-V2	MVP: Multi-view prompt learning with GPT-2.	ROUGE, ROUGE-Sal, Prompt evaluation
Duan et al. (2023) [25]	IMCS21, IMCS-V2	TS-CA: Two-stage compacting-abstractive.	ROUGE, ROUGE-Sal
Li & Yao (2025) [26]	CMDD, MIMIC-III	BART + Pointer Network + CMCH Dictionary.	ROUGE-F1, Concept Recall/Precision
Tsai et al. (2022) [27]	Outpatient visits (Private)	Dialogue Segmentation (BERT) + mT5 transfer.	ROUGE, BLEU, BERTScore
Nguyen et al. (2023) [28]	MEDIQA-Sum 2023	Semantic partitioning (Sentence-BERT) + BART.	ROUGE, ROUGE-LSum
Zafari & Zulkernine (2021) [29]	Private medical chats	Chatsum: Random Forest + TF-IDF + CTAKES.	Salience scoring, Precision, Accuracy

Table 2. Overview of included studies for Information Extraction, Categorization, and ASR.

Authors & Year	Data Source	Technological Focus	Evaluation & Validation
Mani et al. (2020) [30]	Doctor-patient speech	Seq2Seq Machine Translation for ASR error correction.	Word Error Rate (WER), BLEU
Chen et al. (2023) [31]	Telemedicine benchmark (IMCS-21)	ERNIE-Health and MC-BERT for NER, DAC, and SLI.	F1-score, Accuracy, DX-Acc
de Souza et al. (2023) [32]	Telemedicine audio	ASR + fine-tuned GPT-NeoX 20B for RDF triples.	F1-score, Accuracy
Pyne et al. (2023) [33]	OIAM, ICPC-2, NICE CKS	PubMedBERT with distant supervision for ICPC-2.	Macro-F1
Hao et al. (2021) [34]	Online medical consults	Intra-/inter-window attention for dialogue-to-KG.	Precision, Recall, F1-score
He et al. (2021) [35]	Chinese medical dialogue	Document-aware Information Extractor (DIE).	Precision, Recall, F1-score
Beerepoot et al. (2025) [36]	Home-based nursing care	Zero-shot GPT-4o for event log generation.	Expert consensus, Event count

Table 3. Overview of included studies for Evaluation, Safety, and User Experience.

Authors & Year	Data Source	Technological Focus	Evaluation & Validation
Suhas et al. (2025) [37]	ACI-BENCH dialogues	LLM-based fact alignment and hallucination detection.	Pearson correlation
Croxford et al. (2025) [38]	Multi-specialty EHR data	Validation of the PDSQI-9 clinical evaluation instrument.	ICC, Cronbach’s $\alpha$
Fraile Navarro et al. (2025) [39]	-	GenAI with graded automation (Human-in-the-loop).	Likert scales, Interviews
North et al. (2025) [40]	Primary care (Epic EHR)	Abridge: Ambient listening AI clinical deployment.	EHR signal metrics
Quiroz et al. (2020) [10]	GP consultations	NLP sparsity and content analysis.	Word overlap, Cohen’s kappa

Table 4. Overview of Reviews and Methodological Innovation Studies.

Authors & Year	Data Source	Technological Focus	Evaluation & Validation
Li et al. (2025) [26]	Literature (130 papers)	Vision–Language Models (VLMs) in medical image analysis.	Systematic Review
Li et al. (2026) [41]	Literature (124 papers)	Knowledge Distillation (KD) in medical imaging.	Systematic Review
Sun et al. (2026) [42]	Camelyon17, MSPS	PHH-FL: Perceptual Hashing Hypernetwork FL.	Accuracy, Dice Coefficient

Although the reviewed works differ widely in their clinical contexts, spanning from primary care and home-based nursing to emergency departments, they collectively illustrate how research on medical speech processing has expanded from isolated ASR components toward more integrated, end-to-end documentation pipelines leveraging Large Language Models (LLMs) [17,19,31]. Despite substantial methodological diversity, several coherent themes emerge across the literature regarding transcription fidelity, the efficacy of specialized fine-tuning, and the critical need for robust clinical validation frameworks.

3.1. ASR Performance and Its Downstream Impact

This subsection addresses RQ1 by examining speech recognition performance in clinical environments and its downstream impact on automated documentation systems. The fundamental bottleneck for all downstream clinical tasks is the quality of the initial transcription. Raw medical audio is fraught with environmental noise, colloquialisms, and complex terminology, leading to high Word Error Rates (WER) [17,30].

As demonstrated in Table 5, Mani et al. [30] proved that formulating ASR error correction as a Machine Translation (MT) task using Sequence-to-Sequence models significantly improves transcription accuracy. Applying domain adaptation reduced the WER of Google ASR by an absolute 6.9% (from 41.0% to 34.1%). However, these models suffer from severe class imbalance: while they successfully correct frequent medical terms, they consistently fail on rare medical vocabulary, creating a cascading risk for NLP models.

Table 5. Performance Metrics for ASR Error Correction [30].

ASR System	Correction Model	WER (%)	BLEU
Google ASR	Baseline (No correction)	41.0	52.1
	Seq2Seq MT Adapted	34.1	56.4
ASPIRE ASR	Baseline (No correction)	35.8	54.3
	Seq2Seq MT Adapted	34.5	55.8

Capturing patient speech is also essential, as it provides vital clinical coding signals frequently missed in clinician-only transcripts. Pyne et al. [33] proved this empirically during ablation studies for ICPC-2 clinical coding.

As shown in Table 6, when patient speech was excluded, the model’s performance dropped drastically from an F1 of 0.550 to 0.445, demonstrating that aggressive filtering of the “patient voice” destroys essential diagnostic context.

Table 6. Ablation Study on the Impact of Patient Speech in Clinical Coding [33].

Model Architecture	Input Data	Macro-F1
Conventional BERT (PubMedBERT)	GP Speech Only	0.445
Conventional BERT (PubMedBERT)	GP + Patient Speech	0.550

3.2. Information Extraction and Dialogue Understanding

This subsection addresses RQ2 by examining how structured clinical information is extracted from spoken medical dialogues. A persistent challenge is information sparsity. Quiroz et al. [10] manually coded GP transcripts and established that clinical conversations contain a vast amount of superfluous data: less than 20% (averaging exactly 9.11%) of the total words spoken were deemed relevant for documentation.

To extract medical items scattered across such noisy dialogues, approaches based on windowed attention, semantic partitioning, and structured representations have been introduced. Hao et al. [34] and He et al. [35] utilized intra- and inter-window attention models on Chinese datasets, achieving state-of-the-art extraction precision (85.02%). However, this incurs massive computational costs, as the model must iterate over the entire label set during prediction. Additionally, semantic clustering proves far superior to manual heuristics. Nguyen et al. [28] demonstrated that replacing hand-crafted extraction queries with a Sentence-BERT partitioning module improved the ROUGE-1 score to 0.512, massively outperforming traditional Bag-of-Words (BoW) approaches (ROUGE-1: 0.486).

Extracting structured Knowledge Graphs (KGs) exposes another limitation: the complexity of clinical reasoning.

Table 7 shows that abstractive summarization prior to triple extraction removes conversational colloquialisms, boosting F1 [32]. However, Chen et al. [31] highlighted that while models excel at extracting explicitly stated symptoms (F1: 90.54%), performance drops precipitously when inferring implicit symptoms (F1: 64.27%), proving that multi-turn diagnostic reasoning remains largely unsolved. The inherent ambiguity of human dialogue bounds extraction accuracy; Beerepoot et al. [36] found that evaluating 100 zero-shot GPT-4o snippets of nursing care, experts fully agreed on only 9 classifications, marking 17 as entirely “unclear from the text.”

Table 7. Performance Metrics for Semantic Extraction and Reasoning.

Study	Task & Model	F1-Score (%)	Accuracy (%)
de Souza et al. (2023) [32]	RDF Triple Class. (Original Text)	86.7	80.0
	RDF Triple Class. (Summarized Text)	93.3	90.0
Chen et al. (2023) [31]	Explicit Symptom Inference (MC-BERT)	90.54	-
	Implicit Symptom Inference (MC-BERT)	64.27	-

3.3. LLM-Based Summarization and Note Generation

This subsection addresses RQ3 by examining how large language models generate structured clinical notes. Research has increasingly focused on integrated frameworks, such as MediNotes and CLINICSUM, providing substantial administrative relief.

However, lexical metrics often demonstrate algorithmic improvements while obscuring severe clinical risks in real-world deployment. The evaluation of zero-shot ChatGPT-4 on ambulatory encounters by Kernberg et al. [18] revealed an alarming average of 23.6 errors per clinical case, with 86.3% of these being critical omissions. The mean PDQI-9 quality score stood at an unacceptable 29.7 (categorized as “terrible/bad”).

Conversely, methods designed with stringent structural constraints perform robustly. The MedStructGen framework [20] achieved a BLEU-4 of 59.60% and a BERTScore-F1 of 73.20% on real hospital data. More importantly, its strict verification phase kept the hallucination rate tightly controlled at 7.20% and achieved a 73.20% structural compliance rate, safely reducing daily physician documentation time by up to 1.5 h in a live hospital. Factual faithfulness is consistently improved through two-stage architectures that better manage long dialogues than full-note generation [17,21,25].

3.4. LLM Architectures, Parameter Selection, and Fine-Tuning Strategies

A critical theme across the reviewed literature is the performance discrepancy between massive, generalized LLMs utilized in a zero-shot capacity (e.g., GPT-4) and smaller, domain-specifically fine-tuned models (e.g., LLaMA-3-8B, BART).

Table 8 highlights this phenomenon. Neupane et al. [15] proved that incorporating retrieval-based filtering allowed a smaller LLaMA-3-8B model to vastly outperform GPT-4-Turbo (BERTScore 0.84 vs. 0.73). Leong et al. [14] demonstrated a +4.38 point increase in BERTScore F1 when utilizing a fine-tuned LLaMA3-8B model over zero-shot GPT-4o. However, local models can fail on highly unstructured data; Haniff et al. [16] reported that while cloud GPT-4 generated summaries for 100% of mock cases, local models (BART, Flan-T5) failed to produce complete texts.

Table 8. Comparison of Zero-Shot and Fine-Tuned Models on Standardized Datasets.

Study	Model Architecture	ROUGE-1	ROUGE-L	BERTScore F1
Yim et al. (2023) [17]	GPT-4 (Zero-shot)	51.76	45.97	-
	BART + FTSAMSum (Fine-tuned)	53.46	48.62	-
Leong et al. (2024) [14]	GPT-4o (Zero-shot)	50.83	47.39	65.96
	MediNotes (LLaMA3-8B + RAG)	56.16	51.60	70.34
Neupane et al. (2024) [15]	GPT-4-Turbo (Zero-shot)	0.58	0.36	0.73
	CLINICSUM (LLaMA-3-8B + RAG)	0.70	0.55	0.84

3.4.1. Parameter Selection and Fine-Tuning Strategies

To achieve these results while managing computational costs, researchers increasingly rely on Parameter-Efficient Fine-Tuning (PEFT) methods, such as Low-Rank Adaptation (LoRA), rather than full-parameter fine-tuning. This allows 8-billion parameter models like LLaMA-3 to be adapted to clinical jargon using standard hardware [14,15].

Furthermore, parameter selection during the generation phase heavily dictates model reliability. In medical documentation, decoding strategies favor deterministic outputs over creativity. Studies consistently recommend setting the generation temperature to extremely low values (e.g., 0.0 to 0.2) to mitigate the risk of hallucinatory facts, while carefully managing the context window size to prevent the model from truncating essential patient history during long encounters. Despite these established best practices, reproducibility remains a challenge in the field, as many studies fail to report their exact hyperparameter configurations, prompt structures, or specific decoding strategies (e.g., beam search vs. greedy decoding).

3.4.2. Phased Generation and Two-Stage Pipelines

To further overcome the limitations of standard generation algorithms, state-of-the-art systems heavily utilize “phased generation” (two-stage architectures). Medical conversations are inherently sparse and lengthy; forcing an LLM to read a full transcript and generate a note in a single pass often leads to critical omissions.

Phased generation mitigates this by separating information extraction from abstractive generation. For example, the MedStructGen framework employs a strict two-stage LLM pipeline where entities are extracted and verified against templates before the final prose is generated, successfully capping hallucinations at 7.2% [20]. Similarly, on Chinese medical datasets, single-stage models plateau around a ROUGE-1 of 62.59% [22]. In contrast, Table 9 shows that two-stage architectures, such as the TS-CA model (which extracts salient info and compacts remaining noise into a soft context) and the RL-optimized model by Lu et al. (utilizing REINFORCE rewards), push ROUGE-1 scores to nearly 68.0% [21,25]. Isolating extraction from generation structurally prevents the LLM from fabricating non-existent medical terminologies.

Table 9. Performance Comparison on the IMCS21/IMCS-MRG Dataset.

Study	Model Architecture	ROUGE-1	ROUGE-L
Kumar et al. (2024 [22]	BART + Entity Adapters	62.59	58.55
Lu et al. (2023) [21]	Two-Stage Model + REINFORCE	67.75	64.38
Duan et al. (2023) [25]	TS-CA (Compacting-Abstractive)	67.88	66.07

3.5. Clinical Workflow Integration and Usability

Integrating digital scribes into routine clinical workflows remains challenging. Technological capability does not directly translate to clinical adoption. Fraile Navarro et al. [39] evaluated physician perceptions of Generative AI systems operating at different levels of autonomy.

As summarized in Table 10, clinicians fundamentally distrust fully autonomous AI (High Automation) due to fears of automation bias, loss of clinical nuance, and medical-legal liability. The literature strongly identifies “Medium Automation”, where the AI acts as a drafting assistant requiring explicit manual verification, as the most reliable paradigm.

Table 10. Clinician Trust and Adoption Readiness based on Automation Levels [39].

Automation Level	System Role	Clinician Trust	Safety Assessment
Low Automation	Basic spellcheck/Transcription	High	Safe (High Cognitive Load)
Medium Automation	Draft generation requiring MD approval	Very High	Highly Reliable
High Automation	Direct EHR entry without MD review	Very Low	Highly Risky

To illustrate the feasibility of this approach in real clinical scenarios, quantitative data from recent hospital deployments provide critical insights. For instance, the MedStructGen framework, deployed in a real-world hospital workflow, achieved a 73.20% structural compliance rate and actively reduced physician documentation time by 1 to 1.5 h per day [20]. Crucially, this system enforced a strict, template-based verification phase, a hallmark of Medium Automation, which successfully capped the hallucination rate at a manageable 7.20% [20]. Similarly, a real-world implementation of the Abridge ambient listening AI across primary care practices demonstrated a rapid adoption rate, with physician usage surging to 58.4% within weeks, and a measurable 18.6% reduction in documentation time per note [40].

However, real-world deployments also expose inherent risks that necessitate strict human-in-the-loop oversight. Without rigid structural constraints and explicit clinician verification, generative models can produce an alarming average of 23.6 errors per clinical case, with 86.3% of these being critical omissions [18]. Furthermore, AI-generated notes have been empirically shown to increase overall note length by an average of 5.4% post-implementation [40]. This verbosity introduces a new cognitive burden: clinicians must verify longer texts to catch subtle inaccuracies. Therefore, while Medium Automation is highly feasible and offers massive administrative relief, its safety entirely depends on mandatory clinician oversight to mitigate these inevitable algorithmic risks.

3.6. Evaluation and Validation Practices

A profound risk identified in recent literature is the fundamental disconnect between standard algorithmic metrics and true clinical safety. For speech recognition, Word Error Rate (WER) remains the standard measure; however, it assigns equal mathematical weight to all lexical errors. Consequently, the omission of a clinically significant negation (e.g., “not”) is penalized equally to missing a conversational filler word. Similarly, in downstream text generation, Suhas et al. [37] demonstrated that reliance on n-gram overlap metrics (e.g., ROUGE, BLEU) creates a false sense of security. These metrics heavily penalize synonyms but completely fail to capture critical medical inversions (e.g., generating “patient has a fever” instead of “patient does not have a fever”), resulting in high scores for clinically dangerous hallucinations.

For information extraction and clinical coding, classification metrics such as Precision, Recall, and F1-Score are predominantly used. To account for the inherent sparsity of rare medical conditions in datasets, researchers often utilize Macro-F1, ensuring all clinical classes are equally represented regardless of their frequency [33].

To combat the limitations of automated metrics, the field is decisively shifting toward validated, clinician-centered evaluation instruments. Tools such as PDQI-9 [18], modified Q-note scoring [16], MEDCON [17], and LINK/CORRECT [19] abandon strict lexical counting in favor of expert-assessed scales. Most notably, Croxford et al. [38] developed and validated the PDSQI-9 (Physician Documentation Summary Quality Instrument). Unlike automated scripts, PDSQI-9 evaluates dimensions of clinical utility, synthesis, and patient safety, proving highly reliable across diverse medical specialties.

Finally, to ensure the robustness of these human-in-the-loop evaluations, studies increasingly rely on statistical reliability measures. Metrics such as Cohen’s kappa ($\kappa$) for inter-rater agreement, Intra-class Correlation (ICC) for continuous clinician ratings, and Cronbach’s $\alpha$ for internal questionnaire consistency are becoming standard practice. Systematic clinical validation utilizing this combination of structured expert rubrics and rigorous statistical reliability remains the most robust pathway toward safely auditing AI outputs.

3.7. Data Availability, Privacy, and Governance

Access to high-quality, annotated clinical conversation data remains a major bottleneck, as privacy regulations limit large corpora creation [17]. To address this, increasing attention has been directed toward privacy-preserving deployment strategies like Federated Learning (FL). However, standard FL struggles with data heterogeneity across different clinics.

To resolve this, the PHH-FL framework utilizes a perceptual hashing algorithm to generate similarity embeddings without exposing raw data [42]. As demonstrated in Table 11, optimizing these embeddings (e.g., 100-dimensional vectors) strikes the ideal balance between personalization accuracy (93.02% Dice Score) and the communication bandwidth required for decentralized deployment. Such mechanisms are increasingly recognized as integral to trustworthy AI governance.

Table 11. Impact of Dimensionality on Medical Image Segmentation [42].

Embedding Vector Size	Average Dice Score (%)	Computational Cost
Baseline (No Embedding)	90.92	Low
100 Dimensions (Optimal)	93.02	Balanced
128 Dimensions	93.54	Very High (Bandwidth risk)

3.8. Key Observations and Emerging Trends

Across the 31 reviewed studies, five convergent observations summarize the field’s trajectory toward integrated digital scribe systems. First, ASR performance remains the primary technical bottleneck, consistent with the “Pareto principle,” where only 9–20% of conversational content contains clinically relevant information [10,30]. Second, there is a definitive shift toward fine-tuned Transformer architectures (e.g., Llama-3) using retrieval-augmented generation (RAG) to outperform zero-shot cloud LLMs in semantic alignment [14,15]. Third, data scarcity forces reliance on standardized simulated benchmarks (ACI-BENCH) or privacy-preserving federated architectures [17,42]. Fourth, the literature universally identifies “Medium Automation” as the only clinically safe workflow; AI functions as a drafting assistant saving up to 1.5 h daily but mandating explicit physician verification [39,40]. Fifth, evaluation practices are decisively maturing from misleading lexical metrics (ROUGE) toward validated clinical safety instruments such as PDSQI-9 [37,38]. Collectively, these trends indicate that component-level maturity is now being translated into trustworthy, end-to-end documentation pipelines through clinician-in-the-loop design and rigorous structural constraints.

4. Discussion

The findings of this review indicate a clear shift from isolated components toward integrated, end-to-end digital scribe pipelines. While frameworks such as MediNotes and CLINICSUM demonstrate strong generative capabilities, consistent performance across diverse clinical settings remains limited due to persistent technical and institutional fragmentation [14,15]. Automatic speech recognition remains the primary technical bottleneck, as variability in spontaneous clinical dialogue leads to error propagation that undermines factual reliability in downstream summarization [16,30]. Given that less than 20% of conversational content is typically relevant for documentation, effective information filtering is essential, motivating architectures based on semantic partitioning, windowed attention, and structured representations to preserve context and data integrity [10,28].

The adoption of neural and LLM-based approaches improves reasoning capabilities but introduces risks of hallucinations and omissions, particularly in safety-critical domains [18,37]. Consequently, clinicians favor medium-automation workflows in which AI-generated drafts are reviewed and finalized by physicians, balancing efficiency gains with professional accountability [39,40].

Lastly, data scarcity further limits generalizability, as privacy constraints restrict access to real-world clinical audio and encourage reliance on simulated or narrowly scoped datasets [17,33]. Effective deployment therefore requires governance mechanisms that evolve alongside technical advances, including validated evaluation instruments and interdisciplinary collaboration to ensure that digital scribes support rather than replace clinical judgment [24,38].

4.1. The Role of Open-Access Databases in Clinical AI Research

The literature search process highlighted a significant trend regarding the distribution of research across publication platforms. As observed in the initial retrieval phase (see Figure 2), the open-access database (PubMed) accounted for the overwhelming majority of identified records (over 85%) compared to commercial, subscription-based databases such as IEEE Xplore, Scopus, or Web of Science. This stark contrast warrants further discussion. The dominance of open-access repositories demonstrates a strong push toward the open exchange of scientific findings within the medical AI community. In the rapidly evolving fields of large language models (LLMs) and clinical speech recognition, the ability to quickly and freely share findings is critical. Open-access databases not only accelerate the dissemination of novel clinical NLP workflows but also ensure that researchers and healthcare institutions globally, regardless of their financial resources or institutional subscriptions, can access, validate, and build upon these technologies. However, this also suggests a potential divide in publication cultures: while applied clinical research heavily favors open science to drive immediate healthcare innovations, more foundational or strictly algorithmic engineering studies may still be confined to closed-access technical databases. Recognizing this distribution is vital for future studies to ensure a comprehensive understanding of both the technical and clinical advancements in medical AI.

4.2. Limitations

Despite the presented findings, the current review has several limitations. First, the primary evidence is constrained by the prevalence of small, often synthetic or role-played datasets (e.g., ACI-BENCH or staged role-playing), which may not fully capture the multi-speaker complexity, background noise, and emotional nuances of real-world clinical encounters. Second, the narrow clinical scope, centered on primary care, pediatrics, and oncology, limits the generalizability of these findings to hospital settings or complex multi-morbidity cases. Third, language diversity and multilingual deployment represent another important limitation since most reviewed systems were evaluated primarily in English-language clinical settings, often using benchmark datasets collected in controlled environments. Consequently, the generalizability of these approaches to multilingual and real-world healthcare systems remains uncertain. Practical deployment may require support for code-switching, regional accents, low-resource languages, multilingual medical terminology, and country-specific clinical documentation standards. These factors may substantially affect ASR accuracy, dialogue understanding, and downstream summarization reliability. Fourth, the heterogeneity of evaluation metrics and the reliance on clinician-centered assessments, which often show only moderate inter-rater reliability, prevented quantitative synthesis or meta-analysis. Fifth, several studies utilized manually cleaned transcripts, potentially underestimating the significant impact of raw ASR error propagation on downstream LLM performance in live environments. Finally, the rapid pace of LLM development means that our review provides only a temporal snapshot; updates to proprietary models may render specific prompt-engineered strategies outdated relatively quickly. These factors suggest that the synthesized trends should be interpreted with caution across broader medical disciplines.

4.3. Barriers to Real-World Deployment

This subsection addresses RQ4 by examining the key barriers that limit the real-world deployment of automated clinical documentation systems. Despite substantial methodological advances, several technical, organizational, regulatory, and workflow-related challenges continue to constrain clinical adoption. From a technical perspective, maintaining robust performance in real-world clinical environments remains difficult due to background noise, overlapping speech, accented communication, spontaneous dialogue patterns, and variability across medical specialties [19,30]. Errors originating from ASR components frequently propagate into downstream summarization modules, increasing the risk of hallucinations, factual inconsistencies, and omission of clinically important findings [16,18,37].

These limitations become particularly critical in long outpatient consultations and multi-speaker interactions, where contextual dependencies are difficult to preserve reliably. Workflow integration presents an additional barrier. Clinical documentation systems must operate within highly time-constrained healthcare environments and integrate seamlessly with existing EHR infrastructures. In practice, clinicians frequently report that even minor transcription inaccuracies increase cognitive load and require manual correction, potentially offsetting efficiency gains [36,39]. Current evidence suggests that, as a result, clinicians generally prefer medium-automation paradigms in which AI-generated documentation is reviewed and validated by healthcare professionals before finalization.

Regulatory and governance constraints further complicate deployment. Real-world systems must comply with privacy regulations, data retention requirements, informed consent procedures, and institutional cybersecurity policies. In addition, unresolved medico-legal questions regarding accountability for AI-generated documentation remain a substantial concern, particularly when hallucinated or omitted information may affect downstream clinical decisions [14,16]. Ensuring transparency, auditability, and traceability of generated content is therefore essential for trustworthy deployment.

Another major limitation concerns data availability and external validation. Many existing systems are evaluated primarily using simulated consultations, benchmark datasets, or retrospective analyses rather than prospective clinical studies conducted in routine practice [17,18]. Consequently, the generalizability of current models across institutions, specialties, languages, and healthcare systems remains uncertain. Large-scale prospective evaluations involving clinicians, patients, and operational healthcare workflows are still limited.

Finally, economic and organizational factors may significantly influence adoption. Real-world deployment requires substantial investment in infrastructure, integration with commercial EHR platforms, continuous monitoring, clinician training, and ongoing model maintenance. Healthcare organizations must therefore balance potential efficiency gains against implementation costs, operational risks, and the need for sustained human oversight. In summary, these barriers indicate that successful deployment of digital scribes depends not only on improvements in ASR and LLM performance, but also on robust governance frameworks, seamless workflow integration, prospective clinical validation, and careful alignment with real-world healthcare practices.

4.4. Future Research Directions

Future research should prioritize large, high-quality corpora of real clinical conversations that capture multi-speaker and noisy settings. Such datasets require standardized annotation schemes, transparent inter-annotator agreement reporting, and privacy-preserving data sharing approaches such as federated learning or validated synthetic data generation. Methodologically, there remains a need for ASR models optimized for clinical dialogue that jointly address diarization, speaker role identification, and domain-specific error correction.

Another important direction involves multilingual and cross-cultural evaluation of clinical documentation systems. Although existing benchmarks are dominated by English-language datasets, real-world healthcare environments often involve multilingual communication, code-switching, accented speech, and low-resource languages. Developing robust multilingual ASR and LLM pipelines that can adapt to region-specific terminology and documentation practices is essential for equitable and scalable deployment across diverse healthcare systems.

On the LLM and systems side, future work should focus on uncertainty estimation, factuality verification, and retrieval-augmented generation to reduce hallucinations and improve traceability. Integrated pipelines that jointly model speech, semantics, and summarization, while using structured representations such as knowledge graphs, offer a promising direction for mitigating error propagation and managing information sparsity.

To further advance the capabilities of clinical AI, the transition towards multimodal and highly efficient generative architectures is essential. Recent comprehensive reviews highlight two critical trajectories: Vision–Language Models (VLMs) and Knowledge Distillation (KD). The evolution from simple text-based models to general large VLMs enables the seamless integration of visual modalities, such as radiological scans or real-time patient imagery, directly into the clinical reasoning process [43]. This suggests that future digital scribes will not rely solely on conversational transcripts but will generate comprehensive medical notes by jointly analyzing verbal dialogue and visual data [43].

Furthermore, deploying massive LLMs in clinical settings presents significant barriers regarding computational costs and data privacy. To overcome this, KD and teacher-student learning frameworks have emerged as pivotal solutions [41]. By distilling complex reasoning capabilities from massive, cloud-based teacher models into smaller, specialized, and locally-hosted student models, healthcare systems can achieve high-fidelity documentation while maintaining strict data security [41]. This teacher-student paradigm also facilitates semi-supervised learning and domain adaptation, allowing clinics to deploy efficient, privacy-preserving AI scribes without compromising the quality of the generated notes [41].

Another promising direction involves adaptive dialogue management strategies for automated clinical documentation systems. Such approaches could prioritize clinically relevant information, request clarification under uncertainty, and reduce hallucinated or unsupported content. Consequently, these mechanisms may help reduce downstream error propagation originating from ASR inaccuracies and improve the factual reliability of generated SOAP notes [14,15,16,37].

Future research may also investigate automated clinical documentation from a sequential decision-making perspective, in which systems continuously select actions under uncertainty to optimize long-term clinical objectives. In practical deployments, such strategies could support decisions regarding when to request clarification, when to defer uncertain outputs to clinicians, and how to prioritize clinically significant findings. Rather than optimizing only lexical similarity, these approaches may improve the factual reliability and safety of generated SOAP notes throughout the conversational process.

This perspective may be particularly relevant in real-world clinical environments, where documentation systems operate under partial observability due to ASR inaccuracies, incomplete patient information, background noise, and dynamic multi-speaker interactions [19,30]. In such settings, uncertainty-aware strategies may help balance conversational efficiency with factual consistency and clinical safety [16,37].

Furthermore, clinician-in-the-loop supervision may improve alignment between generated documentation and real clinical expectations. Rather than relying exclusively on static summarization objectives, adaptive documentation workflows may optimize the overall quality of the final medical note by incorporating clinician corrections, prioritizing clinically relevant information, and minimizing the omission of critical findings. Accordingly, hybrid architectures that integrate ASR, LLMs, adaptive dialogue management, and human oversight appear to represent a promising direction for trustworthy and deployable digital scribe systems.

5. Conclusions

This review shows that research on automated clinical documentation has advanced substantially across individual components of the pipeline, including speech recognition, information extraction, dialogue understanding, and LLM-based summarization. Despite this progress, the evidence indicates that these components do not yet form a reliable end-to-end system suitable for autonomous clinical use. ASR performance remains the dominant bottleneck, while data scarcity, limited generalizability, and the safety risks associated with LLMs further constrain real-world deployment.

The findings also highlight a clear methodological shift toward neural and LLM-assisted approaches, accompanied by growing attention to governance, transparency, and clinician oversight. Integrated, mixed-initiative workflows, rather than fully automated systems, currently represent the most feasible and clinically acceptable direction. However, the lack of large, diverse corpora of real clinical conversations continues to hinder benchmarking and model robustness.

Overall, automated documentation systems hold significant promise for reducing administrative burden and improving clinical workflows, but realizing this potential will require advances in domain-specific ASR, robust LLM safeguards, and rigorous clinician-in-the-loop evaluation. Continued interdisciplinary collaboration between clinicians, engineers, and policymakers will be essential to ensure that next-generation digital scribe systems are accurate, safe, and aligned with real clinical needs.