Voice, Speech, and Large Language Models in Neurology: From Acoustic Biomarkers to Conversational AI

Abstract

Background: Speech models (wav2vec 2.0, HuBERT, Whisper), large language models (GPT, LLaMA), and conversational AI have expanded computational speech analysis from handcrafted acoustic features to dialogue-based neurological assessment. How well these approaches address clinical practice has not been evaluated. Methods: We conducted a narrative review searching PubMed, Google Scholar, and IEEE Xplore, supplemented by Interspeech and ICASSP proceedings. Findings are organized along three layers: acoustic-motor (voice quality, prosody, articulation), language-transcript (lexical, syntactic, semantic, and discourse analysis), and integrated multimodal-conversational (interactive dialogue systems). Traditional acoustic biomarkers provide background; the primary focus is on foundation models, LLMs, and conversational AI. Findings: Speech foundation models outperform handcrafted features on several classification tasks but degrade on severely impaired speech due to domain mismatch with healthy training data. LLMs classify transcripts and score cognitive tests, but operate on text alone and cannot access acoustic-motor information. Conversational AI can administer cognitive screening through naturalistic dialogue, but validation is limited to small single-centre feasibility studies. Prospective clinical validation remains limited. Cross-linguistic generalizability is untested for most methods. Interpretation: The field is moving toward integrated speech-language assessment, but the gap between technical capability and clinical utility remains wide. Closing it requires diverse multilingual datasets, standardized benchmarks, prospective validation, and ethical governance.

1. Introduction

Speech requires coordinated engagement of respiratory, laryngeal, and articulatory musculature under precise neural control, real-time lexical and syntactic processing, and continuous monitoring of acoustic output [1,2]. Because speech production depends on distributed circuits spanning the motor cortex, basal ganglia, cerebellum, perisylvian language network, and prefrontal systems, disruptions produce characteristic alterations in voice quality, articulation, prosody, fluency, and discourse coherence [3,4]. The hypophonia of Parkinson’s disease, the scanning speech of cerebellar ataxia, the effortful output of Broca’s aphasia, and the empty fluent speech of Alzheimer’s disease are established clinical signs [5,6].

Over two decades, researchers have extracted acoustic features from speech recordings and shown they can discriminate patients from controls in controlled settings [7,8]. NLP applied to transcripts extended this to lexical, syntactic, and discourse-level features reflecting cognitive function [9,10]. These streams established that speech contains extractable diagnostic information, but the translational gap remains substantial: most studies rely on small cohorts, unstandardized features, and retrospective designs [11,12]. Previous reviews examined acoustic biomarkers for specific conditions [12,13] or digital speech methodology [11], but none have traced the trajectory from traditional features through foundation models to conversational AI.

Foundation models have begun to change this. Self-supervised speech models (wav2vec 2.0 [14], HuBERT [15], WavLM [16]) learn acoustic representations from unlabeled speech, replacing handcrafted features with learned representations that capture spectro-temporal patterns inaccessible to traditional measures. Whisper [17] enables reliable transcription across languages. Large language models (GPT [18], LLaMA [19], biomedical adaptations [20]) analyze transcripts at the levels of semantic coherence and clinical reasoning, capabilities beyond earlier NLP.

We organize the reviewed work along three conceptual layers: (1) the acoustic-motor layer, capturing voice quality, prosody, and articulation from audio; (2) the language-transcript layer, analyzing lexical, syntactic, semantic, and discourse features from transcripts; and (3) the integrated multimodal-conversational layer, combining acoustic and linguistic analysis within interactive dialogue systems. The convergence of these technologies has opened a new direction: conversational AI systems that conduct naturalistic clinical dialogues [21,22], though the gap between capability and clinical reality remains wide. Each approach carries characteristic limitations: domain mismatch for speech models [23], text-only access for LLMs [24], and absent clinical validation for conversational AI [25].

This review traces four stages: traditional acoustic biomarkers (as background), speech foundation models and ASR, LLM-based transcript analysis, and conversational AI for neurological assessment. For each, we summarize current evidence, identify limitations, and assess clinical readiness, with particular attention to multimodal integration, the gap between conversational fluency and clinical reasoning, cross-linguistic generalizability, and ethical considerations. Neurological conditions affect an estimated 3.4 billion people worldwide [26], and the global shortage of neurologists means scalable automated assessment tools could address a real unmet need. That estimate includes headache disorders; the burden of neurodegenerative and speech-affecting conditions is smaller but still substantial.

2. Methods

This is a narrative review examining voice and speech analysis, foundation models, and LLMs in neurological assessment. We adopted the narrative format because the topic spans multiple research traditions with fundamentally different methodologies and publication conventions [27,28].

We searched PubMed, Google Scholar, and IEEE Xplore for publications from January 2000 through April 2026, combining three concept domains: (1) neurological conditions; (2) speech and voice analysis; and (3) computational methods (deep learning, foundation models, LLMs, transformers, conversational AI). We supplemented with reference screening, forward citation tracking of seminal papers (wav2vec 2.0, HuBERT, Whisper, GPT-4), and hand-searching of Interspeech (2015 to 2025) and ICASSP (2018 to 2025) proceedings. Traditional acoustic biomarker studies (pre-2020) were included selectively as background; the primary focus was on foundation models, LLMs, and conversational AI from 2020 onward. The complete search strings, database-specific modifications, and applied filters for each database are provided in Supplementary Table S5. A simplified PRISMA-style flow diagram summarizing this process is provided in Figure S1 in the Supplementary Material.

Studies were included if they met at least one of the following criteria: (a) applied speech foundation models or LLMs to neurological assessment; (b) presented benchmark datasets or evaluation frameworks for speech-based neurological analysis; (c) tested conversational AI for clinical cognitive or motor-speech assessment; or (d) addressed translational challenges (regulatory, ethical, clinical workflow) specific to speech-based clinical AI. Pre-2020 acoustic biomarker studies were included selectively as representative exemplars based on citation frequency, methodological influence on subsequent work, and coverage of the major neurological conditions addressed in the review; this section provides historical context rather than exhaustive coverage.

Studies were excluded if they focused on speech therapy outcomes without a diagnostic or monitoring component, addressed psychiatric conditions without neurological framing, reported purely technical speech-processing advances without clinical motivation, or were not available in English.

We organized findings along a conceptual spectrum (Figure 1): (1) traditional acoustic biomarkers (background); (2) speech foundation models and ASR; (3) LLMs and NLP for clinical speech; and (4) conversational AI and dialogue-based assessment.

Figure 1. Conceptual framework for the four stages of computational speech analysis in neurology, corresponding to Section 3, Section 4, Section 5 and Section 6 of this review. The stages progress from traditional acoustic biomarker extraction (Section 3, left) through speech foundation models and automatic speech recognition (Section 4), NLP- and LLM-based clinical speech analysis (Section 5), and conversational AI and dialogue-based assessment (Section 6, right), with increasing analytical depth and contextual understanding. A cross-cutting multimodal integration layer (Section 7) combines acoustic, linguistic, and visual information across stages to support integrated neurological assessment. This figure was generated using the AI tool Google’s Gemini Pro 3 based on author-defined prompts; all values were verified against the original data tables.

The initial database search across PubMed, Google Scholar, and IEEE Xplore yielded approximately 1600 records. After removing duplicates and screening titles and abstracts, approximately 340 articles underwent full-text review. Of these, 128 studies met our inclusion criteria and are cited in the final manuscript; 15 of the 128 were identified through forward citation tracking and hand-searching of conference proceedings rather than the primary database search. A simplified PRISMA-style flow diagram summarizing this process is provided in Figure S1.

We did not apply a formal risk-of-bias tool (e.g., QUADAS-2). Instead, methodological rigour (sample size, validation strategy, dataset limitations) is evaluated for individual studies throughout the text. Study selection was performed by the author with iterative refinement during manuscript preparation. This approach, while standard for narrative reviews, does not carry the reproducibility guarantees of a systematic review, and this limitation should be considered when interpreting the scope and completeness of the evidence presented.

AI Tools Used

Figure 1 and Figure 2 were generated using Google’s Gemini image generation model based on author-defined prompts. The underlying numerical content was prepared in Excel tables, and all values shown in the figures were verified against the original tables to ensure accuracy.

Figure 2. Key future directions for speech and language AI in neurology, derived from the limitations identified in Section 8. The figure summarizes major challenges and opportunities across five domains: data acquisition (Section 8.1 and Section 8.2), validation (Section 8.1), model development (Section 4, Section 5 and Section 7), clinical translation (Section 8.5 and Section 8.6), and clinical impact (Section 8.3 and Section 8.6). Future efforts should focus on larger and multilingual datasets, prospective and cross-corpus validation, multimodal and conversational AI, explainable and deployable systems, and clinically meaningful applications, including early detection, longitudinal monitoring, differential diagnosis, and broader neurological disease coverage. This figure was generated using the AI tool Google’s Gemini Pro 3 based on author-defined prompts; all values were verified against the original data tables.

3. Traditional Acoustic Biomarkers: Background

The computational analysis of speech in neurological disorders has a two-decade history rooted in the extraction of handcrafted acoustic features from controlled recordings. This body of work established that the speech signal contains diagnostically relevant information, but systemic limitations have constrained clinical translation. We summarize the key findings and limitations here as context for the foundation model and LLM-based approaches that follow.

Parkinson’s disease (PD) has received the most attention, reflecting the high prevalence of hypokinetic dysarthria [5,29,30]. Early studies extracted jitter, shimmer, and harmonic-to-noise ratio from sustained phonation, reporting high classification accuracies [7,31], though some used sample-level rather than subject-level cross-validation. Later work on connected speech showed articulatory rate, pause patterns, and prosodic variability were more sensitive to early disease [3,32,33], with cross-linguistic analyses showing substantial influence of language and recording conditions [34].

In Alzheimer’s disease (AD), the changes are cognitive-linguistic: word-finding pauses, reduced lexical diversity, simplified syntax, and loss of discourse coherence [9,35,36]. Linguistic transcript analysis proved more discriminating than acoustic features alone, with Fraser and colleagues achieving over 81% accuracy on the DementiaBank corpus [9]. The ADReSS/ADReSSo challenges established standardized benchmarks [37,38]. Studies of ALS showed that speaking rate is sensitive to early bulbar involvement and can track progression over months [39,40]. Acoustic studies also extended to aphasia [10], autism spectrum disorder [41,42], multiple sclerosis [43,44], Huntington’s disease [45], and traumatic brain injury [46].

Despite these achievements, several systemic limitations have prevented clinical translation. First, handcrafted feature sets are not standardized across laboratories, impeding cross-study comparison [11,13]. Second, the near-universal reliance on small, single-centre datasets (typically fewer than 250 participants) recorded under controlled conditions limits generalizability to real clinical environments [12]. Third, the predominant framework (binary classification of patients versus controls) does not address clinically relevant tasks of disease staging, progression monitoring, or treatment response. Fourth, acoustic and linguistic dimensions have typically been analyzed in separate pipelines despite neurological disorders often affecting both simultaneously. No prospective clinical validation study has been published for any traditional approach. These limitations motivated the foundation model and LLM-based approaches described in the following sections.

4. Speech Foundation Models and Automatic Speech Recognition

4.1. From Handcrafted Features to Learned Representations

Self-supervised learning (SSL) models, trained on large corpora of unlabeled speech, learn general-purpose representations capturing complex spectro-temporal patterns without prior specification of which features to extract [13,47,48]. Neurological speech disorders produce subtle, multidimensional alterations that may be partially captured by individual features but are more completely represented in high-dimensional learned embeddings. The key question is whether representations learned from healthy speech transfer effectively to pathological populations.

4.2. wav2vec 2.0, HuBERT, and WavLM

wav2vec 2.0 [14] learns representations through contrastive self-supervised learning on masked raw audio, producing contextualized representations encoding both local spectral and longer-range temporal dependencies. HuBERT [15] employs offline clustering for masked prediction with more stable layer-wise representations. WavLM [16] incorporates denoising objectives, producing representations more resistant to environmental noise and speaker variation, a property relevant for clinical recordings outside controlled settings.

Several studies have applied SSL models to neurological speech. Bayerl and colleagues showed that SSL representations outperformed acoustic baselines for dysfluency detection [49]. Javanmardi and colleagues reported improvements for dysarthria severity classification [50]. Balagopalan and colleagues found that combining speech representations with linguistic features improved AD detection on ADReSS [51]. Pappagari and colleagues showed competitive performance even with limited labelled data [52]. Favaro and colleagues found that SSL embeddings outperformed handcrafted features across six languages for PD detection [53].

A critical limitation is the domain mismatch between pretraining data (predominantly healthy adult speech from podcasts, audiobooks, and web media) and the target population (patients with dysarthric, aphasic, or cognitively impaired speech). Violeta and colleagues investigated SSL pretraining frameworks for pathological speech recognition and found significant performance degradation on severely disordered speech compared with mild or moderate impairment, suggesting that standard SSL models may require domain adaptation or clinical pretraining to handle the full spectrum of neurological speech disorders [23]. This echoes a broader concern in clinical AI: general-purpose foundation models may not reliably generalize to the distributional shifts encountered in medical populations without explicit adaptation.

4.3. Whisper and Clinical ASR

OpenAI’s Whisper [17], trained on 680,000 h of multilingual weakly supervised data, achieves reliable transcription across languages, accents, and acoustic environments for typical speech. For clinical speech research, Whisper is primarily an enabling technology: automatic transcription removes the bottleneck that has limited the scale of NLP-based speech analysis studies.

However, Whisper’s accuracy on neurological speech is not assured. Rowe and colleagues reported substantially higher word error rates for severe dysarthria compared with mild impairment or healthy speech [54], consistent with the broader ASR literature showing degraded performance on atypical speech. The same distributional mismatch affecting SSL models also affects ASR: models trained predominantly on healthy speech underperform on the populations most in need of automated analysis. Efforts to adapt ASR for pathological speech include fine-tuning on clinical corpora and data augmentation strategies that simulate dysarthric characteristics [55]. The TORGO database [56] and UA-Speech corpus [57] have served as benchmarks, though both are limited in size and linguistic diversity.

4.4. Transfer Learning

The practical value of speech foundation models depends on transfer learning effectiveness. Fine-tuning SSL models on small clinical datasets (n = 50 to 200) can yield performance that is competitive with or superior to traditional pipelines, particularly for subtle distinctions (MCI versus healthy ageing, early PD versus controls) that are poorly captured by handcrafted features [49,51,58].

Transfer learning introduces its own challenges. The high dimensionality of SSL representations (typically 768 or 1024 dimensions per time step) relative to small clinical sample sizes creates substantial overfitting risk, particularly when fine-tuning the full model rather than extracting fixed features from pretrained layers. The optimal SSL layer for feature extraction is itself a consequential hyperparameter: earlier layers encode acoustic-phonetic information while later layers capture more abstract linguistic representations, and the optimal layer depends on whether the condition primarily affects motor speech or language processing [59]. These considerations point to the need for rigorous validation (held-out test sets, cross-corpus evaluation, prospective validation) that the field has not yet consistently adopted.

5. Large Language Models and NLP for Clinical Speech Analysis

5.1. Transformer-Based NLP for Speech Transcripts

The application of NLP to speech transcripts in neurological populations predates the current era of large language models. Early work relied on rule-based analysis, bag-of-words models, and hand-engineered features such as type-token ratio and mean length of utterance [9,60]. The introduction of transformer-based language models, BERT [61] and biomedical variants (BioBERT [62], ClinicalBERT [63]), enabled contextualized semantic representations capturing meaning at a granularity inaccessible to earlier methods. Balagopalan and colleagues showed that BERT representations combined with acoustic features achieved state-of-the-art AD detection on the ADReSS benchmark [51,64]. Yuan and colleagues found BERT-derived pausing patterns predicted cognitive status independent of lexical content [65]. Agbavor and Liang showed that transformers could predict MMSE scores from transcripts with accuracy approaching clinical utility [66].

The transition from feature-extraction NLP to generative language models represents a qualitative shift. BERT-family models excel at classification and regression but cannot generate interpretive clinical text, engage in dialogue, or reason about the relationship between language patterns and neurological mechanisms. These capabilities became possible with large generative models.

5.2. GPT and LLM-Based Analysis

The GPT family [18] introduced open-ended generation and in-context learning, enabling qualitatively new applications. Rather than classifying a transcript as “impaired” or “normal,” an LLM can describe specific linguistic features, generate differential considerations, and produce structured reports.

Agbavor and Liang investigated whether GPT-3 could detect cognitive impairment from speech transcripts, finding that GPT-3 embeddings, used as input to a downstream classifier, distinguished AD from controls with approximately 80% accuracy without fine-tuning the language model itself [66]. B T B and colleagues evaluated ChatGPT-3.5, ChatGPT-4, and Google Bard in zero-shot classification of AD versus cognitively normal individuals, finding all three surpassed chance-level detection but did not reach clinical-application thresholds [67]. Taherinezhad and colleagues systematically compared LLM adaptation strategies, including in-context learning, reasoning-augmented prompting, and fine-tuning, with fine-tuned LLaMA 3B achieving an F1 score of 0.83 and AUC of 0.91 on ADReSSo [68]. Chlasta and colleagues combined GPT-4o-derived linguistic features with HuBERT speech representations for dementia classification, achieving RMSE of 2.78 for cognitive score regression [69]. In the aphasia domain, Kurland and colleagues compared three LLMs to human raters for main concept scoring of story retelling discourse from 96 persons with aphasia, finding strong correlations supporting clinical automation [70]. Fei and colleagues compared GPT-3.5 and GPT-4 to physician-led cognitive evaluations in 60 healthy individuals and 30 stroke survivors, finding that GPT-4 scores aligned closely with clinician ratings [71].

These studies show the flexibility of LLM-based analysis but also reveal fundamental limitations. Five of six relied on transcripts alone as input; only Chlasta and colleagues incorporated audio-derived SSL embeddings. LLMs operate on text and therefore cannot access the acoustic-motor information (voice quality, prosody, speech rate, pause durations) that constitutes a critical dimension of neurological speech assessment. The LLM does not hear the patient; it reads a transcript. The distinction matters. Moreover, ASR errors on dysarthric or aphasic speech propagate through the LLM analysis and may produce spurious findings.

Clinical reliability of LLM-based speech analysis depends on factors beyond classification accuracy. Model architecture, pretraining corpus, instruction tuning, and alignment strategies all influence how an LLM interprets clinical transcripts. Models pretrained on biomedical text may handle clinical terminology more accurately than general-purpose models, but instruction-tuned models may generate more structured outputs. Alignment methods (reinforcement learning from human feedback, constitutional AI) shape whether a model hedges appropriately or produces overconfident clinical inferences. These architectural differences mean that results obtained with one LLM family may not generalize to another, and clinical evaluation must be model-specific.

Retrieval-augmented generation (RAG) offers a potential approach to grounding LLM outputs in external knowledge sources such as clinical guidelines, validated scoring rubrics, or domain-specific literature. By retrieving relevant documents before generating a response, RAG architectures could reduce hallucination and improve the factual accuracy of automated clinical assessments. In neurological speech analysis, a RAG-enabled system could cross-reference a generated interpretation against published diagnostic criteria or normative data for a given speech task. RAG has shown promise in other healthcare applications but remains largely unexplored in speech-based neurological assessment.

5.3. Discourse Coherence Analysis

One area where NLP and LLM methods are well-suited is the analysis of discourse coherence, the logical and thematic connectedness of extended speech, which deteriorates diagnostically in AD, frontotemporal dementia, and thought-disordered states [72,73]. Computational approaches have applied semantic similarity metrics, topic modelling, and graph-based coherence analysis. Embedding-based coherence measures applied to picture descriptions showed that semantic coherence declined in a gradient from healthy controls through MCI to AD [74]. Mota and colleagues showed graph-based speech connectivity could distinguish psychotic from nonpsychotic states [75]. Ilias and Askounis proposed multimodal architectures combining BERT with vision transformers for AD detection [76].

LLMs extend this by enabling assessment of pragmatic appropriateness, inferential coherence, and referential tracking [77], dimensions inaccessible to counting-based metrics, though this remains untested at scale in clinical populations.

5.4. Automated Test Scoring

A practical application of NLP and LLM technologies is automated scoring of standardized cognitive assessments that rely on verbal responses, including the Boston Naming Test, verbal fluency tasks, and story recall [78,79]. Automating this process could reduce administration time, eliminate inter-rater variability, and enable cognitive screening at scale. Toth and colleagues showed automated scoring of verbal fluency tests approaching human rater accuracy [80]. Lehr and colleagues predicted memory scores and detected MCI from story recall transcripts [81]. Fristed and colleagues deployed a remote smartphone-based AI system for daily story recall, achieving AUC 0.85 for MCI/AD detection and 0.78 for amyloid positivity prediction in the MCI/mild AD subgroup [82]; this is also an early example of home-based monitoring (Section 7.2). Automated test scoring is arguably the nearest-term clinical application, as it operates within a structured framework with established ground truth and does not require diagnostic reasoning de novo.

6. Conversational AI and Dialogue-Based Assessment

6.1. From Monologic to Dialogic Assessment

Traditional computational speech analysis relies on monologic samples: sustained phonation, reading passages, picture description, or verbal fluency tasks [11,37]. These are standardized and reproducible but capture only a narrow slice of communicative function. Clinical neurological assessment relies heavily on conversational interaction: the neurologist observes not only what the patient says but how they respond to questions, maintain conversational flow, repair misunderstandings, manage topic transitions, and adapt communication to the demands of dialogue [83,84]. Conversational AI capable of sustaining naturalistic dialogue creates the possibility of extending speech analysis from isolated task performance to interactive conversational assessment.

6.2. Chatbot-Based Cognitive Screening

Mirheidari and colleagues created an automated conversational system that administered questions from standard cognitive assessments within a naturalistic dialogue framework, analyzing both speech content and acoustic characteristics [85,86]. Their system achieved moderate accuracy in distinguishing dementia from healthy controls and from functional memory complaints, with acoustic and linguistic features contributing complementary information. Patients reported high acceptability, suggesting chatbot-based assessment may overcome barriers of clinician time and patient anxiety [86]. Tanaka and colleagues developed a computer-avatar agent for dementia screening that engaged elderly participants in unstructured dialogue and analyzed multimodal features (facial expression, eye gaze, head movement, acoustic features, linguistic content), outperforming unimodal baselines [87]. Takeshige-Amano and colleagues extended this approach using a chatbot to elicit speech and facial expressions from 99 controls and 93 individuals with AD or MCI, achieving an AUC of 0.94 with combined audiovisual features in a single-centre study requiring independent replication [88].

More recently, telephone-based and LLM-powered conversational agents have emerged. Serafimovska and colleagues showed that the TICS-M cognitive test could be autonomously administered by telephone with validity comparable to a psychologist’s assessment [89]. Konig and colleagues validated automated telephone-based semantic verbal fluency testing in the PROSPECT-AD study, finding high agreement with manual gold-standard scoring [90]. Pakhomov and colleagues developed ANNA (Automated Neural Nursing Assistant), an LLM-based agent conducting brief cognitive assessments by telephone multiple times daily to detect neurotoxic symptoms in immunotherapy patients [91]. Ter Huurne and colleagues evaluated the user experience of automated phone-based cognitive testing in 67 memory clinic participants, finding satisfaction comparable to in-person assessment [92].

This body of work is growing but remains early-stage. The gap between conversational fluency and clinical reasoning is a fundamental concern: the system’s ability to conduct an engaging conversation does not imply that its diagnostic inferences are reliable. Clinical validation of these systems has been limited to small, single-centre feasibility studies, and no system has shown diagnostic accuracy sufficient for clinical deployment.

6.3. Dialogue Structure Analysis

The structure of dialogue itself provides diagnostic information. Conversational analysis examines turn-taking, repair sequences, and response timing [93]. AD patients show increased self-repair, topic drift, and failure to respond to adjacency pairs; PD patients exhibit reduced turn-taking initiative and longer response latencies; right hemisphere damage produces tangential responses and failure to maintain conversational coherence [94,95]. Elsey and colleagues found that turn-taking patterns and repair sequences in routine medical consultations could detect cognitive impairment not captured by standard screening instruments [96]. Jones and colleagues showed that automated measures of turn-taking disruption correlated with clinical severity [97]. These studies suggest that the dialogue itself, not just the speech it contains, is a source of clinically actionable information.

6.4. Interactive Voice Assistants for Monitoring

The ubiquity of voice-enabled devices has prompted interest in their potential for ongoing neurological monitoring. Robin and colleagues showed digital speech assessments yield scores sensitive to MCI and early AD [98]. Daily smartphone monitoring in PD yielded acoustic measures with sufficient reliability to detect changes over weeks to months [11]. The appeal lies in ecological validity and scalability, though environmental noise, device variability, and user compliance introduce variance that may obscure the clinical signal [99].

6.5. Toward End-to-End Assessment

The convergence of speech foundation models, LLMs, and multimodal integration approaches the point where naturalistic clinical conversations, administered by AI, analyzed across acoustic and linguistic dimensions, and summarized as structured assessments, become technically feasible. Individual components exist: multimodal fusion [100], automated test administration [89,90], and LLM-based scoring [70]. No system yet combines them end-to-end. The distance between technical feasibility and clinical utility (discussed in Section 8) remains the field’s central challenge.

7. Multimodal Integration and Longitudinal Monitoring

7.1. Multimodal Fusion

The distinction between motor speech analysis and language analysis has organized this review, but clinical reality is that motor and cognitive dimensions are frequently co-affected: PD patients exhibit both dysarthria and executive-mediated discourse deficits; frontotemporal dementia may affect both motor speech and language; stroke patients may have concurrent dysarthria and aphasia [101,102]. Multimodal approaches integrating acoustic, linguistic, and visual features represent the most complete analytical framework. Combining acoustic and linguistic features improved AD detection beyond either modality alone [37]. Rohanian and colleagues implemented a gating-based fusion architecture that learned to weight features adaptively, achieving state-of-the-art ADReSS performance [100]. Tanaka and colleagues showed that visual features (facial expression, eye gaze, head movement) provided additional discriminative information for dementia detection, particularly for patients whose speech remained superficially intact [87].

Multimodal foundation models (GPT-4o [103], Gemini [104]) offer unified processing of speech, text, and visual inputs without separate feature extraction pipelines, but have not been evaluated for neurological speech assessment.

Evaluating multimodal systems requires attention to whether each modality contributes independent clinical value or whether performance gains reflect redundancy. Ablation studies removing one modality at a time are needed to determine marginal contributions. Practical deployment must also address missing modalities: a system trained on audio, text, and video must degrade gracefully when video is unavailable or when ASR quality is poor due to severe dysarthria. Few studies have reported systematic ablation or robustness testing across modality-dropout conditions.

A growing class of native audio-language models can process speech input directly without requiring an intermediate ASR step. Models such as GPT-4o (OpenAI), Gemini (Google), SALMONN, and Qwen-Audio accept audio tokens alongside text, preserving paralinguistic information (prosody, voice quality, timing, speech rate) that is discarded in text-only pipelines. For neurological assessment, where acoustic-motor features carry diagnostic weight independent of linguistic content, this architectural difference matters: a pipeline that first transcribes and then analyzes text cannot recover the breathy voice quality of early vocal fold paresis, the irregular articulatory breakdowns of cerebellar dysarthria, or the monotone prosody of Parkinson’s disease.

These end-to-end models could reduce error propagation (ASR errors compound downstream NLP errors) and detect cross-modal patterns invisible to modular systems. The limitations are equally clear: clinical validation is absent, interpretability is reduced compared to modular pipelines where each component can be evaluated independently, computational requirements are substantial, and general-purpose pretraining on healthy speech may amplify domain mismatch for neurologically impaired speakers. Whether these models represent a practical advance over carefully engineered pipeline approaches or introduce new failure modes specific to disordered speech remains an open empirical question requiring targeted evaluation in neurological populations.

7.2. Home-Based and Remote Monitoring

Most neurodegenerative diseases are progressive, with changes unfolding over weeks to years, a temporal scale poorly captured by clinic visits every 3 to 12 months [105]. Home-based monitoring using consumer devices enables continuous tracking in the patient’s natural environment. Tsanas and colleagues developed a telemonitoring system in which PD patients recorded phonation samples at home using a telephone, achieving remote estimation of UPDRS motor scores [106]. The Roche smartphone study showed passively collected speech features correlated with medication cycles and disease progression in a Phase 1 PD trial [107]. Connaghan and colleagues used a smartphone application for frequent remote speech sampling in ALS, showing automated acoustic measures tracked bulbar decline [108].

The analytical challenges of longitudinal monitoring differ fundamentally from cross-sectional classification. The relevant signal is the rate and pattern of change over time, requiring accounting for within-subject variability, diurnal fluctuations, medication effects, and disease natural history. Mixed-effects models and change-point detection are appropriate frameworks, but require substantially larger longitudinal datasets than currently exist [109].

7.3. Digital Phenotyping

Digital phenotyping extends speech analysis to data captured during routine device use (phone calls, voice commands) rather than to specific assessment tasks [110,111]. Fagherazzi and colleagues proposed a framework for voice as a broad biomarker of health [112]. The ethical implications (consent in cognitively impaired populations, privacy, algorithmic bias) are discussed in Section 8.4 [113,114].

7.4. Toward Integrated Neurological Speech Agents

The technological components reviewed in the preceding sections point toward a future direction: integrated AI agents capable of conducting a full neurological speech assessment through naturalistic interaction, combining real-time acoustic analysis, reliable transcription, linguistic and discourse analysis, adaptive conversational management, and clinical synthesis into a single pipeline [21,115]. No published system achieves this integrated functionality, and the individual components are at varying stages of maturity. We include this projection not as a review of existing systems but to frame the research agenda discussed in Section 8.

8. Discussion

8.1. Methodological Gaps

Despite technological progress, limitations persist across all stages (Figure 2). Clinical speech datasets remain small (DementiaBank ~300 participants [116]; most PD studies fewer than 100 [117]; ALS cohorts 20 to 65 speakers [40,118]). Standardized benchmarks exist only for AD (ADReSS [37,38]). The field remains overwhelmingly retrospective. Prospective clinical validation (deploying a system in a clinical workflow and measuring impact on diagnosis, monitoring, or treatment) is virtually absent. This is the most important barrier to translation.

Overfitting is a pervasive concern. Many studies, particularly those predating the ADReSS challenges, relied on k-fold cross-validation or leave-one-out designs without held-out test sets, creating a risk of inflated performance estimates. The ADReSS challenge [37,38] represents a positive exception: independent training and test partitions enabled more realistic evaluation. Studies using leave-one-out validation on cohorts of fewer than 50 participants should be interpreted with particular caution, as even modest data leakage through shared recording conditions or feature selection can inflate accuracy by 10 to 20 percentage points.

External validation, testing a model on an independent cohort from a different clinical site, recording environment, or demographic population, is nearly absent. Only a handful of studies evaluated models across sites, and none validated a speech-based AI system prospectively. Dataset biases compound this problem: most corpora are demographically homogeneous (predominantly white, educated, English-speaking participants), and models trained on read speech or sustained vowels may not generalize to spontaneous conversational speech. Recording conditions (clinical microphones versus smartphone audio versus telephone) introduce further variability rarely controlled for across studies.

A distinction must be drawn between statistical performance and clinical significance. High classification accuracy (e.g., AUC above 0.90) in a balanced research dataset does not imply clinical readiness. In clinical populations, where disease prevalence, cost of false positives and false negatives, and workflow integration determine real-world utility, even models with strong AUCs may produce unacceptable positive predictive values. No study in this review reported results according to the CLAIM checklist for medical imaging AI or the STARD diagnostic accuracy reporting guidelines, and few discussed clinical significance beyond classification metrics.

8.2. Cross-Linguistic Generalizability

The reviewed literature is dominated by English-speaking populations, limiting generalizability to the majority of the world’s neurological patients. Speech biomarkers are not language-neutral: tonal languages (Mandarin, Cantonese) use pitch contours for lexical meaning, potentially confounding fundamental frequency measures used for PD assessment [119]; agglutinative languages (Turkish, Finnish) produce different pause and articulation patterns, affecting temporal measures for cognitive screening [120]. Linguistic features are even more language-dependent: lexical diversity, syntactic complexity, and discourse coherence scoring all require language-specific NLP tools and normative data. Only two studies in this review addressed cross-linguistic validity directly [34,53]. The DementiaBank corpus, ADReSS challenges, and most conversational AI studies are English-only. Cross-cultural pragmatic norms (turn-taking, directness, use of silence) further complicate conversational AI assessment. A system calibrated to Anglo-American norms could misinterpret culturally normative reticence as cognitive impairment. Until biomarkers are validated across typologically diverse languages, claims of clinical generalizability are not warranted.

8.3. The Fluency-Reasoning Gap

Current LLMs generate fluent, contextually appropriate text that can create an impression of clinical competence. An LLM analyzing a transcript can identify word-finding pauses, reduced lexical diversity, and simplified syntax, and generate an assessment that reads like a clinical note. But the system does not hear the voice, does not observe the patient, does not have access to medical history, and does not possess the clinical training that enables a neurologist to integrate speech observations with motor examination, cognitive testing, and neuroimaging [24,121]. Describing features of dysarthria from a transcript is pattern description; integrating those features into a diagnostic formulation is clinical reasoning. The risk of conflating these capabilities is not theoretical: premature deployment could produce false reassurance or unnecessary anxiety [122,123]. The field must maintain a clear distinction between the capacity to analyze speech data and the capacity to make clinical decisions.

Beyond the fluency-reasoning distinction, LLMs pose specific risks in clinical speech analysis. Hallucination, the generation of plausible but factually incorrect content, could produce fabricated clinical observations or unsupported diagnostic inferences from a speech transcript. Overconfident explanations may present uncertain findings with unwarranted certainty, misleading clinicians who lack the technical background to evaluate the model output. Mitigation strategies under investigation in other clinical AI domains include retrieval-augmented generation to ground outputs in validated sources structured prompting to constrain the response format, uncertainty estimation to flag low-confidence outputs, and clinician-in-the-loop review to ensure that no automated assessment reaches the patient or medical record without expert oversight. These strategies remain largely untested in speech-based neurological assessment.

8.4. Ethical Considerations

Speech recordings are biometric data that can identify speakers, reveal emotional states, and encode demographic characteristics (age, sex, accent, socioeconomic background) beyond the targeted clinical information [113,114]. In neurological conditions that may impair decision-making capacity, data collection requires particular attention to informed consent and data protection [124]. Algorithmic bias is a specific concern: speech models trained on healthy adult English speech degrade on accented, aged, and neurologically impaired speech, potentially disadvantaging the populations most in need [23,54,125]. The privacy implications of passive speech monitoring in homes of patients with dementia raise issues of autonomy and surveillance beyond standard data protection [126]. A related concern is interpretability: SSL embeddings and LLM-generated assessments function as black boxes, and without mechanistic transparency, clinicians cannot judge whether an assessment reflects a genuine clinical pattern or a spurious artifact. Governance must precede deployment.

8.5. Future Directions

Foundational infrastructure: large, diverse, longitudinal clinical speech datasets encompassing multiple conditions, languages, and demographic groups are the single most important investment. The ADReSS benchmarking model should be extended to other conditions and expanded to include acoustic, linguistic, and conversational dimensions. Technical directions: multimodal architectures jointly processing speech audio and linguistic content represent the strongest technical direction. Clinically grounded evaluation frameworks analogous to the CLAIM checklist for medical imaging AI [127] or SPIRIT-AI guidelines [128] would establish minimum reporting standards. Translational requirements: prospective validation studies must evaluate not only accuracy but impact on clinical workflows: whether AI assessments reduce time to diagnosis, improve monitoring, or change treatment decisions. Regulatory readiness (FDA SaMD framework, EU MDR) and ethical governance (consent in cognitively impaired populations, privacy-preserving methods, bias auditing) must advance in parallel. None of these priorities can be achieved by the speech technology community alone; closing the gap requires partnership with neurologists, healthcare systems, and patients [129,130].

8.6. Regulatory Pathways and Clinical Implementation

Translation of speech-based AI tools into clinical neurology requires a regulatory pathway that does not yet exist for this class of biomarker. Under the FDA Software as a Medical Device (SaMD) framework, a speech analysis tool intended for cognitive screening would likely be classified as Class II (moderate risk), requiring a 510(k) or De Novo pathway with clinical performance data. A tool intended to inform diagnostic decisions (e.g., differentiating AD from frontotemporal dementia based on speech patterns) would face higher scrutiny. The EU Medical Device Regulation (MDR) imposes similar tiered requirements. FDA guidance on predetermined change control plans for adaptive AI/ML-based devices is relevant here, as speech models that update with new data or population shifts would need prespecified update protocols. These regulatory expectations differ across intended uses: a screening tool for population-level triage, a symptom monitoring system for longitudinal tracking, an automated scoring aid for cognitive tests, and a clinical decision support system. Each carries distinct evidence requirements, risk classifications, and deployment constraints that must be addressed individually.

Clinical validation must extend beyond discrimination metrics. Validation against established clinical endpoints (specialist-confirmed diagnosis, standardized cognitive scores such as MoCA and MMSE, disease progression on UPDRS for PD or ALSFRS-R for ALS, neuroimaging biomarkers) is a minimum requirement. Concurrent validity with these endpoints, followed by prospective predictive validation demonstrating that the tool improves outcomes or clinical efficiency, would constitute the evidence base needed for regulatory clearance and clinical adoption.

Implementation science considerations are absent from the current literature. Where in the clinical pathway would a speech-based AI tool be deployed: primary care screening, specialist triage, or remote monitoring between visits? How would results be presented to clinicians, and what actions would they trigger? How would false positives (unnecessary referrals, patient anxiety) and false negatives (missed diagnoses) be managed? No reimbursement codes exist for AI-assisted speech assessment, and integration with electronic health records remains an unsolved engineering and governance challenge. Until these practical barriers are addressed, clinical deployment will remain theoretical regardless of technical performance.

8.7. Limitations

This narrative review has four limitations that should be considered when interpreting its findings. First, narrative reviews are inherently subject to selection bias. Although we describe the search strategy, screening process, and inclusion criteria in Section 2, study selection was performed by a single author without independent duplicate screening, and the prioritization of studies relied in part on subjective judgments of relevance and methodological influence. Studies that were not indexed in the searched databases, published in languages other than English, or presented in venues outside the searched conference proceedings may have been missed. Second, publication bias is likely to affect the evidence base: studies reporting positive or high-accuracy results are more likely to be published than null or negative findings, and this review reflects the published literature as it stands. The performance metrics reported across studies should therefore be interpreted as upper-bound estimates. Third, we did not apply a formal quality assessment or risk-of-bias tool (e.g., QUADAS-2 for diagnostic accuracy studies) to the included studies. While methodological limitations of individual studies are discussed throughout the text, the absence of a standardized quality appraisal means that studies of varying rigour are presented alongside one another without formal weighting. Fourth, the rapid pace of development in foundation models and LLMs means that some findings may already be superseded by newer work not captured within our search window.

9. Conclusions

The progression from handcrafted acoustic features through self-supervised speech representations to large language models and conversational AI has expanded what can be extracted from the speech signal. Each stage has advanced analytical capability but introduced new limitations: domain mismatch for SSL models, text-only access for LLMs, and absent clinical validation for conversational AI.

The most important gap identified by this review is not technological but evaluative. Whether these capabilities translate into improved diagnosis, earlier detection, more sensitive monitoring, or better patient outcomes has not been tested in the clinical settings where such tools would be used. No speech-based AI system has yet shown the evidence base required for deployment in neurological clinical practice.

The convergence of speech foundation models and large language models creates the technical possibility of integrated neurological speech agents: systems that can conduct naturalistic clinical conversations, analyze both the voice and its content, and generate structured clinical assessments. Whether this possibility is realized in a clinically responsible manner will depend less on further technical advances than on the willingness of the field to invest in foundational infrastructure (diverse datasets, standardized benchmarks), translational partnerships (clinicians, patients, healthcare systems), and governance frameworks (ethical oversight, regulatory engagement) needed to distinguish genuine utility from impressive demonstration.