Background/Objectives: Development of Patient-Reported Outcome Measures (PROMs) in ophthalmology is limited by high patient burden during early validation. We propose an In Silico Pre-validation Framework using Large Language Models (LLMs) to stress-test instruments before clinical deployment.
Methods: The LLM generated a PROM questionnaire
[...] Read more.
Background/Objectives: Development of Patient-Reported Outcome Measures (PROMs) in ophthalmology is limited by high patient burden during early validation. We propose an In Silico Pre-validation Framework using Large Language Models (LLMs) to stress-test instruments before clinical deployment.
Methods: The LLM generated a PROM questionnaire and a synthetic cohort of 500 distinct patient profiles via a Python-based pipeline. Profiles were instantiated as structured JSON objects with detailed attributes for demographics, lifestyle, and health background, including specific clinical parameters like IOL type (Monofocal, Multifocal, EDOF) and dysphotopsia severity. To eliminate memory bias, a stateless simulation approach was used for test–retest reliability; AI agents were re-instantiated without access to prior conversation history. Psychometric validation included Confirmatory Factor Analysis (CFA) using WLSMV estimation and Differential Item Functioning (DIF).
Results: The model demonstrated excellent fit (CFI = 0.962, TLI = 0.951, RMSEA = 0.048, SRMR = 0.063), confirming structural validity. DIF analysis detected no significant bias based on age, sex, or IOL type (0/20 items flagged). Internal consistency was robust (Cronbach’s alpha > 0.80) and stateless test–retest reliability was high (ICC > 0.90), indicating stability independent of algorithmic memory. Convergent validity was established via significant correlations with NEI-VFQ-25 scores (Spearman’s: −0.425 to −0.652,). While responsive to change, known-groups validity reflected realistic clinical overlap.
Conclusions: LLM-based pre-validation effectively mirrors complex human response patterns through “algorithmic fidelity.” By identifying structural failure points in silico, this framework ensures PROMs are robust and unbiased before clinical trials, reducing the ethical and logistical burden on real-world populations.
Full article
