Design and Field Validation of a VR-Based RAG Historical NPC: Knowledge Grounding and Persona Grounding Can Fail Independently

Featured Application

The self-hosted VR platform supports unscripted spoken interaction with a historical figure in cultural-heritage and intercultural-learning settings. Its documented failure modes, logging requirements, and audit procedure provide a reusable basis for validating persona grounding, knowledge grounding, and deployment reliability in deployed VR–LLM historical NPC systems.

Abstract

Large language model (LLM)-based non-player characters (NPCs) can support unscripted historical dialogue, but their behavior in deployed educational virtual reality (VR) settings remains underexamined. This study reports the design and field validation of a self-hosted VR platform in which a retrieval-augmented generation (RAG) NPC, representing King Sejong the Great, engaged 42 international participants across 126 intercultural-learning sessions. The analytic corpus, reconstructed from retained session logs, comprised 104 sessions from 39 participants (2053 participant turns; 2053 NPC responses). Repetitive NPC response loops occurred in 33 sessions (31.7%), spanning 54 episodes and 190 responses, and co-occurred with learner-side transcript repetition in 31 of these sessions. The NPC maintained its first-person persona throughout the screened corpus, yet produced 30 biographical claims unsupported by the historical record: persona consistency did not ensure factual grounding. Prompt compliance varied by instruction type, and a post hoc implementation audit could not confirm that retrieved content contributed to individual responses. Together, these results identify persona grounding, knowledge grounding, retrieval observability, and session-level instrumentation as validation targets that deployed VR–LLM systems must address separately.

1. Introduction

Virtual reality (VR) offers an immersive medium for representing historical environments and figures [1,2,3,4,5]. Conversational agents can make cultural content more interactive, but many cultural-heritage VR applications still rely on menus, scripted exchanges, or branching dialogue [3,4]. These approaches are predictable, but they limit users to interactions anticipated during authoring. Large language models (LLMs) broaden this interaction space by supporting open-ended dialogue with virtual agents and non-player characters (NPCs) [6,7]. They also shift the design challenge from content authoring alone to runtime control, system integration, and validation. A deployed historical NPC must preserve the intended first-person identity, avoid unsupported historical claims, respond in the required language and format, maintain coherence across turns, and operate within acceptable latency. These behaviors depend on a pipeline that may include speech recognition, retrieval, prompt assembly, generation, speech synthesis, client–server communication, session management, and logging [7,8,9,10,11]. Figure 1 provides an overview of the deployed architecture and data flow before the detailed system description in Section 3.1.

Figure 1. Architecture of the deployed self-hosted VR-based RAG historical NPC platform; “learner” denotes the participant’s interaction role. Learner speech is transcribed, combined with retrieved Korean Sejong Sillok summary material and the persona specification, and answered in English. Knowledge grounding and persona grounding were designed as separate layers. The retrieval-layer annotation marks the query-construction divergence identified in the retained testbed chain, not a confirmed property of the deployed server (Section 5.4). Dialogue turns were written to server-side JSON in a single batch at session end, without per-turn timestamps or retrieved chunks.

Component tests and short demonstrations do not show how an integrated system behaves across repeated sessions and diverse inputs. Failures in generation, session continuity, logging, or prompt realization may become visible only during sustained field use or implementation audit. Field-based technical validation, as defined in Section 2.4, is needed alongside component-level evaluation.

Field logs reveal observable interaction patterns, but they do not necessarily identify their causes. Those causes may lie in speech recognition, retrieval, context construction, generation, client behavior, or logging. This study combines field-log analysis with a post hoc implementation audit of retained client, prompt, retrieval, logging, and server artifacts. Where the available evidence does not establish a direct link between a mechanism and a specific field outcome, the analysis reports plausible mechanisms rather than turn-level causes.

A second challenge concerns multilingual retrieval-augmented generation (RAG), which retrieves external evidence and incorporates it into the generation context [8,11]. The platform examined here used an asymmetric multilingual configuration: participants could speak in English, Korean, or other supported languages, while the historical NPC responded only in English. This asymmetric configuration introduces cross-lingual retrieval and validation requirements [12,13,14]. The presence of a vector index alone does not demonstrate effective knowledge grounding; retrieval must be observable and verifiable within the deployed pipeline, at minimum through per-turn records that link retrieved evidence to generated responses (Section 6.3).

This challenge also requires a distinction between persona grounding and knowledge grounding. A response may preserve the intended King Sejong persona while containing an unsupported historical claim; the two properties therefore require separate validation.

This study examines a self-hosted VR historical NPC representing King Sejong the Great (1397–1450), the monarch who promulgated Hangul, the Korean alphabet. International participants conducted spoken interviews with the embodied NPC in a reconstructed palace environment. The system combined a Unity VR client with speech recognition, RAG over Korean-language Sejong Sillok summary material [15], LLM generation, text-to-speech output, and server-side logging. The platform was deployed across 126 sessions with 42 international participants, yielding a final analytic corpus of 104 sessions from 39 participants (2053 participant turns; 2053 NPC responses), reconstructed as described in Section 5.2.

The paper makes three technical contributions. First, it documents the architecture and field behavior of a self-hosted VR–LLM historical NPC system. Second, it treats persona grounding and knowledge grounding as independently testable properties. Third, it combines field use with post-hoc implementation audit to distinguish observed failures from plausible, but not necessarily turn-level identifiable, implementation mechanisms and to derive validation requirements for future deployments (RQ4). The companion study [16] uses the same deployment but examines learning outcomes with different primary data and research questions. The present paper focuses on system behavior, implementation evidence, and validation requirements. Participant characteristics and conditions are reported only to establish the deployment context (Section 5.1).

The study addresses the following research questions:

• RQ1: What architecture supports unscripted spoken interaction with a historical NPC on a self-hosted open-weights stack, and what technical trade-offs arise from this configuration?
• RQ2: How were multilingual RAG-based knowledge grounding and prompt-based persona grounding implemented, and what evidence is required to validate each layer independently in a deployed system?
• RQ3: What technical and interactional failure modes emerged during development, piloting, and repeated field deployment, and what evidence documents their recurrence or constrains their interpretation?
• RQ4: What validation requirements follow for future deployed VR–LLM historical NPC systems?

2. Related Work

This section situates the study in three areas: generative conversational agents in immersive VR, persona and knowledge grounding, and validation practices for deployed LLM-based systems. These areas frame the technical questions addressed in RQ1–RQ4.

2.1. Conversational Agents in Immersive VR: From Authored Dialogue to Generative NPCs

VR supports interaction within reconstructed environments [1,2]. In cultural heritage applications, immersive environments and virtual characters have been used to represent historical spaces, guide users, and strengthen perceived authenticity and presence [3,4,5]. Spoken interaction adds further requirements for speech processing, interface design, and conversational stability [17,18].

Earlier pedagogical and conversational agents often relied on rule-based dialogue management or pre-authored responses, which kept behavior under designer control but limited interactional flexibility. With LLMs, agents can produce open-ended and context-sensitive responses. Generative agents can sustain apparently coherent behavior [6], and recent LLM-agent architectures increasingly incorporate memory, planning, and role-control mechanisms [7]. Much of this work, however, has been evaluated through benchmarks, simulations, or relatively short demonstrations. Less evidence is available on how self-hosted open-weights models behave when embedded in real-time spoken-interaction VR pipelines that combine speech recognition, retrieval, response generation, synthesis, session management, and logging under repeated participant use. RQ1 addresses this gap.

2.2. Persona Grounding and Prompt-Based Control of LLM NPCs

Prompt-based control is commonly used to assign identity, role, and interaction style to an LLM. Role-playing agents can be guided through natural-language role instructions [19], while in-context examples can further condition model behavior [20]. Such control is not deterministic; models may drift from assigned roles or follow style and behavioral instructions inconsistently. Retrieval-based context adaptation has also been explored as a way to improve role consistency [10].

In small, locally deployed models, less is known about how individual prompt instructions hold up under sustained use. Instruction-following may also depend on where a constraint is implemented in the wider system pipeline, particularly in real-time systems, where prompt behavior interacts with dialogue history, retrieval context, model capacity, and output constraints.

For historical NPCs, persona grounding and knowledge grounding are related but independently testable. A response may remain consistent with the assigned persona while containing unsupported factual content, or convey historically relevant information while failing to speak as the intended character. The literature reviewed here has more often examined persona consistency and factual grounding separately than tested how they may fail independently within the same deployed system. RQ2 therefore examines how the two properties were implemented and what evidence is needed to validate them independently.

2.3. Knowledge Grounding and Multilingual RAG

RAG supplements generation with retrieved external evidence [11] and is widely used in knowledge-intensive applications [8]. For historical NPCs, it offers a way to connect generated dialogue to documentary sources rather than relying only on the model’s parametric knowledge. Unsupported generation and factuality are recognized problems in natural language generation (NLG) [9].

However, adding a retrieval component does not guarantee effective grounding. The quality and use of retrieved context depend on corpus construction, document granularity, chunking, embedding-model suitability, query formulation, retrieval thresholds, reranking, and prompt assembly [8]. A pipeline may contain a functioning retrieval index while still providing weak or repetitive context to the generator. Retrieval should be evaluated at the level of the deployed chain, rather than inferred from the presence of RAG as an architectural label.

Historical applications also involve corpus-granularity trade-offs between contextual detail and retrieval specificity. The platform examined in this study used Korean-language summary material derived from the Korean translation of the Sejong Sillok [15]. The recoverable corpus and index configuration are examined in Section 5.4.

Multilingual RAG adds further constraints when input, corpus, and output languages differ. Cross-lingual RAG requires semantic alignment across languages [12], and multilingual RAG studies have reported output-language drift and language-dependent variation in retrieved evidence [13,14]. In the present asymmetric configuration, the pipeline was required to mediate among multilingual participant input, Korean-language retrieved evidence, and English output. The retained logs do not establish that meaning was preserved consistently across these language transitions. With per-turn retrieval records and a retrieval-free baseline unavailable, this study examines the deployed retrieval configuration and its verifiability rather than treating retrieval as a verified contributor to response quality (Section 5.4 and Section 6.3).

2.4. From Controlled Demonstrations to Field Validation

The validation of LLM-based conversational systems requires a system-level view. Human–computer interaction (HCI) research has long distinguished controlled laboratory studies from in-the-wild deployments, where system behavior is shaped by real users, repeated use, environmental variation, and operational constraints [21]. Recent LLM evaluation work similarly argues that benchmark performance does not fully represent behavior in actual sociotechnical settings [22].

The present study does not provide a comparative benchmark against another LLM-NPC system. It evaluates one deployed system through field-log analysis and retrospective implementation audit; accordingly, the reported failure rates are within-system observations rather than comparative performance estimates.

Component tests and short demonstrations remain useful, but they cannot establish the reliability of an integrated pipeline. Nor can they show whether failures of the kinds examined in this study recur across sessions. In this study, field validation refers to the technical and behavioral assessment of the deployed system, not to measurement of learning outcomes. Relevant evidence includes dialogue logs, implementation artifacts, grounding behavior, logging completeness, interaction stability, and recurrent failure modes. Under this framing, RQ3 asks which failures occurred during deployment, what implementation evidence supports or constrains their interpretation, and which failures remained difficult to detect during development-stage testing.

3. System Design and Implementation

3.1. System Overview and Architecture

The platform was implemented as a self-hosted HTTP client–server system. Three requirements guided its design: unscripted spoken interaction, retrieval of historically relevant source material, and responsiveness judged during pilot testing to be compatible with conversational turn-taking in VR.

A Unity client [23] running on Meta Quest 3 rendered the reconstructed palace environment and the embodied King Sejong NPC, captured participant speech, and transmitted raw 16-kHz audio to a self-hosted Python (version 3.13.3) dialogue server. The client selected an output-mode-specific endpoint according to the mode allocated to the participant. The server exposed REST endpoints for audio upload, response delivery, prompt installation, dialogue-history reset, and session storage.

Locally hosted components and open-weights models were used instead of commercial APIs so that participant speech and interaction data remained within researcher-controlled infrastructure. This choice involved capability and responsiveness constraints judged in pilot testing rather than by formal timing measurement. During development, Gemma 3 12B was replaced by Gemma 3 4B, and the small Whisper model was selected over the medium model, to improve responsiveness for spoken interaction. These decisions prioritized conversational turn-taking on the available hardware rather than maximum model capacity. Table 1 summarizes the deployed hardware and component configuration.

Table 1. Technical configuration of the deployed platform.

Component	Technology or Configuration	Function
VR client	Unity [23] on Meta Quest 3	Environment rendering; embodied NPC; microphone capture and 16-kHz audio upload; modality-specific output
Client–server communication	HTTP REST with output-mode-specific and session-management endpoints	Audio upload; response delivery; prompt installation; dialogue-history management; session storage
Dialogue server	Python on a researcher-managed cloud GPU instance (AWS; 4 vCPU, 16 GiB RAM, NVIDIA A10G GPU)	Server-side transcription, retrieval, generation, speech synthesis, session handling, and logging
Speech recognition	Whisper small multilingual model [24,25], locally hosted	Transcription of participant speech in the learner role
Response generation	Gemma 3 4B [26] via Ollama [27]; LangChain [28]	Persona-conditioned English response generation
Knowledge retrieval	FAISS IndexFlatL2 [29] with nomic-embed-text [30], k = 5	Retrieval from the recoverable Sejong Sillok summary index (Section 3.4)
Speech synthesis	Coqui TTS [31]	Synthesized speech output in the audio mode
Persona specification	Prompt file embedded in the Unity client and configured for installation at session start	Prompt-level specification of persona identity, response style, and output language (Table 2, Appendix A)
Data logging	Server-side JSON written in a single batch at session end; each record stored turn type and text content, without per-turn timestamps or retrieved chunks; session metadata encoded in the filename	Dialogue records used for field analysis

Table 1. Technical configuration of the deployed platform.

Component	Technology or Configuration	Function
VR client	Unity [23] on Meta Quest 3	Environment rendering; embodied NPC; microphone capture and 16-kHz audio upload; modality-specific output
Client–server communication	HTTP REST with output-mode-specific and session-management endpoints	Audio upload; response delivery; prompt installation; dialogue-history management; session storage
Dialogue server	Python on a researcher-managed cloud GPU instance (AWS; 4 vCPU, 16 GiB RAM, NVIDIA A10G GPU)	Server-side transcription, retrieval, generation, speech synthesis, session handling, and logging
Speech recognition	Whisper small multilingual model [24,25], locally hosted	Transcription of participant speech in the learner role
Response generation	Gemma 3 4B [26] via Ollama [27]; LangChain [28]	Persona-conditioned English response generation
Knowledge retrieval	FAISS IndexFlatL2 [29] with nomic-embed-text [30], k = 5	Retrieval from the recoverable Sejong Sillok summary index (Section 3.4)
Speech synthesis	Coqui TTS [31]	Synthesized speech output in the audio mode
Persona specification	Prompt file embedded in the Unity client and configured for installation at session start	Prompt-level specification of persona identity, response style, and output language (Table 2, Appendix A)
Data logging	Server-side JSON written in a single batch at session end; each record stored turn type and text content, without per-turn timestamps or retrieved chunks; session metadata encoded in the filename	Dialogue records used for field analysis

Table 2. Principal elements of the deployed persona prompt.

Prompt Element	Intended Technical Function	Targeted Output Aspect
First-person King Sejong identity; no self-reference as a digital system or language model	Prompt-level persona specification	Response content and perspective
Three to six sentences, 10–20 s of spoken output; one main idea	Prompt-level response-length and focus guidance	Response structure
English-only response with a fixed redirection line	Prompt-level output-language instruction	Response language
Three alternative closing invitations	Prompt-level dialogue-continuation cue	Response ending
No repetition of prior responses or phrasing; new details when a topic recurs	Prompt-level repetition-avoidance instruction	Intended response variation across turns
Topic, curiosity, and follow-up tags [Topic:…], [Curious], [Follow-up]	Dialogue-continuity cueing	Tag use and continuity cues
Three example question-answer pairs	Answer-style scaffolding	Response form
Personal and warm interactional style (including phrases such as “my people,” “in my heart”)	Persona vividness	Tone and direct address

Table 2. Principal elements of the deployed persona prompt.

Prompt Element	Intended Technical Function	Targeted Output Aspect
First-person King Sejong identity; no self-reference as a digital system or language model	Prompt-level persona specification	Response content and perspective
Three to six sentences, 10–20 s of spoken output; one main idea	Prompt-level response-length and focus guidance	Response structure
English-only response with a fixed redirection line	Prompt-level output-language instruction	Response language
Three alternative closing invitations	Prompt-level dialogue-continuation cue	Response ending
No repetition of prior responses or phrasing; new details when a topic recurs	Prompt-level repetition-avoidance instruction	Intended response variation across turns
Topic, curiosity, and follow-up tags [Topic:…], [Curious], [Follow-up]	Dialogue-continuity cueing	Tag use and continuity cues
Three example question-answer pairs	Answer-style scaffolding	Response form
Personal and warm interactional style (including phrases such as “my people,” “in my heart”)	Persona vividness	Tone and direct address

3.2. VR Client and Development Testbed

During development, a lightweight Godot-based desktop client [32] served as a pipeline testbed and communicated with a separate WebSocket server [33] retained in the project archive. That server, its port, and its protocol did not appear in the deployed Unity client, indicating that the participant-facing system did not communicate with the retained testbed server. All field sessions used the final Unity VR build.

Participants entered a self-chosen nickname at session start. Dialogue history was managed on the server and keyed by this nickname. The system did not assign an immutable server-generated participant or session identifier. The resulting reconciliation requirements are described in Section 5.2. The Unity client held no transcript and sent only the current participant utterance as raw audio. After transcription, the server appended the participant utterance and generated NPC response to the session history. Source audio was not retained after transcription. At session start, the client reset the history through a dedicated endpoint; at session end, the accumulated dialogue was written to a JSON file in a single batch. The resulting logging constraints are examined in Section 5.2 and Section 5.4.

3.3. Server-Side Turn Processing

In the recoverable server chain, each turn proceeded through four stages: transcription, retrieval of five corpus chunks, context assembly, and response generation. The recovered client defines the server interface as a set of HTTP endpoints, including separate text-to-text, audio-to-text, and audio-to-audio turn routes, together with set_prompt for installing the persona specification, reset_history for clearing server-held dialogue state, remake_rag for rebuilding the retrieval index, and store_user_history for persisting the session log. The recovered client configuration specified a self-hosted HTTP deployment address on port 5000.

The recoverable chain assembled the model input from a Korean-language retrieval scaffold, the client-supplied English persona specification, five retrieved Korean corpus chunks, the server-held dialogue history, and the current learner utterance. Gemma 3 4B generated the English response, which was synthesized in the audio condition or displayed in the persistent-text condition.

Since the exact server build and retrieval index active during field deployment cannot be identified, this sequence is reported as the recoverable implementation rather than as a complete reconstruction of every deployment-time turn. The effective retrieval-query construction and associated evidential limitations are examined in Section 5.4.

3.4. RAG over the Korean Sejong Sillok Corpus

The corpus prepared for retrieval consisted of official Korean article-level summaries derived from the Korean translation of the Sejong Sillok, the section of the Annals of the Joseon Dynasty documenting King Sejong’s reign [15]. The summaries were collected from the National Institute of Korean History’s online database and organized as 33 year-level PDF documents covering the accession year through the thirty-second year of the reign. The corpus contained 31,340 entries and approximately 817,000 Korean characters, with each entry representing a one-sentence summary of an individual annal article; the resource therefore operated at event-summary rather than full-text granularity (Section 2.3).

Retrospective inspection of the recoverable index identified 4179 chunks derived from the 33 year-level documents. The surviving configuration specified recursive splitting with a maximum chunk length of 300 characters and a 50-character overlap. The audited document store contained no extraneous source material and only two duplicated chunk texts, corresponding to four chunks (0.1%). Ten byte-identical local copies of the audited 4179-vector index were identified. However, a separate earlier build containing 1337 vectors and a different content hash was also found under the same directory name. The client also exposed a remake_rag route that could rebuild and overwrite the index at runtime, so the exact build served during field deployment cannot be identified (Section 3.3). The index-comparison and document-store audits are reported in Section 5.4.

The recoverable configuration used nomic-embed-text [30], with vectors stored in a FAISS IndexFlatL2 index [29]. Nomic-embed-text was not designed or validated specifically for Korean or cross-lingual retrieval in the retained project documentation, and no pre-deployment test established its suitability for this Korean corpus or for potentially multilingual learner queries. Language and model fit were therefore unvalidated design assumptions. In the recoverable chain, the five nearest chunks were returned without a distance threshold or reranking, so every query supplied five chunks to the generation stage regardless of retrieval distance. The effective retrieval-query construction and final context-assembly process are examined in Section 5.4.

3.5. Language Policy and Output Modality

The NPC generated English-only responses in both output modes. In the audio mode, responses were delivered as synthesized English speech without subtitles. In the persistent-text mode, responses were displayed in English without speech and remained visible for rereading.

Participants could speak English, Korean, or another language supported by the speech recognition model. The English-only response policy was specified in the deployed prompt but was not enforced through deterministic pipeline-level filtering. The prompt instructed the NPC to reply in English regardless of input language and to redirect non-English input toward continued English interaction (Section 3.6).

The two output modes used the same virtual environment, embodied character, and visual character animations. Output mode was allocated sequentially by the researcher rather than randomly assigned. At session start, each participant selected the button corresponding to the allocated audio or persistent-text mode; this loaded the appropriate client scene and directed requests to the associated server endpoint (Section 3.1). The deployed prompt contained no output-mode-specific instruction. The modes differed in output channel while retaining the same model specification and visual embodiment. The rendering instability identified during piloting in the persistent-text mode is described in Section 4.6.

3.6. Persona Prompt and Deployed Specification

Three prompt artifacts were retained from development: an exploratory specification that incorporated learning-construct labels for intercultural communicative competence (ICC) [34] and cultural self-awareness (CSA) [35], an intermediate persona-and-style file dated 4 June 2025, and the YAML-structured prompt dated 23 July 2025 reproduced in Appendix A. Comparison of their distinctive markers with the retained field responses supports treating the first two files as development-stage revisions and the July file as the deployed specification. The detailed marker audit is reported in Section 5.3. The repeated gratitude response beginning “It is my pleasure entirely” appears in neither the intermediate nor the deployed prompt file and is therefore treated as observed model behavior rather than direct prompt-template reuse (Section 6.2).

The retained Unity client contains the July prompt as a text asset and installs its contents through the server’s set_prompt endpoint at the start of each session. This establishes that the persona specification was supplied by the client rather than stored as a server-side constant. The server-side prompt state was not logged for each session, however, so the exact prompt state applied in every field session cannot be independently verified. The specification provided prompt-level guidance for persona identity, English-only output, response length and style, dialogue-continuation cues, repetition avoidance, and answer form. These requirements were expressed as natural-language instructions. No enforcement through constrained decoding, repetition filtering, or deterministic post-processing was identified in the recovered prompt, client, or testbed artifacts; additional enforcement in the unrecovered deployed HTTP server cannot be ruled out. Table 2 summarizes the principal prompt elements and their intended functions; their field realization is analyzed in Section 5.3.

4. Iterative Development and Failure Modes

4.1. Development Timeline

Retained project artifacts are consistent with this sequence: an initial retrieval index dates to 19 April 2025, the Unity client project to May 2025, the server code and the audited retrieval index to June 2025, and the deployed client prompt specification to 23 July 2025. The coexistence of an April index with the June audited build, together with the client’s runtime index-rebuild route, indicates that the retrieval configuration changed during development, consistent with the index-uncertainty already established in Section 3.4. Table 3 summarizes the principal development phases and pre-deployment design responses, while Figure 2 traces how the five failure modes were first detected, mitigated, or carried into field deployment. The figure is intended as a trajectory summary; detailed field prevalence and implementation-audit evidence are reported in Section 5.

Table 3. Development timeline and pre-deployment design responses (2025).

Phase	Main Activity	Issue Observed	Design Response/Outcome
April	Architecture, model selection, Sejong Sillok corpus preparation	Gemma 3 12B was judged insufficiently responsive during qualitative pilot testing on the available hardware; no formal latency benchmark was retained.	Model reduced to Gemma 3 4B; persona specification redesigned
May	Unity VR client prototype; prompt iteration	VR integration complexity; slow test cycles	Godot desktop testbed for independent pipeline testing
June	Server deployment (Python/AWS); FAISS index; IRB submission (25 June), testbed prompt finalized	None isolated	Knowledge (retrieval) and persona (prompt) layers integrated
July	Integrated and concurrent-connection testing; final Unity prompt	Pseudo-lexical generation (FM2); concurrent session/connection conflicts (FM4)	Lexical instructions drafted, not deployed (Section 4.3); session and connection handling revised
August	System stabilization; IRB approval (13 August); internal technical pilot sessions	Persistent-text rendering instability (FM5)	Rendering corrected before persistent-text field data collection

FM = failure mode. FM3 (repetitive response loops) is not listed because it was not detected before field use.

Figure 2. Trajectories of the five failure modes across development, integrated testing, piloting, and field deployment. FM1 and FM2 were identified during development; FM1 was not detected in the pre-recovery screening, whereas FM2 showed no clear recurrence in the analyzed field logs. FM3 was first identified during field deployment; FM4 was identified during integrated testing, with residual session-continuity issues observed in the field (Section 4.5); and FM5 was resolved before persistent-text field data collection. Detailed field outcomes and implementation-audit evidence are reported in Section 5.

4.2. Failure Mode 1: Persona Self-Recognition Failure

Early development versions occasionally described King Sejong in the third person or responded from the perspective of an external narrator rather than maintaining the intended first-person identity. This failure indicated that providing historical content alone was insufficient to preserve first-person character identity, motivating a stronger persona specification.

Mitigation was applied at the prompt layer. The deployed specification explicitly defined the NPC’s first-person King Sejong identity, prohibited the model from mentioning being an AI or digital system, and instructed it to remain in character as the interviewed historical figure. No instance of the predefined persona self-recognition failure was identified in the screened retained corpus (Section 5.3). This screening is reported separately from the newly reconstructed repetition analysis.

4.3. Failure Mode 2: Pseudo-Lexical Generation

Development testing also revealed occasional pseudo-lexical generation: plausible-sounding romanized Korean expressions were presented as if they were authentic Korean terminology, despite lacking a verifiable source form. Such output posed a validation risk: learners could interpret invented vocabulary as culturally or historically valid information.

An exploratory prompt track introduced instructions prohibiting invented romanized Korean terms and requiring Korean expressions to be explained in English. These clauses were not included in the prompt ultimately deployed in the Unity system (Section 3.6).

No clear recurrence of this failure was identified in the analyzed field logs. Since the drafted mitigation was absent from the deployed specification, its disappearance cannot be attributed to that prompt intervention.

4.4. Failure Mode 3: Repetitive Response Loops

Repetitive response loops were not detected during development, integrated testing, or piloting, and no repetition-specific mitigation was implemented before deployment. The failure was first identified in the field logs. In the final session-level reconstruction, 33 of 104 sessions contained at least one repetition episode (31.7%), comprising 54 episodes and 190 NPC responses (Section 5.3).

The available records do not establish whether repeated learner transcripts caused individual NPC repetitions or whether those inputs originated from learner behavior, speech recognition, or another upstream process. The recovered client logic used an interaction flag to block the recording control while a response was pending, making immediate re-pressing during the wait an unlikely explanation under the recovered implementation. Retrieved chunks, source audio, and per-turn timestamps were not retained, limiting turn-level causal tracing. Section 5.3 reports the field distribution, and Section 5.4 examines the remaining plausible mechanisms.

4.5. Failure Mode 4: Session and Connection Fragility

Integrated testing exposed session and connection-handling problems under concurrent use, prompting revisions before field deployment. No cross-session dialogue contamination was identified in the field logs. In the final retained-file reconstruction, one participant-session unit (P038 S03) was represented by two JSON files, which were ordered by timestamp and merged before analysis; no exact duplicated turn was detected at the merge boundary. Content-level verification further showed that two files that label-based matching had treated as same-session fragments belonged to other participants: one was the first session of P016, and the other was a P025 session conducted concurrently with P020 S03 and saved one minute earlier (Section 5.2). These cases demonstrate that session continuity and file persistence were not fully robust, but the retained evidence does not identify the cause of any individual split.

The deployment used a wireless client connection from 20 to 28 August 2025 and a wired connection from 29 August 2025 onward. The single multi-file session unit and the three reassigned files all came from sessions conducted after the change, so the retained cases do not support a wireless-versus-wired comparison or a causal interpretation of network type. The batch-at-end logging design made completeness vulnerable to interruption. The deployed HTTP server was not recovered, and the separate WebSocket testbed is not treated as evidence of the deployment-time failure mechanism (Section 5.2 and Section 5.4).

4.6. Failure Mode 5: Persistent-Text Rendering Instability

The persistent-text condition introduced interface requirements not present in the audio condition: responses had to remain legible, correctly positioned, and available for rereading throughout the interaction.

Rendering instability was detected during internal team piloting, conducted before participant sessions and excluded from the 126-session count. Persistent-text field data collection was delayed until the issue had been corrected; after resolution, the audio and persistent-text conditions were run in parallel (Section 5.1).

FM5 differs from the other failure modes in that it was identified and resolved before persistent-text field data collection, representing a pre-deployment validation case rather than a field failure.

5. Field Deployment, Performance, and Implementation Audit

5.1. Deployment Context

Following stabilization and institutional review board approval, 48 international participants were recruited from the university community. Forty-two completed three VR interviews each (126 conducted sessions) between 13 August and 31 October 2025, with at least five days between sessions. They represented 17 nationalities and ranged in age from 18 to 54 years. Language background was classified by prespecified criteria (20 monolingual-dominant, 22 active bilingual). The researcher allocated output mode sequentially, without randomization (21 audio, 21 persistent text). Logs from 39 completers were included in the final session-level analysis; records for three completers could not be recovered (Section 5.2). All field sessions used the final Unity client, but the retained records do not establish that the same server build, retrieval index, or server-side prompt state was active throughout deployment. This technical validation analyzes dialogue logs and implementation artifacts; learner outcomes are reported separately [16].

5.2. Interaction Logging and Data Preparation

Each field session was intended to produce a server-side JSON history with speaker type (human or AI) and text for each turn. The filename encoded the participant identifier, session number, language-background group, output mode, and timestamp (recorded in UTC and converted to UTC+9 for analysis). The recovered client called the session-storage endpoint only at exit (Section 3.2; Figure 1), so an interruption before that request could leave no analyzable record. Source audio, per-turn timestamps, latency, retrieved chunks, detected input languages, and error events were not retained.

Participant-entered identifiers required archive-level reconciliation. Labels were associated with 45 individuals: 42 completed the protocol, and 39 were represented in the final analytic corpus. Among the 45 archive-linked individuals, 33 had at least two raw label variants, comprising 56 non-canonical instances. Case-insensitive normalization and fuzzy matching were followed by manual review of participant and session records, timestamps, and file contents. The supplementary reconciliation record documents the adjudications.

The retained ledger contained 106 JSON files. One pair belonging to the same participant-session (P038 S03) was ordered by timestamp and merged; no exact normalized turn overlap was found at the merge boundary. Because participant-entered labels were unreliable, the attribution of each file was also verified against dialogue content, file timestamps, output-condition labels, and field records. This verification reassigned three files that label-based matching had attributed to other sessions: a six-turn file appended to P015 S03 was the first session of P016; an eleven-turn file merged into P020 S03 was a P025 session conducted concurrently and saved one minute earlier; and a 22-turn file assigned to P025 S02 was the second session of P030. One unsupported one-turn P044 fragment between the documented S02 and S03 records was excluded because no fourth or replacement session was documented. This yielded 104 analytic sessions from 39 participants, with 2053 participant turns and 2053 NPC responses. Text was normalized with NFKC, standardized quotation marks, collapsed whitespace, and case-insensitive matching. Closing expressions were counted only at response end; sentence counts excluded bracketed tags and parenthetical content.

Repetition was recalculated on the 104 reconstructed session histories. An episode comprised two or more consecutive NPC responses identical after normalization. The same rule was applied independently to participant transcripts for the co-occurrence analysis. The session-level file, merge audit, exclusion record, and derivation script are described in the Supplementary Materials; rerunning the script from source requires authorized access to the protected JSON histories.

Persona self-recognition was screened for artificial-identity disclosures and third-person references to King Sejong. Non-Latin transcripts in the speech-recognition audit were treated as possible recognition anomalies, not proof of the spoken language or its cause.

5.3. Field Performance Indicators

Repetitive NPC responses occurred in 33 of 104 sessions (31.7%), comprising 54 episodes and 190 of 2053 NPC responses (9.3%); eight episodes exceeded five responses (Figure 3). Session prevalence was 35.0% (7/20) for bilingual-audio, 37.0% (10/27) for monolingual-audio, 39.3% (11/28) for bilingual-text, and 17.2% (5/29) for monolingual-text. The four-cell comparison was not significant, chi-square(3) = 3.998, p = 0.262. The modality by language-background interaction was also not significant in logistic regression (OR = 0.637, 95% CI [0.096, 4.212], p = 0.640) or participant-clustered GEE (p = 0.621). Output mode was allocated sequentially, so these comparisons are exploratory associations, not causal estimates. NPC response count was associated with observing at least one repetition episode in logistic regression (OR = 1.082 per response, 95% CI [1.034, 1.133], p < 0.001) and participant-clustered GEE (OR = 1.083, 95% CI [1.038, 1.130], p < 0.001). We interpret this association as reflecting greater opportunity to observe repetition in longer histories; it does not establish that session length caused repetition.

Figure 3. Repetition patterns in the 104-session analytic corpus. (a) Length distribution of 54 NPC repetition episodes. (b) Session-level prevalence across language-background × output-mode cells: bilingual-audio 35.0%, bilingual-text 39.3%, monolingual-audio 37.0%, and monolingual-text 17.2%. Overall prevalence was 33/104 (31.7%), indicated by the dashed line in (b).

Among the 33 repetition-positive sessions, the first episode began at a median of the seventh NPC response. Total response count was not significantly associated with onset position (Spearman’s ρ = 0.230, p = 0.199, N = 33). This analysis includes only repetition-positive sessions and does not treat unaffected sessions as censored observations. It is therefore an affected-session descriptive analysis rather than a formal time-to-event model.

Consecutive participant-transcript repetition occurred in 55 sessions, comprising 92 episodes and 262 of 2053 participant turns (12.8%). Independently applying the same rule showed co-occurrence in 31 of the 33 NPC-repetition-positive sessions. This session-level overlap does not establish causation; the logs cannot distinguish repeated speech from recognition or other upstream behavior. Table 4 shows the longest NPC episode.

Table 4. Longest NPC repetition episode: responses 4–16 of 43 NPC responses in one session.

Response Position	Speaker	Utterance
3–16	Participant	“Thank you” repeated across 14 participant turns; punctuation varied.
4–16	NPC	“It is my pleasure entirely. I am glad you found our conversation agreeable. Would you like to know more?” repeated across 13 NPC responses.

Participant A, Session 1 (active bilingual, persistent-text condition). Participant and NPC positions are counted separately within each speaker’s sequence. Repetition was determined after text normalization.

Prompt compliance varied (Figure 4). Response-length compliance was inconsistent, with non-compliant responses occurring predominantly above the specified upper bound. The three exact closing exemplars appeared in 129 responses (6.3%); a response-final “know more” form appeared in 700 (34.1%), and a response-final “Would you like…?” invitation in 1838 (89.5%).

Figure 4. Field realization of selected deployed prompt instructions in the final 2053-response corpus. (a) Learner-visible prompt tags; per-tag counts are non-exclusive. (b) Response-final closing forms: the three exact exemplars accounted for 129 responses (6.3%; 92, 25, and 12 occurrences), the response-final “know more” form for 700 (34.1%), and a response-final “Would you like…?” invitation for 1838 (89.5%).

All 2053 NPC responses were in English, and the fixed redirection sentence appeared verbatim in 289 (14.1%). The predefined screen found no persona self-recognition failure. Nevertheless, 30 unsupported references to Prince Jeongjo occurred across 12 sessions, including 17 responses calling him “my son.” First-person persona identity therefore did not ensure historical accuracy.

Across 135 log files, 30 NPC references to Jeongjo were identified in 12 sessions. No learner utterance explicitly mentioned Jeongjo, Jeong-jo, 정조, or another identified spelling variant; all 30 references were introduced first by the NPC. Classification used the preceding learner turn together with the immediately preceding NPC turn when the learner utterance was fragmentary or referential. Under this context-aware coding, the references occurred in family or succession threads (learner turns asking about, or continuing a thread on, King Sejong’s children or succession; 14 occurrences), verse, poetry, or music threads (8 occurrences), and open-ended historical threads (8 occurrences). Three fragmentary follow-up turns accounted for the difference from a conservative learner-turn-only coding, which yielded 11, 8, and 11 occurrences, respectively. Among the 18 within-session intervals between successive references, 13 continued or revisited the conversational thread associated with the preceding reference. These results are consistent with a local conversational-context association, but they do not identify whether the unsupported relationship arose from retrieval, dialogue history, generation, or another component.

Prompt-specified tags [Topic], [Curious], and [Follow-up] were visible to participants in 1023 responses (49.8%). Their respective response counts were 724, 725, and 413; the counts overlap because one response could contain multiple tags. The recovered client exposed no separate channel for these tags (Section 6.2).

5.4. Post Hoc Implementation Audit

The audit examined the recovered Unity client, deployed prompt file, FAISS index and document store, raw logs, development artifacts, and surviving WebSocket testbed. The deployed HTTP server was not recovered. The client directly documents prompt installation, client-side turn gating, and the exit-triggered storage request; the testbed documents a retrieval and context-assembly chain that cannot be confirmed as deployed. The audited index matches the testbed chunking configuration (300-character maximum, 50-character overlap) and contains 4179 chunks from 33 sources, but this does not identify the server or index build used in every field session. Retrieval replay therefore evaluates a recoverable configuration, not the historical deployment.

First, no retained pre-deployment test established the embedding configuration’s suitability for Korean or cross-lingual retrieval, and deployment-time retrieved chunks were not logged. A replay of the recoverable index used 49 sampled queries and produced 245 top-five query-chunk judgments: 225 irrelevant, 20 partially relevant, and none fully relevant. Under the strict score-2-only criterion, Precision@5 and MRR were both 0.000. Under a relaxed criterion that counted scores of 1 or 2 as relevant, Precision@5 was 0.082, MRR was 0.130, hit rate was 0.204 (10/49 queries), and conventional query-normalized nDCG@5 was 0.151. An additional absolute-normalized nDCG@5, using an ideal list with score 2 at all five ranks, was 0.028. For the 40 category-stratified queries, the corresponding values were 0.095, 0.147, 0.225, 0.169, and 0.033; for the nine Jeongjo queries, they were 0.022, 0.056, 0.111, 0.070, and 0.008. Per-query results and both nDCG definitions are reported in the Supplementary File S1 retrieval_per_query_relaxed.csv.

Relevance labels were assigned by the OpenCode big-pickle language model using the fixed 0–2 rubric. A researcher independently rated a blind subsample of 30 items using a binary 0/1 scale; agreement with the binarized model labels was 70.0% (21/30), with Cohen’s kappa = 0.18. The model labeled 27 of 30 items irrelevant, compared with 20 of 30 for the researcher, indicating conservative bias in the model labels relative to human judgment. No 245-item second-rater file was produced; accordingly, inter-rater agreement for the full retrieval audit is not reported, and the audit remains exploratory.

A consistent-encoding replay over the 4179-chunk corpus found a mean pairwise cosine similarity of 0.965 for nomic-embed-text, with 99.9% of sampled pairs at or above 0.9. Rebuilding the embeddings and IndexFlatL2 retrieval structure with BGE-M3 while holding the 49 queries, corpus, chunking, and top-five retrieval procedure constant reduced mean pairwise similarity to 0.634 and the share at or above 0.9 to 0.0%. The maximum share of top-1 results assigned to one chunk fell from 38.8% to 12.2%, and the number of distinct chunks represented in the pooled top-five results increased from 88 to 172. These diagnostics show that BGE-M3 produced more discriminative representations for this corpus; they do not provide comparative relevance effectiveness because the new top-five results were not independently relevance-rated.

Full diagnostics are reported in the Supplementary File S1 replay_diagnostics.json.

The corpus search was also expanded beyond the Hangul string 정조. All 87 occurrences of 정조 referred to 正朝 (New Year’s Day court assembly) or 正朝使 (New Year’s tribute envoy), not King Jeongjo (正祖). Searches for 正祖, 李祘, 이산, and 정조대왕 returned zero hits. Because the Sejong Sillok predates King Jeongjo by approximately three centuries, this absence confirms the scope of the corpus rather than a retrieval failure.

Second, instrumented re-execution of the testbed showed that retrieval used serialized model input, including persona specification and dialogue history, rather than the current utterance alone. The testbed also contained an unlogged retrieval-free fallback after index-loading errors. Because neither fallback activation nor retrieved chunks were recorded, retained sessions cannot be classified reliably as retrieval-active or retrieval-free; a valid baseline comparison cannot be reconstructed post hoc. The recovered client also included a route that could rebuild and overwrite the index. These behaviors identify risks in retained artifacts, but none can be verified for the unrecovered deployed HTTP server.

Third, the recovered client issued a single storage request at exit, leaving histories vulnerable to interruption. The 106 retained JSON files required one same-session pair merge, three content-verified file reassignments, and one fragment exclusion (Section 5.2). Neither the unrecovered HTTP server nor the WebSocket testbed can establish the server-side cause of an individual missing session. No cross-session content contamination was found.

Fourth, participant-entered identifiers were neither unique nor stable across records, requiring the reconciliation in Section 5.2; label-based matching alone misattributed three files, which were corrected only through content-level verification. Future deployments should use an immutable server-generated session key across logs, surveys, and participant records.

Fifth, the logs contained 54 empty transcripts, non-Latin scripts in participant turns, and transcripts without a plausible speech source. The recovered client captured audio between button press and release and blocked another request while a response was pending. Repeated transcripts could reflect recognition, repeated speech after the NPC response, or another upstream process; the logs cannot distinguish them. The client also contained an always-true guard in server-address construction, but no individual connection failure can be attributed to it.

Sixth, no retained deployment record linked the server code and active index to each session. The index and prompt provided partial version evidence, but the deployed HTTP server code could not be reconstructed.

The audit exposed limitations in research instrumentation as well as conversation behavior. In particular, batch-at-end logging, unstable identifiers, and missing version records limit recovery and attribution of field evidence.

No deployment-time timing measurements were retained; the 12B-to-4B model change was based on qualitative pilot observations and its effect on response time cannot be quantified (Section 7).

5.5. Summary of Validation Outcomes

Development-stage persona self-recognition and pseudo-lexical generation showed no clear recurrence under the retained field screening. Repetitive responses and unsupported historical relationships emerged during sustained use. The audit identified separate risks in retrieval, logging, session identity, speech recognition, and versioning. Table 5 summarizes the observations and their design implications.

Table 5. Summary of failure modes and implementation-audit findings.

Finding	Stage/Mitigation	Field Evidence	Transferable Implication
FM1 persona self-recognition	Development; prompt redesign	No predefined recurrence in the screened retained corpus	Screen persona identity rather than inferring it from fluent dialogue
FM2 pseudo-lexical generation	Development; lexical constraint drafted but not deployed	No clear field recurrence	Reassess development failures during field use
FM3 repetitive response loops	First identified in field use; no pre-deployment mitigation	33/104 sessions (31.7%); 190/2053 NPC responses (9.3%); participant repetition co-occurred in 31/33 affected sessions	Inspect input, context, and generation separately; co-occurrence does not establish cause
FM4 concurrency/connection instability	Integrated testing; server/pipeline revision	No cross-session contamination found; one same-session file pair merged, three misattributed files reassigned, and one unsupported fragment excluded (Section 4.5 and Section 5.2)	Test concurrent connections and persist recoverable session logs
FM5 persistent-text rendering instability	Piloting; client fix before field data collection	No identified recurrence after the fix	Validate modality-specific rendering
Unsupported biographical claims	Field logs; none applied	30 unsupported Prince Jeongjo references across 12 sessions	Validate historical grounding separately from persona identity
Retrieval-configuration and chain integrity	Post hoc audit; none applied	Testbed query-construction and fallback risks; no turn-level retrieval trace in field logs (Section 5.4 and Section 6.3)	Check embedder-language fit and log retrieved chunks
Batch-at-end logging	Post hoc audit; none applied	22/126 conducted sessions (17.5%) had no analyzable session-level record	Persist logs incrementally
Session-identifier integrity	Post hoc audit; none applied	Archive labels for 33 of 45 individuals had multiple variants (56 non-canonical instances; 7 capitalization/spacing-only); content-level verification corrected three misattributed files	Use immutable server-generated identifiers
Speech-recognition vulnerability	Post hoc audit; none applied	54 empty transcripts; transcripts without a plausible speech source; non-Latin-script anomalies	Log recognition metadata and filter low-information input
Deployment versioning	Post hoc audit; none applied	Deployed server implementation could not be reconstructed	Record code, model, prompt, and index versions

FM = failure mode (Section 4.2, Section 4.3, Section 4.4, Section 4.5 and Section 4.6). Audit rows correspond to the findings in Section 5.4.

6. Discussion

6.1. Persona and Knowledge Grounding

Persona consistency and factual reliability are distinct properties [8,9,11,19]. During development, the NPC sometimes discussed King Sejong without maintaining his first-person identity (FM1). In the screened field corpus, no predefined persona self-recognition failure was detected, but 30 unsupported references to Prince Jeongjo occurred across 12 sessions. Persona consistency therefore did not ensure historical accuracy. Fluent presentation may also make unsupported claims harder to detect [3,4].

Six of the twelve affected sessions contained multiple references. Of the 18 intervals between successive references within a session, 13 continued or revisited the conversational thread associated with the preceding reference. The median reference position was 61% of the NPC-response sequence. The references followed or continued discussions of family or verse. This pattern suggests a relation to conversational context but does not identify a cause.

A replay of nine selected field questions returned 45 top-five chunks from the recoverable FAISS index. None were rated fully relevant, and none referred to King Jeongjo, the crown prince, or succession. A corpus search identified 87 occurrences of “정조” across 80 chunks, all referring to 正朝 or 正朝使 rather than King Jeongjo (正祖). Neither check identified the audited corpus as the source of the references. The chunks retrieved during field deployment were not logged, so the source of individual claims remains unknown.

6.2. Prompt Compliance and Pipeline Enforcement

Prompt instructions were followed unevenly in field use. Responses outside the specified length range were predominantly too long. The three exact closing exemplars appeared in 129 of 2053 responses (6.3%). Responses ending with a “Would you like …?” invitation numbered 1838 (89.5%). The phrase “my people” appeared in 845 responses (41.2%), “my friend” in 110 (5.4%), and “You are most welcome” in 59 (2.9%). Neither of the latter two expressions was present in the retained prompt files. Broad conversational patterns were followed more often than exact wording.

The retained prompt specified response length, closing forms, and repetition avoidance, but no enforcement mechanism was identified in the recovered client or testbed. Repetition occurred in 33 of the 104 retained sessions (31.7%). The logs do not establish whether repeated participant input contributed to individual cases (Section 5.3). The deployed HTTP server was not recovered, so additional server-side controls could not be assessed. Requirements such as length limits and repetition suppression should be implemented and tested in the processing pipeline rather than left to prompt instructions alone [7,19,20].

6.3. Multilingual RAG and Retrieval Observability

RAG effectiveness depends on retrieval quality, query construction, and language alignment [8,11,12,13,14]. In this deployment, the corpus was Korean, participant input could be multilingual, and NPC responses were in English. The retained testbed indicated potential query-construction and retrieval-free fallback risks, but these could not be confirmed for the unrecovered deployed HTTP server. Retrieved chunks were not logged during field use, so their contribution to individual responses could not be assessed.

The retrieval audit in Section 5.4 found that strict relevance was zero and relaxed relevance remained low for the recoverable nomic-embed-text configuration. The BGE-M3 replay yielded a less concentrated embedding space and more diverse retrieved chunks, but the absence of relevance judgments for its new result set prevents a comparative effectiveness claim. The audit therefore supports a model-and-language-fit concern for the recoverable configuration while leaving deployment-time retrieval performance unresolved.

The audited index contained 4179 vectors. Ten byte-identical local copies of the audited 4179-vector index were identified. An earlier 1337-vector build with a different content hash was also retained, and the client could rebuild and overwrite the index (Section 3.4). The matching copies therefore do not establish which build was active during deployment.

These replay diagnostics should not be interpreted as proving that Korean language alone caused the observed retrieval weakness. Model choice, training coverage, query construction, corpus granularity, and the unrecovered deployment state remain competing explanations. Future evaluations should pair multilingual or Korean-capable embedding models with independent relevance judgments under the same query set and corpus.

The audit raises concern about retrieval quality in the recoverable configuration. It does not establish deployment-time retrieval performance or explain individual factual errors. These results are consistent with retrieval contributing little or no relevant content in the audited configuration, rather than retrieval functioning imperfectly. This bears on whether the deployed system is accurately described as retrieval-augmented.

6.4. Field Validation and Implementation Audit

Controlled testing does not necessarily show how an integrated system behaves during sustained field use [1,17,21,22]. Field sessions revealed response loops and unsupported biographical claims, while retained artifacts showed implementation risks and gaps in the deployment record.

Field logs documented interaction patterns, and artifact inspection identified possible explanations and limits on causal attribution. Failures may arise from speech recognition, retrieval, context construction, generation, or client behavior, but incomplete logging limited attribution of individual failures.

Missing records, fragmented histories, and unstable identifiers complicated reconstruction of the analytic corpus. Missing deployment versions also limited interpretation of the audit. Future systems should save logs incrementally, use immutable server-generated session identifiers, and record the code, model, prompt, and index versions used in each session.

Future deployments should also record component-level timing for speech recognition, retrieval, response generation, speech synthesis, and network transfer, together with end-to-end latency and failure events. This instrumentation is necessary to locate bottlenecks and to support reproducible latency comparisons between model or pipeline configurations.

Persona grounding, knowledge grounding, generation stability, and deployment reliability should be assessed separately through field testing and implementation audit. Future work could evaluate whether contrastive relevance checking before generation reduces unsupported historical claims [36]. Its effectiveness for the small self-hosted model used here remains to be evaluated.

7. Limitations

This study examined one historical NPC, one Korean corpus, and one deployment at a single university. The findings may not generalize to other characters, populations, or technical configurations. Output mode was allocated sequentially without randomization, and the network configuration changed during deployment. Any comparisons between conditions are therefore exploratory. No alternative LLM–NPC system or retrieval-free baseline was evaluated under the same protocol.

The final corpus contained 104 of the 126 conducted sessions. Missingness may have been systematic, so differences between retained and unavailable sessions cannot be excluded. Participant-entered identifiers required manual reconciliation; content-level verification identified and corrected three misattributed files, but residual attribution errors cannot be excluded where records were incomplete. Repetition occurred in 33 of the 104 retained sessions (31.7%). Across all 126 sessions, the corresponding bounds are 33/126 (26.2%) if none of the 22 unavailable sessions contained repetition and 55/126 (43.7%) if all did. All repetition analyses used the same final 104-session dataset.

Source audio, per-turn timestamps, retrieved chunks, reliable input-language records, and error events were not retained, limiting attribution of individual failures. The deployed HTTP server and active index could not be verified. Findings from the retained testbed therefore cannot establish the behavior of the deployed server. The retrieval audit used the recoverable index and consolidated relevance labels, but the relevance labels were assigned by a large language model; a researcher reviewed a 30-item blind subsample (agreement 70.0%, κ = 0.18). Its results remain exploratory and do not establish retrieval performance during field use.

No full second-rater file was produced for the 245 query–chunk judgments; accordingly, inter-rater agreement for the complete retrieval audit is not reported.

A separate function-coding exercise included 190 doubly coded participant utterances from 19 participants. Agreement before adjudication was 69.0%, with Cohen’s κ = 0.576, below the prespecified threshold of 0.75. Disagreements were adjudicated, but the function labels were not used in the repetition counts or statistical analyses.

No deployment-time latency measurements were retained. The change from a 12B to a 4B model was based on pilot observations, so its effect on response time cannot be quantified. The findings concern the retained 2025 system and prompt configuration; failure patterns may differ with other models or configurations. Learning outcomes are examined separately in the companion study [16].

Future work should compare multiple historical-NPC systems under a shared protocol, including retrieval-free and retrieval-augmented baselines, alternative embedding models, and component-level latency measurement. Such comparisons require prospective instrumentation because the unlogged fallback path and missing timing records prevent valid reconstruction from the present deployment.

8. Conclusions

This study presented the design, field deployment, and technical validation of a self-hosted VR historical NPC integrating multilingual speech recognition, retrieval-augmented generation, and LLM-based dialogue. The system was deployed across 126 field sessions with 42 international participants. The final technical analysis used 104 reconstructed sessions from 39 participants, containing 2053 participant turns and 2053 NPC responses, together with post hoc implementation and retrieval audits.

First, persona grounding and historical knowledge grounding require separate assessment: no predefined persona self-recognition failure was detected in the screened field corpus, yet unsupported biographical claims occurred. Second, prompt instructions were followed unevenly, supporting explicit pipeline controls for requirements such as response length and repetition suppression. Third, field logs and artifact inspection provided complementary evidence. Field use revealed repetition and factual failures, while retained artifacts identified implementation risks and gaps in the deployment record. The retained evidence could not establish the causes of individual failures.

The architecture, failure records, and audit procedure provide a basis for assessing persona grounding, knowledge grounding, generation stability, and deployment reliability in future VR–LLM historical NPC systems. The proposed assessment approach requires further evaluation in other systems and deployment settings.