The Home as an Active Caregiving Partner: Scaling Zero-Interface Audiovisual Connectivity for “Aging in Place” with Dementia

Abstract

Effective dementia care is often hindered by fragmented communication among patients, informal caregivers, and clinicians. To address this, we introduce an ambient assisted living (AAL) framework designed to establish a continuous, virtual, and unobtrusive connection between an elder’s home and external guardians or medical staff (virtual rounds). The system enables guardians to communicate directly within the home environment, without requiring the older adult to manually accept calls or activate the connection using wearable devices, buttons, or other interfaces. The elders can activate the connection verbally. The structural core of this system relies on three novel hardware configurations designed for zero-interface operation: a remote audio announcement device, a bidirectional intercom, and a “zero-interface mirror” enabling stream-only, real-time video co-presence between patients and guardians. Crucially, the system utilizes a privacy-preserving, staged edge-AI architecture to process data. By default, it operates without long-term persistent storage, selectively transmitting abstracted audio-based behavioral metrics to a secure dashboard. For advanced dementia stages, the system employs ephemeral data retention—specifically a highly restricted, 24 h rolling audio buffer—allowing authorized guardians to verify acute events without permanently exfiltrating raw data. We evaluate this infrastructure through a 10-month longitudinal, single-home feasibility deployment, augmented with historical verified fall data to rigorously test the detection of rare acute events. The study validates the framework’s technical viability, system uptime, and privacy-first architecture in continuously tracking long-term proxy behavioral indicators under real-world conditions. Rather than asserting generalized clinical efficacy, this work demonstrates the operational feasibility of a novel, affordable, technical blueprint for dignified, remote digital care coordination.

1. Introduction

The convergence of smart home technology and elderly caregiving represents a critical response to one of the most pressing demographic shifts of our time. With the global population of individuals aged 60 and older projected to exceed two billion in the coming decades, healthcare systems and families are facing an unprecedented shortage of caregivers and reliable elderly-care facilities, underscoring the need for technology that helps seniors maintain their independence [1]. The primary problem this technological intersection seeks to solve is enabling older adults to “age in place” safely and autonomously, mitigating the risks of undetected falls, sudden medical emergencies, and gradual cognitive decline [2]. By transforming the home into an active care partner through the integration of ambient, affordable hardware and intelligent software, we can provide continuous, unobtrusive monitoring [3] and decompress elderly care facilities. This is vitally important because it not only drastically reduces emergency response times and hospital readmissions but also preserves the independence and psychological well-being of the elderly, alleviating the logistical and emotional strain placed on their human caregivers.

Ambient assisted living (AAL) has a long-standing presence [4], but over the past five years (2021–2026), the state of the art has rapidly transitioned away from reliance on easily forgotten wearable devices toward highly sophisticated environmental monitoring [5,6,7]. Recent advancements heavily emphasize the fusion of audio and camera technologies to track behaviors and detect anomalies in real time without continuous user compliance [7]. Modern computer vision frameworks utilize advanced models like YOLO object detection algorithms linked with spatio-temporal graph convolutional networks (ST-GCN) to achieve high-precision pose estimation and autonomous fall recognition directly from camera feeds [8,9,10]. Simultaneously, audio monitoring has evolved through the use of deep learning anomaly detection frameworks capable of analyzing acoustic signatures and environmental Doppler variations to instantly recognize specific events, such as heavy falls or cries for help [10,11]. A defining hallmark of the current era is the shift toward edge AI; by processing raw video frames, spatial data, and acoustic arrays directly on local smart home hubs rather than transmitting them to a cloud server, modern systems extract only essential behavioral metadata to ensure rapid, localized emergency responses [4,12].

Despite these technological leaps, the integration of audiovisual monitoring in domestic caregiving is still constrained by several critical bottlenecks that demand research focus. Current AAL research oscillates between broad architectural frameworks and niche hardware implementations. The most profound pending challenge lies in the inherent privacy trade-off: cameras and microphones capture highly sensitive, intimate data [13,14,15,16,17]. The intrusive nature of cameras has prompted the development of custom, secure audiovisual systems designed specifically for unobtrusiveness [15]. While anonymization techniques like frame differencing, real-time avatar replacement, and the use of ultra-wideband ambient signals have improved, establishing deep user acceptance and trust among the elderly remains a fragile endeavor [16,17]. Furthermore, current audiovisual systems frequently struggle with real-world environmental variability, dynamic challenges such as poor or fluctuating lighting, complex scenes, physical occlusion in cluttered rooms, and overlapping background noises, which severely degrade the reliability of anomaly detection [12].

Complementing these dedicated systems is the rise of ubiquitous consumer hardware, such as smart speakers, which provide a familiar interface for monitoring health and well-being [18]. Moving forward, the field must heavily focus on developing privacy-by-design sensor fusion that operates flawlessly under unpredictable real-world conditions, alongside establishing interoperable software frameworks that reduce the prohibitive costs of current AAL setups. Bridging these gaps is essential to delivering human-centric AI systems that guarantee both the physical safety and the dignity of aging populations.

Our objective is to engineer affordable, all-inclusive hardware and software solutions that facilitate “aging in place,” mitigate domestic risks, and push back hospitalizations due to timely interventions [19,20,21,22,23]. We propose that a smart home for vulnerable populations must transcend remote handling of appliances or energy efficiency. Instead, it should function as an integrated ecosystem where voice agents analyze internal activities to ensure safety, while remaining strictly moderated to protect resident privacy from external data leakage.

Our dataset is not designed to represent a large cohort and addresses research questions that are fundamentally different from those giving emphasis to video [7,8,9,24,25,26,27]. Rather than prioritizing scale, this work deliberately emphasizes longitudinal depth and technological contribution, offering several important and distinctive advantages.

First, the data is collected from individuals residing in their own home environment, thereby capturing realistic, everyday living conditions. We contrast that with studies where patients are gathered in a common room, or elders’ falls are simulated by actors and researchers. The central question is how a typical household can be augmented with affordable sensing technologies—specifically audio-based solutions, as opposed to wearables or video systems—to support and prolong safe and independent living for people with dementia.

Second, the dataset is longitudinal, spanning nearly one year while gathering data on a 24/7 basis. This extended duration enables the development of analytical methods capable of identifying behavioral patterns, routines, and temporal dynamics that are inherently inaccessible in larger but short-term cohort studies.

Third, the study focuses on a particularly relevant and common real-world scenario: a married couple cohabiting in the same household. This setting allows for the investigation of interpersonal dynamics, interaction patterns, and mutual influence, which are largely overlooked in datasets that consider individuals in isolation, yet are critical for understanding and supporting dementia care in practice.

We emphasize the integration of novel hardware components, the use of open-source software, and the combination of heterogeneous technologies to deliver affordable, scalable, and practically deployable end-to-end solutions. The code of the sentinels is made openly available so that the community can apply it to a larger cohort.

This work extends our prior research in several ways. First, we introduce three novel hardware solutions beyond those presented in [28], designed to create a direct virtual link between in-home infrastructure and the outside world. These units establish a remote audio connection, enabling an external guardian to monitor a household’s soundscape and provide real-time responses to voice-activated links. Another device establishes a bilateral continuous but non-storable video stream between a home with patients and outside medical staff or guardians. Subsequently, we present a protocol that gradually changes intervention depending on the clinical status of the patients with dementia, and finally, we present data analytics from a continuous deployment of 10 months in a home, emphasizing the practical questions that can be answered contingent upon this data. This paper presents a deployable infrastructure and operational blueprint for continuous home-based dementia support, not a validated clinical intervention.

Accordingly, this work does not attempt to clinically validate psychosocial outcomes, dementia biomarkers, or caregiver bonding effects. Instead, it presents and evaluates the technical feasibility of an integrated, low-cost infrastructure intended to enable future clinical and behavioral research. We intentionally prioritize ecological validity and longitudinal deployment over controlled experimentation.

2. Materials and Methods

Our approach is based on engineering affordable, duplex audio communication sentinels. By ensuring cost-effectiveness, a practical system can deploy a unit in every room—including bathrooms—to provide comprehensive acoustic coverage across all living spaces. These devices generate audio class distributions and heatmaps, which are accessible to remote guardians via a mobile application. Crucially, when speech is detected, the system performs keyword analysis. If an emergency keyword or an expression of pain is identified, a live audio stream is automatically established between the guardian and the resident for immediate verification. As in [28], the study involves a couple living independently at home—an 88-year-old woman and a 91-year-old man (2026), both diagnosed with Stage-2 dementia and mobility limitations. They are supported by two caregivers working in 8 h shifts, with occasional additional care. The home is equipped with audio-enabled CCTV cameras monitored overnight. The camera is not part of our system and is used only for verification.

In our previous work, we introduced audio sentinels that stream short audio events to an audio classifier of 527 audio event labels, speaker identity and emotion are extracted from speech dialogues and summarized by an LLM. The aim was to produce reports on events that occurred during the absence of caregivers from their shift. These sentinels are stateless by design, meaning that they boot from flash and cannot store audio to adhere to ethical standards; therefore, any processing must be done “on the fly.” In this work, we employed three ESP32 microcontrollers (Espressif Systems, Shanghai, China) installed in the toilet, kitchen and living room, respectively, and two ESP32-BOX3 (Espressif Systems, Shanghai, China) (positioned at conversational endpoints in the bedroom and near the couch facing the TV) (see Figure 1 for the devices employed in this work).

This research introduces scalable hardware for unobtrusive virtual connectivity, bridging the domestic environment with external support networks and preparing the home to handle autonomous agents that listen, see and respond (see Figure 2 for a depiction of the whole concept). Engineered for broad demographic accessibility, the system prioritizes affordability and a zero-interface design philosophy, intentionally omitting physical buttons or complex controls to ensure seamless integration into the lives of non-technical users or elderly users with dementia and disabilities. Upon installation into a standard power outlet, these devices establish an automated, persistent stream of acoustic telemetry. The sentinels are designed so that no data can be stored in them. Regarding the home server, raw data never leaves the local edge server by default, utilizing an encrypted, transient 24 h review window strictly limited to authorized proxies during Stage-2 acute events (see Ethics section later). This architectural foundation facilitates a spectrum of interventions ranging from clinical-grade longitudinal assessments for dementia by remote medical professionals and informal caregiving by family members to the integration of AI agents.

2.1. Hardware

2.1.1. One-Way Audio-Based Announcements

The first device in our ecosystem is a lightweight, wireless, one-way audio announcement system built using commodity hardware and open-source software. The system enables the delivery of spoken messages within the home environment, originating either from human caregivers (e.g., family members or medical personnel) or from automated software agents (Figure 3). Messages may be triggered remotely through mobile networks or generated locally using Large Language Models (LLMs) combined with Text-to-Speech (TTS) synthesis. We have successfully carried out the gentle delivery of short contextual prompts, including greetings, medication reminders, meal notifications, and bedtime cues. Beyond their practical utility, these interactions are designed to reinforce a sense of connectedness, care, and ongoing social presence for older adults living at home. Music streaming and audiobook narration are also possibilities.

In our setting, the system bypasses firewalls by utilizing Taildrop, a peer-to-peer (P2P) file transfer feature of the Tailscale v1.98.3 (Tailscale Inc., Toronto, ON, Canada), a mesh virtual private network (VPN). Tailscale operates transparently across network address translation (NAT) boundaries and heterogeneous networks without requiring port forwarding or a dedicated relay. On the local home server, a Python 3.11.13 script continuously monitors the “Downloads” folder; upon detecting a new audio file, it automatically triggers the VideoLAN Client (VLC) media player. VLC then routes the audio through the CS-HIFI-ESPARAGUS audio module (Sonocotta, China) to connected loudspeakers, completing a one-way voice message pipeline that requires no proprietary applications or subscription services (see also Appendix A.1). Latency is bounded solely by the file transfer speed over the VPN, and the total Bill of Materials (BOM) cost remains under $30 USD.

2.1.2. Bidirectional Audio-Based Communication (Remote Intercom)

The current device extends previous passive sentinels of [28] by establishing a bidirectional communication loop between older adults and their caregivers. By integrating a dual-microphone capture system with a synchronized loudspeaker, the sentinel transitions from a passive monitor to a remote intercom device, allowing a guardian outside or an autonomous conversational agent inside the home to provide immediate, context-aware feedback to the user (see Figure 4). The primary function is that a guardian sees on the dashboard in real time that a speech event has occurred, can listen inside the home, and respond or converse with the elderly remotely, in near-real-time. The embedded acoustic monitoring and bidirectional intercom system is built on the Espressif ESP32-S3-BOX-3 development platform. New firmware written in C implements all processing stages at the edge (see also Appendix A.2 and a demo video in Supplementary Materials).

The ESP32-S3-BOX-3 integrates an ESP32-S3 dual-core Xtensa LX7 processor at 240 MHz, 16 MB of Pseudo-Static Random-Access Memory (PSRAM), 16 MB of flash memory, a dual-microphone array with an ES7210 Analog-to-Digital Converter (ADC), a speaker output with an ES8311 Digital-to-Analog Converter (DAC), a 2.4-inch 320 × 240 In-Plane Switching (IPS) touchscreen, a microSD card slot, and 802.11 b/g/n Wi-Fi. The Board Support Package (BSP) from Espressif provides a unified codec abstraction layer through the esp_codec_dev Application Programming Interface (API), which exposes open/read/write/close operations over the Inter-IC Sound (I2S) and Inter-Integrated Circuit (I2C) audio paths.

The firmware uses the Espressif IoT Development Framework (ESP-IDF) v5.3+ and is organized into four concurrent Free Real-Time Operating System (FreeRTOS) tasks: a sound activity detection (SAD) monitor task (priority 10), a player task (priority 8), an upload task (priority 5), and a time-update task (priority 3). The system follows a five-state machine: 1: Monitoring the RMS level, 2: Recording upon triggering, 3: Streaming raw PCM audio chunks via MQTT, 4: Cooldown and return to monitoring, 5: Playing (the reverse audio path for inbound speech).

Upon detection of a speech event by the on-device SAD algorithm that is based on simple RMS thresholding, the firmware initiates a live outbound stream without buffering the full recording. Raw PCM frames of 1024 samples, captured at 16 kHz with 16-bit resolution, are published to the broker within approximately 64 ms of the trigger event. Each frame is transmitted as a binary MQTT chunk over a mutually authenticated TLS session, with session metadata—including sample rate, bit depth, channel count, and format—conveyed in a preceding QoS 1 control message to guarantee delivery. The inbound path is symmetric: the dashboard or conversational agent publishes a reciprocal PCM stream to a device-specific command topic, which the ESP32-S3-BOX-3 subscribes to continuously. The on-device player task opens the speaker codec upon receipt of the very first incoming chunk, eliminating the latency penalty of reassembling a complete audio file prior to playback. This design results in a full-duplex intercom loop whose end-to-end latency is bounded primarily by network round-trip time rather than file transfer overhead, making near-real-time voice interaction feasible over standard residential broadband infrastructure. All processing stages—SAD, streaming, playback scheduling, and MQTT session management—execute concurrently as independent FreeRTOS tasks on the dual-core microcontroller, with no reliance on cloud-side computation.

The system operates at a sampling frequency of 16,000 Hz, 16-bit mono. At this configuration, the raw PCM data rate is 32 KB/s, yielding approximately 640 KB for a 20 s recording, transmitted without compression as a sequence of MQTT binary chunks. The first chunk reaches the broker within approximately 64 ms of the SAD trigger. The idle current draw with Wi-Fi active is approximately 150 mA, and the total Bill of Materials (BOM) cost is under $50 USD.

2.1.3. Bidirectional Real-Time Video Streaming Between Distributed Devices

To mitigate social detachment and foster emotional connectivity between elders with mobility problems and their guardians/relatives, we propose a “zero-interface” intervention centered on the concept of ambient visual co-presence. This affordable system comprises a pair of buttonless, mirrored screens—one situated in the patient’s domestic environment and the other in the guardian’s—providing a continuous, real-time visual window without the cognitive burden of synchronous audio. Technological adoption varies significantly across generations, with older adults often disengaging from complex digital interfaces over time. We emphasize that the proposed framework is not a surveillance system; rather, it is a privacy-centric “bonding” tool. These “plug-and-play” devices are portable and buttonless—allowing for deployment even during hospitalization—and facilitate a direct visual link between the patient and guardian without persistent data storage and without the stress of “making/accepting” a call. To ensure privacy, camera fields of view are strictly constrained to specific areas, for instance, a child’s playground for the guardian’s side or a resting armchair for the older adults. The system removes the need to prioritize application-based navigation, allowing busy guardians to engage via a brief glance as they pass by the device or at work. Over time, this continuous, low-friction visual contact establishes a bond and provides elders with a sense of presence of their loved ones. By intentionally omitting audio transmission and internal storage and excluding all external output sockets, the device adheres to a strict privacy-by-design protocol, ensuring that no data is captured or exfiltrated. The key challenge we address is simultaneity: each device must function as both a video producer and a video consumer at the same time, without one role blocking or degrading the other (see Figure 5). Although other costly solutions can be currently found, our contribution is the integration of both roles into a single, unified script that runs identically on both low-cost machines, employing Python’s 3.11.13 threading module to decouple the send and receive paths into independent execution contexts. The system consists of two mini-PCs (see Figure 1d), but a Raspberry Pi 5 could also carry out the task. Each node runs a single Python process containing two threads.

Unlike conventional client-server video architectures that handle transmission and reception as separate applications or sequential processes, our approach executes both tasks simultaneously on each node using a multithreaded design. Each device concurrently captures and publishes its local camera feed while subscribing to and rendering the remote peer’s stream within a single Python script running identical code on both endpoints. The system leverages the ZeroMQ (ZMQ) asynchronous messaging library for low-latency frame transport and OpenCV for image capture, compression, and display. When combined with a virtual private network overlay like Tailscale, the system operates transparently across geographically distributed locations without requiring port forwarding, static IPs, or centralized relay infrastructure.

Note that for elders suffering from mobility limitations and Stage-2 dementia, traditional telecommunication devices designed for the general population—for example, smartphones or tablets or even simplified phones—often present insurmountable interface barriers and are ill-suited for the bonding task we are interested in. Real-time video communication between two endpoints is a well-studied problem. What we offer is an open, affordable, minimal, symmetric system that can be deployed identically on both machines with no architectural distinction between sender and receiver.

This work focuses on the engineering problem of maintaining continuous, low-interaction visual connectivity between distributed domestic environments under privacy and low-cost deployment constraints. The system is intended to enable infrastructure for future behavioral, clinical, or psychosocial studies. Evaluation of emotional bonding, caregiver burden, loneliness reduction, or quality-of-life outcomes lies outside the scope of the present engineering study. This device aims to enable sustained ambient visual co-presence between guardians and older adults, particularly in geographically distributed caregiving settings.

2.2. Software

2.2.1. The Audio Event Recognizer

Audio snippets generated by distributed “audio sentinels”—including the ESP32 and ESP32-S3-BOX3—are routed to an in-home server for processing by a dedicated audio event recognizer. The core of this system is the Audio Spectrogram Transformer (AST), which treats audio classification as a visual recognition task and was trained on the extensive AudioSet database. In this work, the AST operates in a zero-shot configuration without fine-tuning to specific household data. This preserves the model’s generality and adheres to ethical restrictions regarding the sharing of raw data. The process begins by converting raw 1D audio waveforms into 2D Log-Mel spectrograms. This spectrogram is then partitioned into a sequence of overlapping patches, similar to how a Vision Transformer (ViT) processes an image. These patches are linearly projected into embeddings and augmented with positional encoding to preserve the temporal and spectral structure of the sound. By utilizing multi-head self-attention, the AST captures global dependencies across the entire audio clip, allowing it to distinguish between mundane household noises and speech. The system remains performant on a localized home server without cloud reliance. While its accuracy was analyzed in [28], we observed over almost a year of continuous operation that the model’s massive training and the limited variety of domestic sounds led to high general reliability. However, it tends to classify many vocal events as “speech,” thereby failing to distinguish specific distress signals like yelling, agitation, or pain. Conversely, it is highly efficient at classifying TV broadcasts, coughing, and snoring, a capability that proves useful for our specific application.

The novel element in this work is the grouping and collective visualization of labels of audio events from long-term 24/7 operation that allow the remote guardian to immediately see patterns of clinical interest through an app that accesses the home server’s dashboard.

The integration currently operates in two modes: (A) In the semi-automated mode, the server aggregates the classified acoustic signatures (hits/thuds, knocks, speech content, speaker ID, speaker’s emotional state) into high-level visualizations and historical data processing, enabling guardians to grasp behavioral trends and frequency reports via a secure connection to the home server and respond verbally (Esparagus and ESP32-BOX3 can broadcast voice messages inside the home originating from outside). (B) In the automated mode, upon being triggered by an audio event, the system utilizes a configurable 30 min rolling window to aggregate all dashboard textual data. This data is processed locally by a large language model (LLM), exemplified by the Gemma 2 9B model running through Ollama, which analyzes incoming audio events for predefined keywords and indicators of emotional distress. The dashboard data comprises speech transcriptions (Faster Whisper), emotional tagging (Hubert-large-superb-er), speaker diarization (pyannote), and CLAP analysis for indicators of pain or stress (see Dashboard.mp4 video in Supportive Materials). All models are configured in zero-shot mode and have been analyzed extensively in prior work [29,30]. Based on our analysis of real fall recordings, the typical sequence consists of a scream or yell followed by a thud or impact sound. Subsequently, older adults often produce spontaneous verbal and non-verbal expressions of distress, such as calling for help or describing their condition involuntarily and without conscious deliberation. If such an event is detected, a live communication stream is immediately established with the guardian, while the system automatically sends an email notification containing timestamped transcriptions via the Brevo service (see Appendix A.4). This architecture provides a non-intrusive approach for monitoring inhabitant safety in every room of the house, particularly for people with dementia who typically at Stage 2 and beyond do not remember that they have wearable emergency buttons or how to use them. For languages other than Greek, simpler specialized wake-word detection solutions such as Picovoice Porcupine may also be employed.

2.2.2. A Fall Detector

This work is based on audio only. However, we are preparing the system for the integration of vision to handle complex scenes (e.g., falls, unattended cooking). Therefore, while computer vision is not the primary focus of the proposed system, it is incorporated into the processing pipeline to support fall detection as an application scenario. Detection of falls is a well-established and independent research area due to its critical importance [31,32,33,34,35]. Here, it is included to augment the functionality of the in-home agent. Future work will further exploit computer vision and local audiovisual foundation models to identify potential hazards within the home environment in real time. At present, comprehensive scene understanding remains computationally demanding for continuous domestic deployment, particularly under privacy-preserving edge constraints. To facilitate the autonomous monitoring of the elderly with Wi-Fi cameras, we developed a long-lie prevention pipeline centered on an open-source vision-language model (VLM). Recognizing that native video inference is computationally prohibitive for high-volume screening on consumer-grade hardware, the system employs a dual-stage architecture that prioritizes efficiency by partially sacrificing semantic depth. In the first stage, a lightweight motion-detection pre-processor—utilizing MOG2 background subtraction—identifies significant activity within 1 min clips, triggering the extraction of a stratified sample of six high-salience frames. These frames represent a temporal sequence capturing the onset, progression, and final posture of a detected event (see Figure 6).

In the second stage, these discrete images are fed into a quantized version of the Qwen3-VL-2B-Instruct model, hosted on a local server to ensure data privacy. The model is steered via a multimodal zero-shot prompt, which tasks the VLM with providing a binary fall verdict (“yes”/“no”) and a final postural state classification (e.g., “floor,” “bed,” or “standing”). By reducing dense video streams into a concise visual-textual inquiry, this methodology allows for the rapid, automated auditing of thousands of video clips, effectively identifying accidental falls while minimizing the hardware overhead typically associated with deep-learning-based video analysis. The postural instability module could be integrated into the bidirectional real-time video streaming system to deliver alerts on the guardian’s interface. However, it was intentionally omitted from this work to maintain conceptual clarity and modular independence.

2.2.3. Linguistic and Semantic Biomarkers (ASR-Derived) forCognitive Decline Tracking on Dashboard

Speech is transcribed but transcriptions are accessible to guardians for cases involving moderate-stage dementia and beyond. Ten linguistic biomarkers were extracted from the daily spontaneous speech transcripts using an automated natural language processing pipeline built on the spaCy Greek language model (el_core_news_lg). Each metric targets a distinct dimension of language production known to deteriorate along the Alzheimer’s disease continuum. Metrics are meant to be applied over months to years and uploaded to the dashboard to track cognitive decline and are visible upon consensus only to authorized external medical staff. From the text we extract the following metrics previously reported to be correlated with dementia:

(1) Idea Density measures the proportion of content words—verbs, nouns, adjectives, and adverbs—relative to total word count and serves as an index of propositional richness per unit of speech. Lower values reflect impoverished semantic content and reduced cognitive elaboration. This measure was established as a longitudinal predictor of Alzheimer’s disease (AD) (see [36] in the seminal Nun Study, where low idea density in early-life autobiographical writing predicted late-life dementia with high accuracy).

(2) Mean Dependency Distance (MDD) quantifies syntactic complexity by computing the mean linear distance between syntactically dependent word pairs within the dependency parse tree. Greater distances correspond to more elaborate, hierarchically structured sentences, while reduced MDD reflects a flattening of syntactic architecture consistent with the working memory and executive function deficits characteristic of AD. The diagnostic validity of MDD as a marker of syntactic decline in AD was demonstrated in [37].

(3) Type-Token Ratio (TTR) expresses lexical diversity as the ratio of unique lemmas to total tokens. A declining TTR indicates a shrinking active vocabulary and increased lexical repetition, both hallmarks of the anomic and semantic memory deficits observed in [38]. Because TTR is sensitive to text length—it decreases mechanically as samples grow longer—it is supplemented here by the Moving-Average Type-Token Ratio (MATTR), which computes TTR over a sliding window of fixed size (here, 25 tokens) and averages across all windows. MATTR provides a length-robust estimate of lexical richness and has been validated specifically on spontaneous AD speech and applied to dementia corpora [39].

(4) Light Verb Ratio captures the proportion of verbs that belong to a set of semantically bleached, high-frequency verbs (e.g., κάνω, έχω, παίρνω, γίνομαι, βάζω, βγάζω, δίνω, φέρνω) relative to all verbs produced. Increased reliance on such generic verbs at the expense of specific lexical verbs reflects the lexical retrieval failures and verb-finding difficulties that accompany progressive semantic degradation in AD, as documented in [40].

(5) Noun-to-Verb Ratio measures the balance between nominal and verbal elements in the transcript. In typical language production, nouns and verbs appear in a roughly balanced ratio. In AD, anomia disproportionately affects verb retrieval relative to noun retrieval, driving the ratio below unity. Values substantially below 1.0 thus index the selective verbal anomia described in [41] in both oral and written narrative production.

(6) Coordinate Conjunction Rate reflects the frequency of coordinating conjunctions (e.g., και, αλλά, ή) relative to total tokens. As syntactic competence declines, speakers increasingly substitute subordinate and embedded clause structures with paratactic coordination—stringing clauses together with “and” rather than embedding them hierarchically. A high coordinate conjunction rate therefore signals syntactic simplification and the regression from hypotaxis to parataxis observed in moderate-stage AD, as identified in [42], in their comprehensive linguistic feature analysis.

(7) Pronoun-to-Noun Ratio indexes referential vagueness by comparing pronoun frequency to noun frequency. Speakers with AD progressively substitute specific nominal referents with pronouns and demonstratives (e.g., αυτό, εκεί, το), producing speech that is semantically underspecified and difficult to interpret without shared context. In [43] it was demonstrated that this shift reflects a breakdown in the inferential mechanisms governing referential choice and that an elevated pronoun-to-noun ratio distinguishes AD speech from age-matched controls.

(8) Mean Utterance Length (MUL) is computed as the mean number of words per speaker turn across all utterances in the transcript. Progressive reduction in utterance length reflects the convergence of multiple deficits in AD—diminished working memory capacity, word-finding failures, and reduced discourse planning—and has been associated with disease severity in [44], where the authors demonstrated systematic MUL reduction across stages of AD-associated cognitive decline.

(9) Perseveration Index quantifies pathological repetition by calculating the proportion of unique content lemmas—drawn from verbs, nouns, adjectives, and adverbs—that occur three or more times within the transcript. Unlike ordinary topic coherence, perseveration in AD reflects an impaired ability to inhibit previously activated lexical representations, resulting in the compulsive reuse of a narrow set of words and phrases irrespective of communicative need. In [45], the authors established perseveration as a clinically significant feature of moderate-to-severe AD, distinguishing it from the repetition patterns observed in normal aging and other dementia subtypes.

(10) The light verb inventory used in this study was defined according to four criteria applied jointly. First, semantic bleaching: only verbs that contribute minimal referential content of their own and function primarily as carriers of tense, aspect, or modality were included, consistent with the operational definition of light verbs in the Greek linguistic literature [46]. Second, constructional versatility: each verb must participate in a demonstrably wide range of light verb constructions (LVCs) in Greek. Third, register appropriateness: the set was restricted to verbs with high attestation frequency in informal spoken Greek, excluding verbs whose light verb uses are predominantly confined to formal or written registers—specifically βγάζω, δίνω, and φέρνω, which, despite forming LVCs in standard Greek, are rare in spontaneous colloquial dialogue and would inflate the ratio without diagnostic warrant in the present corpus. Fourth, semi-grammaticalization: two verbs—πάω and λέω—were included on the basis of their quasi-auxiliary behavior in periphrastic constructions (πάω να + infinitive, λέω να + infinitive), in which the speaker’s intended action is expressed entirely by the infinitival complement while the matrix verb functions as a prospective or volitional carrier. This pattern is diagnostically relevant in Alzheimer’s disease because it represents a productive strategy by which speakers with impaired specific verb retrieval substitute a bleached, high-frequency carrier for the target lexical verb, a mechanism documented in [47] in their analysis of light verb overuse in semantic dementia and Alzheimer’s disease. The final set—κάνω, έχω, παίρνω, γίνομαι, πάω, λέω, βάζω—represents the intersection of theoretical well-formedness and register validity for spontaneous Greek dementia dialogue. All biomarkers are gathered in

Table 1. Data transformation pipeline for clinical monitoring. The table illustrates the extraction of linguistic biomarkers from raw acoustic dialogue transcribed text, and their final synthesis into a clinician-facing dashboard for longitudinal in-home health tracking.

Biomarker	Definition	Clinical Implication
1. Idea Density [36]	Proportion of content words (verbs, nouns, adjectives, adverbs) relative to total word count.	Measures propositional richness; low values predict early-stage AD.
2. Mean Dependency Distance (MDD) [37]	Mean linear distance between syntactically dependent word pairs.	Quantifies syntactic complexity; reduced MDD reflects executive function deficits.
3. MATTR (Lexical Diversity) [39]	Moving-Average Type-Token Ratio computed over a sliding 25-token window.	Length-robust measure of vocabulary attrition and lexical repetition.
4. Light Verb Ratio [40]	Proportion of semantically “bleached” verbs (e.g., κάνω, έχω) relative to total verbs.	High ratio indicates failures in retrieving specific lexical verbs.
5. Noun-to-Verb Ratio [41]	Balance of nominal elements relative to verbal elements.	Values < 1.0 index selective verbal anomia (naming deficits).
6. Coordinate Conjunction Rate [42]	Frequency of coordinating conjunctions (e.g., και, αλλά) relative to total tokens.	Signals a regression from complex hierarchy (hypotaxis) to simple stringing (parataxis).
7. Pronoun-to-Noun Ratio [43]	Frequency of pronouns/demonstratives relative to specific nouns.	Measures referential vagueness and breakdown in inferential mechanisms.
8. Mean Utterance Length (MUL) [44]	Average number of words per speaker turn.	Reflects diminished working memory and reduced discourse planning.
9. Perseveration Index [45]	Proportion of content lemmas occurring three or more times in a transcript.	Quantifies pathological repetition and inability to inhibit activated representations.
10. Light Verb Inventory [46,47]	Selection of high-frequency carrier verbs (κάνω, έχω, παίρνω, γίνομαι, πάω, λέω, βάζω).	Specifically calibrated for Greek colloquial dialogue and semi-grammaticalization.

.

3. Results

In this section, we gather practical results and an evaluation of several stages that, from our point of view, have elements of novelty and are rarely met in the literature. Crucially, each audio sensor (i.e., ESP32 and ESP32-BOX3) was configured to respond to near-field audio from the room in which it was placed (20–30 m²), minimizing the capture of reverberant signals from adjacent spaces that degrade downstream signal processing and classification accuracy. Near-field audio capture is implemented by applying elevated thresholds within the audio activity detection loop, ensuring that only high-intensity local signals trigger the system. Furthermore, unique device identifiers are transmitted along the audio to enable precise spatial localization on the dashboard, allowing the system to map patient activity to specific domestic zones (e.g., kitchen vs. bedroom). We prioritized the following results in light of impact:

1. Re-establishing the Caregiver/Guardian-Patient Bond. The system facilitates an auditory connection between patients and geographically distant relatives (typically adult children) and/or caregivers. Our findings suggest that this unobtrusive “acoustic window” and the bilateral video device allow busy guardians to reconnect with the elderly, fostering a renewed emotional bond through shared situational awareness that less invasive methods (e.g., wearables) fail to provide.

2. We posit that optimal dementia management necessitates a cohesive consensus within a clinical triad: the patient, the guardian (typically adult children of patients or other close relatives), and medical personnel. Current care models are frequently compromised by a significant informational asymmetry. Patients often exhibit anosognosia, masking or a lack of insight into the severity of their cognitive decline. The level of denial is often so high that patients will view recordings of their own actions and still deny that they are the person in the video. Meanwhile, geographically distant guardians often lack situational awareness regarding the patient’s domestic environment, which may involve acute medical incidents or immediate physical hazards. Consequently, clinicians are often forced to make critical pharmacological or placement decisions predicated on fragmented, subjective anecdotes reported during brief consultations. Our monitoring framework reconfigures this dynamic by introducing objective acoustic evidence into the domestic environment. By quantifying previously “invisible” behavioral markers, including nocturnal wandering, falls, acute agitation, toileting frequency, silence periods, and counting of delusional episodes or even hallucinations (this requires camera confirmation), the system provides the empirical foundation for a data-driven therapeutic alliance. This ensures that clinical interventions, or the transition to institutional care, are predicated on documented longitudinal facts rather than observational, fragmented and partly subjective assessments.

3. Clinical Intervention and Stressor Mitigation. In cases of advanced dementia, direct auditory access has proven essential for identifying and resolving acute stressors, reassuring and redirecting. Caregivers working in shifts or guardians can have a wider picture of the situation during their absence, especially at night, by looking at the dashboard that is updated on a 24/7 basis. The dashboard immediately exposes the situation to medical staff by virtue of long-term data analytics. Specifically, the system supports the identification of clinically and behaviorally relevant patterns, notably goal-directed nighttime activity, exit-seeking, adherence to expected toileting routines, and the presence of vocalized distress or calls for assistance. It further enables the quantification of phenomena such as delusional or disorganized speech, as well as the extraction and longitudinal tracking of digital biomarkers derived from transcribed text. In addition, aggregate measures—such as the proportion of silence within the home—provide further context regarding daily activity and engagement. Finally, by analyzing both the linguistic content of transcribed speech and speech prosody in relation to sentiment, the system can capture subtle changes in affective state, offering a richer and more nuanced understanding of the inhabitants’ condition over time.

3.1. Data Analytics of Audio Events Reveal Incidents and Patterns

Regarding the visualizations of long-term data and the patterns that they reveal, we report the following:

1. Nocturnal ambulation and safety. The spatial distribution of audio events (see Figure 7 for a single day and Figure 8 for the whole observational period) and the analysis of non-speech sounds—for instance, footsteps, door/switches interactions, and the characteristic rhythmical “thuds/knocks” of mobility aids—have enabled the reliable automated detection of nighttime activity. Wandering is defined as atypical, repetitive, usually purposeless and disoriented movement patterns that deviate from an individual’s baseline behavior and may compromise safety. Goal-directed nighttime activity represents a critical safety concern, primarily driven by the patient’s loss of temporal orientation and circadian dysregulation, which persists despite the presence of environmental cues, including clocks. This behavior is particularly hazardous due to the compounding effects of sedative-hypnotic medications, which often result in residual drowsiness and compromised gait stability during the night. In the absence of a live-in caregiver, these unobserved episodes significantly elevate the risk of falls and “long-lie” events, where the patient may remain on the floor for extended periods without assistance, leading to severe complications, notably rhabdomyolysis, pressure ulcers and ischemic incidents. In the context of our approach, talking back through Wi-Fi loudspeakers has several times allowed redirection to bed and reassurance during falls.

2. Respiratory Health and Sleep. The acoustic monitoring of respiratory patterns—specifically coughing (see Figure 9a) and snoring (see Figure 9b)—offers essential insights into both acute and chronic conditions. While specialized polysomnography provides a more granular analysis of sleep apnea, the unobtrusive detection of snoring patterns serves as a highly accessible proxy for sleep-disordered breathing, which is known to exacerbate cognitive decline and daytime agitation in dementia patients. Similarly, the longitudinal tracking of cough frequency is vital for identifying early signs of respiratory tract infections or aspiration pneumonia, a leading cause of mortality in late-stage dementia. Automated cough monitoring also indirectly assesses aftertreatment effects. By establishing an acoustic baseline for the patient, the system can flag sudden increases in cough frequency or changes in snore intensity, allowing for early medical assessment before the condition escalates into a systemic infection or acute respiratory distress. See Figure 10 for long-term monitoring of these effects.

3. Gastrointestinal Monitoring and Autonomy. The monitoring of bathroom activity serves as a critical, non-invasive proxy for gastrointestinal health, specifically regarding the detection of chronic constipation (see Figure 11). In patients with dementia, constipation is a clinically significant and potentially dangerous condition due to its multifactorial impact on physical, cognitive, and behavioral health. Reduced mobility, inadequate fluid intake, autonomic dysfunction, and the anticholinergic effects of commonly prescribed medications increase constipation risk in this population. In these patients, constipation may precipitate delirium, agitation, pain, and behavioral disturbances that are often misattributed to psychiatric progression. Severe constipation can lead to fecal impaction, bowel obstruction, urinary retention, or aspiration risk due to nausea and vomiting. Doctors need this information, but caregivers in shifts can fail to deliver this information, as it requires careful documentation that is hardly met in practice, and elderly people often lack the cognitive capacity to accurately report their bowel habits. Our approach provides an informative proxy by clustering acoustic events, like toilet flushes and localized bathroom sounds, into 10–15 min episodes. While these clusters do not offer direct clinical confirmation of defecation, they provide a reliable, automated approximate indicator that preserves patient dignity, as we remind the reader that audio is classified on the fly. Data is not processed on commercial servers. Guardians and authorized personnel have temporary access to the raw audio data, which are subsequently deleted. Timely detection supports preventive interventions, reduces avoidable hospitalizations, and helps clinicians distinguish somatic causes of behavioral change from neuropsychiatric deterioration. In Figure 12, a heatmap of events is depicted for the whole monitoring period that allows caregivers to see a pattern of visits to the toilet. Notably, there were no missed detections or false alarms regarding bathroom visits, as each sentinel is located directly inside the bathroom with its specific placement logged on the dashboard.

4. Nutritional Intake and Activities of Daily Living. Finally, an acoustic node in the kitchen can monitor nutritional adherence through the acoustic detection of culinary activities and meal consumption. The sounds associated with dishes, pots and pans, plate handling, utensil use, and kitchen appliance operation serve as proxies for eating habits, which are frequently disrupted in dementia. Malnutrition and dehydration are common contributors to “sundowning” and accelerated cognitive regression, yet they are difficult to track accurately through traditional self-reporting. By identifying clusters of culinary-related noises in the kitchen area (refrigerator opening/closing, electrical appliances, drawers/closets sounds, and cleaning), the system can verify whether a patient or caregiver is engaging in regular meal preparation and consumption. Along this line, we can detect and count the sounds of a cane and other mobility aids to assess mobility and nocturnal activity (see Figure 13). These data allow healthcare providers to correlate periods of agitation or lethargy with potential caloric or fluid deficits, enabling targeted interventions, specifically dietary adjustments or assisted feeding protocols, to maintain the patient’s physical resilience.

Monitoring TV usage (see Figure 14) and silence periods (Figure 15) and uploading measurements on the household’s dashboard can provide valuable insights into daily routines and the well-being of elderly people living with dementia. Patterns in TV use may reflect levels of engagement, cognitive function, and adherence to habitual behaviors, with sudden increases or decreases potentially indicating changes in mood, confusion, or apathy.

A significant increase in the duration and frequency of silence during times when the patient was previously social or active can indicate social disengagement, where the patient is losing the motivation or ability to initiate conversation. Long periods of silence are also key markers for depression, especially when combined with physical inactivity.

Similarly, prolonged periods of silence can signal possible issues, notably social withdrawal, disrupted sleep, or lack of activity, while deviations from typical patterns may help identify risks such as falls, distress, or medical events. Together, these indicators offer a non-intrusive way to track behavioral changes over time and support early intervention when needed.

3.2. Fall Detector

To evaluate the efficacy of the proposed VLM-based pipeline, we conducted a large-scale validation study using a dataset of 3000 one-minute video clips taken from the Wi-Fi camera. Because true fall events are rare, the 10-month continuous observation dataset was augmented with 10 verified fall clips from the same participants recorded over the preceding two years to adequately evaluate the detection model. The resulting dataset is highly imbalanced, reflecting real-world conditions with a low frequency of actual fall events. Out of the 3000 videos processed, the ground truth identified 10 positive fall events and 2990 negative (normal activity) events. The system successfully identified all 10 true-positive falls, achieving a perfect sensitivity rate. However, the model produced 12 false-positive alerts, where non-fall activities were misclassified as falls. These errors stem from the loss of temporal context when utilizing static images rather than video sequences for analysis. Therefore, several cases where a caregiver was lying on the floor, usually to clean under a bed, were misclassified as falls.

The metrics can be found in

Table 2. Metrics and their definitions.

Metric	Formula
Accuracy	(TP + TN)/(TP + TN + FP + FN)
Sensitivity/Recall/True-Positive Rate	TP/(TP + FN)
Specificity/True-Negative Rate	TN/(TN + FP)
Precision	TP/(TP + FP)
F1 Score	2 × (Precision × Recall)/(Precision + Recall)
False-Positive Rate	FP/(FP + TN)
False-Negative Rate	FN/(FN + TP)

, whereas the classification performance is summarized in

Table 3. Performance metrics with Wilson 95% confidence intervals (CI) for the VLM-based fall-detection pipeline. Evaluation utilizing a validation set of 3000 video clips (highly imbalanced set as true falls are rare).

Metric	Value	95% CI
Accuracy	0.996	[0.993–0.998]
Sensitivity/Recall/True Positive	1.0	[0.722–1.000]
Specificity/True-Negative Rate	0.996	[0.993–0.998]
Precision	0.454	[0.266–0.654]
F1 Score	0.625
False-Positive Rate	0.004
False-Negative Rate	0.0

below:

Running the code on the corpus of video clips produces the following results gathered in Table 3:

The proposed VLM-based fall-detection pipeline achieved high overall accuracy, 0.996, and high specificity, 0.996, on the 3000-video validation set. However, these values should be interpreted in the context of the strong class imbalance, since only 10 of the 3000 clips corresponded to true fall events. Although the system detected all 10 falls, yielding an observed sensitivity of 1.000, the Wilson 95% confidence interval was wide (0.722–1.000), reflecting the uncertainty caused by the small number of positive cases. Similarly, the false-negative rate was 0.000, but its confidence interval extended up to 0.278. The system generated 12 false-positive alerts among 2990 non-fall clips, corresponding to a low false-positive rate of 0.004. Nevertheless, because fall events were rare, these false positives substantially reduced the precision to 0.455 and the F1-score to 0.625. This indicates that while the system shows promise in avoiding missed falls, further work is needed to reduce false alarms, particularly in scenarios where non-fall activities, such as a caregiver lying on the floor to clean under a bed, visually resemble fall events.

Since ethical restrictions limit the distribution of complete video databases involving vulnerable people, we share the code used in our experiments (see Appendix A.3) and evaluate the method on an open fall-detection dataset [35] that resembles our target problem. The dataset contains 192 video clips, of which 176 contain falls (folders 1–22) and 16 correspond to non-fall situations (folders 23–24). Importantly, the non-fall subset is small and includes several activities or postures that may visually resemble falls, such as kneeling, stumbling on furniture, bending over, falling onto a couch, sweeping the floor, and picking up items from the floor. Therefore, performance estimates, particularly specificity and precision, should be interpreted with caution.

The results are summarized in

Table 4. Performance metrics with Wilson 95% confidence intervals for the Qwen3-VL-2B-Instruct on the fall-detection task. Evaluation utilizing a validation set of 192 video clips of the Multiple Cameras Fall Dataset of the University of Montreal.

Metric	Value	95% CI
Accuracy	0.891	[0.839, 0.927]
Sensitivity/Recall/True Positive	0.898	[0.844, 0.934]
Specificity/True-Negative Rate	0.813	[0.570, 0.934]
Precision	0.981	[0.947, 0.994]
F1 Score	0.936
False-Positive Rate	0.187
False-Negative Rate	0.102

. Wilson 95% confidence intervals were used because they provide more reliable coverage than Wald intervals for small sample sizes and proportions close to 0 or 1. This is particularly relevant for specificity, which is estimated from only 16 non-fall clips. Owing to the limited number of negative examples, the confidence interval for specificity is relatively wide, highlighting substantial uncertainty in the estimated true-negative performance.

3.3. Delusion Detection and Automatic Counting of Events

Due to ethical constraints, we are unable to provide a large corpus of transcribed audio. However, to demonstrate the potential of our approach, we provide an open sample analyzing a specific incident in which repetitive delusional urges and exit-seeking behavior, lasting nearly two hours, led the non-delusional partner to extreme agitation and near burnout. The incident, recorded on 14 February 2026, between 02:00 and 04:00, highlighted the system’s potential utility in identifying behavioral changes associated with the missed administration of prescribed neuropsychiatric and sedative medications to the female participant.

This incident is of particular interest for several reasons: (a) we acknowledge that abuse and violence can stem from burnout among untrained caregivers or those living with dementia patients long-term, who are frequently exposed to repetitive questioning, constant pleas to leave the home, agitation, and delusional discourse, (b) guardians (usually the children of the elders)—even those living nearby—often lack visibility into the daily environment of elders living alone, leading them to underestimate how quickly situations can become acute, and (c) we provide a case study demonstrating our approach, which utilizes local speech transcription combined with LLM filtering monitoring delusional and repetitive exit-seeking behaviors, when expressed verbally and transcribed accurately. The detection and clinical management of delusional clusters represent a critical component of the proposed system, as these episodes—characterized by persecutory delusions regarding theft, or loss of property—frequently contradict the documented history and immediate reality of the patient.

An LLM-based analysis of the resulting 123-turn dialogue between the elders, contextualized against the couple’s history, was performed. Regarding the annotation protocol, transcripts were automatically generated by employing the Faster Whisper ASR and diarization. However, due to dysarthric speech patterns, Greek dialogues were manually corrected and annotated with speaker-diarization tags for male and female speakers ([M]/[F]), as well as agitation indicators ([A]). A binary classification of “Delusional” versus “Non-delusional” was applied, consistent with the clinical definition of delusions as fixed false beliefs resistant to contradictory evidence. Strict decision rules were enforced to distinguish delusions from simple confusion or memory gaps. Delusional patterns were identified by cross-referencing patient narratives against validated family background data to isolate factual contradictions. We are confident in the ground truth of the dialogue labels: delusions in patients with dementia tend to follow predictable patterns. In particular, the most prevalent pattern observed in our corpus is the misidentification of the home as an alien space, which led to repetitive exit-seeking behavior [47]. Another pattern is the belief that her husband is simultaneously present in their current home but also waiting for her in her fictitious home. Note that the LLM can justify its decision and report which facts from the small history of patients are contradicted and therefore is auditable.

This dataset, beyond its practical value, also has research interest. The main difficulties the LLM has to cope with are:

(A) The classification of a dialogue as “delusional” is not an isolated decision. It is inherently dependent on the preceding conversational context. For instance, a speaker describing a store is not, on its own, a delusional act. However, if that same speaker established moments earlier that she was at home, her subsequent description of being in a store represents a break from objective reality. In this case, the statement continues a documented delusional pattern and must be assigned a ground-truth label of “delusion.” This demonstrates why accurate classification requires a rolling context window—such as the three-phrase sequence illustrated in Figure 16—to capture the narrative arc rather than just individual sentences.

(B) Human verbal interaction is rarely straightforward, particularly when clinical histories are involved. Consider a male subject who has been officially diagnosed as non-delusional. If this diagnostic baseline is excluded from the model’s input, the LLM might misidentify his statements as delusional, as in the dialogue corpus we examine he often expresses “false beliefs” that contradict objective facts. However, a nuanced analysis of the dialogue reveals that these are not clinical delusions, but rather confabulatory reassurances—intentional untruths used as a de-escalation tactic to comfort the female speaker (e.g., “yes we will leave house in the morning, sleep now is very late”). By analyzing the subtext of the interaction, the LLM is able to distinguish between a pathological lack of insight and a socially motivated departure from the truth, correctly identifying the speaker’s intent.

We present detection results in Figure 16, using a local model (Gemma2:9b) and a commercial one (GPT-4.1). Confusion matrices for the same task follow in Figure 17 and in

Table 5. Automatic delusion detection accuracy on a corpus of 123 short dialogues, with 70 positive and 53 negative cases, using a local Gemma2:9b model and a GPT-4.1 API. Results are for a window context of three.

	Gemma2:9b	GPT-4.1
Accuracy	0.813	0.805
Sensitivity/Recall/True Positive Rate	0.943	1
Specificity/True-Negative Rate	0.642	0.547
Precision	0.776	0.745
F1 Score	0.852	0.854
False-Positive Rate	0.358	0.453
False-Negative Rate	0.057	0

for comparison. Each LLM is presented with a short dialogue, and the LLM classifies the dialogue as delusional or not, leveraging the history of the couple (see Appendix A.3), the current dialogue and three previous ones for context. GPT-4.1 is not part of the deployment pipeline. It is utilized as a performance benchmark during the evaluation phase to assess the accuracy boundaries of the local model As regards its practical use, not all patients have dysarthria. Generally speaking, dementia at Stage-1 begins at 65 years, and we have analyzed healthy elders at 83 without a significant drop in word error rates.

A key function of the system is to automate behavioral data collection for clinicians, helping them guide targeted interventions that support the home care environment. Systematically documenting these daily behavioral patterns provides objective insights that can assist families in managing care at home longer, potentially delaying the need for institutional placement. We clarify that the proposed system is not intended to provide autonomous clinical decision support or automated pharmacological recommendations.

3.4. Analyzing the Linguistic Content of Dialogue of People with Dementia

We view spontaneous speech not just as a communication medium, but as a continuous stream of clinical data that provides insights into a patient’s cognitive state and immediate safety. We analyze the same corpus in Section 3.3 as an example of spontaneous speech. The analysis of spontaneous speech has been suggested for the quantification of dementia. In fact, it is more revealing than reading text or reciting written text from memory [48]. The transcript is automatically split into female and male counterparts, which are then analyzed independently. At its core, the system monitors dialogues, attributes utterances to specific speakers, and performs transcription and biomarker extraction to generate speaker-dependent metrics for dashboard display (see

Table 6. Transcribing dialogues and deriving biomarkers that are appended on dashboard. Example case: Extracting biomarkers from a dialogue of 123 short dialogues between two patients with dementia. Text is grouped per speaker, and 10 biomarkers are extracted. Green: Normal, Red: Abnormal (Flagged).

Metric	Threshold	Female (F)	Male (M)	Normal Range	F Status	M Status
SEMANTIC CONTENT
Idea density	≥0.45	0.46	0.501	0.50–0.65	Marginal	Normal
Noun-verb ratio	≥1.0	0.595	0.649	1.00–1.50	Flagged	Flagged
Light verb ratio	≤0.15	0.088	0.1	0.05–0.15	Normal	Normal
SYNTACTIC COMPLEXITY
Mean dependency distance	≥2.25	1.326	1.357	2.25–3.50	Flagged	Flagged
Coordinate conjunction rate	≤0.10	0.034	0.027	0.03–0.10	Normal	Normal
Mean utterance length	≥7.0 words	5.52	6.19	7–12 words	Flagged	Flagged
LEXICAL DIVERSITY
TTR–type-token ratio	≥0.45	0.189	0.17	0.45–0.70	Flagged	Flagged
MATTR (window = 25)	≥0.65	0.74	0.742	0.65–0.85	Normal	Normal
REFERENTIAL & REPETITIVE BEHAVIOUR
Pronoun-noun ratio	≤0.72	1.243	1.328	0.30–0.72	Flagged	Flagged
Perseveration index	≤0.20	0.221	0.314	0.05–0.20	Flagged	Flagged

).

The female speaker flags six of ten metrics. The one near-miss is idea density (0.460 vs. threshold 0.45), which, as discussed, is structurally inflated by short turns—it is better read as marginal than normal. The pattern is consistent with moderate-stage AD: syntactic flattening (MDD), utterance fragmentation (MUL), anomic verb loss (NVR), referential vagueness (PNR), and pathological repetition (PI). The male speaker also flags six metrics—the same six—which is clinically important context. He is speaking under severe nocturnal stress, and his speech in this recording is itself compressed, repetitive, and pragmatically reduced. His perseveration index (0.314) is notably higher than hers (0.221), reflecting the looping, pleading nature of his turns (“ξημερώσει... ξημερώσει... ξημερώσει”). This does not indicate pathology, but the interaction itself is distorting both speakers’ profiles.

TTR vs. MATTR—the contrast is methodologically instructive: TTR flags both speakers severely (0.189/0.170) while MATTR returns normal (0.74/0.742). This is the length-sensitivity artifact in action. MATTR is the more valid measure here. TTR is retained as a supplementary comparison to illustrate exactly this point.

Notably, the biomarkers correspond to clinical reality. The female indeed has a sharper mental decline than her male partner, as confirmed by caregivers and medical staff. Table 6 metrics can be derived daily and logged directly into a dashboard, enabling medical staff to track dementia progression over long-term intervals spanning months or years.

3.5. Graph-Based Representation of the Linguistic Content of Dialogue of People with Dementia over Long Horizons

In Section 3.3, we analyzed an incident to detect and count delusional events based on transcribed speech, the couple’s medical history, and an LLM. We employed short-term memory via a rolling window of transcribed content to provide context for classifying the current sentence. However, an online system operating over long horizons (months to years) cannot retain referential coherence, even if it is augmented with long-term memory and summarization. The primary motivation remains to provide caregivers and guardians with immediate access to dashboard patterns that reveal clinically significant incidents. These patterns are identified whenever they are associated with verbal expressions such as agitation, delusion, apathy, depression, hallucinations, exit-seeking, perseveration, pain, or social engagement. This provides structured clinical insight into the patient’s evolving perception of reality—a necessity for informed intervention and care planning. Consequently, we suggest parsing transcribed context into graphs (see Figure 18).

Analyzing longitudinal dialogues of individuals with dementia using a GraphRAG framework offers substantial methodological advantages over conventional LLM-based retrieval-augmented generation (RAG) systems [49,50]. In contrast to standard RAG, which relies on vector similarity to fetch isolated “snippets” of past conversations, a GraphRAG approach constructs an evolving knowledge graph that encodes entities (e.g., speakers, actions, symptoms, and recurring themes) and their relationships over time. A key insight is that in elderly care environments, there are few inhabitants, and dementia often leads to repetitive actions and transitions. A graph-based system exploits this repetition to achieve extreme data compression. The graph can be subsequently traced through an LLM to get answers on complex questions.

By aggregating dialogue data into a persistent graph, the system not only enormously compresses the text but also captures subtle longitudinal patterns—such as shifts in social connectedness or the emergence of repetitive narrative loops—that would be difficult to detect through isolated retrievals. Temporal or pattern-based queries become graph traversals rather than full-text re-reads. Moreover, GraphRAG supports multi-hop reasoning over extended periods, enabling the model to link distant conversational elements and identify clinically relevant trajectories rather than mere surface-level similarities.

This persistence is essential for deployments spanning months or years, as it avoids the instability and context loss inherent in memory-based RAG systems, whose representations can degrade or become inconsistent over long horizons. Additionally, the graph structure enhances interpretability and auditability—key requirements in scientific and clinical settings—by allowing researchers to trace how specific inferences arise from underlying relational data. Common queries include reporting and counting delusional, exit-seeking, or agitation incidents per month. In the code section (see Appendix A), the text is first converted into nodes and edges and then traced using a local Gemma2:9b model. In

Table 7. The transcribed dialogues are transformed into Subject–Action–Object triplets. Querying a GraphRAG representation of 123 dialogue turns for evidence of exit-seeking, delusion and agitation.

		Subject–Action–Object	Seeking an Exit
#	Subject	Action	Object
1	Female	wants to finish and leave	to go to her home
2	Female	she will get up and leave	to catch the bus
3	Female	she will pull	her legs (to get up and leave)
4	Female	she will take	Take a taxi and leave.
5	Female	demands	Male opened the door
6	Female	demands	the key
		Delusional/Disorientational Disorder
7	Female	says	“Your room has moved”
8	Female	refuses	This is her home.
9	Female	says	her home is “up there”
10	Female	says	She came here by mistake.
11	Female	believes	It is not night.
		Agitation/Urgency
12	Female	says	They don’t know she is here.
13	Female	says	“Everyone will go crazy looking for me”
14	Male	tells Female	to sit there
15	Male	says	There is no taxi or public transport.
16	Male	says	It is cold outside.
17	Male	says	cannot (physically) help her get up
18	Male	tells Female	“Pull yourself together”
19	Male	Female asks	“Who is your husband?” (attempt at reorientation)
21	Male	says	He has reached his limit.
22	Male	shouts	“LEAVE ME ALONE!” (distressed)
23	Male	says	“You are torturing me”
24	Male	tells Female	to lie down and sleep

, we present a translated sample where the system detects exit-seeking, delusion, and agitation while providing supporting evidence. Ultimately, GraphRAG serves as a framework for the fully automated agent responses of the near future. Using the audio sentinels shown in Figure 2, these agents will be able to broadcast messages, provide task reminders, offer reassurance, or redirect residents to bed during nocturnal walks.

In our implementation, GraphRAG data entry is slow, and weights should be updated at night.

4. Discussion

Our position is that privacy can be rigorously preserved by design by ensuring no raw multimedia streams or unencrypted telemetry are continuously transmitted to, or processed by, centralized third-party commercial cloud servers or unauthorized users. When a system enforces this strict boundary, its constituent ‘sentinel’ agents can no longer be regarded as invasive observers. Instead, they function as enablers of a qualitatively different living environment. In this paradigm, the home is transformed from a passive physical shell into a responsive and context-aware setting that understands the needs of and can sensitively support its vulnerable inhabitants. Crucially, these systems are conceived to integrate seamlessly into the domestic environment as ambient, unobtrusive elements of the household—akin to furniture—thereby avoiding disruption while enhancing safety, awareness, and quality of life.

Reports of fatal incidents involving older adults are common in the media, with fires, gas-related accidents, suicide and falls among the most frequent causes. While not all adverse events involving older adults are directly attributable to dementia or lack of supervision, individuals with cognitive impairment are at increased risk of hazardous situations due to memory loss, impaired judgment, and reduced awareness of danger. Furthermore, the repetitive behaviors frequently associated with dementia—including perseveration (asking the same questions repeatedly), denial of facts, agitation, and behavioral disturbances—can place significant emotional strain on caregivers or partners. Such prolonged stress often leads to caregiver burnout, which, in extreme but documented cases, may escalate into neglect, psychological distress, or physical violence. In this context, carefully designed monitoring systems can provide an added layer of support by detecting and documenting early signs of risk, exemplified by unusual inactivity, distress-related sounds, deviations from established routines, nocturnal activity and delusions. By enabling timely intervention and support in a form that depends on clinical status, these systems can help reduce the likelihood of life-threatening situations, complementing human care while preserving dignity and independence for both the individual and their caregiver.

4.1. Ethics

The proposed system raises an inherent ethical tension between the protection of personal privacy (see [51] for a deep analysis of this issue) and the justification for continuous monitoring of a vulnerable population [52]. This work adopts the position that ethical guardrails should depend on proportionality, data minimization, and clinical necessity.

The proposed methodology is ethically justified by the balance of its clinical utility against privacy risks. The benefits—including injury prevention, emergency detection, and the facilitation of informed decision-making among stakeholders—outweigh the limited and strictly controlled intrusions into private life. Privacy risks are further mitigated by restricting access to healthcare professionals bound by medical confidentiality or consenting close relatives. A staged monitoring strategy is implemented to align data access and processing with the progression of cognitive impairment:

At Dementia-1 (Mild Dementia), a minimally intrusive, privacy-preserving approach is adopted, in which an on-site, consumer-grade edge server processes audio locally without uploading data, ensuring that all data remain in situ. Clinical staff access through a secure connection a dashboard hosted in an in-home server that presents only text of classified audio events and aggregated statistics, with no access to raw audio streams, speech transcripts, or live audio connections. Automatic speech recognition (ASR) operates entirely on-device and is restricted to identifying filler words and predefined emergency-related keywords (“help,” “fallen,” “My [hip/leg/back]!,” “Can anyone hear me?”). At this stage, staff can review strictly clinically relevant indicators, including the temporal distribution of speech activity, coughing, snoring, mobility patterns, and emergency incidents detected via keyword-based speech transcription. LLMs then synthesize these event labels into structured summaries describing household activity to support baseline health monitoring, without exposing any identifiable speech content either in transcribed or raw form. Audio data of the elders are immediately deleted from the in-home server after text labels are extracted from audio recognizers, whereas sentinels are configured in a stateless configuration. Bilateral communication is possible, but external users can only see labels’ distributions and heatmaps and cannot listen to raw data.

At early Stage-2 dementia (Early Progression), additional analytical layers are introduced. Full ASR outputs and emotion tagging extracted from text and audio features are appended to the dashboard. Speech is then subjected to granular linguistic analysis—tracking vocabulary diversity, part-of-speech ratios (e.g., nominalization shifts), and prosodic features. Biomarkers extracted from transcribed speech quantify the rate of cognitive decline in the long run. Given the increased probability of falls, confusion, and delusional episodes, the system initiates rolling event-based LLM assessment of all information on the dashboard covering the most recent ten-minute window. When risk thresholds are exceeded, automated notifications are sent to caregivers or nursing staff via email. Importantly, access to raw audio remains restricted, maintaining privacy while enhancing situational awareness. Original audio is deleted after being processed, and only abstractions derived from it in the form of reports are preserved. External users still do not have access to raw audio.

Upon progression to severe Stage-2 Dementia, the system transitions to a high-fidelity monitoring protocol, ethically justified by a substantially elevated clinical risk profile. At this juncture, the duty of beneficence—ensuring the patient’s physical safety—necessitates a broader scope of system access. This is facilitated through bidirectional audiovisual channels, incorporating a single integrated unit equipped with night-vision capabilities, a microphone, and a loudspeaker operated by a human. The data stream is processed by Large Audiovisual Models (LAVMs) to provide contextual awareness and to tag potential hazards that audio alone may fail to capture, such as falls, choking, nocturnal falls, or hazardous activities of daily living (ADLs) like exit seeking, unattended cooking, smoking while falling asleep, and agitation. Currently, only a fall detector has been implemented in local processing. Audio is retained in a database for one day, and the medical staff or guardian by law has direct access to it by clicking the corresponding timestamp on the dashboard. To preserve the privacy of the care circle, speaker diarization is employed to isolate patient speech from that of caregivers or visitors. By systematically documenting delusional patterns, exit-seeking behaviors, and repetitive questioning, the system provides an empirical basis for adjusting pharmacological treatments and implementing caregiver support interventions, thereby preventing caregiver burnout and early hospitalization and providing evidence to reach consensus on potential institutionalization (see also

Table 8. Following the Least Restrictive Alternative for elders with dementia/Alzheimer’s spectrum.

Dementia Stage	Technical Implementation	Privacy Preservation Status	Primary Safety & Clinical Objective
Stage 1 (Mild)	Local edge processing, Dashboard of event labels (e.g., “Cough,” “Toilet”) and LLM-generated summaries.	High: No raw audio leaves the home, no live connection, speech recognition of keywords.	Baseline health tracking, detection of abnormal physical patterns (sleep, bathroom frequency, safety flags).
Stage 2 (Early Progression)	Automated Speech Recognition (ASR), 10 min rolling buffer for event context, Linguistic/Emotion analysis.	Medium: ASR used for clinical biomarkers, direct staff access to audio remains restricted.	Monitoring cognitive decline (vocabulary/syntax), automated alerts for falls or delusional episodes.
Stage 2 (Severe)	Multimodal (Audio/Video), Night-vision camera, Bidirectional intercom, Live stream access for verified alerts.	Low: Integrated AV surveillance in common areas, Speaker diarization protects caregiver/visitor privacy. Direct access to raw audio.	Life-saving intervention (choking, fire, falls), mitigating caregiver burnout, managing severe delusions and hallucinations.

).

4.2. Data Perimeter and Network Isolation Architecture

To ensure the dignity and privacy of residents experiencing progressive cognitive decline, the system enforces a strict localized data perimeter. Rather than operating an open cloud-connected pipeline, the framework adopts a local-first edge inference model. Under this paradigm, continuous multimodal data ingestion, acoustic event detection (via Audio Spectrogram Transformer), and speech processing are executed entirely on-premise on the local edge hardware using localized models (e.g., Gemma 2). External communication boundaries are tiered according to user clearance depending on the clinical stage and network protocols:

Abstracted Telemetry (Outbound): Outbound notification bridges (e.g., Brevo email alerts; see also Appendix A.4) are restricted to transactional, low-bandwidth semantic metadata (such as timestamped emergency alert tokens) and do not stream raw or unencrypted background data layers.

Peer-to-Peer Encryption (Remote Monitoring): External access by clinicians or designated family guardians via the dashboard is mediated through an end-to-end encrypted virtual overlay network (Tailscale). This configuration bypasses intermediate centralized cloud servers, facilitating a direct, secure peer-to-peer tunnel to the home server.

Access Controls for Advanced Cognitive Decline: While remote views are restricted to abstracted behavioral event labels by default, the framework accommodates an escalated access tier for patients at Stage-2 dementia and above. To assist legal proxies (e.g., a primary family caregiver) in verifying critical incidents without requiring immediate physical travel, the system allows the authenticated proxy to securely retrieve a localized, rolling 24 h raw audio buffer via the encrypted P2P link. Raw multimedia storage is strictly transient and automatically purged on the local edge server post-window expiration.

4.3. Practical Aspects

While dementia remains irreversible, its rate of progression is not fixed. We believe technology can significantly enhance the quality of life for seniors with dementia and mobility problems by identifying early warning signs that allow for timely intervention. Intervention resolves problems and pushes back hospitalization. Our core insight is simple: a home cannot meet the evolving needs of its residents if it remains a static shell as decades pass. By deploying a network of AI-powered, plug-and-play “sentinels,” we transform the home into a clinically aware environment. This approach aligns with the goals of families while alleviating the burden on overstretched elder-care facilities. We advocate for voice-activated intercoms because patients in Stage 2 dementia often lack the cognitive presence to use wearable “SOS” buttons or bracelets. Unlike those with only mobility impairments, these individuals face cognitive decline. They may not remember a crisis or be able to report symptoms the following day, so we must identify risks in real time. Because hospitalization rarely happens “out of thin air,” timely detection of needs and subsequent interventions can successfully postpone major medical events.

The following case scenarios are categorized by their verification status. The first scenario has been validated through multi-member simulations, while the second and third have been confirmed during the 10-month monitoring period. The fourth scenario remains a pending task.

(1)

Immediate Fall Response

• The Problem: George (40) received an automated app notification indicating his father had fallen in the bathroom. Through his mobile, he connected and confirmed this.
• The Action: George initiated a live voice connection, reassuring his father and keeping him calm for the 10 min it took to drive home.
• The Result: His father was assisted within the “golden hour” of the fall. By avoiding a long period of immobility on the floor, they prevented both post-traumatic shock and a high-risk hospitalization.

(2)

Mobility Intervention

• The Problem: While reviewing speech-event logs, Anna (57) noticed her 90-year-old father frequently expressing frustration and saying “My legs won’t hold me up.”
• The Action: Anna immediately scheduled a physiotherapist for a home assessment.
• The Result: Targeted physical therapy improved his strength, preventing a predicted fall and a permanent loss of mobility. The problem is not eliminated, but action will be repeated as needed, pushing back in time a possible fall.

(3)

Managing Agitation and Delusions

• The Problem: Mary (54) used the app to monitor speech events and discovered her mother was experiencing “exit-seeking” delusions—insisting on “going home” and becoming agitated. She was also wandering at night and becoming stressed by minor factors (bedsheets, humidity, furniture blockage, etc.).
• The Action: Mary sought a consultation with her primary care physician to refine the pharmacological management of her delusional symptoms. Simultaneously, she introduced environmental safety modifications, such as a hospital-grade bed with side rails to discourage unassisted ambulation and the use of floor mats. Additionally, she cleared pathways by rearranging the furniture. In coordination with her part-time caregiver, she tuned climate control and optimized the home layout by repositioning furniture and securing exit points.
• The Result: Nighttime activity ceased and agitation decreased. By addressing small but frequently occurring stressors and increasing “reminiscence time” with family albums, her mother became more stable.

(4)

Professional Supervision and Social Connection

• The Professional: Paraskevi, a healthcare professional, monitors 20 remote residences in Crete via the central dashboard (commercial application).
• The Action: Each day, Paraskevi performs a “virtual round.” A gentle chime announces her “arrival,” and she engages each elder in a 5 min “micro-chat” directly in the room where the elder resides. She has a small history notepad for each case prefilled by their child-guardian; therefore, she can form questions in a way that elders find engaging (“Your daughter Helen has two kids, have you heard from them lately?,” “Is your leg getting better?”). As a trained professional, she assesses their mental state during the conversation, updates the database with notes, and alerts families to specific needs or emergencies before moving to the next case.
• The Result: This “digital bridge” combats social exclusion. These brief, daily cognitive stimulations motivate the mind and reduce boredom and depression, ensuring the elder feels valued and watched over by society. The elderly users look forward to this daily conversation, as it is brief enough not to tire them while still providing a meaningful human connection. Some of them would otherwise go long periods without speaking to anyone, especially those living alone.

4.4. Commercial Prospects

To ensure the system is viable for real-world implementation, we have supplemented our technological framework with a foundational economic model. For individual residential use, we propose a subscription-based model of a mobile app priced at 50€ per month under a three-year contract. This allows the system to reach a net profit threshold after the first year, offsetting the hardware cost of approximately 600€. For commercial applications, such as professional care networks, the subscription is set at 200€ per month, yielding a net profit of 4000€ per 10 clients annually (6000€ hardware cost and 14,000€ annual salary of the supervisor). This price variance is justified by the service level: the professional tier includes a medical supervisor who performs daily check-ins and coordinates information updates for all stakeholders, whereas the residential tier is managed directly by the user. Prices are indicative, and the tiered structure aims at accessibility for private households while providing a sustainable growth path for commercial healthcare providers.

Existing commercial solutions address these challenges through various modalities, most notably via vision-based systems. While effective, the transmission of raw video from within a private residence presents insurmountable ethical and privacy concerns. To mitigate this, some commercial systems employ “edge intelligence” to map raw feeds into privacy-preserving abstractions, such as skeletal stick figures or avatars, while utilizing AI to recognize emergencies such as falls. However, these methods significantly increase computational requirements and are limited by line-of-sight. Providing comprehensive coverage for an entire home would require multiple units, severely increasing the total cost of deployment. Other alternatives, such as Doppler (mmWave) radar and Wi-Fi scattering, offer high privacy but are generally narrow in scope, focusing almost exclusively on fall detection. Wearable devices, such as bracelets and emergency buttons, are largely insufficient for individuals with Stage 2 dementia or beyond; patients often forget to wear them, and accidental activations frequently trigger false alarms. In practice, maintaining the efficacy of wearable accelerometers is challenging, as the high incidence of false positives can overwhelm external supervisors with extraneous notifications. In contrast, our approach targets elders with dementia and is specifically designed for global scalability and mass adoption, particularly in developing regions and emerging economies where high-cost hardware is not a viable option. By utilizing low-cost audio components rather than expensive sensors or high-compute vision systems, we cover all rooms and provide holistic services. Mobile applications can be developed to support both informal, family-based caregiving and professional, commercial-grade clinical monitoring. Dementia care must not become a luxury, but an accessible reality for the millions of families in countries where the need for affordable, wide-scale elderly support is most acute.

5. Challenges and Future Prospects

In future commercial iterations, the system will integrate an intelligent voice agent designed to conduct routine check-ins and extract longitudinal clinical observations directly from daily discourse. By analyzing conversation, the agent can populate the database with summaries and objective indicators of cognitive decline without requiring manual entry. To maintain ethical and safety standards, the AI’s role would be strictly limited to greetings, reassurance and social small talk. It will be intentionally restricted from engaging with the patient’s deeper psychological realm or complex clinical needs. Should an interaction move beyond these predefined boundaries, or speech recognition return poor results due to dysarthria, local dialects or use of slang, the system automatically triggers a seamless hand-off to a medical supervisor. This hybrid architecture allows for massive scalability, enabling a single clinical center to provide personalized, high-frequency oversight to hundreds of households in parallel while keeping that human expertise central for critical care.

Audio data alone lacks the capacity to resolve complex everyday situations automatically; ASR limitations—particularly high false-alarm rates in the Greek language—preclude the use of a fully autonomous, audio-only system. Accuracy is further compromised by dysarthria and local dialects, especially in languages that are underrepresented in large, reliable corpora. Sentinels need to fine-tune their activation thresholds to respond to proximal audio activity, as reverberation can cause ASR and speaker diarization errors that are propagated to text analysis and biomarkers (see also Appendix A.5).

System resilience is critical in AAL environments. In the event of a power failure, the distributed components of the integrated system may lose synchronization or network connectivity upon reboot. To address this, we propose the integration of “agentic” frameworks, exemplified by OpenClaw, hosted on the local server. Unlike traditional static scripts, an agentic approach can autonomously perform diagnostic sweeps, re-establish handshake protocols between “audio sentinels” and the central hub, and verify the integrity of the stateless data pipeline without manual intervention. By acting as an autonomous system administrator, such an agent ensures that the “caregiving partner” resumes full monitoring capabilities immediately after power restoration.

We have carried out a number of informal deliveries of messages inside the home of the elders regarding reassurance and redirection, and they were well received, raising further questions about whether cloned voices of close relatives may provide a more effective and emotionally resonant medium for messaging. We recognize, however, that this feasibility study relies on a single case of a couple in their home.

Informed consent in the context of dementia should be understood as a dynamic and ongoing process rather than a one-time decision. In the early stages of cognitive decline, individuals typically retain sufficient capacity to understand the purpose, benefits, and potential risks of living in an ‘augmented’ space, and their autonomous decision-making should be fully respected. As the condition progresses and decisional capacity diminishes, responsibility gradually shifts toward a collaborative “triad” involving the patient, their legal guardian or family, and the clinical team, with increasing weight placed on guardians and healthcare professionals to act in the patient’s best interests. However, even in later stages, the patient’s expressed wishes—verbal or behavioral—remain critically important. Any clear indication of discomfort, distress, or refusal directed toward the monitoring devices or the act of being monitored should be taken as withdrawal of assent, and participation must be discontinued immediately. Clinical distress arising from the progression of dementia or acute medical incidents is precisely what the system aims to detect to alert guardians and does not constitute a withdrawal of consent. Moreover, to respect privacy boundaries, monitoring functions are temporarily deactivated during social gatherings because external guests fall outside the scope of established participant consent. This is not a serious limitation, as elders with dementia are in danger mainly during the absence of other people from their home.

Finally, clinical guardrails are paramount [53]. For patients with conditions like schizophrenia or mania, a proactive, disembodied voice or the fact that they are being monitored, even if information does not leave home, can dangerously blur the lines between reality and hallucinations. Consequently, a truly “smart” home must be capable of receding into the background, adjusting its proactive intensity in light of the resident’s real-time psychiatric history and current state.

6. Conclusions

This paper presents a holistic ecosystem for dementia care mediated by an edge-AI architecture, designed to transition the domestic environment from a passive shelter into an active, affective partner. By deploying a zero-interface hardware configuration that eliminates the need for wearables, the system maintains a resilient emotional and physical bond between patients and external guardians without imposing a cognitive or technical burden on the elderly. Crucially, this research demonstrates that multimodal monitoring can coexist alongside strict data privacy. Hosting all inference—from VLM-based visual analysis to AST-based acoustic recognition—on a localized home server eliminates cloud-based data transmission, successfully mitigating the surveillance concerns traditionally associated with ambient assisted living technologies.

The longitudinal data gathered over a 10-month period underscore the system’s value in providing automated, objective reporting with direct clinical utility. This deployment yielded insights that are highly sparse in the existing literature, demonstrating the capability to continuously track verbally expressed delusions, goal-directed nighttime activity, bathroom usage frequency, and linguistic biomarkers. By parsing these automatically transcribed corpora with GraphRAG frameworks, the system enables multi-hop reasoning over long-term behavioral trends. This data-driven approach provides clinicians and families with a quantitative map of a patient’s cognitive trajectory, allowing for timely interventions that can extend the period during which an individual can safely and with dignity “age in place.”

Moving forward, the ecosystem is designed to follow a strategic, staged progression. Having validated the core connectivity and privacy infrastructure, immediate next steps will involve transitioning peripheral tasks—such as daily check-ins, medication reminders, and nocturnal redirection messaging—to autonomous handling, alongside expanding the commercial capability for brief, automated five-minute dialogues. Future research will also explore the physical dimension of ambient intelligence by integrating proactive edge agents with socially assistive robotics. Specifically, utilizing a localized robotic platform (e.g., Reachy Mini by Hugging Face) as an embodied medication station will allow for the synergy of voice-led reminders and vision-based adherence monitoring, ultimately bridging the gap between ambient digital surveillance and physical caregiving orchestration.