Mindful Waters: An Interactive Digital Aquarium for People with Dementia

Abstract

Dementia can be associated with social withdrawal, mood changes, and decreased interaction. Animal-assisted therapies and robotic companions have shown potential in enhancing well-being but come with limitations like high maintenance costs and complexity. This research presents an interactive digital aquarium called Mindful Waters, which was developed to promote social interaction and engagement among People with Dementia. The pilot study involved interactive sessions at a community center and a care facility, with situated observations, video and audio recordings, and interviews to assess user engagement motivation, behavior, and user experience with Mindful Waters. The study revealed that Mindful Waters functioned well with People with Dementia and stimulated conversational topics about aquariums through engagement. User feedback was generally positive, with participants appreciating the visual appeal and simplicity. However, some participants with advanced dementia found it challenging to interact due to their mobility limitations, cognitive impairments, and the limited duration of interaction sessions. The overall results suggest that Mindful Waters can benefit dementia care; further research is needed to optimize its design and functionality for long-term placement in care facilities.

1. Introduction

Dementia is a progressive disease that affects an increasing population worldwide [1]. People with Dementia (PwD) suffer from cognitive and psychological symptoms that affect their daily activities and cognitive functioning and later result in their withdrawal from social activities, changes in moods and personality, decreased interactions, and problems with words when speaking or writing [2]. PwD in care facilities often have opportunities for initiating social interactions; however, they tend not to interact [3] likely due to the physical environment, presence of pain, and lack of social stimulation [4]. Moreover, regular interactions tend to occur only in small groups when initiated by staff members, which can be sub-optimal, as it increases staff member workload and is mostly limited to assisting PwD [3]. The cognitive and psychological symptoms, in combination with a low amount of social interaction, further influence the mental well-being of people with the disease, resulting in an increase in restless, tense behavior, a decrease in self-image [5] and, additionally, significant depressive symptoms [6].

Unfortunately, a cure does not exist for later-stage dementia. Nevertheless, it is possible to mitigate these symptoms and thus improve the Quality of Life (QoL) of PwD. For instance, researchers found that introducing an aquarium into a care facility can increase relaxation and lower stress [7]. Some believe this can be explained by social support and attachment since PwD may consider fish a source of companionship and build emotional bonds with it [8]; while others believe that the non-threatening nature itself is restorative [9,10]. However, like other animal-assisted therapy/activities, installing and maintaining an aquarium is costly. Alternatively, social robots, like PARO, were designed to offer benefits in alleviating stress and anxiety through interactive companionship [11], with PwD often caring for them, talking to them, and picking them up [12]. Although generally effective, some PwD may experience difficulty identifying with certain forms of social robots [12], necessitating interaction mediated by care staff [13]. Moreover, a few digital public interactive displays enable both individual, shared, and collaborative interactions, showing potential for enhancing engagement and communication among PwD [14,15], but they may also be experienced as overly complex [15].

To address these limitations and further extend the range of technologies that may add to the QoL of PwD, we propose Mindful Waters, which combines features of aquariums and public interactive displays. We expect that introducing Mindful Waters to care facilities may produce many positive effects for the residents. Specifically, the aquarium environments of Mindful Waters can have restorative effects on PwD, decrease their restless and tense behavior, and positively affect their mood. Research has shown that exposure to natural environments, even digital representations, can reduce stress [16,17]. In addition, a digital aquarium can add to aesthetic pleasure and engagement [18,19]. Moreover, Mindful Waters supports both individual and multi-person interactions, thus stimulating social interactions around the virtual aquarium. It can act as a conversation trigger that enhances social connectedness beyond direct interactions with the display itself. In this sense, the display may become a “triangulation object” —that is, a common point of interest that can draw people together and initiate conversations or interactions among them. The design of Mindful Waters adheres to the philosophy of warm technology, which emphasizes improving the QoL and well-being of PwD by using technology that is person-centered, multi-modal and focuses on supporting and extending a person’s capabilities, enhancing personal autonomy and social connectedness, and preserving dignity [20]. This approach also involves including PwD throughout the design process. Examples of warm technologies include VitaPillow, an interactive cushion that enables people with dementia to play music, and Geurenpalet, a tool that evokes memories through scent [21].

In this work, we designed and implemented Mindful Waters and conducted a field study to assess its effectiveness with the target users, i.e., PwD, in a local community center and a care facility. This pilot study focuses on understanding the feasibility and effectiveness of Mindful Waters in facilitating social interactions and physical activities for PwD. During the study, the researchers closely observed the interactions of PwD with the system and with one another, aiming to identify the engagement motivations, behaviors, and patterns of PwD based on the Participant Journey Map (PJM) that defines user engagement stages when interacting with public displays [22]. Additionally, we explore factors that may facilitate shared interaction, co-discovery, and conversations, as well as factors that may impede engagements, including environmental and system features. In particular, we designed the pilot study and analyzed the results to understand the following research questions:

• (RQ1) What are the users’ motivations for (not) interacting with Mindful Waters?
• (RQ2) What levels and kinds of engagement do people with dementia have with and around Mindful Waters, and to what extent can this be well captured by engagement models such as the PJM model?
• (RQ3) How usable is Mindful Waters for PwD?

The remainder of the paper is structured as follows. In Section 2, we review previous work on robotic animal-assisted therapy and activities, interactive displays and activities in nursing homes, and user behavior models for public displays, which are related to our research topic and questions. Then, we introduce Mindful Waters in Section 3, demonstrating the system design, the characteristic features, and their implementation. In Section 4, we illustrate the pilot study in detail, including participant information, experiment set-up, measurements, and procedures. In Section 5, we showcase the results of the pilot study, supplemented with graphs and tables for better visualization. In Section 6 and Section 7, we discuss the experiment results and study limitations, and future work, respectively. Eventually, we summarize this study in Section 8.

2. Related Work

To design a proper, effective, feasible interactive aquarium for PwD, we review existing work in (robotic) animal-assisted therapy/activities and interactive (semi-)public displays in nursing homes. Then, to understand engagement behaviors and patterns with public displays, we delve into user behavior models for describing interactions with (semi-)public displays.

2.1. Robotic Animal-Assisted Therapy and Activities

Animal-assisted therapy harnesses the positive influence of pets on the health and well-being of care home residents, including PwD [23]. By introducing pets into the care environment, researchers have observed improvements in aggressive, agitated, and disruptive behaviors, social interaction, self-care ability, and day-night disturbances or wandering [24]. These therapy pets can include traditional animals like cats [25] and dogs [26], as well as alternatives such as fish in aquariums [27,28]. However, a significant downside of traditional pets is the constant expense of food and maintenance and issues of hygiene and animal health. Therefore, robotic alternatives, like PARO [11], aibo [12], and JoyForAll [29], were developed to offer similar benefits in alleviating stress and anxiety through interactive companionship.

Possessing a user-friendly design resembling a baby seal, PARO [11] is among a list of well-known animal care robots [13]. Commonly utilized in the care of patients with autism, down syndrome, and dementia, PARO can mimic being alive by moving its head and legs and making sounds. PARO is able to respond to human behaviors, such as petting, holding, and talking, and can increase or decrease certain behaviors based on human responses using Artificial Intelligence (AI). PwD can pet PARO when it moves, and it will respond by moving more; PwD could also hit PARO if it makes a sound, which results in a decrease in the production of that sound. According to [13], PARO still suffers from the commonplace difficulties of human-robot interaction, i.e., the elderly can be either passive or confused with PARO and its behavior. PwD might not understand why it makes certain sounds and movements and decided not to interact with it. Therefore, care staff must mediate most interactions, as they require close guidance, leading to an increased workload for caretakers [13].

aibo is a robotic dog developed and manufactured by Sony [30]. aibo has a white plastic exterior with rounded corners and shapes that resemble a dog. It is designed to offer its users companionship through various interactions, such as petting, talking, and giving commands. aibo can learn from its owner using AI and can be taught various tricks and behaviors. Unlike many other robotic dogs designed for assisted therapy/activity, aibo, despite its dog-like exterior, can be clearly identified as a robot due to the lack of a fluffy exterior. aibo is programmed to behave like a highly intelligent dog that can “understand” its surroundings via vision, auditory, and haptic sensors. Supported by advanced hardware and AI, it constantly learns about the users, responds, and interacts with them via expressive dog-like facial expressions and flexible tricks and gestures. Moreover, it automatically explores the environments and learns from the users to form new skills. While dog robots, in general, have positive impacts on wandering and agitation, with patients often caring for them, talking to them, and picking them up, aibo’s unnatural appearance can make it difficult for patients to identify it as a dog [12]. Tamura et al. (2004) [12] also found that patients with a strong bond with dogs would only interact with aibo when prompted, and aibo with a fluffy dog-like exterior improved its acceptability and enhanced its positive effects.

The JoyforAll cats and dogs were designed by Ageless Innovation to be cheaper robotic therapy pets for the elderly [31]. Compared with advanced robotic pets like aibo, the JoyForAll cats are designed to look like cats with soft fur and feline vocalizations but with less complex functions. They can also respond to users’ motion with basic head movements. The JoyforAll pets provide companionship to the users and were found to be conversation starters [29]. Thunberg et al. (2020) [29] found that JoyForAll offered similar results to more expensive robotic companions like PARO and aibo. Additionally, the robotic cats seemed to contribute positively to the overall mood of the group and were sometimes regarded as family members. During a 9-month trial with JoyforAll cats, Thunberg et al. (2020) [29] observed more than a novelty effect, noting increased use within care departments over time.

In one study, researchers observed the effects of immediate and long-term (12 months) exposure to robotic animals on the QoL and well-being of PwD using a robotic dinosaur and a robotic cat. Researchers found through interviews and observations that robotic animals can provide affective presence, strong emotional attachment, and significantly enhance the immediate and long-term psycho-social health of individuals with middle- to late-stage dementia [32]. Although the research was limited by its duration and sample size, it indicates promising positive effects of robotic animals for PwD. However, despite the benefits, caretakers and care facilities face several barriers to adopting robotic animals [33]. Through semi-structured interviews with care staff, Koh et al. (2022) [33] found that high costs, organizational independence, insufficient evidence of long-term engagement, and the sustainability of interventions requiring additional manpower were significant determinants for care facilities.

2.2. Interactive Displays/Activities in Nursing Homes

Interactive displays are seen in places such as museums, arcades, fast food chains, and shopping malls, but they also find their place in more intimate settings such as nursing homes. In these care facilities, residents’ well-being and engagement can be enhanced through the use of public displays. The adoption of interactive displays in nursing homes signifies a shift towards utilizing technology to address the needs of PwD.

Switch2Move is a TV interface with a tangible controller designed to engage PwD in music-supported exercises [34]. Using a specialized controller, the interface allows PwD to control play, pause, and stop the music exercise video. Switch2Move reduces the stigma and feelings of embarrassment associated with operating digital devices by providing an easy-to-use alternative. The system was co-designed with PwD in multiple workshops and later tested in a one-week deployment. The researchers found that the dance exercises provided enjoyment and facilitated social activity, supporting couples affected by dementia. Overall, Switch2Move offers rich experiences through its user-friendly, tangible interface.

The Move and Paint System is an interactive drawing device designed for older adults [35]. Move and Paint allows users to draw on a digital white canvas using body recognition and a digital brush with a wide range of colors. When evaluating the system, Lee et al. (2020) [35] compared the unique behavioral characteristics of older adults toward interactive technologies with those of the local college student population. They found that older adults are more timid and tend to observe before using, indicating their concerns about self-efficacy. Additionally, older adults required printed instructions next to the system due to their limited experience with similar technologies. They also observed little creative and intentional use by PwD, suggesting that they remained in a learning mode and never fully transitioned out of it. Additionally, they exhibited a different nature of engagement, mostly focusing on the interaction itself rather than the content, affecting their overall engagement with the system [35].

“Active cues magic table”, also known in Dutch as “Tovertafel”, researched by Bruil et al., allows multiple users to interact with light projections [5]. These projections are designed to create different games that users can play with, e.g., swatting flies, catching butterflies, and touching flowers. They also designed and co-developed several experiences with PwD, including “Nostalgia Puzzle”, which allows PwD to complete puzzles featuring vintage objects and pictures, and “Record Player”, which plays songs from a collection of old music, encouraging PwD to sing along. The magic table has shown moderate improvements in negative affect, restless, tense behavior, and positive self-image [5].

OutLook is a gallery-like display set that allows shared images of real-time views and triggers additional communications through an interaction similar to sending a postcard [36]. OutLook was used in a field trial setting, with an introduction and conclusion sessions at the start and end, respectively. After interviewing 13 PwD, the researchers concluded that Outlook was less effective than expected. The overall user experience was positive for all PwD, but Kang et al. (2018) [37] found that the low levels of involvement between the sessions led to a misunderstanding in the intention and perception of Outlook. PwD did not remember how it functioned and did not often engage with it.

Reading-to-sharing is a tabletop display system aiming to enhance the social interaction between residents by augmenting their reading experience [15]. Reading-to-sharing comprises three components: ISticker, IStamp, and a digital display. The ISticker can be placed on printed media, such as newspapers, to select interesting content; the IStamp reads the content marked by IStickers and gives visual and auditory feedback to users via a digital display. Researchers found that reading-to-sharing succeeds in engaging users in viewing and sharing content and triggering social interactions. However, the PwD can often feel uncertain due to the system’s complexity.

Intermem offers interactive, joyful memories in immersive virtual 3D environments for PwD [38]. The researchers created an interactive memory designed to look like an old country house with either a cat or a dog as a pet avatar. They conducted two interactive sessions to compare the effectiveness of two control options. In the first session, they used Leap Motion [39], a motion-tracking device. In the second session, they used a Microsft Kinect and a Myoelectric bracelet. Using gesture-based commands, participants could use these control options to move around and interact with objects. Bejan et al. observed during an interaction that PwD displayed facial expressions of pleasure and interest, and participants made several statements that reminded them of their past.

Many interactive displays have been proven effective at engaging PwD with engaging and interesting interactions and positively influencing behavior and mood. Reading-to-sharing engages users and stimulates social interaction, and the magic table decreases restless and tense behavior. However, while the effects are generally positive, user acceptance and successful deployment in care practice can still prove challenging. With OutLook, PwD had low involvement, which resulted in misunderstanding about the functioning and decreased use; similarly, with reading-to-sharing, PwD felt uncertain due to the system’s complexity. Adopting these interactive displays can prove difficult; however, it is crucial for care staff to integrate the system’s daily activities and workflows [40].

2.3. User Behavior Models for Public Displays

When encountering interactive systems, users move through different stages of engagement. To describe these stages of engagement, researchers create frameworks and trajectories. Several frameworks exist in the Human-Computer Interaction domain, including the participant journey map and honeypot model, which can accurately describe people’s interaction with public interactive systems. It is important to highlight and compare these models, showcasing their benefits and downsides, to decide which model suits our research purposes and contexts.

The Honeypot Effect is a phenomenon observed in the context of public interactive systems, where the presence of active users engaging with a system attracts more bystanders and observers to also engage with it. Essentially, it describes how the interaction of initial users can serve as a catalyst, drawing in additional participants who might otherwise have remained passive [14]. The honeypot framework was created to model participants’ behavior and to define what factors can trigger audience engagement. The framework was based on observations and semi-structured interviews conducted over four consecutive days during the Melbourne Summer Festival, using a playful interactive installation called Encounters. In this framework, audiences can be in one of six different roles/phases: passer-by, bystander, audience member, participant, actor, and dropout. Using this model, Wouters et al. looked into how to get people to change from passers-by to interactive actors through the different defined phases with field experiments.

The spectator experience plays a crucial role in the design of interactive systems. While there are numerous approaches to designing various types of interactions, Reeves and colleagues identified four key design strategies: secretive, expressive, magical, and suspenseful [41]. In the secretive approach, both manipulations and effects are hidden, whereas in the expressive approach, both manipulations and effects are revealed. The magical approach reveals effects while keeping manipulations hidden, while the suspenseful approach hides effects but reveals manipulations when spectators have their turn. These strategies offer valuable insights for enhancing interactive systems and experiences, providing designers with tools to engage and captivate users in different design types.

TheAudience Funnel is a framework focusing on interaction with gesture-based public display systems [42]. It is based on an observation with Magical Mirrors, a set of four large public displays with gesture-based interaction. The framework defines different phases audiences take when passing an interactive public display. The steps of passing by a display, viewing and reacting, subtle interaction, direct interaction, multiple interactions, and follow-up actions were defined using quantitative data from 660 people. The framework tries to support a systematic investigation of public display systems.

The Participant Journey Map (PJM) [22] is a framework designed to define participation through interactive play in (semi-)public environments. The PJM was defined using observations of the naturalistic behavior of 687 visitors at a museum exhibit. The researchers used observational data to plot visualization using the PJM. Participants can be in either the onboarding or participation phases, each with its own state. The journey begins with the ’transit’ state, followed by the onboarding phase, which includes awareness, interest, and intention. Participants then transition to the participation phase, which comprises exploration, continuation, and completion. This framework represents participant behavior and can be utilized to identify moments of stagnation and progression in audience participation.

Focusing specifically on social play experiences, the work by De Kort and IJsselsteijn [43] emphasizes the environmental and social psychological aspects of physical spaces. For example, based on the way a space is structured, tables and seating configurations are arranged, and lines of sight and sound are supported, a space may be more or less conducive to social interactions. While their work is not focused on public displays as such, it does offer insight into the importance of the social affordances of the socio-physical space around a public display in order to be conducive to social interaction patterns.

In general, many different frameworks and models exist to define the interactive and audience experience with public displays. These models have different focuses and strengths. For instance, the PJM model defines a participant’s journey during an interaction, while the spectator experience focuses on designing for the audience. These models have been tested with various audiences, mostly in naturalistic settings, but little research has been conducted to apply them to PwD in care facilities. Due to increased mobility issues, unpredictability, and a wide variety of symptoms [44], PwD can exhibit behaviors that do not always align with existing models and frameworks.

3. System Design

Mindful Waters is a multi-user digital interactive system designed for PwD that is aimed to foster individual engagement as well as social interaction. The system operates mainly within the Unity game engine, using its broadly applicable 3D rendering software and coding libraries based on C#. Figure 1 shows a screenshot of the Mindful Waters aquarium, which is mainly composed of various fish, decorative features, and ambient sounds. Mindful Waters supports user interactions, including “playing with the fish” and “feeding the fish” as illustrated in Figure 2. The former relies on hand recognition and tracking algorithms [45] to calculate the hand coordinates in the aquarium environment in real-time, which becomes a target of fish movements. The latter incorporates a fish-feeding device containing an ESP32 microcontroller [46] with a push button, which activates the fish-feeding process. In Mindful Waters, we designed different types of fish and programmed various fish behaviors and interactions according to how they behave in real aquariums. The passive environment enhances the system’s realism and entertainment value by incorporating additional movements and sensory elements. It features decorative elements such as a chest and a scuba helmet, along with moving plants that create the illusion of oxygen bubbles. Furthermore, ambient sounds are integrated into the environment to further immerse users. The rest of this section explains the main features of Mindful Waters.

3.1. Hand Movement-Based Interaction

Mindful Waters allows the user(s) to interact with the aquarium through hand movements. We implemented such interactions based on CVZone, an open-source software package that identifies and recognizes objects and body parts [45]. Once Mindful Waters starts running, the connected webcam (either external or built-in) will be activated. The algorithm will then process the real-time feed from the webcam and look for human-hand objects in the video.

As shown in Figure 3, once the algorithm recognizes a hand, it identifies it as either the left or right hand and then creates a visual representation of the hand structures with dots and lines. This representation enables hand gesture/command recognition and can be utilized to develop more advanced hand movement-based interactions. In this work, we do not consider different hand gestures; instead, after the algorithm recognizes the human hand in any gesture, it always sees the hand as a single target and uses one coordinate (screen coordination system) to represent the hand. This design enables PwD with decreased fine motor skills to interact with the system. Indeed, in the experiments, we observed that several participants interacted with Mindful Waters with their fists. The code can recognize two hands at most, due to hardware limitations, and output their coordinates in real-time. The two users could interact collaboratively with Mindful Waters by their hand movements.

During the interaction, Unity receives a constant, real-time data stream of hand coordinates and translates them into a virtual hand in Mindful Waters. Every time the user moves the hand, the location of this virtual hand changes. The velocity of the virtual hand is measured constantly. If the velocity becomes bigger than zero, it will change the virtual hand into a target for the fish; if the velocity of the virtual hand stays zero for a certain amount of time, it stops being a target.

3.2. Feeding Bottle-Based Interaction

Mindful Waters allows users to feed the fish with a physical feeding bottle. The device, designed as an interactive fish-feeding mechanism, features a straightforward tactile button connected via wires to an ESP32 microcontroller. The ESP32, a low-cost, low-power system-on-chip with integrated Wi-Fi and dual-mode Bluetooth capabilities, sends a signal to Mindful Waters upon button press. All components are housed in a plastic tube, originally used as a fish food storage container.

When the button is pressed, it triggers a signal to the ESP32, which then relays the message to Mindful Waters. Mindful Waters constantly checks this message, based on which the particle effect is activated, i.e., fish feeding, from the top of the aquarium as shown in Figure 4. The feedings drive all fish targets to converge at the feeding location. Upon contact with the particles, they are destroyed, simulating the action of the fish eating them. If the particles are not destroyed by the fish, they will disappear after 10 s as they dissolve in the water.

3.3. Fish Appearance Design and Behaviour Modelling

In Mindful Waters, fish are objects described by 3D visualizations, controller scripts, animators, and Navmesh agents [47]. We introduced six types of fish: angelfish, clown loach, neon tetra, red-tailed black shark, rummy-nose tetra, and the Siamese algae eater. These fish and their 3D visualization in Mindful Waters can be observed in Figure 5. The designed 3D visualizations keep the salient features of each type of fish. Different fish can interact with each other and the environment; all fish interact with the fish feeding. For each fish, we call its interactive objects targets, which drive the fish’s behaviors. The controller scripts control the fish’s selection and orientation towards a target, while the Navmesh agent, a navigational component designed for moving objects in Unity, decides the fish’s movement path and speed.

Each fish has its invisible movement target. Fish always follow their targets, swimming as close to them as possible. Targets have two defined modes: movement and feeder mode. Movement mode is the default mode; the target remains in movement mode unless the feeder mode is activated. In movement mode, targets move from the current location to a randomized destination at randomized intervals. Specified boundaries restrict the target’s movement within certain areas using X, Y, and Z coordinates in the 3D simulation Mindful Waters system. Feeder mode is activated once the button on the fish feeder is pressed and the fish-feeding particles appear. In feeder mode, the target moves to the predefined food destination and stays there for a randomized interval before switching back to movement mode.

Moreover, some fish possess unique behaviors specific to their species. While mostly subtle, these behaviors have been designed to increase realism and provide users with various interactions to discover.

Hunting: Compared to other fish targets, the Red-Tailed Black Shark randomly follows other fish around. Red-Tailed Black Sharks are often seen as very territorial and will, therefore, follow and sometimes attack other fish. We designed a unique movement mode for the Black Shark when implementing this in Mindful Waters. Instead of moving toward randomized destinations, the Black Shark chases after other fish (targets) and switches its targets at randomized intervals.

Hiding: Algae eaters typically go to places with a lot of algae, like plants or rocks. In Mindful Waters, the algae eater mimics this behavior by moving between predefined destinations. Examples of these destinations include the plants and decorations. Therefore, when the algae eater moves between and behind these destinations, it seems like the fish is hiding.

Multi-Hand Interaction: The angelfish can additionally follow moving hands. If a moving hand is recognized by the software, it becomes a target for the angelfish, if the hand stops moving for 8 s, it stops becoming a target. This allows the user to interact with the digital angelfish using their hands.

Schooling: Fish in aquariums can often be seen schooling together, as observed with species like neon tetra or rummy-nose tetra. Similarly, in Mindful Waters, specific fish will school briefly when they swim less than 1 cm close to each other. Once activated, these fish will follow the same targets. After a randomized interval, they will revert to their original targets and resume their previous behavior. Additionally, users can guide fish near each other through hand interactions or by feeding them, which similarly triggers the schooling behavior.

3.4. Aquarium Environment

We added decorations, plants, bubbles, and ambient sounds to Mindful Waters to create a more realistic and fun aquarium environment.

Decorations: Decorations are a common addition to aquariums. People often have statues, plants, or wood in their aquariums, which can be used as entertainment for the fish or for purely decorative purposes. As seen in Figure 6, we created a diver helmet, treasure chest, and ${\mathrm{CO}}_2$ tube. Additionally, reflective mirrors on the left and right sides of the simulation mimic the light reflection in a real aquarium.

Plants: The water plants in Mindful Waters are programmed to move and sway in the water. Moreover, the plants produce oxygen bubbles randomly, like real aquarium plants. These features are introduced to add vitality and realism to the simulation.

Sound: Sound plays an important role in digital experiences, so Mindful Waters incorporates a constant water-running sound to enhance immersion. In fact, sound is a key component of the Honeypot effect, where participants are prompted to engage due to the involvement of others. We implement this effect within the hand movement-based interaction when the bubbling sound is played upon the user’s interaction with the fish. This sound is designed to pique the interest of other participants and encourage their engagement with the simulation.

4. Materials and Methods

In this study, we conducted a series of interactive field study sessions with PwD. Our primary objectives were to evaluate Mindful Waters regarding user experience, the validity of the PJM model, and to understand user motivation. To achieve this, we established a recording set-up along with a questionnaire to gather extensive data while keeping the sessions manageable for the PwD participants. The data analysis primarily focuses on qualitative data supported by quantitative data.

4.1. Participants

Participants were sourced from two distinct settings. Group A comprises visitors with dementia who gather at the local community center for various weekly activities. The community center organizes these activities every Wednesday for PwD and their partners. Anyone with dementia can come to the community center and take part in any of the activities of their own volition. In Group A, we had eight participants, four females and four males. Two of them were caregivers; five of the six PwD participants could walk without assistance.

Group B consists of residents of an independent living care facility that is part of a larger care company. The residents reside in rental apartments with access to diverse communal areas. Several weekly activities are organized for the residents, such as music, painting, or exercise. Several care staff members work at the facility and help the residents daily. Concerning the experiment, we did not set specific research criteria for participants. Three participants with dementia, one female and two males, participated in the experiments. Compared with participants in Group A, those in Group B are in the later stages of dementia. Only one of them can walk without external assistance.

Participants could vary in age, background, progression, and stage of dementia. We had participants in very early stages and also those in more extreme stages of dementia. Additionally, some participants were still very healthy, while others were hard of hearing or wheelchair-bound. This choice was made to test the device with a larger variety of participants; therefore, the only stipulation for participation was a dementia diagnosis. Prior to the study, the study proposal was reviewed and approved by the institutional ethical review board. In addition, participants and (in)formal carers were always asked for informed consent, and participation was—also by the very nature of the study—entirely voluntary.

4.2. Experimental Set-Up

Two field evaluations were conducted in 45-min interactive group sessions. The introduced stimuli during the interactive session were a screen that showed the Mindful Waters user interface, the fish-feeding device, and 3 cameras with tripods. An instructional page (Figure 7) was placed near the screen to provide detailed instructions to the participants [35]. In the instructional page, the participants were suggested to stand within 2 m of the screen, move their hands to interact, interact with a maximum of 2 simultaneous users, and feed fish by grabbing the fish can and pressing the button. All participants could see and read the instructional page during the experiments with the researchers nearby, ready to answer any questions the potential participants might have.

During the experiment, we set up the experiments as illustrated in Figure 8: Mindful Waters was running locally on a computer connected to a TV screen during the experiment. The large TV screen made it easier for all the participants to observe and interact with the system. The webcam for hand recognition and tracking was connected to the computer and placed on top of the television; by placing it on top of the TV, it could not be obstructed and could clearly see and recognize users. Cameras A and B were placed at the back of the room, located on two different sides of the room, so that their recordings could cover the whole room and, hence, record the participants’ movement/participation trajectories. Camera C was placed near the system, facing the participants to record their interactions with Mindful Waters closely. Figure 9 shows our set-up of Mindful Waters in the community rooms. Similar to Figure 8, we positioned a screen with a webcam on top in the center of the room, arranging chairs in an arc around the screen for PwD. In front of the screen, we placed the instructional image and fish-feeding bottle on a table within the engagement zone. The three cameras were positioned around the participants in the corners of the room to capture different user interactions: one near the screen, the second viewing the participants, and the third behind the participants. Finally, the laptop running Mindful Waters was placed behind the screen and hidden from the participants.

4.3. Measurements

Observation: One or two researchers were always present for observations during the interactive sessions. The first researcher focused on the participants’ user engagement by looking at the social interaction and listening to the conversational topics. The second researcher focused on the interaction of the users with the Mindful Waters system. Otherwise, the single researcher handled both.

Interview: The participants were asked a set of questions in an interview setting after the interactive session. The interview was conducted using a semi-structured method and consisted of the questionnaire below and additional unscripted questions based on their answers to the questionnaire. Exemplary questions included “What did you think about the design of Mindful Waters?”, “Did you find Mindful Waters easy to use?”, “Did you find Mindful Waters too complex?”. These questions were asked to better understand the system’s usability and attractiveness and to recognize improvable features. Participants and caretakers were all invited to participate in the interview to gather data from different perspectives. All questions were asked and answered in Dutch.

Questionnaire: The questionnaire’s function was to record a wide range of additional information that is difficult to measure during an interactive session. We used Attrak Diff Short [48], a 7-point Likert scale with 11 questions measuring the quality and attractiveness of an interactive product. Attrak Diff Short consists of word pairs with opposite meanings, like “ugly-attractive” and “cheap-premium”, “complicated-simple”, etc. The participants were asked to decide on their preferred words to describe Mindful Waters. Due to the specialty of the target users, the researchers read out the questions for the PwD and filled in the questionnaires based on their answers. All questionnaire questions were asked in Dutch.

Recordings: Audio and video recordings were made during the interactive sessions. They were used to record the participants’ movements, engagement patterns, and the participants’ specific behaviors during their interactive sessions, which might have been missed during the observation. The review was also used to look at user engagement with the system, the closeness between the users during the interaction, and the system’s usability.

4.4. Experimental Procedure

Potential participants were given an introduction to the research and an explanation of the device and its functionalities. After signing the consent forms, a short demonstration was given, demonstrating the different features and preparing the participants for interaction with the system. The participants were then suggested to read the instructional page (Figure 7) and interact openly with the system for 30 min, where the researchers made observations. Upon completion, participants took part in an interview. After the experiment, an informational pamphlet was provided. This pamphlet explains the goal, method, and researchers involved in the experiment and provides enough information for participants to review the experiment later. It was left with caregivers in both research environments.

Researchers observed participants’ behaviors during the experiments and noted their interactions with the system and others. Cameras A and B recorded the general movements of all potential participants; camera C focused on user interaction and recorded the video and audio of those who interacted with Mindful Waters.

4.5. Data Analysis

The researchers imported, translated, and transcribed each of the recorded interactive sessions. The semi-structured interviews were not audio recorded; a single researcher collected and translated the responses. The questionnaire results were digitized for further analysis. The Attrak Diff Short data were averaged among all participants. The video recording was analyzed by comparing it to the PJM model. The researcher noted the times of the phase change, which can be used to determine the duration of each phase and the total interaction length.

5. Results

Two interactive sessions were successfully performed at the care facilities. Although we expected all participants to fill in the questionnaire, only 4 out of 11 participants could fully complete it. Partially, this was due to their disinterest in the questionnaires, while some participants did not comprehend the questions as a consequence of their advanced stage of dementia. We recorded all the interactions and conversations during the experiment with permission from the care facility, caretakers, and the participants.

5.1. Participation Motives

At the care facility, participants initially hesitated to interact with Mindful Waters. The video recording showed that all participants observed the system but did not want to interact initially. The first participant had to be nudged by one of the researchers to interact. After the first interaction, participants were more likely to want to interact but sometimes required help from a caretaker. Participants often started observing, then walked towards the system with help from a caretaker and observed from nearby. Participants then tried to interact while talking to the researchers about their experiences with aquariums or fishing. Only one participant chose not to interact with Mindful Waters; all others either fed the fish, interacted using their hands, or both.

Participants tried motivating other participants to partake in their interaction. One participant stood up to observe and asked, “Can I convince you as well?” to another participant. The second participant decided to decline the offer and stay seated. These participants were verbally interested in motivating other participants but did not always have success.

Participants also seemed interested in observing the system itself and were focused more on its visual aspect. One participant said, “I would like to look at your masterpiece. Have the fish already been fed?” Similarly, another participant said, “I came to watch if the food keeps lying on the ground. No, I guess it disappears”. Thus, participants showed interest in the aquarium features and interactions that Mindful Waters offers.

Not all participants were interested in interacting. One female participant in an advanced stage of dementia showed fear when asked to interact with a caretaker. She did not want to reach out or feed the fish but was happy to observe. She stated to the caretaker, “You can do it, but I don’t want to.”

Due to the positioning of the webcam, the camera did not always correctly recognize participants’ hands. Nonetheless, the participants often thought they were interacting and followed the fish with their hands, thinking that the fish was following them. The video recording shows that participants sometimes used the feeding bottle to guide the fish around, thinking they would follow the food.

A few participants sometimes decided not to feed the fish, fearing it might be bad for them. The participants made statements like “I think I gave them too much food” and “But would the fish even like it?” Since the participants believed that the fish could think and have opinions, they were inhibited and refrained from feeding the fish.

Based on both interactive sessions, ten out of eleven participants decided to interact with Mindful Waters. We calculated an average interactive time of 3 min and 43 s for the first interactive session. The times for the second interactive session were not measured, mainly due to difficulty with mobility. Participants in the second interactive session required either a wheelchair or walker to stand and walk around; standing to interact exhausted the participants significantly, thereby prolonging interaction.

5.2. Participant Journey

The PJM model defines the participation journey of an interactive participant into several states, from “Awareness”, to “Interest”, “Intention”, “Exploration”, “Continuation”, and “Finishing”. Using the framework, we can identify the stagnation and progression of PwD participation and understand their participation motivation, engagement behaviors and patterns, and the usability of Mindful Waters.

Figure 10 aggregates the representation of all PwD participants’ engagement journeys using the PJM model. We decided to define “Awareness” as point zero. Due to our introduction to the interactive session, which is needed to ease PwD into the experiment and help them understand the goal and functionality of the system, all participants were already aware of the system and its functionality. Similarly, “Finishing” was considered the interaction’s endpoint. The numbers in the circles represent the frequency of participants transitioning to those states, with the arrows indicating the paths that participants took. We also made it clear that they could decide if they want to interact with Mindful Waters or not, and they could decide to leave anytime during the interaction. We observed that five out of ten participants followed the same order of the original PJM states, while others skipped a few states or revisited previous states. Figure 11 shows an example of one participant who moved from “Awareness” to “Interest”, “Intention”, “Exploration”, and “Continuation” but then moved back to “Interest”, and from “Interest” directly to “Finishing”. Eventually, as indicated in Figure 10, the eight PwD participants showed interest in Mindful Waters eleven times, and the “Interest” transited into ten times of “Intention”, which led to seven “Exploration” and five “Continuation”. These numbers suggest that the PwD participants were highly interested in interacting with Mindful Waters, which resulted in the continuation of interactions.

The PJM model focuses on the transitions of participant journeys across different states. Nevertheless, this model does not show more details about the interactions. To tackle this, we noted each participant’s time in every state (see

Table 1. Participant journey and state duration.

Participant	Awareness	Interest	Intention	Exploration	Continuation	Finishing
P1	x	0	0	130	0	x
P2	x	10	10	60	45	x
P3	x	20	105	0	0	x
P4	x	15	30	35	155	x
P5_1	x	20 & 115	10	40	20	x
P5_2	x	10	15 & 45	10	0	x
P5_3	x	10	430	0	0	x
P6	x	10	10	40	65	x
P7	x	50 & 50	45	15	100	x
P8	x	20	485	0	0	x
Average	x	30	119	47	77	x

). The table shows that the shortest phase was “Interest”, when a participant showed interest in Mindful Waters by observing or commenting on others’ interactions. After the first interaction, P5 entered the state “Interest” again and stayed there for 115 s. On average, participants spent 30 s in this state. We can also observe that some participants took longer (e.g., P5 took 430 s and P8 took 485 s) in “Intention” since, despite their intention to start the interaction, they needed to wait for the chance to interact with Mindful Waters. These two participants also spent much time talking during this time. Additionally, we observed that, on average, participants spent 47 and 77 s in “Exploration” and “Continuation”, respectively. One participant, P5, interacted with Mindful Waters three times.

5.3. Social Interactions

Participants at the care facility had positive experiences with Mindful Waters in general. When asked about their experience, eight out of ten participants tended to have positive opinions. Two participants stated that their interaction was “very fun” and “there is a lot of pretty fish in the aquarium”. These responses suggest that the interactive session with Mindful Waters was enjoyable and that the visuals were attractive. Two out of ten participants had less positive experiences; the first participant rejected Mindful Waters entirety and stated, “Well, sorry, but no. Only if there would be more fish”.

The fish and aquariums became a big conversational topic; participants talked about their experiences with aquariums, which fish they owned, how big their aquarium was, and about their experiences with sports fishing. Most stories reminisced about the past, with participants stating, “I also had Angelfish, and they always stored their eggs in their mouths. But another fish always managed to eat the eggs”. “We also had fish in the past, like this one” and “Yes, at our house, we always overfed the fish”. The stories had positive connotations, and participants were excited to share them with each other and with the researchers.

Similarly, some participants started conversing about their experience with fishing. One participant stated, “With fish, I do different stuff; I always catch them on my free days and release them again”. Another participant said, “I always catch big ones; I once caught one that was 520 kilos. Those were big carpers a meter long”. The stories related to fishing were relatively more recent than those about aquariums. This is likely due to participants still being actively engaged in fishing and continuing to have recent experiences.

Finally, some participants praised the system concept and started showing an interest in the system. One participant said he would like an aquarium in his house, but there is not enough space anymore; the participant stated, “It would be very handy because our house is quite small, and this you can just put on the wall”. This response suggests that this participant would see Mindful Waters as a possible solution for not having enough space in his house. Similar opinions were observed from other participants, who showed identical interest in having it in their house.

5.4. User Experience

The results from the Attrak Diff Short support the statements made by the PwD participants about Mindful Waters.

Table 2. Attrak Diff Short results.

Category	P1	P2	P3	P4	Average
Simple	3	5	6	1	3.75
Practical	7	7	7	7	7
Clear	6	6	6	1	4.75
Interesting	3	5	5	7	5
Attractive	6	6	7	7	6.50
Good	5	6	6	7	6
Motivating	4	5	5	7	5.25
Cheap	1	2	2	1	1.50
Unpredictable	2	2	2	1	1.75
Unimaginative	1	1	2	1	1.25
Repulsive	1	2	2	1	1.50

explains details of the Attrak Diff Short answers from four participants with the average of each category at the end of the table. Most averages are near the extremes of the Likert scale. For example, participants found Mindful Waters “practical” to use, very “clear”, “attractive”, “good”. They also rated 1 or 2 points to “cheap”, “unpredictable”, “unimaginative”, and “repulsive”. Two of the four participants rated Mindful Waters as simple to use, while the other two did not agree. Moreover, most participants had similar opinions except for P4, who gave 1 point to “simple” and “clear” and nevertheless gave the highest score (7 points) to “interesting”. This implies that although P4 might have some difficulties starting the interaction, he/she did find Mindful Waters attractive and interesting. Additionally, it is expected that participants possess various user experiences, as they could be in different types/stages of dementia and hence maintain varying cognitive functioning.

6. Discussion

This paper presents Mindful Waters, an interactive, multi-modal digital aquarium for people with dementia (PwD). We described the rationale and technical characteristics of the system and positioned it within the design space of robotic companion animals and interactive public displays, both aimed at PwD.

A pilot study was executed primarily aimed at evaluating the effectiveness and functionalities of a prototype version of Mindful Waters. The focus was on understanding user motivation, participant behavior, and overall user experience during supervised field trials. Key insights from the study suggest that Mindful Waters can positively influence social interaction (i.e., trigger joyful conversations and collaborative interactions) and provide a meaningful and engaging experience (i.e., attract repeated, prolonged interactions) for PwD. However, certain usability challenges were identified, particularly for individuals with advanced stages of dementia, indicating areas for further development, as well as long-term placement strategies.

Firstly, the expectation that users would find it challenging to interact due to unfamiliarity seems incorrect. Users instead seemed enticed to interact after an initial reluctance period. Most participants were happy to try an interaction with Mindful Waters while simultaneously talking to the researchers. The participants were often curious and interested in Mindful Waters, which helped motivate them. Conversations between participants were mainly based on past experiences with aquariums or fishing, which helped increase participants’ interests. Participants also tried motivating other participants to partake in the interactions. Motivating was mainly performed through verbal encouragement, such as asking to join the interaction. While not always successful, it does show that participants are interested in trying out the system together. Research by Reeves and colleagues [41] on designing for spectator experiences also emphasizes the fact that active participants can take on the role of orchestrators, enhancing the spectator experience around public displays. Their involvement ensures that the display not only attracts attention but also engages and retains spectators, turning passive observers into active participants. While most participants decided to interact, some showed disinterest or fear and observed the system from a distance instead. According to these PwD, the disinterest mainly stemmed from features lacking in the aquarium, such as specific fish and certain amounts of fish. Moreover, the fear stemmed from their lack of understanding of the system, resulting in hesitation, which is mitigated by observing instead of interacting. In short, users were often motivated by curiosity and interest due to their previous experiences of owning or maintaining an aquarium. However, a lack of experience and advanced dementia became a debilitating factor in the interaction, especially when considering the target audience.

Secondly, while useful, engagement models like the PJM model appear to have slight shortcomings when applied to the target group during short interactive sessions. The PJM model is designed around participants being in transit; however, the required introduction to Mindful Waters meant that users were seated and became aware of the system in a stationary manner. Moreover, we observed that several participants moved from “Intention” to “Finishing”, indicating that while they were interested in interacting, they decided not to for various reasons. We believe this might be due to the aspects of the environment, such as the fact that most people were already seated and that the space to move around was confined, which limited users’ freedom of movement. This aligns with the insights of De Kort and IJsselsteijn [43], who emphasize the importance of social affordances of physical space and physical design elements (e.g., placement of tables and chairs) to enable social interaction and engagement. Additionally, external distractions were a possible explanation for skipping phases. When PwD were engaged and interacting with Mindful Waters, other PwD often approached and talked to them, interrupting the interactions and their intention to interact. After these conversations, the PwD frequently ended the interaction and immediately transitioned to “finishing”. Despite these challenges, most participants felt confident with the interaction, and we did not observe concerns about self-efficacy as indicated in other interactive activities/displays designed for PwD [15]. Some participants also moved back to previous phases; for example, participants would transition from interacting to observing. The current model, however, does not clearly indicate the order in which participants take certain steps. In summary, while the behavior of five out of eleven participants followed the logic of the PJM model, an adapted model could be more suitable for defining the engagement behavior of PwD in smaller nursing home settings.

Thirdly, it was anticipated that Mindful Waters’s usability might have some limitations due to missing functionalities or challenges in interaction. However, it was expected that PwD would still find the system interesting and novel. Based on the video recordings and observations, similar results can be determined. PwD found the experience positive and complimented the functionality of the system. No statements were made about difficulty with interacting or missing functionalities, as the participants never noticed any possible shortcomings. Participants often thought the fish were following their hands, but this was not always the case. Similarly, the participant did not want to feed the fish, thinking the fish would not like it. This suggests that participants are likely to interpret the fish’s behavior and attribute meaning to it. These results seem to indicate that participants did not always adequately recognize the role of digital technology in generating the aquarium experience. This is known as the ’perceptual illusion of non-mediation’ [49], and has also been described by Tummers et al. [32] in the context of robotic companion animals. While most human beings tend to engage with digital social entities (e.g., chatbots, robots, and avatars) as if they possess some level of intentionality or sentience (see [50]), this behavior may be more pronounced in PwD whose cognitive limitations will also likely affect the accuracy and sophistication of their mental models of digital technologies. Additionally, Mindful Waters became an unexpectedly effective conversational trigger for the participants, with many stories being told about past aquariums and fishing trips. Participants started sharing their stories with each other, with the researchers showing enthusiasm and clearly reminiscing about their past. Both participants and informal carers also indicated that Mindful Waters, compared to normal aquariums, solves the problem of cost and space. Several residents could not keep the aquarium due to space constraints and mentioned that changing the water and maintaining the aquarium is a lot of effort and expensive. We, therefore, find that according to the interactions of PwD with Mindful Waters, it can be seen as a cheap and easy alternative to an aquarium. Based on these results, we find that Mindful Waters is seen and can be used as a pleasant experience in an interactive session.

7. Limitations and Future Works

While this pilot study demonstrated the potential benefits of “Mindful Waters” for engaging people with dementia (PwD), several limitations were identified. The short duration of the interactive sessions proved sub-optimal for participants with advanced dementia, who often required more time to understand and interact with the system. Additionally, the small sample size and limited number of interactive sessions may affect the generalizability of the results. Future research should include longer-term placements and a larger, more diverse participant pool to obtain more representative data.

Moreover, the study highlighted the necessity for adapting the interaction design and setup for the target users. During the interactive sessions, participants with advanced dementia faced challenges due to mobility and comprehension issues. Enhancing the physical design of interactive elements, such as a stationary fish feeder, improving the realism of fish models, and expanding the variety of interactions could make “Mindful Waters” more accessible and engaging for a broader range of PwD. Also, in this context, more explicit considerations need to be given to the social affordances of the physical positioning of the Mindful Waters display, as well as the structuring of the larger socio-physical context around the display. These non-digital elements are crucial to the accessibility and sociability of the design.

Despite these limitations, the feedback from participants and caretakers was largely positive, with many noting the system’s potential to substitute traditional aquariums, which are often costly and space-consuming. The positive reception suggests that “Mindful Waters” can offer a valuable, low-maintenance alternative that fosters social interaction and mental stimulation.

Looking ahead, the next steps involve refining the system based on user feedback and exploring opportunities for long-term integration into care facilities. By continuing to develop and optimize “Mindful Waters”, we aim to enhance the quality of life for PwD, promoting greater social connectedness and well-being. This pilot study marks a promising beginning, setting the stage for future innovations in digital therapeutic environments for dementia care.

8. Conclusions

This paper presents Mindful Waters, an easy-to-use interactive display with promising results in offering PwD engaging activities and promoting their social interactions. The pilot study, involving eleven participants (nine PwD and two caregivers) at a care facility, has provided valuable insights into the motivation, engagement patterns, and user experiences of PwD with Mindful Waters, as well as into the design and setting of interactive displays for PwD in general.

During the experiments, ten out of eleven participants interacted with Mindful Waters, who mostly started with observations and then were encouraged to join by researchers, caregivers, or other active participants. The interactions with Mindful Waters triggered joyful conversations around aquariums, fish, and fishing. Eventually, most participants praised the design of Mindful Waters as practical, clear, interesting, and attractive, reinforcing the positive impact of the interactive display. However, it was also noted that a few participants were reluctant to interact with Mindful Waters due to the lack of desired features or had difficulties understanding the system due to their cognitive declines. Moreover, the interactions and engagement patterns were possibly affected by the social affordances of the physical space.

Overall, the pilot study proved Mindful Waters’ potential as an effective trigger for meaningful engagement and social interactions for PwD and provided essential information for improvements. Future long-term studies are necessary to understand the effects, feasibility, and usability of Mindful Waters in care homes.