Development of Hand–Eye Coordination Analysis System Using Mixed-Reality Technology ^†

Abstract

As Taiwan is about to enter a super-elderly society (the elderly over 65 years old account for more than 20% of the total population), how to delay disability and dementia in the elderly has become an important issue. Increasing the use of toys and games for the elderly delays their disability. Therefore, we developed three 3D interactive games for the elderly using mixed-reality (MR) technology. The three games were designed for hand-eye coordination, including digital cognition, object shape recognition, and color recognition ability. The reaction time was recorded to analyze the elderly’s reaction ability when playing the games. The questionnaire survey results showed that more than 83% of the elderly were satisfied with the use of MR equipment. More than 87% of the elderly accepted the interactive mechanism of MR while more than 80% felt that the three games improved their reaction ability.

1. Introduction

According to the Taiwan National Development Council, Taiwan entered an elderly society in 2018 and is expected to enter a super-elderly society in 2025. In other words, one in five people will be over 65 years old in Taiwan soon [1,2,3]. The advent of the super-elderly society denotes the following problems:

• Increasing life expectancy increases the cost of living after retirement.
• According to the statistics of the Ministry of Health and Welfare, the average annual medical expenses for seniors over the age of 60 are about NT$97,000 (gradually increasing with age).
• The burden on young and middle-aged people is increasing.
• The demand for long-term care (disabled) in Taiwan continues to increase. The allocation of medical resources and the demand for long-term care manpower have exploded.

The elderly over 65 years old usually need rehabilitation with assistive devices. However, the pain and medical resources spent are large. By playing with toys, the prefrontal lobe and central nervous system of the brain improve to have good working situations. Then, the body and mind operate properly, delay the deterioration of body functions, and avoid Alzheimer’s disease. This practice has been used in Nordic countries for many years. It helps the elderly maintain their physical and mental health and keep a good quality of life after retirement.

In this study, we developed a diverse interaction method for the elderly based on the characteristics of play using mixed reality (MR) to reduce the risk of dementia and disability.

2. Related Works

Brill et al. [4] stated that cognitive training based on serious games effectively reduces dementia in the elderly and confirmed the efficacy and feasibility of computerized cognitive training (CCT). In addition, he conducted in-depth research on the extent to which the elderly are affected by dementia in three aspects: episodic memory, semantic memory, and spatial abilities.

Virtual reality (VR) is an immersive and interactive human–machine interface. VR improves cognitive function. Its flexibility helps in personalized and safe relevant training [5]. Its dynamic and real-time feedback provides a high level of labor intensity for elderly caregivers. In addition, high-resolution data generated through VR is used to track changes in the cognitive data of the elderly. Pratt et al. [6] introduced Hololens into computed tomography (CT) in 2018. Augmented reality (AR) assists physicians in accurately identifying, dissecting, and performing vascular pedicle flaps during reconstructive surgery. The approach has two phases. First, Pratt et al. used preoperative computed tomography (CT) angiography to distinguish bone, blood vessels, skin, soft tissue structures, and associated vascular perforations and constructed the required 3D image information. This 3D information was converted into polygonal 3D information and imported into Hololens. In this study, Hololens presented the 3D model information in three dimensions, as shown in Figure 1.

3. Game Development

To delay dementia in the elderly [7], we developed a “Hand–Eye Coordination Online Analysis System Based on MR3D Elderly Toy Games”. It employs the Meta Quest headset as the main execution platform and Unity 3D as the main development environment, with OpenXR for game system design.

3.1. VR Headset

The Meta Quest [8] is a set of headsets that combine ergonomics with a sleek design (Figure 2). It provides a more comfortable wearing experience than previous VR/AR headset systems. Therefore, it is appropriate for long-term wear. Meta Quest 3 features stereoscopic MR penetration and allows users to work, create, and collaborate in the virtual world while keeping an eye on the real world. In other words, the MR-class headset combines VR and AR.

3.2. Cross-Platform Tools—Unity

Unity is a suite of 3D cross-platform development tools [9]. Its cross-platform range includes Windows, Android, iOS, Windows Phone, Gear VR, PS4, XBOX, Wii, and more (Figure 3). Developers only need to develop the system once and can release it into a variety of different execution environments according to user needs. This shortens the development time for developers in different execution environments. Unity3D is a development software designed to meet the needs of 3D digital content. Most developers use Unity for 3D system development. Its development environment includes an integrated editor, asset management, scene construction, real-time execution, system program script editor, and network connection, allowing developers to complete the construction of 3D interactive systems in the shortest possible time.

3.3. OpenXR

OpenXR [10] emerged in the development of standardized VR and AR applications. It makes it easy for users and developers to interact with each other. OpenXR is supported by most of the major VR hardware, platforms, and game engines, including AR, of course. Unity has been a long-standing public supporter of OpenXR. OpenXR has become an industry standard. It allows users to develop VR applications that easily support a wide range of headsets, as shown in Figure 4.

3.4. Game Design

In terms of game system design, we integrated MR tracking, color recognition, number ranking, shape pairing, time recording, and time data transmission (Figure 5).

• MR tracking: The main function is to track movement changes in the elderly person’s body.
• Color Recognition: This module analyzes the color recognition ability of the elderly person.
• Number Ranking: This module analyzes the elderly person’s ability to recognize the logical response to numbers.
• Shape Pairing: The main function is to analyze the elderly person’s ability to recognize the shape of objects.
• Time recording: This module records time data when every operation action is made by the elderly person.
• Time data transmission: This module transmits time data.

4. System Implementation

We implemented three games, namely “Color Recognition”, “Number Ranking”, and “Shape Pairing”. To allow the elderly to choose the games, a game menu was designed in this study. From left to right, the menu included the icons of three games: “Color Recognition”, “Number Ranking”, and “Shape Pairing”. The elderly clicked on the icon on Meta Quest to start the game. The design screen is shown in Figure 6, and the menu program fragment is shown in Figure 7.

4.1. Color Recognition

In the “Color Recognition” game, three color plates were designed for the difficulty: 3 × 3, 4 × 4, and 5 × 5. This design allows the elderly to interact with the game at different levels of difficulty, from easy to difficult. The elderly can choose a different color button on the color plate. The screen design is shown in Figure 8. It is easy to remember the same color plate configuration and configuration content after using it. This directly or indirectly affected the results of the analysis of the data. Therefore, the conversion of the color plate was designed. Once the user started the game, the color plate was randomly configured. When the elderly played the game, the system automatically determined whether the color was correctly chosen.

4.2. Number Ranking

In this game, three levels of number plates were designed in “Number Ranking”: 3 × 3, 4 × 4, and 5 × 5. This allowed the elderly to perform the game operation in order from easy to difficult. In this game, the elderly clicked on the number plate on each level from smallest to largest until all the numbers in the level were correctly selected and then moved on to the next level (Figure 9). If the content of the “Number Ranking” was fixed every time, the elderly person would be likely to memorize the configuration content of the number plate. Therefore, the number plate was randomly changed in this study to make the game enjoyable and the recorded data credible.

4.3. Shape Pairing

The “Shape Pairing” game was designed as 3 × 3, 4 × 4, and 5 × 5 levels on a shape-recognition board. The elderly played the game from easy to difficult levels. The elderly selected the specified geometry on the shape-recognition board until all the geometry in the level was correctly selected before moving on to the next level. The screen design is shown in Figure 10. If the content of the layout of the shape recognition board was the same, it would be easy for the elderly to remember the configuration of the discernment disk. As a result, the results of the analysis of reaction time data would be affected. Therefore, the board changed randomly.

5. Conclusions

With the rapid development of 3D and information technology, the application of 3D MR is becoming popular. As Taiwan enters a super-elderly society, delaying the risk of dementia in the elderly is important. According to numerous studies, increasing the use of toys and games for the elderly is critical. Therefore, we developed three games for the elderly using MR: “Color Recognition”, “Number Ranking”, and “Shape Pairing”. All three games were developed using Unity and OpenXR to interact with the MR interactive device (Meta Quest system). When the elderly played the game, the reaction time of their operation was recorded, and the elderly people’s reaction ability was analyzed. After the operation of the system, the elderly filled in the questionnaire. In total, 83% of the elderly were satisfied with the MR equipment, while 87% accepted the interaction mechanism of MR. More than 80% of the elderly believed that playing these three games multiple times effectively improved their reflexes. On the contrary, several users reported 3D dizziness, which necessitates the improvement of the games.