Implementing Finger Movement Measure System with Music-Gamification Elements ^†

Abstract

Dexterity of the fingers is crucial in physical function, as it directly impacts daily activities and is closely connected to cognitive function. The production of brain-derived neurotrophic factor (BDNF) is related to the fingertips in motion. In previous research, we developed a finger motion measurement system for the elderly by integrating image recognition technology with a touch panel. However, despite the system’s ability to capture fine-grained coordinate changes at the moment when fingers touch the panel, the experiment was unengaging for participants. Therefore, we improved the system for measuring finger motion to be less exhausting and more enjoyable. We incorporated music and gamification elements at the moments of finger touch. We obtained a selection of representative rhythm tracks and implemented animated materials in gamification. The participants’ fatigue and enjoyment were measured based on “responsiveness” and “focus” using a quantitative evaluation method.

1. Introduction

In the aging society, assessing cognitive function and finger dexterity in the elderly is critical. Finger movement serves as a key indicator of neural health and cognitive performance, but traditional methods, such as the pegboard and tapping tests, often suffer from monotony and lack of interactivity, leading to participant fatigue and data accuracy reduction. While recent systems incorporate image recognition and touch interfaces and improve their precision, they still lack personalized designs and fail to sustain user engagement.

Therefore, we developed an innovative system that integrates multi-sensory feedback, gamified design, and generative artificial intelligence (AI)-driven music recommendations. Self-selected background music significantly reduces mind-wandering and increases task-focus states, highlighting its potential for creating engaging test environments [1]. Integrating visual feedback and rhythmic music enhances the user’s focus and dynamic task adjustments and ensures good data reliability, personalized recommendations, and user experience. The system addresses the limitations of traditional approaches. Its objectives were to improve user engagement, reduce fatigue, and enhance the accuracy of data collection. The developed system alleviates user fatigue, enhances testing engagement, and ensures scientific rigor in data collection. With its modular and adaptive design, the system can be used as a next-generation cognitive assessment tool for diverse user groups.

2. Related Works

The flexibility of the fingers is a crucial physical function that directly impacts daily activities and is closely interconnected with cognitive functions. This connection partly stems from the release of brain-derived neurotrophic factor (BDNF) during finger movements, which promotes brain health and functionality [2]. Furthermore, traditional methods for measuring finger dexterity are used to develop a system that combines image recognition with touchscreen input [3,4].

2.1. Technical Challenges

To address the issues of participant fatigue and a lack of engagement in existing systems, we integrated musical rhythm corresponding to finger touch to increase participant enjoyment and focus (

Table 1. Example of local music tracks.

Category	Music	Composer/Performer
Classical music	Air on the G String Moonlight Sonata—1st Movement Träumerei/Dreaming	J.S. Bach Ludwig van Beethoven Robert Schumann
Light jazz	Waltz for Debby The Girl from Ipanema Take Five Autumn Leaves	Bill Evans Stan Getz Dave Brubeck Chet Baker
New age music	River Flows in You Nuvole Bianche Silk Road Longing Love	Yiruma Ludovico Einaudi Kitaro George Winston
Game OSTs	Aerith’s Theme Breath of the Wild (Main Theme) Nascence	Nobuo Uematsu (Final Fantasy Series) Koji Kondo (The Legend of Zelda Series) Austin Wintory (Journey)
Traditional music	High Mountain and Flowing Water (Guzheng) Sakura (Koto) Greensleeves (Celtic Harp) Raga Hamsadhvani (Indian Classical)	Traditional Chinese Traditional Japanese Traditional Celtic Traditional Indian

). Different types of music significantly influence the information processing speed and memory task performance of older adults [5]. Gamification was also introduced, as it has been introduced to health-related fields. Deterring et al. [6] introduced the concept of gamification, which incorporated game elements into non-game contexts to enhance engagement.

2.2. Rhythmic Music and Gamification for Older Adults

Incorporating rhythmic music and gamification elements into finger movement measurement systems is essential to enhance the elderly’s engagement and focus. Se defined the criteria for selecting rhythmic music and designing gamified elements tailored to older users. Considering the individual differences in motor skills and cognitive functions of the elderly, the personalized selection of rhythmic music is crucial. Moderate tempos attract their attention and reduce fatigue. Choosing music familiar to the elderly evokes emotional resonance, thereby increasing their willingness to participate. Examples of these local music tracks are listed in Table 1.

3. Design Elements

3.1. Design Goals and System Requirements

Existing finger movement measurement systems conduct tests and collect data in a monotonous environment. The lack of engaging and personalized design often leads to participant fatigue, which compromises the accuracy of the measurement results. Therefore, the developed system was designed to optimize the user experience through a multi-sensory approach with the following requirements:

• Enhancing participant engagement: The system employed visual and auditory transmission to reduce the psychological fatigue caused by repetitive testing, thereby increasing the attractiveness and enjoyment of the testing process.
• Ensuring the data collection accuracy: By incorporating real-time visual feedback and synchronized rhythmic music, the system enhances the participants’ focus, reduces errors, and improves the reliability of the data collection.
• Achieving adaptability and personalization: Leveraging AI-assisted music recommendations, the system dynamically adjusts the test content based on the participants’ preferences and cognitive abilities, delivering a personalized testing experience.

3.2. Visual Transmission with Gamified Design

The system actively interacts with users through visual and auditory dimensions, functioning as a proactive transmitter of sensory information. Visual elements guide the participants’ actions while providing immediate feedback and progress updates in a gamified format:

• Immediate feedback: Dynamic animations (e.g., ripple effects upon tapping) are used to convey operation results in real-time, giving participants a sense of accomplishment.
• Task-progress visualization: Progress is visualized through unlockable task goals, helping participants understand their current status and motivating them to complete challenges.
• Thematic scenery: Relaxing visual themes, such as blue skies and oceans, reduce visual fatigue and create a comfortable testing environment.

3.3. Auditory Transmission with BGM Design

Personalized background music enhances the auditory enjoyment and focus:

• Synchronization with tapping actions: The rhythm of the background music is precisely matched with tapping actions, improving the smoothness and enjoyment of the testing process.
• Dynamic music recommendations: Background music is selected based on the participants’ preferences (e.g., familiar melodies or relaxing rhythms), fostering emotional resonance and increasing willingness to participate.
• Fatigue reduction: The system avoids overly intense rhythms, prioritizing light music, slow-tempo classical piano pieces, or traditional melodies to help the participants maintain focus while alleviating fatigue.

3.4. System Architecture

The proposed testing system was designed with a multi-sensory transmission architecture at its core, combining visual and auditory interactions to guide users’ focus on the testing tasks while reducing the cognitive load and enhancing the user experience. The system was structured to improve focus, alleviate fatigue, and enhance engagement. Through dynamic interactions across visual transmission, auditory transmission, and task control, the system provides a comfortable and enjoyable environment while maintaining the scientific rigor and reliability of the testing process.

4. Module and System Structure

The system is composed of four key modules, each with distinct roles and seamless integration, focusing on visual transmission, auditory transmission, task control, and data analysis. The functionalities of each module and the integration of user workflows are described below.

4.1. Visual Transmission Module

The visual transmission module interacts directly with participants through dynamic feedback, task progress visualization, and calming thematic designs. Upon completing a touch operation, the system provides immediate visual feedback, such as dynamic “ripple effects” at the touchpoint, enhancing the intuitiveness of operations and the participants’ sense of accomplishment. The task progress is displayed in real-time through progress bars and phased objectives, enabling participants to track their current progress and goal completion status. To alleviate visual fatigue from the prolonged operation, the system adopts relaxing visual themes, such as “blue sky and clouds” or “ocean waves”, to create a comfortable and stress-free testing environment.

4.2. Auditory Transmission Module

The auditory module enhances the testing experience through background music. Before testing, the system gathers the participants’ music preferences (e.g., genre, tempo) through an AI-driven questionnaire dialogue. Based on these preferences, generative AI creates personalized music recommendations. Once the participants select their preferred track, the system dynamically synchronizes the music with the testing tasks, ensuring smooth and natural touch interactions. The rhythm of the background music adapts to the participants’ performance, balancing increased engagement with reduced fatigue.

4.3. Task Control Module

The task control module offers two types of tasks: standard touch tasks and n-back tasks, allowing participants to choose based on their preferences. During the test, the system dynamically adjusts the task difficulty based on the participants’ performance metrics, such as touch accuracy and reaction time. Adjustments are performed to reduce the size of target points or shorten prompt intervals to maintain an appropriate level of challenge. Each task is assigned clear completion objectives—for instance, achieving a specific number of accurate touches may trigger animated rewards, such as a “Perfect” stamp. Upon completing a task, participants receive motivational feedback in the form of animated rewards, such as “star rewards”, encouraging continued participation.

4.4. Data Collection and Analysis Module

The data collection and analysis module operates throughout the testing process, ensuring the extraction of precise and meaningful results from user interactions. The system records metrics such as the touch response times, touch accuracy, task completion rates, and finger movement trajectories in real-time. These data points evaluate the user’s focus (e.g., delays in response times) and fatigue (e.g., decreases in task efficiency) while providing scientific insights for optimizing future tests. Upon test completion, the experimental results are automatically saved in the JavaScript object notation (JSON) format, enabling administrators to easily download and analyze the data. The results are presented visually on graphs and charts, facilitating comparisons with previous data and tracking changes in participants’ finger movement abilities over time (Figure 1).

5. Implementation

5.1. Technologies and Tools

The developed system was implemented using the following technologies and tools (Figure 2):

• User interface: Hypertext markup language (HTML) was used to construct the basic structure of the user interface. JavaScript was adopted to facilitate dynamic user interactions and page transitions. Cascading style sheets (CSS) was used to enhance the visual consistency and user experience through advanced styling techniques.
• Server-side technology: Node.js was employed for the server-side implementation to handle client requests and support real-time data collection effectively.
• Computer vision technology: Mediapipe and Hands were used to track finger movements, enabling real-time detection of hand landmarks and precise capture of user motion data.
• Music management technology: The music playback functionality leverages user preferences to recommend personalized tracks. Dynamic playback and synchronization are achieved through a JavaScript-based audio library.
• Hardware devices: A computer acts as the core system for executing the program and hosting the server. A wide-angle camera tracks users’ finger movements across a broad range, ensuring accurate and flexible detection. A tablet computer serves as the primary interaction device for users, providing intuitive and seamless operations.

Figure 2. User interface and components of developed system.

Figure 2. User interface and components of developed system.

5.2. Display Features

The developed system consists of three key screens, each designed to guide the user through the testing process while ensuring a seamless and user-friendly experience. The functionality and elements of each screen are detailed as follows.

The configuration screen collects user preferences and sets up testing parameters to enable a personalized experience. The title is displayed at the top of the screen as “Configuration” or “Settings”, indicating the current step. Users can select their preferred background music (e.g., light music, piano tracks, or traditional melodies) from a dropdown menu or radio buttons. Brief instructions or examples are provided to guide their selection. Users can adjust the testing difficulty (e.g., Easy or Hard) using a slider or buttons. Higher difficulty levels correspond to faster tasks and more precise interaction requirements. A prominently displayed “Confirm” button is used to save the user’s preferences and transition to the testing screen.

The testing screen is the essential interactive interface where users perform the testing tasks with real-time feedback and synchronized background music. Central dynamic target points (e.g., circular targets) require users to perform specific interactions, such as tapping. Successful operations trigger dynamic visual effects, such as ripples or color changes on the targets, enhancing the engagement. At the bottom of the screen, the track name and playback progress of the background music are displayed. A dynamic progress bar at the top indicates the percentage of the task completed. Instant feedback messages (e.g., “Great!” or “Perfect!”) appear to motivate users and encourage continuous participation. Located in the top-right corner, the pause/exit button is used to enable users to pause or exit the test at any time.

The results screen displays the testing outcomes in a visually intuitive manner and collects user feedback for system improvement. The “Results” or “Test Results” are displayed at the top of the screen, indicating the current stage. Bar charts or line graphs are drawn to present performance metrics, such as the reaction time distribution or task completion rates. A concise summary of the user’s performance, such as “Your focus score: 85/100” is provided for evaluation. A simple form with multiple-choice questions or a text box is used to collect user feedback and suggestions. The “Restart” button leads users to the Welcome Screen for a new session. “Exit” closes the system.

Each screen was designed to ensure a smooth transition and provide a cohesive user experience, supporting the system’s objectives of enhancing engagement, reducing fatigue, and improving data reliability.

6. Discussion and Conclusions

By integrating visual feedback and background music, the developed system enhances user engagement and alleviates fatigue during prolonged testing sessions. Dynamic task adjustments and personalized music recommendations further improve the user’s focus and the enjoyment of the testing process. The requirement for high-performance hardware, such as wide-angle cameras and high-response touchscreens, limit its deployment in resource-constrained environments. The developed system demonstrates advantages by integrating multi-sensory feedback through visual and auditory transmission to enhance the user experience and reduce testing fatigue. In the future, comprehensive user experiments need to be conducted to evaluate the system’s effectiveness and usability. Furthermore, optimizing the music recommendation algorithms and refining interaction designs are necessary to enhance the system’s generalizability and practical applications.