Motion Capture System in Performance Assessment of Playing Piano: Establishing the Center for Music Performance Science and Musicians’ Medicine in China ^†

Abstract

This article introduces China’s first Center for Music Performance Science and Musicians’ Medicine. In the center, motion capture (MoCap) technology is used to study piano performance and musicians’ health. An idea and methodology to assess the performance of piano performance are developed in the center. The center uses high-precision MoCap system to analyze movement efficiency, posture, joint angles, and coordination of pianists. By addressing physical challenges, the center promotes healthier, more efficient practice ways, especially for adolescent piano learners. The pioneering research results bridge the gap between music performance (art) and science, positioning China as a leader in music performance science and musicians’ health.

1. Introduction

Integrating high technology into music education has altered instructional methodologies, and the results offer unprecedented experiences for educators and learners. Artificial intelligence (AI) facilitates individual learning, breaking the limitations of traditional methods that lack interest and enthusiasm. Automatic accompaniment software, tone error detection systems, and machine learning algorithms are used to analyze performance data to provide real-time feedback and personalized learning experiences [1]. These technological tools are also used to develop music analysis methods to analyze pitch, tempo, and dynamics [2], which are significant in teaching and researching music performance. A motion capture (MoCap) system provides a visualized and analytical method to assess body movements in music performance. By capturing instrument players’ movement, MoCap visualizes feedback on posture, technique, and music expressive gestures [3], according to the data analyzed in biomechanics. MoCap addresses technical challenges and enhances expressive ability [4]. Such tools help musicians understand their physical alignment and reduce the risk of injury by identifying potentially harmful body use.

Besides its applications in music performance, these technologies are also utilized in music education. Virtual tutors simulate students playing and guide students through restructured practice models and patterns [5]. Music information retrieval (MIR) helps researchers categorize extensive music libraries and study music patterns, genres, and historical courses [6]. Digital instruments and virtual reality (VR) learning experiences bridge the gap between theories and practical experiences [7]. While technology in music education shows great potential and advantages, challenges such as the system’s cost and the need for specialized training also exist. However, with interdisciplinary collaboration and investment from technological companies, “Art + Science” overcomes these challenges by introducing technologies such as AI or MoCap into music education and performance.

European countries, Japan, and the United States are constructing specialized institutions for music performance research to exploit new ways for the use of the body to play instruments. They are helping students, teachers, and musicians train body coordination to improve playing techniques and decrease the risk of occupational diseases. Although China has rapidly developed technology in music education, it is still lacking practical application in music performance education. Many students and teachers are using traditional methods, thus which leads to inefficient practice and body pain, even playing-related musculoskeletal disorders (PRMDs). Therefore, systematic research for music performance is necessary. Chinese academics in music performance science and musicians pay attention to performance science and health. In the first Center for Music Performance Science and Musicians’ Medicine in China, MoCap is used as the primary equipment in piano performance and kinematics research, to promote music performance with scientific, natural, and healthy consciousness.

2. MoCap in Piano Performance Research

The application of motion capture technology in piano performance research has significantly enhanced the recognition of pianists’ movements and impacted on artistry and physical health. MoCap offers multiple approaches to analyze sight-reading (eye movement), body posture, hand position, expressive gestures and injury prevention, which collectively provide a scientific foundation for advancing music education and performance practices.

Wristen et al. [8] examined sight-reading and repertoire playing through high-speed MoCap and identified distinct movement patterns that reflected the varied cognitive and physical demands of these tasks. Similarly, Mora et al. [9] demonstrated how 3D visual feedback aided posture correction and technique refinement, enabling students to align their physical movements with ergonomic principles and musical goals.

Building on this foundation, there are studies that have shifted focus toward exploring the relationship between expressive and musical structure, examining how physical gestures contribute to interpret artistry. Thompson and Luck [10] revealed that pianists’ body movements often corresponded to expressive intentions and the structural elements of the music, offering data to understand how physical gestures enhanced interpretative depth. Complementarily, MacRitchie et al. [11] tracked performance gestures and found the relationship between physical movements and music expression; they provided a new dimension to understanding music interpretative choices. These studies illustrated how MoCap has filled the gap between physical performance and artistic expression as an invaluable tool for musicians and educators.

Furthermore, kinematic studies have demonstrated the precision of MoCap in capturing fine motor movements that are important to piano performance. Researchers examined finger gestures, timing, and key velocity, uncovering how pianists maintain consistent sound quality through controlled movements [12,13]. These findings emphasized MoCap’s ability to analyze and teach intricate technical elements of piano performance.

In addition to performance enhancement, MoCap assists pianists in addressing physical challenges. MoCap is used to identify harmful hand postures and investigate the causes of hand injuries to develop preventive strategies and ergonomic interventions [14,15]. Additionally, musicians’ well-being is supported by innovative data collection systems and integration with eye-tracking technologies [16,17] to enhance the precision of movement analysis and broaden the scope of research applications.

China’s rapidly growing population of piano learners and professionals underscores the significance of the use of technological innovations. In response, the establishment of the country’s first Centre for Music Performance Science and Musicians’ Medicine is important. The center integrates MoCap technology focusing on occupational disease prevention, ergonomics, and the enhancement of pedagogical methods. By addressing the physical and technical challenges, the center ensures that China’s vibrant music community thrives artistically while maintaining physical health. MoCap technology is used by the center to reform piano performance research through these efforts.

3. China’s First Centre for Music Performance Science and Musicians’ Medicine

Kurt Singer headed a medical counseling center at the MusikHochschule in Berlin from 1923 to 1932 [18]. The center began medical and scientific research for musicians. Since the 1970s, research on occupational diseases among musicians has been conducted in Germany, laying the foundation for a specialized study. Over the past 50 years, institutions and academic programs across Europe, North America, and Japan have played a crucial role in enhancing musicians’ performance techniques and mitigating the risks of occupational ailments.

Currently, research centers are introducing technological equipment. For example, the Kurt Singer Institute for Music Physiology and Musicians’ Health in Berlin is renowned for combining physiological research with music education to support musicians’ health [19]. Similarly, the University of Florida’s partnering with the National Endowment for the Arts has demonstrated how interdisciplinary research positively impacts musicians’ mental and physical well-being [20]. Stanford University’s Performing Arts Medicine Program integrates multiple disciplines such as orthopedics, psychiatry, and physical therapy to provide comprehensive care for musicians [21]. These centers use motion capture systems and other experimental equipment to examine the relationship between movement and music, and subtle hand and finger movements to optimize techniques and prevent injury. Through collaboration across disciplines, medical health is integrated with musical pedagogy. This integration is observed in centers worldwide, creating healthy, efficient, and scientific music performance.

In China, many musicians hope to optimize body movements and improve their performance to prevent occupational diseases such as tenosynovitis and joint pain. However, hospitals and clinics are the only choices for treatment; the “pianist’s hand” or “musician’s hand” is treated with conventional medical methods rather than “music-specialized” methods. Medical treatments often fail to recognize the pathogenesis, even leading to a high recurrence rate. As a result, a dedicated research center for music performance science and musicians’ medicine in China is necessary.

The Research Centre for Music Performance Science and Musicians’ Medicine is located in Chengdu, China. By using advanced equipment such as an optitrack motion capture system, the center researches the biomechanics of music performance and musicians’ health problems. The center aims to advance interdisciplinary research and promote musicians’ health to help musicians in China for treatment. Musicians, students, and teachers are provided with physical and psychological suggestions to enhance music performance and pedagogy.

4. Methodologies

4.1. Motion Capture System

Motion capture systems adopt optical, inertial, or markerless technologies to precisely track and record human motion. In piano performance research, this system captures kinematic data on finger, wrist, arm, and torso movements, providing data to increase technical efficiency and improve body coordination. Generally, there are two types of MoCap systems: the marker-based system and the markerless system.

In the marker-based system, optical MoCap systems such as Vicon, Qualysis, or Optitrack use reflective markers as anatomical landmarks. Reflective markers (Figure 1) have various types and sizes, tailored to the anatomical location and motion precision. Typically, smaller markers, such as 6.4 mm in diameter, are placed on finger joints to capture fine motor movements with high accuracy. Larger markers, 9 mm or 14 mm X-base markers, are positioned on larger body joints to facilitate the capture of broader motion ranges. This differentiation allows for comprehensive motion analysis across different body parts, accommodating the varying scale and complexity of movements involved. Infrared cameras track these markers, generating 3D data on joint angles, velocities, and movement trajectories. The obtained details allow for an objective analysis of biomechanical patterns in playing the piano such as finger independence [22], wrist rotations [23], and posture alignment [24]. However, these devices lack portability and require high-performance computers for operation, making them challenging to use. Additionally, they are susceptible to environmental factors such as temperature and humidity, which can affect their functionality.

In the markerless system, emerging technologies such as kinetic vision algorithms are used to capture movements without physical reflective markers. Although less precise, they are more accessible and appropriate for larger objectives. Wang [25] proposed a method to generate natural, dexterous hand motions for piano performance by using markerless MoCap. It is also used in animation, AI, biomechanics, and interactive media, especially in enhancing virtual reality (VR)/augmented reality (AR) experiences.

The research center has installed a motion capture marker-based system equipped with eight infrared cameras (Flex 13, OptiTrack, Corvallis, OR, USA) to capture three-dimensional movement with high precision (Figure 2). This system allows for a detailed analysis of dynamic motion in piano performance, providing important data on movement efficiency and coordination. A digital piano (YDP-105, YAMAHA, Tokyo, Japan) is integrated into the setup to facilitate experiments related to piano performance, ensuring realistic and replicable conditions for study. Additionally, “calibration” is necessary before performing motion capture. The Calibration Wand Kit is used for spatial calibration, while the Calibration Square is used for calibrating the horizontal plane. Spatial calibration helps ensure more precise capturing, resulting in a more accurate 3D model.

Sound data is captured using a high-fidelity linear pulse code modulation (PCM) recorder (PCM-A10, Sony, Tokyo, Japan), enabling audio synchronization with motion data to assess performance quality. Reflective markers and the essential components of the system are strategically placed at key anatomical landmarks to track body movements accurately. In previous studies, additional reflective markers are positioned on the piano or keyboard to measure parameters such as hammer velocity, keystroke timing, or accuracy. These markers help establish a frame of reference, allowing for consistent and standardized data collection.

4.2. Participants

Pianists at various skill levels, particularly adolescents, were recruited in this study. In China, adolescent piano beginners, typically aged 10–18, make up the majority of learners because they increasingly participate in the entrance examinations of the universities. Most of them have learned the piano but need to improve their techniques in a short period. Therefore, they have to practice and learn to play the piano extensively. This results in body pain, inefficient practice, and other issues. Therefore, a detailed analysis of physiological developments is required in scientific pedagogies [26]. Based on previous research, we targeted this demographic group to observe the formation of foundational techniques and habits.

4.3. Data Analysis

Kinematic analysis was used in this study as it is the primary method for interpreting the movement data captured by MoCap in piano performance (Figure 3). Focusing on postures, joint angles, and motion efficiency, it provides suggestions for biomechanics improvement related to piano performance and motor skill optimization methods.

• Posture analysis

Posture analysis is conducted to analyze sitting posture and hand position during performance. The alignment of the pianist’s body is also examined. MoCap system reflective markers (large size) are placed along the spinal curve and shoulder girdle to measure angles such as head flexion and shoulder symmetry [27,28,29]. Incorrect postures or misalignment that may lead to chronic pain are detected in the analysis. Mora et al. [9] used 3D visualizations of postures to provide real-time feedback to pianists, reducing long-term musculoskeletal risks.

2.

Joint angle analysis

This analysis is conducted for the wrist, elbow, and fingers by evaluating movement precision and adaptability in piano performance. These parts influence musical expression, such as tone color and dynamic control. Angular velocities and joint ranges are calculated based on the positional data of reflective markers. Data analysis focuses on key striking and release by placing small markers on the phalangeal joints, and finger movement is tracked for efficiency and timing to assess performance quality and prevent injury [13]. Li et al. [15] developed an analysis method for wrist joint angles that can detect harmful over-extension associated with repetitive strain injuries.

3.

Movement efficiency

Movement efficiency is assessed for the smoothness, accuracy, coordination, and energy expenditure of movements in piano performance. Finger and hand movement trajectories [13,30] and body acceleration patterns related to tempo [31] are traced to assess motion efficiency. The efficiency, economy, and coordination of body movements are accurately evaluated during a performance. By evaluating these factors, how physical mechanics influence both the quality and sustainability of musical performance is understood.

Additionally, Piano Touch Analysis Toolbox developed by Bernays and Traube is used to describe performance based on high-resolution keyboard and pedaling data. It analyzes delicate differences in articulation, timing, dynamics, and pedaling, providing a comprehensive understanding of a pianist’s key touch and playing gesture [32]. The MoCap Toolbox facilitates the computational analysis of movement data. It is not exclusively designed for piano performance, as it has been applied to study musicians’ movements and analyze ergonomics in performance techniques [33]. Zhou [34] integrated MATLAB into video analysis with audio and motion capture data to develop a framework for analyzing musicians’ body movements during performance. This framework is applied to piano performance research to examine the relationship between movement, technique, and musical expression.

5. Conclusions

The MoCap system plays an important role in learning and practicing music performance. The system is utilized for analyzing the three-dimensional movements of pianists and other musicians while playing, including joint angles, motion trajectories, and postural alignment. It offers a visualized method for uncovering the secrets of efficient and economical piano performance. It also helps innovate current piano teaching methods and potentially all instrumental education. Additionally, it helps prolong musicians’ healthy performance careers, along with strategies for treating and preventing occupational diseases.

The Centre for Music Performance Science and Musicians’ Medicine marks a significant milestone in the integration of disciplines within music performance and education. The center helps overcome the physical challenges musicians face, such as playing-related musculoskeletal disorders (PRMDs) and ergonomic errors. It provides local services so that music-related patients do not have to go to other countries for consulting. On the other hand, the center focuses on piano pedagogy innovation, especially to increase the number of adolescent piano beginners and improve techniques efficiently within a certain time for the entrance examination. Furthermore, constructing a scientific, natural, efficient, and healthy music career for musicians, music students, and teachers is the center’s goal to achieve.

Future advancements in MoCap technology are necessary to combine AI and VR to enhance music performance science and musicians’ medicine. Li et al. [15] indicated that AI-driven motion analysis can be used to identify subtle deviations of playing technique and provide real-time feedback or guidance, allowing musicians to optimize their performance efficiently while reducing injury risks. At the same time, immersive VR environments can simulate realistic performance, offering interactive experiences for skill development [9]. These technologies provide opportunities for enhancing technique and refining pedagogy in music education.

The Centre for Music Performance Science and Musicians’ Medicine plans to collaborate with international institutions, promoting communication and interdisciplinary development. The center also focuses on amateur musicians and instrument educators by providing natural practice methods and pedagogies to promote healthy performance habits. Additionally, more investments are necessary to adopt AI-related systems or VR equipment to enhance the center’s research capabilities and position to be a global leader in music performance science.

The MoCap system applied to music performance and musicians’ health research is an innovative technology to visualize body coordination, body use, and physiological mechanisms. As the most important device of the first Center for Music Performance Science and Musicians’ Medicine in China, the MoCap system plays a pivotal role in combining music performance (art) and science (technology) to improve China’s scientific research in music performance development.