Real-Time Recognition of Korean Traditional Dance Movements Using BlazePose and a Metadata-Enhanced Framework

Abstract

This study presents the implementation of an AI-based prototype for recognizing Korean traditional dance movements using a metadata-enhanced dataset. The research was conducted in three stages. First, a classification framework was developed to reflect the unique characteristics of Korean traditional dance. Second, video data were collected from existing and newly filmed sources, and a metadata set was created by labeling five fundamental movements for training. Third, the BlazePose model was applied to generate real-time skeletal key points, which were integrated with the metadata-enhanced dataset and processed using a customized approach to recognize dance movements in real time. The developed prototype successfully recognizes five fundamental Korean traditional dance movements and demonstrates the potential of AI technology in analyzing complex motion patterns. By integrating existing AI models with a domain-specific dataset, this study provides a systematic approach to the digital preservation and modern reinterpretation of traditional arts. Furthermore, the methodology can be extended to recognize dance movements from other cultures, offering new possibilities for the preservation and transmission of intangible cultural heritage through digital technology.

1. Introduction

The movements of Korean traditional dance are not merely physical motions but are expressions of cultural context and historical background. These movements have continuously evolved over time and across regions, with each community developing its own unique styles and techniques [1,2,3,4,5,6,7]. Therefore, preserving intangible cultural heritage, such as traditional dance, through digital means requires capturing these complex and layered characteristics.

With recent advancements in artificial intelligence (AI), research on applying deep learning to dance recognition has gained momentum [8,9,10,11,12,13]. Machine learning techniques, such as body detection, provide new methods for analyzing and understanding traditional art forms by creating real-time visual models that capture body movements [14]. Unlike static images, dance involves a flow of various movements over time, which requires precise recognition and tracking of multiple body parts in motion. AI algorithms can analyze these movement patterns to recognize and classify actions and even generate new movements.

To enable real-time recognition of Korean traditional dance, this study implemented an AI-based system that integrates a structured classification framework and a metadata-enhanced dataset reflecting the unique attributes of these movements. A systematic classification framework was first developed through literature review and expert consultation, creating a classification system of 36 fundamental movements that characterize Korean traditional dance. Using this framework, video data of these movements were collected from multiple angles, and a subset of 521 metadata-enhanced samples was constructed to train the AI model.

For real-time pose estimation, the open-source BlazePose model was employed to generate skeletal key points, which were then processed using a customized Bi-LSTM (bidirectional long short-term memory) architecture to analyze sequential motion data. This integration enabled the prototype AI model to recognize five fundamental Korean traditional dance movements accurately and efficiently.

This study demonstrates the potential of AI in the systematic analysis and digital preservation of complex dance movements. By combining BlazePose with domain-specific adaptations, such as a tailored classification framework and a metadata-enhanced dataset, the research successfully addresses the challenges of capturing and interpreting the intricate patterns of Korean traditional dance. This approach highlights the role of AI in preserving intangible cultural heritage while offering new possibilities for its reinterpretation and transmission in contemporary digital contexts.

2. Background Works

2.1. Classification and Characteristics of Korean Traditional Dance Movements

To identify and systematize the fundamental and common movements in Korean dance, relevant theories were reviewed [1,2,3,4]. This investigation revealed that the basic movements (chum-sawi) in Korean traditional dance can be categorized into four primary methods, which are summarized in Table 1.

The first is a body-based classification, categorizing movements by body parts such as the head, shoulders, hands, feet, and knees [15,16,17,18,19]. The second is an element-based classification, dividing movements into upper-body and lower-body actions [20,21,22,23]. The third approach involves categorizing movements by types of traditional dances, and the fourth method standardizes movement elements based on Labanotation, a Western dance notation system [24,25,26].

Among the various classification methods, Heo Soon-seon’s system, which emphasizes fundamental dance movement elements applicable across multiple dance forms, was selected. Heo’s classification emphasizes that the body’s movements align with the natural flow of breathing, categorizing movement elements based on the harmonious integration of breath and dance movements. When inhaling, the movements involve lifting, opening, and expanding; when exhaling, they involve lowering, closing, and bending. In other words, upper body movements express opening, extending, and descending with the breath, while lower body movements focus on closing and bending actions.

Table 1. Methods of dance movement classification in Korean traditional dance.

Dance Movement Classification Method	Classification	Description
Body-Based Movement Classification	Classification by body parts (head, shoulders, hands, feet, knees)	Lee Eun-joo (1996) proposed a method for categorizing dance movements by body parts, such as the head, arms, feet, and legs, into applied movements [5]. Later, Lee Bo-reum (2022) refined this method, systematizing a detailed classification based on specific body parts suitable for Korean court dance education [27].
Dance Movement Element-Based Classification	Upper body dance movement—48 items/ Lower body dance movement—48 items	This is a classification method that subdivides the upper and lower body into 48 items each, utilizing the breathing techniques of Korean dance and the concept of yin and yang. Seo Hee-joo (2003) systematized this [15], while Cha Su-jeong (2005) constructed 48 movements distinguishing the lower body and upper body for traditional dance education [16].
	Upper body dance Movement—18 items/ Lower body dance movement—18 items	Heo Sun-sun developed 18 upper and lower body movements based on traditional philosophy, including the concepts of heaven, earth, and humanity, as well as yin-yang and the five elements, emphasizing their connection to breathing [17,18,19,28].
Dance Type-Based Movement Classification	Movement classification by dance type	The traditional Korean fan dance, known as “buchaechum,” is a modern dance created by dancers and has developed under the influence of other traditional dances, such as the sword dance. The fan dance consists of a total of 21 movements [20], while the Honam sword dance consists of 11 movements [21,23].
Labanotation-Based Movement Element Classification	Standardization of traditional dance movement elements based on Western techniques (LMA, Laban Movement Analysis)	Kang Sung-beom (2004) analyzed the key elements of Korean traditional dance through Laban Movement Analysis (LMA), and subsequently [24], Choi Won-sun (2016, 2018) utilized LMA analysis as a method to reassess the value of the intangible cultural heritage of Jindo [25,26].

Table 1. Methods of dance movement classification in Korean traditional dance.

Dance Movement Classification Method	Classification	Description
Body-Based Movement Classification	Classification by body parts (head, shoulders, hands, feet, knees)	Lee Eun-joo (1996) proposed a method for categorizing dance movements by body parts, such as the head, arms, feet, and legs, into applied movements [5]. Later, Lee Bo-reum (2022) refined this method, systematizing a detailed classification based on specific body parts suitable for Korean court dance education [27].
Dance Movement Element-Based Classification	Upper body dance movement—48 items/ Lower body dance movement—48 items	This is a classification method that subdivides the upper and lower body into 48 items each, utilizing the breathing techniques of Korean dance and the concept of yin and yang. Seo Hee-joo (2003) systematized this [15], while Cha Su-jeong (2005) constructed 48 movements distinguishing the lower body and upper body for traditional dance education [16].
	Upper body dance Movement—18 items/ Lower body dance movement—18 items	Heo Sun-sun developed 18 upper and lower body movements based on traditional philosophy, including the concepts of heaven, earth, and humanity, as well as yin-yang and the five elements, emphasizing their connection to breathing [17,18,19,28].
Dance Type-Based Movement Classification	Movement classification by dance type	The traditional Korean fan dance, known as “buchaechum,” is a modern dance created by dancers and has developed under the influence of other traditional dances, such as the sword dance. The fan dance consists of a total of 21 movements [20], while the Honam sword dance consists of 11 movements [21,23].
Labanotation-Based Movement Element Classification	Standardization of traditional dance movement elements based on Western techniques (LMA, Laban Movement Analysis)	Kang Sung-beom (2004) analyzed the key elements of Korean traditional dance through Laban Movement Analysis (LMA), and subsequently [24], Choi Won-sun (2016, 2018) utilized LMA analysis as a method to reassess the value of the intangible cultural heritage of Jindo [25,26].

As shown in Figure 1, changes in movements associated with breathing cause ‘upper body dance movements’ and ‘lower body dance movements’ to interact with each other, creating a flow in dance that represents the harmony of the entire body, known as ‘whole-body dance movement’.

Figure 1. Structural principles of Korean dance movements by Heo Sun-seon.

Additionally, the foundational principle of movement in Korean traditional dance emphasizes footwork, where stepping motions—achieved through bending and straightening the knees—play a crucial role in not only ensuring stability but also defining the distinctive characteristics of the dance. The terminology used for Korean traditional dance movements encapsulates the essence, rhythm, and character of each movement, conveying the unique sense and energy inherent to each action. For example, the term ‘Balddidim-sawi’ (foot stepping dance movement) represents the action of stepping onto the ground, and the sound of ‘ddidim’ intuitively conveys the sense of rhythm and weight felt at the moment the step is taken. In this way, the names of these movements are not merely labels; they play a significant role in reflecting the distinct sensations and energy embodied in each movement [28,29,30,31].

Thus, the classification framework for Korean traditional dance movement elements used in this study was organized based on previous research on dance movements conducted by prior scholars.

2.2. Applications of AI in Dance Movement Analysis and Creation

Research in dance movement analysis and generation through machine learning and artificial intelligence has demonstrated the potential to transform choreography and performance, with various studies exploring these applications. For example, Luka Crnkovic-Friis and Louise Crnkovic-Friis introduced Chor-RNN, a system that generates choreography reflecting a dancer’s nuanced choreographic language and style. Rather than merely generating simple movement sequences, Chor-RNN produces structurally cohesive choreography, showing promising results. The core of this system is a deep recurrent neural network, trained on raw motion capture data to create new dance sequences for solo dancers. This approach extracts high-level features from data, unlocking new possibilities for computer-generated choreography and indicating the potential of machine learning in transforming dance creation and performance [14].

Furthermore, Google Arts & Culture Lab collaborated with renowned choreographer Wayne McGregor to develop an AI-based choreographic tool utilizing machine learning. This project, known as Living Archive, used U-MAP to map and organize approximately 500,000 moments from McGregor’s 25-year choreography archive based on visual similarity. In a related study, Chan et al. proposed a “Do As I Do” motion transfer technique [8]. This approach enables the transfer of standard movements from a source dancer to a new (often amateur) target dancer. Using poses as intermediate representations, this method extracts poses from the source and applies a learned pose-to-appearance mapping to the target, effectively translating movements across subjects. Collectively, these studies illustrate how machine learning can influence the creation and performance of dance.

The availability of extensive datasets is essential in machine learning, particularly for dance studies. Dance performances provide abundant motion data that enable algorithms to learn from diverse dance video collections. Such data allow algorithms to identify inherent patterns and structures within various dance styles, fostering a deeper understanding of movement diversity. For instance, a dataset titled Martial Arts, Dancing, and Sports Dataset for 3D Human Pose Estimation includes challenging movements from tai chi, karate, jazz, hip-hop, and sports, captured across 30 multi-view and stereo depth videos. This dataset, recorded using multiple cameras and involving dancers and martial artists, is valuable for examining complex movements under diverse scenarios [32].

In the realm of 2D pose recognition, datasets such as Leeds Sports Pose (LSP), containing approximately 25,000 images with 40,000 individuals in sports contexts, have been widely used, with images sourced from YouTube videos. The MPII Human Pose dataset includes 25,000 images with detailed annotations, such as joint coordinates, body part occlusion, 3D torso and head orientation, and 410 activity labels. Larger datasets, such as MS COCO, containing 60,000 images and 150,000 individuals, and AI Challenger, with 300,000 images and 700,000 individuals, offer extensive resources for pose estimation tasks [33,34]. The Human3.6M dataset provides 3.6 million images from a lab setting, captured using markers on 11 subjects performing various scenarios [35]. Similarly, datasets like MPI-INF-3DHP and CMU Panoptic offer diverse scenarios and sophisticated setups, serving as essential resources for complex human pose research.

Studies on Indian classical dance have also utilized convolutional neural networks (CNNs) for movement classification. A CNN architecture, designed with four convolutional layers and different filter window sizes, was proposed to improve recognition speed and accuracy. This architecture employs probabilistic pooling, combining max and average pooling techniques, and was trained on a dataset of 10 dancers performing 200 mudras/poses across various backgrounds [9,10,11]. This CNN model achieved an average recognition rate of 93.33%, outperforming previously proposed Indian classical dance (ICD) classification models and other state-of-the-art classifiers.

Despite the advancement in human pose recognition and the increasing availability of datasets, research specifically targeting dance movements remains limited due to the complexity of dance motion. While the application of machine learning in dance analysis is expanding, challenges persist in achieving accurate analysis and understanding of intricate, varied dance sequences [27,36].

2.3. AI Model Research on Motion Recognition

AI models for estimating dance movements use deep learning algorithms to recognize and analyze complex human motions. These models identify essential features of dance movements, extracting patterns that help the AI understand the intricacies of dance sequences. Dance movement estimation models typically use deep learning architectures, such as CNNs and recurrent neural networks (RNNs), to extract features from visual data. CNNs, through repeated convolution and pooling, identify key invariant features and feed them into a fully connected network for classification. CNNs effectively learn spatial hierarchies within images, while RNNs are suited to temporal data, making them particularly useful for modeling the continuity of dance movements [37].

RNNs can retain and update previous information, enabling them to recognize patterns within continuous sequences. Long short-term memory (LSTM), a specialized RNN variant, is adept at handling long-term dependencies, crucial for recognizing dance sequences that require maintaining relevant information across multiple movements. LSTM networks can retain essential information while discarding unnecessary details, making them instrumental in dance movement recognition [38,39].

For video data, 3D convolutional neural networks (3D-CNNs) are designed to process temporal and spatial data simultaneously, capturing the dynamic nature of dance. Advanced models, such as graph neural networks (GNNs), are also used for 3D pose estimation, extending understanding beyond 2D image data into 3D spaces [40].

Dance movement estimation models use various training approaches, including supervised learning, unsupervised learning, and reinforcement learning. Supervised learning utilizes labeled data to guide models toward accurate predictions, while unsupervised learning allows the model to uncover structure and patterns within unlabeled data. Reinforcement learning enables the model to discover optimal actions within a given environment [41,42,43,44]. These methods enhance the model’s ability to recognize dance movements with greater accuracy and to analyze movements in real time. In this study, supervised learning was chosen to recognize traditional dance movements through labeled data.

Deep residual networks (ResNet), as introduced in the study by He et al., incorporated shortcut connections to address the vanishing gradient problem through the concept of residual learning. This design significantly increased the depth of CNNs, improving their ability to learn complex features [45]. In particular, the architecture of ResNet demonstrates the effectiveness of increasing network depth, which is expected to accurately capture the fine movements of traditional dance, such as the subtle motions of fingertips, toes, and gaze. This deep structure of ResNet aligns with approaches aimed at improving feature extraction to precisely identify subtle differences. Gated CNN and Hybrid Connectivity, as developed in the study by Yang et al., achieved enhanced feature representation by integrating hybrid connections into convolutional operations. This method showed significant improvements in tasks requiring the differentiation of fine features and provided a foundation for optimizing convolutional networks through connection strategies [46]. This approach is expected to enable the modeling of complex correlations between dance movements based on their continuity and connectivity. The gate mechanism in Gated CNN helps efficiently distinguish related dance movements while simultaneously learning their continuity. Additionally, Hybrid Connectivity selectively emphasizes key features, which are expected to be useful in learning core elements of traditional dance, such as the directionality of hand movements, the position of the feet, and the angles of the arms. By integrating these discussions, it can be emphasized that such CNN architectures provide a foundation for advanced neural network development and significantly impact the proposed methodology.

Pose estimation models extract key points of the body from videos, which are then provided as input to other models. For example, models like OpenPose and PoseNet are used to identify precise positions within dance movements [47,48]. Each model presents specific strengths and limitations, and researchers can select appropriate models based on their goals and datasets. Combining multiple models through ensemble methods can also improve dance movement recognition accuracy. Motion and gesture recognition algorithms in AI broadly fall into three categories, with motion recognition models particularly focused on handling temporal data. In supervised learning, a labeled dataset enables the model to learn correct outputs for given inputs, such as classifying images of specific dance movements. Through this process, the model minimizes a loss function, which measures the difference between predicted and actual outcomes, ultimately refining the model’s accuracy [49,50].

Previous studies on AI-based dance movement recognition showed promising results, particularly in specific dance styles such as Indian classical dance and general choreography generation. However, these approaches often struggle to effectively handle complex and culturally unique movements due to the lack of specialized datasets for traditional dances [9,10]. Furthermore, many existing methods face challenges in accurately capturing the dynamic and temporal flow of movements, especially in real-time scenarios [11,14].

This study addresses these limitations by developing a domain-specific classification framework tailored to Korean traditional dance, establishing a systematic framework for 36 fundamental dance movements, and constructing a metadata-enhanced dataset for selected movements. Additionally, BlazePose was integrated with a Bi-LSTM architecture to achieve accurate real-time recognition while preserving the temporal continuity and granularity of dance sequences. By systematically addressing these challenges, this research presents a novel and effective solution for recognizing and preserving the intricate movements of Korean traditional dance.

3. Materials and Methods

The study was conducted in three stages to develop a prototype capable of automatically recognizing Korean traditional dance movements using AI technology. In the first stage, a classification framework for Korean traditional dance movement elements was established based on existing research on Korean dance. In the second stage, original data were collected by obtaining or filming video materials related to Korean traditional dance, and these data were labeled according to the classification framework, forming a metadata set essential for AI training. In the final stage, various AI models for movement recognition were tested to identify the most suitable model for traditional dance. The chosen model was then combined with the Korean dance metadata to complete a prototype for real-time recognition of Korean traditional dance movements.

The entire process is visualized in the diagram in Figure 2 illustrating each stage from data collection, preprocessing, model training, to the final development of the AI model prototype.

Figure 2. Process for developing a Korean dance-specialized AI model prototype.

3.1. Establishing a Classification Framework for Korean Traditional Dance Movements

After a comparative review of eight classification frameworks for Korean traditional dance movements proposed by various sources, including Jung Byung-ho, Heo Soon-seon, Kim Young-hee, Seo Hee-joo, Bae Jung-hye, Cha Soo-jung, the Korean Traditional Dance Education Association, and the high school arts curriculum, it was determined that Heo Soon-seon’s classification method, based on essential dance elements, was the most suitable for general application across different dance types. Following this, direct consultation with Heo Soon-seon was conducted to establish a classification framework for traditional dance movements. This system categorizes 36 fundamental dance elements into primary and secondary categories, including upper-body movements, lower-body movements, finger movements, and footwork.

The validity of this organized classification framework was further reviewed with input from other Korean dance experts. To verify the system’s applicability, 184 videos encompassing 14 types of traditional dances—such as Jin-do Bukchum, Samdo Talchum, Bukku-chum, Chun-aeng-mu, Salpuri-chum, Ganggangsullae, Honam Geommu, Seungmu, Cheoyongmu, and Hallyangmu—were collected and analyzed. As shown in the example in Figure 3, this analysis revealed that upper-body movements were divided into 20 main categories and 58 sub-movements, while lower-body movements were classified into 32 main categories and 67 sub-movements. All 14 traditional dances were found to adhere to the classification framework of 36 fundamental dance movements, composed of 18 upper-body and 18 lower-body actions.

Figure 3. Example of Seungmu dance analysis applied for classification framework Verification.

Consequently, as shown in Table 2 and Table 3, a classification framework for fundamental Korean dance movements was defined, consisting of 18 upper body movements and 18 lower body movements, totaling 36 basic dance movements

Table 2. The 18 upper body movements.

1. In-Sawi (인사위)		2. Palppeodeum-Sawi (팔뻗음사위)		3. Sonmoyang Sawi (손모양사위)
	Greeting Gesture: A gesture of placing the hands in front, to the sides, behind, above, or below close to the body.		Arm Stretch Gesture: A movement of stretching the arm forward, backward, to the side, up, or down.		Hand Shape Gesture: A motion in which the fingertips are directed up, down, or back and forth.
4. Sonbadakmodeum-Sawi (손바닥모듬사위)		5. Aneum-Sawi (안음사위)		6. Seongju-Sawi (성주사위)
	Palm Combination Gesture: A movement of bringing the palms of both hands together.		Hug Gesture: An action of placing one’s arm across the chest, with fingertips reaching to the opposite shoulder.		Guardian Spirit Gesture: A movement representing the shape of a house, with arms forming an “L” or “D” shape in front.
7. Mem-Sawi (멤사위)		8. Hwal-Sawi (활사위)		9. Palkkeokkeum-Sawi (팔꺽음(께끼)사위)
	Carrying Gesture: An action of carrying an object on one’s shoulder.		Bow Gesture: A movement where the arm is rounded, symbolizing a bow.		Arm Bending Gesture: A movement of bending the arm up, down, forward, sideways, or backward.
10. Momguphim-Sawwi (몸굽힘사위)		11. Heumshin Sawwi (흠신사위)		12. Palollim-Sawwi (팔올림사위)
	Body Bending Gesture: A movement of bending the body forward, sideways, or backward, with emphasis on bending forward.		Body Bowing Gesture: Heumshin refers to a movement of bending the body forward as a sign of respect.		Arm Raising Gesture: A motion of raising the arm.
13. Palpyeoollim-Sawi (팔펴올림사위)		14. Palnaerim-Sawi (팔내림사위)		15. (Eoreum-Sawi (어름사위)
	Extended Arm Raising Gesture: A movement of lifting the arm while bending and straightening it.		Arm Lowering Gesture: A motion of lowering the arm.		Affectionate Gesture: A movement of pushing, pulling, or shaking the body, as if treating a child or animal fondly.
16. Palgeondeuri-Sawi (팔건들이사위)		17. Paldolm-Sawi (팔돎사위)		18. Yeodadi-Sawi (여닫이사위)
	Arm Shaking Gesture: A movement of shaking the arm.		Arm Rotating Gesture: A movement that rotates the arm using the shoulder, elbow, and wrist joints.		Opening and Closing Gesture: A movement that rotates the arm using the shoulder, elbow, and wrist joints

Table 3. 18 lower body movements.

1. Baldidim-Ssawi (발디딤사위)		2. Baldodum-Sawi (발돋움사위)		3. Eutteum-Sawi (으뜸사위)
	Foot Pressing Gesture: “Didim” refers to standing firmly with one foot and pressing down, derived from the term “Didida”.		Toe-Raising Gesture: “Dodum” refers to the act of standing on tiptoe with heels raised, focusing balance on the toes.		Primary Gesture: This primary leg movement combines stepping with one foot and rising with the other, creating a sweeping motion in the lower body.
4. Balbatchim-Sawi (발받침사위)		5. Balmoyang-Sawi (발모양사위)		6. Balmodum & Balbeollim -Sawi (발모둠, 발벌림사위)
	Foot Support Gesture: A movement of supporting an object with the feet, lifting the feet and knees in response.		Foot Shape Gesture: Movements where the feet point up, down, inward, outward, or where the soles face up or down.		Feet Together and Apart Gesture: Movements in which the feet come closer together or move farther apart.
7.Mureupguphim & Mureuppyeom-Sawi (무릎굽힘, 무릎폄사위)		8. Mureupgulheoanjeum-Sawi (무릎밃, 뗌사위)		9. Muleupkkulheoanjeum-Sawi (무릎꿇어앉음사위)
	Knee Bending and Extending Gesture: Movements involving bending and extending the knees.		Knee Pushing and Lifting Gesture: A motion of pushing against the floor with the knee and then lifting the knee from the floor.		Kneeling Gesture: The action of kneeling down on the floor.
10. Meam-Sawi (맴사위)		11. Balgeonneom-Sawi (발건넘사위)		12. Baljjigeum-Sawi (발찍음사위)
	Spinning Gesture: A movement in which the body spins in place.		Foot Crossing Gesture: A movement representing the act of crossing over, with one foot leaping forward and landing.		Stamping Gesture: A stamping motion using the heel or toe.
13. Gyeopdari-Sawi (겹다리사위)		14. Georeum-Sawi (걸음사위)		15. Balgureum-Sawi (발구름사위)
	Leg Overlapping Gesture: A movement of overlapping the legs.		Walking Gesture: Dance movements that depict walking.		Foot Stamping Gesture: A Movement of stepping firmly on the floor with the foot so that the ground resounds.
16. Jiseum-Sawi (짓음사위)		17. Momdolm-Sawi (몸돎사위)		18. Momteulm-Ssawi (몸틈사위)
	Excitement Gesture: A movement that signifies playfully showing off in delight, with the center of gravity shifting charmingly.		Torso Turning Gesture: A movement of rotating the torso.		Body Twisting Gesture: A twisting motion of the body.

3.2. Establishing a Metadata Set for Traditional Dance Training

Based on the established classification framework for traditional Korean dance, it was necessary to develop an AI training metadata set to support various movement recognition technologies. To this end, a wide range of raw data on traditional dance movements was extensively collected. This raw data were then annotated with the defined classification framework and basic information for Korean dance, resulting in a complete metadata set for AI training purposes.

3.2.1. Collecting Video Data on Traditional Dance

A substantial number of video materials distinguishing various types of traditional Korean dance were required for AI training metadata development. The sources utilized included publicly available online videos, archives from national institutions like the National Intangible Heritage Center, materials held by individuals in the dance and performing arts fields, and data acquired through field surveys. Additionally, to ensure a comprehensive dataset, original footage of traditional dancers performing was filmed, resulting in the acquisition of 497 unique video files [51,52,53,54,55].

After standardizing the size and format of the collected videos, some preprocessing tasks were carried out to optimize the data for training. These tasks included extracting joint coordinates, skeleton extraction, and generating 3D model data, thereby converting the data into a format suitable for AI training. This preprocessing highlighted considerations and potential challenges, such as the importance of capturing each dance movement from multiple camera angles to provide a variety of perspectives. Without multi-angle footage, occlusion issues—where parts of the body are hidden—could lower training accuracy. Consequently, dancers were hired, and each of the 36 basic movements, along with other dance types, was captured from multiple angles to prevent occlusion issues and to ensure comprehensive visibility of each movement. Figure 4 and Figure 5 below illustrate these processes.

Figure 4. Sequential motion display of basic traditional dance movements.

Figure 5. Standardization process for video files, including size and format adjustment for consistency.

Additionally, as shown in Figure 6, the collected video data were standardized in size and format, followed by a preprocessing phase involving the extraction of joint coordinates, skeleton data, and 3D model generation, transforming the data into a format suitable for training.

3.2.2. Generating a Metadata Set for Traditional Dance Training

For labeling the training data on basic traditional dance movements, several key criteria were defined, as shown in Figure 6. These criteria included movement classification, lighting conditions, camera type, clothing type, dancer name, and numbering for differentiation. Each video filename was encoded with these information elements to form a structured label.

Figure 6. Labeling code structure for traditional dance training data.

As exemplified in Figure 7, a file labeled as ‘A407C003B002D002P001S001’ denotes that the video captures the “Kneeling Dance Movement” performed by dancer Kim Sori, in a brightly lit environment with casual attire, recorded with a GoPro camera in a square aspect ratio, marked as the first take.

Figure 7. Example of labeled dance video file.

As summarized in Table 4, a total of 521 metadata entries were created, covering thirteen core movements, each with standardized labeling to support AI model training.

Table 4. Quantity of labeled metadata by classification framework.

Dance Movements			Quantity
Upper body dance movements	1	In-Sawi (인사위)	48
	2	palppeodeum-Sawi (팔뻗움사위)	44
	3	Sonbadakmodum-Sawi (손바닥모듬사위)	45
	4	Momguphim-Sawi (몸굽힘사위)	36
	5	Eoreumi-Sawi (어름사위)	45
	6	Yeodad-Sawi (여닫이사위)	46
Lower body dance movements	1	Baldidim-Sawi (발디딤사위)	36
	2	Baldodum-Sawi (발돋움사위)	38
	3	Balbatch-Sawi (발받침사위)	36
	4	Balmodum, balbeollim-Sawi (발모음발벌림사위)	40
	5	Mureupguphim, Mureuppyeom-Sawi (무릎굽힘, 무릎폄)	36
	6	Balgeonneom-Sawi (발건넘사위)	35
	7	muleupkkulheoanjeum-Sawi (무릎꿇어앉음사위)	36
Total			521

3.3. Development of an AI Prototype Specialized in Traditional Dance Movements

To develop an AI model specialized in recognizing traditional Korean dance movements, we conducted tests on four models and progressively improved the model’s performance, including methods for constructing the traditional dance dataset and training strategies. The main attempts at each stage are as shown in Table 5.

Table 5. Analysis of four model tests for developing an AI model specialized in recognizing traditional dance movements.

Model/Methodology	Key Achievements	Limitations/Identified Issues
LRCN Model (combining CNN and LSTM)	- Learned motion patterns from traditional dance video data. - Enabled early motion recognition.	- High error rate due to datasets with diverse backgrounds. - Requires standardized datasets.
SlowFast Network [56]	- Parallel analysis of static and dynamic content. - Improved accuracy in recognizing upper body and arm movements.	- Additional data needed for recognizing specific detailed movements.
Two-stream Action Recognition Network [57]	- Simultaneously analyzed appearance and motion information using 2D and 3D CNN. - Improved accuracy for upper-body-centric movements.	- Poor recognition rate for lower body movements, necessitating further training.
Key point R-CNN (based on COCO dataset)	- Achieved high recognition accuracy (over 95%) even for various angles and rapid movements. - Delivered stable performance for both individual and group movements.	- Not capable of real-time processing.

As shown in Table 5, experiments were conducted to find the most suitable AI model (CNN + LSTM, SlowFast Network, Two-stream Action Recognition Network, Key point R-CNN) for recognizing traditional dance postures. This process involved comparing the performance of various artificial intelligence models to determine the optimal methodology. Specifically, CNN + LSTM was used to learn sequential data (continuous movements), and R-CNN was employed to identify key joint positions in the human body. This allowed us to verify how different networks contribute to the accurate recognition of traditional dance poses.

The key achievements, improvements, and limitations identified at each stage were thoroughly analyzed, and ultimately, a lightweight structure and an efficient algorithm capable of real-time processing were chosen. An optimized model based on BlazePose was selected to improve the accuracy and efficiency of traditional dance motion recognition simultaneously.

BlazePose utilizes a two-stage ’detection-tracking’ ML pipeline to locate the human body area within a frame and estimate 33 key joint points, as shown in Figure 8. Inspired by Leonardo’s Vitruvian Man, this system predicts the human body’s center, rotation, and size by estimating the midpoints of the hips, the radius of the surrounding circle, and the angles of lines connecting the shoulders and hips. The key points extracted from BlazePose were trained on labeled traditional dance data using supervised learning to accurately predict specific movements [58,59,60].

Figure 8. BlazePose key point.

Although the COCO model’s 17 landmark points are suitable for basic pose tracking, it lacks detailed information for regions like hands and feet, limiting its ability to capture the subtle details inherent in traditional dance movements. Consequently, Google’s MediaPipe BlazePose was chosen as the AI model for this study.

As shown in Table 6, upper body movements like In-Swai, which focus on arm motions, are identified using Arm key points (11–14), while hand-focused movements like Sonmoeum-Sawi are recognized through Hand key points (15–22). For lower body movements, knee-focused motions like Muleupkkulheoanjeum-Sawi utilize Leg key points (25–26), and foot-focused movements like Balbatchim-Sawi are identified using Foot key points (27–31). This key point-based motion analysis enables a systematic understanding of the structural characteristics of traditional dance movements.

Table 6. Analysis of key points for each traditional dance movement type.

Division	Upper Body Movement		Lower Body Movement
Dance Movements
Types of dance movements	In-Swai	Sonmoeum-Sawi	Muleupkkulheoanjeum-Sawi	Balbatchim-Sawi
Body division	Arm key point	Hand key point	Leg key point	Foot key point
Key point	11~14	15~22	25~26	27~31

The motion analysis functionality of BlazePose was integrated with the traditional dance dataset (Traditional Dance PoseNet) and combined with the sequential motion processing capability of Bi-LSTM to develop an AI model capable of precise movement recognition.

As shown in the first step of the development workflow in Figure 9, key joint coordinates were extracted from real-time captured motion data. Vectors between joints were then generated, and cosine values were used to calculate angles, quantifying the motion characteristics. The extracted data were normalized, and skeleton data were created as input for the Bi-LSTM model.

Figure 9. Development workflow for traditional dance recognition AI model prototype using Mediapipe BlazePose.

In the second step, Bi-LSTM was employed to analyze dance movements. Bi-LSTM’s ability to learn information from both forward and backward time sequences enables it to effectively capture continuous movements inherent in dance. As illustrated in Figure 10, the Bi-LSTM model, trained on a metadata set of 521 traditional Korean dance movements, compares extracted user motion data (left) with the training dataset, classifying movements into distinct categories as depicted in the result (right).

Figure 10. Process of analyzing traditional Korean dance movements using Bi-LSTM.

In Step 3, the results of the comparison and analysis with the metadata set were visualized by overlaying the user’s skeleton motion image onto the movement images, with the movement names and confidence scores displayed (e.g., “PAL: 0.99996723”, “INSA: 0.99992804”). This model was developed using state-of-the-art technologies such as Mediapipe 0.8.9.1, Python 3.9.0, and TensorFlow 2.7.0, along with additional libraries including Protobuf, Pillow, OpenCV, and NumPy (Table 7). These tools effectively facilitated BlazePose-based pose estimation, data preprocessing, and the training and evaluation of the Bi-LSTM model.

Table 7. AI prototype development code for recognizing dance movements.

Item	Description	Key Roles/Actions	Related Libraries/ Modules
Initialization	Initialization of MediaPipe and TensorFlow models	- Initialize the BlazePose model (mp_pose.Pose) and load the pre-trained LSTM model (model_4.h5).	mediapipe, tensorflow
Landmark Extraction	Extract landmarks from video frames using BlazePose	- Use pose.process(img)to extract relative position data (landmarks): 33 key points with their (x, y, z, visibility) values.	mediapipe
Feature Calculation	Calculate relative vectors and angles using landmark data	- Compute relative vectors (v) between landmarks. - Calculate angles between vectors (using np.arccosand np.einsum). - Generate feature vectors including angles (in radians).	numpy
Preparing LSTM Input	Store specific vector sequences (seq) and input them to the LSTM model	- Create sequences composed of 40 frames. - Adjust input dimensions for the model using np.expand_dims.	tensorflow.keras.models
Comparison Analysis	Compare BlazePose results with traditional classifications and templates	- Use a Bi-LSTM model to analyze real-time data sequences and compare them with pre-trained datasets and templates.	train.ipynb
Motion Prediction	Predict motion classes for input sequences using the LSTM model	- Identify the class with the highest probability from model output (y_pred). - Exclude results with low confidence based on the confidence threshold.	tensorflow

In conclusion, a prototype AI model specialized in recognizing traditional dance movements was successfully developed by integrating BlazePose with an innovative traditional dance classification framework. This model goes beyond generic pose estimation models by learning from traditional dance motion data, enabling precise recognition of Korean traditional dance movements. It has been designated as the ‘AI Model Specialized in Korean Traditional Dance.

4. Result

Summarizing the entire development process in Figure 11, the trained dataset consists of quantified values of joint coordinates and angles for each traditional dance movement. These were used to create a traditional dance classification-based labeled dataset, which compares real-time input data along the time axis to predict the most similar training data for specific movements. A new AI prototype model specialized in traditional dance pose recognition was developed. By integrating BlazePose with the unique traditional dance classification system and templates, it became possible to create an executable file specialized in recognizing traditional dance movements.

Figure 11. Development process of a traditional dance-specialized AI prototype.

The executable file uses BlazePose as the foundational technology for extracting and processing data, and it is fused with the Traditional Dance PoseNetmodel, a dataset of traditional dance movements categorized by the classification system. This integration resulted in the creation of a new AI model specialized in recognizing traditional dance movements. By applying traditional dance pose labels to the general-purpose pose estimation AI model BlazePose, the new model was termed a ’Traditional Dance Specialized AI Learning Model’.

Unlike the original BlazePose, this model was trained specifically on the postures and movements of Korean traditional dance. Therefore, referring to it as a ’new traditional dance specialized AI learning model’ signifies that this model has been newly defined beyond simple pose estimation to recognize traditional dance. While BlazePose is a general-purpose pose estimation model, it has been expanded into an AI model specialized for Korean dance by learning the unique movement elements of Korean traditional dance.

The additional traditional dance dataset customized for BlazePose indicates that this model has been tailored to recognize Korean traditional dance movements. Thus, it emphasizes that this is a ’specialized’ AI learning model, distinct from existing pose estimation models.

As shown in Table 8, BlazePose was trained using metadata for five basic Korean traditional dance movements. Through this, an algorithm was developed to recognize three upper-body dance movements and two lower-body dance movements, leading to the development of a functional AI model prototype. Through real-time dance movement recognition testing, this final model confirmed its capability to accurately identify detailed traditional movements in real-time, such as In-Sawi, Sonbadakmodeum-Sawi, Palppeodeum-Sawi, Muleupkkulheanjeum-Sawi, and Balbatchim-Sawi.

Table 8. Real-time recognition results of dance movements.

1. In-Sawi	2. Sonbadakmodeum-Sawi	3. Palppeodeum-Sawi	4. Muleupkkulheanjeum-Sawi	5. Balbatchim-Sawi

The proposed approach was compared and evaluated against another motion recognition AI model, DD-Net, for performance. The performance of the proposed model, Bi-LSTM, and DD-Net was compared under identical datasets and conditions. The experiments were conducted on a system equipped with an Intel i9-9900K CPU, Nvidia RTX 2080Ti GPU, and 48 GB of RAM. The dataset consisted of a total of 80,171 sequences, with each sequence containing 40 frames. The data were split into 90% for training and 10% for testing. The models were trained for 30 epochs with a learning rate of 1 × 10⁻³, and these conditions were consistently applied to all models to ensure a fair evaluation of performance metrics. The comparison metrics used were Accuracy, Computational Time, and Frames Per Second (FPS) for processing.

As shown in Table 9, DD-Net achieved the highest accuracy of 100%, while the proposed Bi-LSTM model attained a comparable accuracy of 99.35%. In terms of real-time processing capabilities, the Bi-LSTM model outperformed DD-Net with an inference time of 0.0804 ms per frame and a processing speed of 12,440 FPS, demonstrating significantly superior real-time performance. These results confirm that the proposed method is not only accurate but also highly efficient in processing speed, making it particularly effective for real-time applications.

Table 9. Performance comparison between Bi-LSTM and DD-Net.

Model	Accuracy (%)	Inference Time (ms)	FPS
Bi-LSTM (Proposed)	99.35	0.0804	12,440
DD-Net	100	0.5417	1846

5. Limitations and Future Work

This study proposed an AI-based prototype model for recognizing Korean traditional dance movements, focusing on five fundamental movements. However, several limitations must be addressed to advance this research further.

First, although the established classification framework includes 36 traditional dance movements, metadata was created for only 13 movements due to time constraints, and the current model was trained on five movements. Expanding the dataset to include all 36 movements is essential for comprehensive recognition.

Second, the current AI model is trained on a limited number of samples, which may reduce its ability to generalize across diverse or modified movement patterns. Increasing both the quantity and diversity of the training data, including different dancers, styles, and recording conditions, is critical to improving model robustness and performance.

Third, this study primarily focused on developing an initial prototype, and real-time performance was evaluated in controlled environments with limited testing under varied conditions. Future research should include extensive testing in real-world environments with varying lighting, backgrounds, and occlusions to enhance the system’s adaptability and accuracy.

To overcome these limitations, future work will involve designing a more sophisticated AI model to capture both the spatial and temporal features of dance movements. Specifically, the integration of ResNet for stable feature extraction and Gated CNN for selectively emphasizing relevant information will be explored. Additionally, incorporating 3D CNNs or LSTM-based temporal modeling techniques will allow the model to learn the sequential flow and interdependencies between movements more effectively.

Further efforts will focus on expanding the dataset through advanced data augmentation techniques and detailed labeling, enabling the recognition of diverse movement styles and regional characteristics. A comprehensive recognition system encompassing all 36 movements will ultimately be developed.

Finally, a real-time recognition and analysis platform will be proposed to facilitate practical applications, such as educational tools for Korean traditional dance and systems for the digital preservation and transmission of cultural heritage. By advancing movement recognition technology, this research aims to enhance both the preservation and contemporary utilization of traditional arts.

6. Conclusions

This study aimed to implement an AI-based prototype capable of automatically recognizing movements in Korean traditional dance. A systematic classification framework reflecting the characteristics of traditional dance was established, and video recordings of movements were collected to create a metadata-enhanced dataset. Using the open-source BlazePose model for real-time skeletal key point detection and a customized Bi-LSTM architecture for sequential analysis, the study successfully adapted and optimized existing AI techniques to recognize five fundamental dance movements.

Unlike previous research that focused on general pose recognition, the key contribution of this study lies in the domain-specific adaptation of existing AI tools. By combining BlazePose with a tailored metadata-enhanced dataset, this research demonstrated how existing models can be effectively applied to recognize the subtle and complex movements unique to Korean traditional dance.

The findings highlight the potential of AI for the digitalization and preservation of cultural heritage, offering a new methodology for the analysis and reinterpretation of intangible cultural heritage. The systematic creation of a metadata-enhanced dataset and the adaptation of AI models to cultural contexts mark significant progress in applying modern technology to traditional arts.

The proposed method is specialized for Korean traditional dance movements. However, with the establishment of an appropriate dataset and classification system, it can be extended to other dance movements. Since it is based on the general-purpose pose estimation model BlazePose, it is capable of learning different types of dance movements. However, to accurately recognize other dance movements, it is necessary to newly design metadata sets and classification systems that reflect the unique characteristics and movements of each dance. Through this process, the model can learn the patterns and continuity of dance movements, making it effectively applicable to various dance styles.

While this study achieved promising results with a prototype model, it does not involve the development of a new recognition algorithm. Instead, the research demonstrates how existing tools can be adapted and optimized for culturally specific tasks.

Ultimately, this study lays a critical foundation for the preservation, analysis, and modern reinterpretation of Korean traditional dance. By leveraging existing technologies and tailoring them to cultural heritage, this research fosters new opportunities for the global exchange and appreciation of traditional dances in contemporary digital contexts.