L-DGC: LLM-Based Dance Generative Control

Abstract

The global expansion of K-pop has increased demand for AI-driven choreography learning. However, existing motion recognition models often struggle to capture fine-grained rhythm patterns and dynamic motion transitions across consecutive frames, limiting their ability to provide accurate and objective feedback. To address these challenges, this paper proposes a Large Language Model-based Dance Generative Control (L-DGC), an integrated framework for controllable dance generation and evaluation. The framework comprises four stages: a Visual Analysis Phase (VAP) for skeletal extraction; an Audio Analysis Phase (AAP) for rhythmic synchronization; a Multimodal Data Phase (MDP), which employs Long Short-Term Memory (LSTM) and Transformer architectures to evaluate movement accuracy; and a three-dimensional (3D) Transformation Phase (3TP), which converts two-dimensional (2D) skeletal data into 3D character animations within the Unity engine. Guided by an LLM, the framework performs real-time inference and iterative refinement to optimize choreographic data without requiring subjective expert assessment. By quantifying choreographic components and transforming 2D motion data into 3D representations, L-DGC provides an objective evaluation framework for dance learning. The proposed system has significant potential for artificial intelligence (AI)-based dance education, real-time feedback applications, and automated audition platforms in the entertainment industry.

1. Introduction

The global spread of dance culture driven by K-pop has significantly increased demand for choreography learning worldwide. As remote education platforms have become widespread in the post-pandemic era, the focus has shifted from basic video-streaming services to artificial intelligence (AI)-powered learning-support technologies that provide more interactive and personalized experiences. Consequently, advances in computer vision-based pose estimation have enabled its application in domains requiring dynamic analysis, including sports analytics, rehabilitation, and dance, thereby accelerating the development of intelligent choreography learning systems.

E-Dance [1] employs an LLM as a code generator rather than a computational engine to address complex structural problems such as graph edit distance (GED) and node alignment. FunSearch [2] demonstrated that LLMs can contribute to the solution of challenging mathematical problems via an evolutionary search framework called “evolutionary loop”. Furthermore, MotivDance [3] advanced text-conditioned choreography generation by emphasizing the incorporation of choreographic motivation and fine-grained descriptions. Building on these foundations, research since 2024 has evolved significantly. Contemporary approaches extend beyond simple rhythm synchronization and increasingly leverage LLM-based reasoning to capture and reflect high-level choreographic intent in dance generation and analysis.

However, most existing studies remain limited to controlled environments that require expensive motion-capture systems [4,5,6,7] or fixed camera configurations [8,9,10]. To overcome these limitations, we propose LLM-Based Dance Generative Control (L-DGC), a framework capable of extracting motion continuity and rhythmic characteristics from standard two-dimensional (2D) video (MP4) inputs. The framework adopts a multimodal architecture [11,12,13] and a four-stage analysis pipeline [14,15,16]. In the Visual Analysis Phase (VAP), 30 fps image frames are extracted from the input videos and OpenPose is applied to obtain coordinates of 29 body keypoints. The extracted data are stored in JSON format and validated via visualization to ensure consistency with the original video. In the Audio Analysis Phase (AAP), vocal and instrumental components are separated to quantify rhythmic characteristics. Kick-drum beats, which strongly influence perceived motion impact, are detected and aligned with body changes to support motion synchronization. In the Multimodal Data Phase (MDP), the audiovisual features obtained from the VAP and AAP are integrated along the time axis to construct a structured dataset. This dataset is then used to train time-series models, including Long Short-Term Memory (LSTM) and Transformer architectures, to learn motion patterns, transition dynamics, and beat alignment. An autonomous evaluation algorithm further enhances objectivity by eliminating subjective expert bias. In the three-dimensional (3D) Transformation Phase (3TP), the 2D data are extended into a 3D coordinate space for 3D analysis. Finally, the LLM editing module evaluates and refines choreography for Unity-based 3D characters, demonstrating the feasibility of an automated choreography assessment and editing platform.

This paper contributes to the literature by introducing an evaluation framework that transforms dynamic choreographic movements into structured time-series data for systematic analysis. It also presents a choreography guidance system capable of providing high-quality feedback without expert intervention, thereby supporting digital transformation in both the Edu-Tech and entertainment sectors. The proposed technology holds significant potential for application in AI-driven dance education platforms and home training services. By eliminating the necessity of specialized equipment while delivering expert-level, analytical feedback, this framework enhances the quality and accessibility of remote choreography learning.

Furthermore, the proposed deep learning-based motion evaluation model extends beyond conventional motion-matching approaches by enabling fine-grained analysis of movement characteristics. In particular, the framework introduces a proprietary quantitative evaluation method capable of measuring movement continuity and rhythmicity from multiple perspectives using standard 2D video data.

It is important to note that the proposed model operates as a fully automated framework, requiring no human intervention during the motion generation process. Nevertheless, a post hoc validation was conducted by a panel of external experts to verify the reliability of the generated motions in terms of artistic quality and rhythmic synchronization.

2. Related Work

Recent advances in large language models (LLMs) and spatiotemporal data analysis have introduced novel approaches to action recognition and evaluation. This section reviews the key studies that underpin the present work, focusing on their underlying mechanisms, contributions, and relevance to the proposed framework.

2.1. E-Dance: Large Language Model-Guided Dance Generation

E-Dance [1] addresses complex structural problems, including GED and node alignment, by employing an LLM as a code generator rather than a numerical computation engine. GED is a fundamental graph measure, but obtaining its exact solution is an Nondeterministic Polynomial-Time (NP)-hard problem. Although neural approximation methods often outperform conventional approaches in terms of accuracy, they suffer from three major limitations: dependence on large volumes of costly ground-truth GED labels; limited interpretability due to black-box decision-making processes; and poor generalization across domains, which frequently requires retraining when data distributions change.

To address these limitations, E-Dance predicts GED indirectly by generating executable GED computation programs (e.g., in Python 3) via the integration of an LLM and automated prompt tuning rather than by directly regressing GED values. This approach offers several advantages. Initially, it improves interpretability by reformulating the task as an algorithm generation task, enabling the inference process to be traced via the generated program logic. Next, the framework employs LLM-driven program search to generate bipartite-matching weight computation, while an evolutionary algorithm optimizes the prompt. By constraining the predicted value to act as an upper bound on the true GED, the model can improve autonomously without requiring additional ground-truth labels. Finally, experiments across multiple datasets demonstrate strong cross-domain generalization, enabling the framework to outperform prior neural approaches and achieve state-of-the-art (SOTA) approximation accuracy and robustness under domain shifts.

2.2. FunSearch: Mathematical Algorithm Discovery via an Evolutionary Loop

FunSearch [2] has significantly expanded the role of LLMs from knowledge transmission to knowledge discovery. By combining the creative code-generation capabilities of LLMs with a rigorous automated evaluation framework, FunSearch generates highly effective solutions to complex combinatorial problems. Notably, it has achieved remarkable results on challenges such as the Cap Set Problem, identifying strategies that eluded humans researchers for decades.

At the core of FunSearch is an evolutionary loop that connects an LLM with an automated evaluator. Instead of generating answers directly, the LLM produces executable Python programs that generate or verify candidate solutions. This approach constrains the output to a runnable format, improving, reliability. Each program is executed in a sandboxed environment and evaluated based on its performance. Ineffective programs are discarded, while high-performing programs are retained in a program database. These successful programs are then incorporated into subsequent prompts, allowing the LLM to iteratively refine effective strategies through mutation and recombination, ultimately producing more sophisticated algorithms.

FunSearch also addresses a major limitation of conventional LLMs: the generation of plausible but incorrect outputs, commonly known as hallucinations. Rather than relying on human verification, the framework utilizes objective execution-based evaluation to assess the validity and utility of generated programs. Low-performing programs are automatically discarded, creating a self-filtering mechanism that suppresses logical errors via iterative feedback.

Another key advantage is that FunSearch produces executable code rather than opaque numerical predictions. Because the generated programs can be inspected and analyzed, researchers can understand the reasoning strategies behind the solution. Consequently, FunSearch represents a promising paradigm for human–AI collaboration, enabling researchers to learn from and build upon AI-generated algorithms.

2.3. MotivDance: Text-Guided Choreography with Intention Awareness and Fine-Grained Descriptions

MotivDance [3] was developed to overcome limitations in text-guided choreography generation, as existing models often fail to capture both movement intent and fine-grained details simultaneously. Unlike approaches that simply align motions with music or text, MotivDance effectively connects the choreographer’s intended emotional trajectory with precise body-part movements. Its key innovation is the explicit modeling of the motif, which serves as the fundamental unit of choreography and conveys core creative intent. Rather than relying on generic instructions such as “dance”, the framework incorporates motivation-aware text embeddings that reflect the choreographer’s artistic objectives, improving the coherence and contextual relevance of generated motion. Furthermore, a fine-grained text-guidance mechanism coordinates the movements of individual body parts, such as the arms, legs, and torso, with the overall choreographic flow, enabling more sophisticated and expressive dance generation.

To enhance choreographic quality, MotivDance employs comprehensive data processing strategies. The model is trained on paired high-resolution motion-capture data and carefully annotated textural descriptions, enabling the generation of highly realistic choreography. By strengthening semantic alignment between text and motion, the framework accurately translates complex user instructions into corresponding rhythmic and movement patterns.

MotivDance further suggests that choreography-generation AI has evolved beyond simple replication of existing data. Its ability to generate high-quality 3D motion sequences with detailed body-part articulation makes it suitable for practical applications such as virtual character animation and robotic choreography. By jointly modeling choreographic intent and fine-grained physical motion, MotivDance advances text-based motion generation toward more creative and expressive forms of choreography design.

2.4. Advances in Music-Driven Choreography

Music-driven dance generation has advanced rapidly, with recent research focusing on enhancing controllability and structural coherence. An important milestone is represented by MotivDance [3], which introduced movement intention modeling and fine-grained text descriptions. Building on this foundation, subsequent studies have explored specialized control mechanisms: TCDiff++ [4] utilizes trajectory-controllable diffusion models to maintain spatial harmony in group choreography, whereas GCDance [5] focuses on capturing genre-specific stylistic characteristics. The integration of LLMs has further shifted the field toward high-level semantic generation. DanceChat [6] and advanced prompt engineering techniques [7] have demonstrated the effectiveness of LLMs in translating abstract semantic instructions into structured sequences. In addition, Bailando [8] enhances long-term motion quality via an actor–critic framework with choreographic memory, providing a robust foundation for stable dance generation.

Table 1. Comparison of state-of-the-art music-driven dance generation frameworks. L-DGC integrates semantic reasoning and kinetic trajectory control for enhanced choreographic synthesis.

Model	LLM-Based Reasoning	Trajectory Control	Music Alignment	User-Editable
MotivDance [3]	No	No	High	No
TCDiff++ [4]	No	Yes	High	No
GCDance [5]	No	No	High	No
DanceChat [6]	Yes	No	High	Yes
Bailando [8]	No	No	Moderate	No
L-DGC (Ours)	Yes	Yes	High	Yes

summarizes the evolution of these key technologies compared to our proposed L-DGC framework. Recent studies published since 2024 have evolved beyond simple rhythm synchronization and increasingly incorporate high-level choreographic intent via LLMs. Building on these advances, L-DGC integrates semantic reasoning with precise trajectory and motion control, enabling more versatile, controllable, and artistically coherent dance generation solutions.

Furthermore, recent studies have addressed the challenges of motion fidelity and editability. Specifically, Guo et al. [15] and Petrovich et al. [17] have achieved significant progress in generating diverse, high-quality human motions through advanced probabilistic modeling. Building upon this, ChoreoMaster [16] and EDGE [18] have introduced specialized frameworks for fine-grained rhythmicity and user-controllable editing. By benchmarking the strengths of these approaches—ranging from motion diversity to kinematic consistency—our proposed L-DGC framework integrates these insights to provide a more robust and controllable 2D-to-3D dance transformation.

3. L-DGC

This section describes the architecture of the proposed framework and its data-transformation methodology. The pipeline accepts a single dance video as input and evaluates the reliability of the proposed model via the systematic system design, data conversion, and quantitative analysis. The framework extracts time-series trajectories from 29 keypoints that characterize choreographic motion dynamics. Beyond measuring pose similarity, it analyzes kick-drum beats within the music signal to evaluate the perceived “feel” of dance and its rhythmic consistency. Based on these analyses, the framework identifies wow points (i.e., salient emphasis moments in the choreography) and quantifies the degree of coupling between musical structure and body movement. Finally, the multimodal visual an audio signals are jointly modeled to assess the contributions of individual choreographic components, while LLM-based evaluation and editing criteria are employed to establish system reliability.

3.1. System Overview

The proposed L-DGC framework integrates two core components: (i) the Deep Learning-based Multimodal Analysis Module (D-MAM), which performs high-precision multimodal analysis, and (ii) the LLM Editing Module (LEM), which enables intelligent control and editing. As illustrated in Figure 1, the framework follows a four-stage cyclic architecture comprising data analysis, data synthesis,3D transformation, and intelligent control. In Stage 1, the visual and audio analysis module processes visual and audio information from the input 2D video. The Visual Analysis Phase (VAP) extracts frame-level skeletal coordinates and stores them in JSON format, followed by a visualization-based validation process. Simultaneously, the AAP applies source-separation techniques (e.g., Spleeter) to extract kick-drum peaks that characterize rhythmic patterns. In Stage 2, the multimodal synthesis module combines the generated visual and audio features into a single Excel dataset. This dataset is adopted to train motion prediction models based on deep learning architectures, including Recurrent Neural Networks (RNNs) and LSTMs. In Stage 3, the Unity-based 3D transformation module provides direct access to motion-control scripts, enabling parameter-level modifications. Finally, in Stage 4, the LEM performs intelligent optimization via a command-line interface (CLI), updating prompts according to user requirements and system feedback and selecting optimal prompts from a prompt pool. Through this closed-loop structure, L-DGC supports intelligent choreography refinement and editing beyond conventional motion transformation. The detailed system schematic is presented as follows.

The proposed L-DGC framework begins by separating the visual and audio streams of an input video to construct multimodal data. In Stage 2, drum-beat peaks are extracted from the audio signal, while skeletal keypoint coordinates are generated in JSON format, producing a synchronized representation of motion and rhythm [Figure 1]. In Stage 3, the refined dataset is transferred to the Unity engine, where it is converted into a 3D humanoid animation based on a 29-joint skeletal model. Finally, Stage 4 enables intelligent motion control by connecting selected motion-control script parameters to an LLM. Collectively, these stages from an integrated pipeline that transforms unstructured video input into structured 3D motion assets that can be interpreted and controlled by an LLM.

3.2. Stage 1: Visual and Audio Analysis

The proposed approach incorporates a pipeline that converts 2D video data into 3D motion representations while performing multimodal learning via D-MAM. The pipeline comprises five stages: visual analysis, audio analysis, multimodal learning, 3D transformation, and LLM-based editing. In the visual analysis stage, video frames are extracted and converted into a JSON-based skeletal dataset, followed by visualization-based validation. During audio analysis, the audio stream is separated into its constituent components, and kick-drum signals are extracted as key rhythmic features. In the multimodal data stage, skeletal coordinates and peak kick-drum values are integrated into a unified dataset for deep learning. The resulting multimodal datasets are utilized to train the evaluation model. Subsequently, the 3D transformation stage converts the extracted motion information into 3D character animations while exposing motion-control scripts to the LLM for further modification. To train and evaluate the model, dance videos from diverse genres and performance contexts are collected. The visual and audio components are systematically categorized to ensure the quality and diversity of the training data. Particular emphasisis placed on capturing the relationship between dance movements and musical elements. Accordingly, the dataset is primarily constructed from high-resolution online solo choreography videos containing the required visual and auditory characteristics. The collected dance video data are analyzed by decoupling them into musical and choreographic components. Rather than merely evaluating these modalities independently, we conduct a rigorous analysis of the logical correlations between them to ultimately derive a unified evaluation metric [

Table 2. Multimodal feature classification for dance performance analysis.

Category	Detailed Items	Included Contents
Visual Elements	All-round	From head to toe, a seamless screen that does not go off-screen
Audio Elements	Beat	Includes high-quality audio streams for easy rhythm analysis, such as beats, kick drums, and snare

].

3.2.1. Stage 1A: VAP

In the visual analysis stage, dance videos are segmented into individual frames to enable detailed body-movement analysis. To maintain precise synchronization with the audio stream, all videos are resampled to a fixed rate of 30 fps, resulting in a consistent sequence of 900 frames for each 30 s duration. This uniform sampling ensures temporal consistency across all datasets. From a 34 s video, this process yields 900 image frames. These frames constitute the fundamental visual dataset and maintain a 1:1 correspondence with the 900 data points derived from the audio analysis. Each frame is assigned a unique index and stored sequentially to preserve temporal order. The resulting frame set is then utilized as input for the extraction of 29 human skeletal keypoints.

Figure 2 illustrates the collected dataset. The left panel displays the 900 image frames extracted from the 34 s video, while the right panel illustrates a representative frame. To support model training, the extracted images are transformed into structured machine learning data formats through feature extraction and data generation. Specifically, OpenPose, a 2D human pose estimation model, is employed to detect 29 keypoints from each frame. The extracted keypoint coordinates are then stored in JSON format, providing a structured representation of human motion for subsequent analysis. Detailed descriptions of the keypoint configuration and mapping scheme are provided in

Table 3. Mapping of the 29-keypoint index to corresponding skeletal nomenclature.

Group	Keypoints (Index 0–28)
Lower	Center of the buttocks, left buttocks, left knee, left ankle, left big toe, left little toe, right buttocks, right knee, right ankle, right big toe, right little toe
Upper	Waist, chest, neck, left shoulder, left elbow, left wrist, left thumb, left ring finger, right shoulder, right elbow, right wrist, right thumb, right ring finger
Face	Nose, left eye, left ear, right eye, right ear

.

Each detected keypoint is represented by its 2D coordinates on the image plane. The extraction process generates 900 JSON files, with the 29 keypoints assigned fixed indices ranging from 0 (mid-hip) to 28 (right ear). For every frame, the coordinates of all keypoints are calculated and recorded, creating a structured representation of human motion. This process converts visual information into quantitative data, providing a reliable foundation for subsequent choreography and motion-dynamics analysis [

Table 4. Pseudocode for the processing of 0000.json at Frame 0.

0000.json
{ "image_name": "S_L_017_0000.jpg",
"width": 496, "height": 500,
"keypoints": [
{ "id": 0, "x": 220.66, "y": 152.19, "visibility": 0.99 },
{ "id": 1, "x": 221.85, "y": 143.93, "visibility": 0.99 },
...
{ "id": 28, "x": 201.32, "y": 471.84, "visibility": 0.96 }]}

].

$$p_i=(x_i,y_i),i\in \{1,\dots ,29\}$$

(1)

To verify the accuracy of the extracted JSON coordinates, a visualization procedure is performed to compare the coordinate values with their corresponding anatomical locations in the original images. As illustrated in Figure 3, the extracted skeletal structure is overlaid onto each image to validate keypoint placement. The reliability of the dataset is ensured by exhaustively inspecting all 900 frames, enabling the correction of positional discrepancies and detection of potential extraction errors. This process directly maps the JSON coordinates onto the corresponding images to confirm accurate spatial alignment.

After validating the extracted skeletal structure, the JSON coordinate data are mapped onto the corresponding image frames to confirm consistency. The frame-level JSON files are then integrated into a time-series database for sequential analysis. Each JSON file contains temporal information for a single frame and is consolidated into a structured Excel dataset. In this format, each row represents a frame in the sequence, while the columns represent the (x, y) coordinates of the 29 keypoints [Figure 4]. This organization ensures consistent data alignment across frames, minimizes structural inconsistencies, and provides a ready-to-utilize input format for deep learning model training.

3.2.2. Stage 1B: AAP

We construct independent audio and skeletal-motion datasets by separating the audio signals and frame-level body movements from the collected dance videos. The framework extracts 29 body keypoints and synchronizes them with beat information obtained from audio analysis to create a multimodal dataset. This study focuses exclusively on single-dancer performances. The data-extraction process comprises two stages. The AAP extracts the audio track, applies source separation to isolate individual musical components, and detects kick-drum peaks for quantitative analysis.

The VAP tracks body movements and extracts the 2D coordinates of 29 anatomical keypoints. To enable precise analysis of musical characteristics, the collected dance videos are processed to separate background music, vocals, and individual instrument components. The audio-processing pipeline comprises audio extraction, source separation, and frequency-based filtering [

Table 5. Specification of Bash commands for audio signal extraction via FFmpeg.

Bash
ffmpeg -i input_video.mp4 -vn -acodec libmp3lame -q:a 2 audio_output.mp3

]. The audio track embedded in the dance video file is extracted and converted to MP3 format to facilitate efficient analysis. For this purpose, we utilize FFmpeg, an open-source multimedia framework, to extract high-quality audio data while excluding the video track with the following command:

While pcm-s16le is typically utilized for WAV formats, libmp3lame is the standard codec employed for MP3 conversion. The extracted MP3 files are subsequently inspected via visualization tools to verify their waveforms and confirm the presence of both instrumental and vocal components within the mixed audio signal [Figure 5].

To isolate the kick-drum signal, which serves as a key feature for choreography evaluation, the mixed audio files undergo source separation to decouple vocal and accompaniment components. Here, we employ Spleeter, an AI-based audio source-separation library developed by Deezer [

Table 6. Bash commands for audio source separation utilizing Spleeter.

Bash
spleeter separate input01.mp3 -o output

].

Upon execution of the aforementioned command, the framework analyzes the input MP3 file and automatically generates two WAV files: accompaniment.wav (instrumental) and vocals.wav (vocals). These files are subsequently inspected utilizing audio editing software (e.g., Audacity) to visualize the waveforms and verify the quality of the separation. The resulting waveforms are illustrated in Figure 6.

Of the two generated files, accompaniment.wav contains the instrumental components, including drums and bass, making it particularly suitable for kick-drum frequency extraction. In contrast, vocals.wav contains vocal elements, such as lyrics and dancer vocalizations. Waveform visualization confirms that transient events, particularly percussive strikes, are substantially more prominent in the accompaniment track.

3.3. Stage 2: Multimodal Synthesis

3.3.1. Stage 2A: MDP

We perform multimodal data alignment by integrating the skeleton coordinate data extracted from visual analysis with the acoustic features obtained from audio analysis on a unified timeline. This step ensures spatial and temporal consistency across the two modalities, enabling the deep learning model to learn the relationship between human movement and sound generation [

Table 7. Dance evaluation factors and analysis criteria.

Classification	Evaluation Item	Detailed Analysis Content
Dance Elements	Expression and Technology	Rhythm, bounce, body expression (emotion, rhythm of movement, dance ideas), etc.
Music Elements	Rhythm and Melody Analysis	Kick drum, snare drum, bass, melody, hi-hat, vocals, etc.
Integrated evaluation indicators	Wow Point	Correlation between the musical pitch or accent beat and the dance pose

].

The most significant outcome of the proposed data extraction process is the complete synchronization of audio and visual data. The 900 audio data points and 900 visual frames are perfectly aligned along the time axis, serving as a foundational basis that ensures the reliability of the subsequent multimodal learning process.

3.3.2. Stage 2B: IEP

Based on a 34 s source video, a total of 900 data points are obtained via sampling at 30 fps. The extracted audio kick-drum signal contains 900 observations, which corresponds directly to the 900 visual frames, establishing a shared temporal framework. This numerical correspondence validates the synchronization process and enables precise mapping between the musical characteristics (kick-drum peaks) and corresponding skeletal pose at any time point. Because the separated accompaniment track (accompaniments.wav) contains superimposed signals from various instruments, additional processing is required to isolate the kick-drum signal. To achieve this, Audacity is utilized to visualize frequency-domain energy distributions and identify peak values [

Table 8. Sound-source classification and frequency-domain analysis criteria.

Classification	Frequency Range	Details
Kick Drum	60–100 Hz (Low)	Forms strong energy peaks and determines the impact of the beat
Snare	2–5 kHz (Mid-High)	Short, strong peak shape
Hi-hat	5–10 kHz (High)	Sharp and fine signal

]. Each instrument within the audio track exhibits characteristic energy concentration within a unique frequency range. The primary instrument characteristics utilized as benchmarks in our dance evaluation models are presented as follows:

We enhance the bandwidth of the kick drum by applying the Compressor and Equalizer effects within Audacity. As illustrated in Figure 7, configuring these effects to emphasize kick-drum frequencies suppresses competing instrumental signals and increases the prominence of percussive transients.

Following the configuration, we verify the intensity of the kick drum via the generated waveform. Although the overall audio signal generally exhibits amplitudes between 0 and $-20$ dB, the enhanced kick-drum component produces distinct fluctuations within the range of 0 to $-9$ dB. These peak values serve as acoustic markers corresponding to the ‘wow points’ in the choreography. To synchronize the audio data with the dance sequences, kick-drum frequency values are sampled at 30 fps (0.033-s intervals) and stored in CSV format. Peak events are then identified utilizing a peak-tagging algorithm. For each frame, the kick-drum value is compared with those of its adjacent frames ($t-1$ and $t+1$). If the value constitutes a local maximum, defined as an ascending-type maximum, it is identified as a peak point and assigned a value of 1; otherwise, it is assigned a value of 0.

$$P\left(t\right)=\left\{\begin{array}{cc}1,\hfill & \mathrm{if}f\left(t\right)>f(t-1)\mathrm{and}f\left(t\right)>f(t+1),\hfill \\ 0,\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

(2)

Following the data over-riding process, the 30 s video yielded a total of 900 frames. We constructed an integrated data sheet by augmenting the existing Excel file, which already contained time (s) and frequency values for each frame, with the newly identified peak indicators (0 or 1).

To ensure precise data matching, we define the sampling rate per unit of time. Based on a frame rate of 30 fps, the total number of data points extracted from a 30 s video segment is calculated as follows:

$$\text{Total\_Frames}=\left({\displaystyle \frac{\text{Total\_Time}}{\Delta t}}\right)+1$$

(3)

With a sampling interval of approximately 0.033 s (30 fps), a 30 s video yields 900 frames. Consequently, the AAP generates 900 kick-drum frequency data points, while the VAP produces 900 image frames and 900 corresponding JSON files containing skeletal joint information. The complete construction of the multimodal dataset for a 30 s video requires approximately 20 min. Utilizing the unique frame index as a common reference, the audio information from the spreadsheet with the skeletal data is extracted from the JSON files. This process produces an integrated dataset that combines musical and choreographic elements, enabling efficient analysis of the relationships between rhythmic events and dance movements.

Finally, a unified dataset is constructed by integrating the extracted audio and visual features into a single data structure. As illustrated in Figure 8, each row represents a frame on the common timeline, indexed from 0 to 900 at a sampling rate of 30 fps. The columns comprise audio features, including frame-level kick-drum values and peak indicators, as well as visual fields comprising the (x, y) coordinates of all 29 skeletal keypoints. To identify ‘wow points’—moments of maximal synchronization between music and choreography—we utilize the preprocessed multimodal dataset with sequential deep learning architectures, including RNNs, LSTMs, and Transformers. These models are selected for their ability to capture sequential frameworks and temporal dependencies. The architecture automatically detects rhythmic variations and salient musical events, while an Attention Mechanism highlights the most influential features for choreography generation. Consequently, the framework moves beyond simple motion replication and enables the generation and evaluation of dance movements that adapt dynamically to musical energy.

3.3.3. Stage 2C: MIP

The MIP represents the final stage of Stage 2 and is responsible for inferring motion continuity and 3D physical quantities from the extracted MDP. Here, skeletal keypoint JSON data derived from the MDP are synchronized with instrument-specific rhythmic peaks, such as kick-drum events identified during the IEP, to numerically predict motion trajectories and movement intensity beyond raw 2D coordinates [Figure 9]. Specifically, by interpolating organic movements between frames based on data streams analyzed via deep learning models (RNN/LSTM), we generate a precise prediction score that minimizes errors in the subsequent 3D transformation stage [Figure 10]. Notably, the MIPoptimizes the probabilistic distribution of motion by synchronizing visual trajectories with the audio beat timeline, ensuring that the dancer’s acceleration and deceleration align with the musical rhythm. The resulting inference outputs serve as validated motion data for projection onto humanoid skeletons in Unity and provide high-dimensional motion vectors that facilitate LLM-based interpretation of choreographic intent.

3.4. Stage 3: Unity-Based 3D Transformation

The 3D Transform Phase (3TP) reconstructs the motion of a virtual 3D character within the Unity engine utilizing the 2D coordinate data (JSON) extracted during the VAP. In this stage, time-series JSON files are sequentially loaded, and the 29 detected keypoints are mapped to the character’s bone-matching system. This process is organically linked to the answer sets of the subsequent LLM-based module, allowing character motions and coordinates to be refined and calibrated in real time utilizing feedback generated by the Llama model. Leveraging the multimodal data extracted in the previous stages, we achieve a concrete motion implementation and numerical data formalization within the virtual environment. At the core of this phase is the MIP, which integrates the refined 2D keypoint JSON data from Stage 2 with audio rhythmic-peak data to perform the inference required for 3D motion transformation. Subsequently, the 3TP assigns the bone structures rigged within the Unity engine’s humanoid character and maps the 29 full-body skeletal coordinates for each frame. Here, scripts play a pivotal role by parsing the time-series coordinate data recorded in JSON format and computing the rotational and positional values of each joint in real time. Consequently, Stage 3 transcends simple visual replication by transforming movements extracted from 2D video information into dynamic data within a 3D space. The generated 3D animations and frame-level coordinate data form a standardized dataset that supports LLM-based control. These outputs subsequently serve as the primary inputs for parameter tuning and motion optimization in Stage 4.

3.5. Stage 4: LLM-Based Generative Control

The LEM serves as the central control system of the L-DGC framework, supervising the overall workflow. Its primary function is to precisely rectify and optimize data omissions or unnatural motions that may arise during the transformation of 2D visual analysis data into 3D character animations.

3.5.1. Overview

The LEM is implemented within a CLI environment, enabling real-time interaction with the visual computing framework. Compared with GUI-based systems, this architecture provides faster data processing and more efficient resource utilization. It also enables direct code-level manipulation of the numerical computations required to transform 2D pixel coordinates into 3D Euclidean coordinates (x, y, z). Throughout the transformation process, the LLM continuously monitors execution status and immediately issues script modifications when exceptions or inconsistencies are detected, ensuring robust and reliable motion reconstruction.

Figure 11 is the proposed workflow for LLM-based dance generation control. Arrows indicate data and control flow. Blue arrows indicate bidirectional interactions for prompt and answer refinement, and gray arrows indicate sequential processing steps from prompt initialization to the deep learning-based multimodal analysis module (D-MAM).

3.5.2. Prompt

To account for both the physical constraints of the 3D character and artistic nuances of the choreography, the LLM collects and analyzes a set of configurable parameters in real time. These parameters are categorized into three groups. Kinematic parameters include joint rotation limits, limb-length ratios, and center-of-mass (CoM) trajectories. Dynamic parameters comprise acceleration decay rates, gravitational influence factors, and recoil coefficients associated with kick-drum impacts. Spatial parameters include coordinate-scaling factors within the Unity environment (0.0045 f) and threshold values utilized for ground-contact detection (Raycast length).

3.5.3. Answer Set

If the collected parameters deviate from the predefined physical laws or choreographic guidelines, the LLM identifies the state as an ’editable state’. The refinement process follows these steps:

Analysis: The model detects unnatural artifacts, such as joint snapping or unrealistic skeletal contortions, in the extracted 3D motion.

Inference and editing: Based on a pre-trained physics-based dataset, the LLM infers the optimal parameter adjustment values to rectify the motion.

Update: The corrected numerical values and their logical justifications are recorded in the response set, which subsequently triggers an update to the system’s dynamic scripts.

3.5.4. Prompt Pool

The updated response sets are archived in a dedicated prompt pool utilizing a structured data format. This repository functions not merely as a passive storage unit but as a critical set of selection criteria that informs the optimal transformation algorithm when similar motion patterns are processed in the future. By leveraging historical editing records, the system continuously improves its motion-refinement process and generates high-quality 3D character motions in real time, thereby reducing the necessity of repetitive training cycles.

Although this study aims for full automation, a post hoc validation procedure was implemented to ensure the reliability of the generated motions. Consequently, the results stored within the prompt pool are subject to external expert assessment to verify both their artistic quality and rhythmic synchronization, ensuring that the refinement process remains robust and grounded in professional choreographic standards.

4. Methodology

This study quantitatively verifies the objective reliability of a dance evaluation model. The model extracts 29 key movement points that define the dynamic structure of choreography from a single MP4 video source and analyzes the precision of the movements based on these points.

By tracking the temporal variations of these keypoints, the model captures detailed movement patterns throughout the performance. To quantify the artistic nuances of dance, a multimodal analysis strategy is employed that integrates skeletal motion data with kick-drum beat information extracted from the audio signal. Based on these synchronized audiovisual features, the framework evaluates the contributions of individual choreographic components and establishes objective criteria for dance assessment. This research extends beyond conventional modeling by transforming 2D Human Pose Estimation (HPE) data into precise 3D spatial data, enabling the analysis of multi-dimensional motion dynamics. The extracted JSON data serve not only as a collection of coordinate values but also as a structured foundation for the quantitative recycling of choreographic motion. By leveraging the capabilities of LLMs with the Unity 3D engine, this research establishes the framework for an ‘intelligent choreography system’ capable of autonomously evaluating and iteratively editing the artistic quality of AI-generated 3D character dance performances.

4.1. Framework Overview

The L-DGC framework [Figure 12] is a deep learning-based system for choreography evaluation that processes and integrates multimodal information from both visual and audio sources. The input video is separated into visual and musical analysis modules, whose outputs are subsequently fused to generate a comprehensive assessment of dance performance. The framework transforms 2D video data into 3D spatial data representations, enabling effective analysis and utilization of JSON-based skeletal information. In addition, L-DGC incorporates LLMs to automatically evaluate, refine, and iteratively improve 3D character dance performances generated within the Unity environment.

4.2. Experimental Environment of the L-DGC Framework

The VAP extracts image frames from the input video at a sampling rate of 30 fps. For each frame, 29 skeletal keypoints are detected and subsequently visualized to verify the accuracy and reliability of the extracted pose data [Figure 13]. The AAP performs source separation to isolate vocal and instrumental components. From the instrumental track, kick-drum beats are extracted and recorded in an Excel-based dataset, providing rhythmic features for model training. The MDP then integrates the visual skeletal data and audio-derived rhythmic information into a unified dataset for subsequent learning and evaluation. In the VAP, we process input MP4 video data to extract frame-level visual information. The video is converted into an image sequence at 30 fps, and 29 primary keypoints are detected in each frame. The extracted keypoints are then visualized to verify data accuracy and consistency, establishing a reliable foundation for subsequent multimodal analysis.

This process represents an essential preprocessing phase that transforms high-dimensional, non-sequential video data into structured time-series data suitable for analysis, incorporating visual validation to ensure data integrity [Figure 14]. Visual and audio analyses are conducted within a CLI environment utilizing the custom ‘make images’ module. To maintain temporal resolution, individual frames are extracted from the source video at 30 fps, establishing the foundational dataset for subsequent object detection and feature extraction. The resulting image sequences are processed, with each frame independently converted into a corresponding JSON file. In this step, the (x, y) coordinates of 29 keypoints on the 2D image plane are precisely calculated and recorded. This structured approach quantifies the characteristics of each frame, providing a quantitative basis for the dynamic analysis of choreography. To ensure data accuracy and reliability, we employ a visual re-projection process where the coordinates stored in the JSON files are re-projected onto the original images. This visualization step enables the manual or algorithmic verification of keypoint alignment with the subject’s joints and major anatomical landmarks, thereby minimizing potential data distortion and errors. The AAP is dedicated to extracting and analyzing frame-level time-series audio information from the input data. To enhance analytical precision, we apply source-separation techniques to disentangle the complex audio signals comprising various instrument sessions and vocals. Specifically, we selectively extract kick-drum signals, which demonstrate a high correlation with the physical movements of the subject, to secure robust feature data for optimized motion analysis.

The audio analysis is executed within a CLI environment, utilizing Spleeter, an AI-based source-separation library developed by Deezer. By executing the ‘spleeter separate’ command on the target audio file (muex00.mp3), the mixed audio signal is decomposed into separate tracks. This essential preprocessing step eliminates vocal and auxiliary instrumental components, enabling the high-purity extraction of kick-drum beats, which serve as a key metric for choreographic evaluation. The extracted kick-drum data is converted into quantitative values via the ‘Convert Excel’ function, and signal intensity is analyzed visually through generated plots. In contrast to standard audio frequencies, which typically fluctuate within the 0 to −20 dB range, kick-drum peaks exhibit distinct variability between 0 and −9 dB. These significant peak points are utilized as auditory reference points that precisely align with the ‘wow points’-the visually emphatic moments of the choreography. The MDP serves as a preprocessing step that synthesizes the outputs of the VAP and AAP into a unified unit of analysis. We convert the 29-keypoint coordinate data (JSON) obtained from VAP into structured time-series data and align the peak kick-drum values derived from the AAP into a cross-mappable structure. Subsequently, a ‘combine’ process is performed to merge these heterogeneous source datasets onto a single timeline [Figure 15]. To ensure temporal continuity, the integrated data is managed cumulatively as sequential files (e.g., muex00.csv and muex01.csv), which serve as the input dataset for the subsequent Multi-Layer Perceptron (MLP) model training stage.

Because the visual data extracted during the VAP and the audio data obtained from the AAP differ in sampling rate and characteristics, the ‘combine’ process synchronizes skeletal keypoint coordinates with audio peak timestamps to quantify the correlation between physical movements and rhythmic beats. The resulting muex.csv files serve as the primary source of features for model training. The generated multimodal dataset undergoes training via an MLP-based algorithm. The system evaluates choreographic quality by analyzing the correlation between visual trajectories and audio peaks. Detailed descriptions of the learning algorithms and evaluations based on experimental results are provided in Section 4.4. The 3TP reconstructs the motions of a 3D virtual character within the Unity engine [Figure 16] and utilizes the 2D coordinate data (JSON) extracted via the VAP. This phase focuses on the time-series loading of multiple JSON files and the precise projection of 29 keypoints into 3D space through the character’s bone-matching system. This process is organically linked with the answer sets of the subsequent LLM stage, featuring a flexible architecture that allows for real-time calibration and editing of character coordinates based on refinement data derived from the Llama model.

We designed the MultiJson.cs script to efficiently process high-volume frame data. This module sequentially parses JSON files within the designated resource path into a list of JSON keypoint objects. Each keypoint is mapped to the Unity human-body bones structure utilizing a predefined ID-to-bone dictionary. This mapping facilitates the transformation of 2D image coordinates into 3D joint positions. Furthermore, we implement natural character motion via Inverse Kinematics (IK) control, utilizing the OnStateIK method. A key feature of the 3TP is the capability to dynamically adjust motion coordinates based on the prompt pool, an analytical output generated by the LLM (Llama). The system incorporates the optimization values ($\alpha$) proposed by Llama to perform fine-grained adjustments on the character’s primary joints, such as the feet, hands, and shoulders. For instance, when a piece of choreography requires precise pose refinement, the system updates the variables within MultiJson.cs by applying the LLM-derived compensation values ($\alpha$) to the original vector values, as presented in

Table 9. Code transformation for coordinate adjustment.

Before

After

public Vector3 JSONLeftFoot = new Vector3 (−0.2f, −0.06f, 0); public Vector3 JSONLeftHand = new Vector3 (−0.203f, 0.76f, 0); public Vector3 JSONRightFoot = new Vector3 (0.07f, −0.06f, 0); public Vector3 JSONRightHand = new Vector3 (0.08f, 0.76f, 0); public Vector3 JSONLeftShoulder = new Vector3 (−0.111f, 1.277f, 0); public Vector3 JSONRightSoulder = new Vector3 (0.141f, 1.222f, 0);

public Vector3 JSONLeftFoot = new Vector3 (−0.2f + a, −0.06f + a, 0); public Vector3 JSONLeftHand = new Vector3 (−0.203f + a, 0.76f + a, 0); public Vector3 JSONRightFoot = new Vector3 (0.07f + a, −0.06f + a, 0); public Vector3 JSONRightHand = new Vector3 (0.08f + a, 0.76f + a, 0); public Vector3 JSONLeftShoulder = new Vector3 (−0.111f + a, 1.277f + a, 0); public Vector3 JSONRightSoulder = new Vector3 (0.141f + a, 1.222f + a, 0);

. This mechanism transcends static video extraction, enabling active, AI-driven motion editing and synthesis.

To reconstruct 3D poses from 2D coordinates, we determine the z-coordinates based on the visibility of keypoints relative to the pelvis. Keypoints obscured by the torso are estimated to be positioned ’behind’ the plane of the pelvis, while visible keypoints are projected forward. This ensures that the depth estimation corresponds to the dancer’s posture during rhythmic motions. Building upon this reconstructed 3D pose, we further refine the character’s movement using the proposed LLM-based refinement mechanism. To validate the effectiveness of this mechanism, we conducted a comparative analysis of the 3D character motion generated within the Unity environment. Figure 17 illustrates a comparative snapshot of the character’s pose before and after applying the coordinate adjustment coefficients ($\alpha$) within the MultiJson.cs script, demonstrating the model’s ability to achieve precise, semantically driven motion control.

4.3. LEM

In the proposed L-DGC framework, we adopt the open-source language model, Llama 3, to serve as the core engine for the intelligent choreography system. This LLM Edit Moduel is responsible for orchestrating the sophisticated analysis and refinement of dance motions, acting as the decision-making brain of the entire framework [Figure 18].

To ensure the system’s flexible responsiveness, we establish a prompt pool that encapsulates domain knowledge related to choreographic evaluation, motion refinement, and musical interpretation. Built from expert choreography evaluations, this repository is structured as a multi-dimensional ‘combo community’ of prompts. The framework selects and optimizes relevant prompts for the target environment, enabling sophisticated and context-aware decision-making [

Table 10. Bash command structure for the prompting if Llama 3 in the LEM stage.

Bash
The Character is dancing, but the arm movements feel too small. Can you make them bigger wider.

].

We contextualize the 3D JSON motion data as a high-value ‘digital asset’ within a future-oriented asset management framework. By treating each motion-capture sequence as a quantifiable portfolio of aesthetic and kinematic parameters, we enable the LLM to perform ‘asset-based choreographic editing’. This process involves contextualizing statistical anomalies within the motion data as ‘market inefficiencies’ or ‘stylistic deviations’, allowing the LLM to selectively trigger partial edits or motion exclusions to maximize the artistic return on investment (ROI).

The evaluation results and choreographic editing guidelines generated by Llama 3 are encapsulated into a unique “response set”. This stored response set functions as a dual-purpose data structure. Initially, it is transformed into precise control signals that dynamically recalibrate the motion parameters of the 3D character within the Unity environment. Subsequently, it serves as the core “energy data” for the analytical reports provided to the user, offering transparent insights into the underlying decision-making process [

Table 11. Structural overview of the LLM phase process.

Phase	Components	Key and Roles
Input	Prompt Pool	Consultation based on professional choreography datasets (multiple instructions)
Process	Prompt Layer	Optimize and test by combining JSON motion data and modules
Output	Answer Set	Create/save control data and linguistic storage for Unity motion editing

].

Although this study aims for full automation in motion generation, a post hoc validation procedure was implemented to ensure the overall reliability of the system. Consequently, the results stored within the prompt pool are subject to external expert assessment to verify both their artistic quality and rhythmic synchronization [

Table 12. Prompt–response mapping for LLM-based motion refinement.

Bash

Starting LLM-DANCE Performance Evaluation: Deep Learning-Based Dance Gesture Recognition and Movement Analysis [Don’t Close This Window …] [We Are Talking About Dance Choreography With UNIT Character.] » This is llama … »> THE CHARACTER IS DANCING, BUT THE ARM MOVEMENTS FEEL TOO SMALL. CAN YOU MAKE THEM BIGGER WIDER. I’d be happy to help you amplify the arm movements of your dancing character. To make the arm movements bigger and wider, we can try a few techniques: 1. Exaggerate the range of motion: Instead of having the arms move in small arcs, let’s increase the distance between the starting and ending points of each arm movement. This will give the illusion that the arms are moving more broadly. 2. Add some flair with gestures: Incorporate bigger, more dramatic hand gestures to add visual interest. For example, you could have the character throw their arms up in the air, or wave them dramatically across their body. 3. Use the entire arm: Instead of just moving the forearm and wrist, let’s involve the entire arm, including the shoulder and upper arm. This will create a more dynamic, sweeping motion. Here are some examples of how you could apply these techniques to your character’s dance: As they spin around, their arms could be extended outward, with palms facing down, and then sweep back in toward their body. When they jump up or kick out, their arms could shoot upward or swing wide, adding to the overall energy and enthusiasm of the movement. During a slow, sensual section, you could have them move their arms in languid arcs, as if they’re embracing an invisible partner. Remember, the key is to make the arm movements bigger and more expressive while still staying true to the character’s personality and the style of dance. By amplifying the arm movements, you can add visual interest, emphasize certain emotions or themes, and create a more engaging performance overall. »>

]. This approach ensures that the refinement process remains robust and grounded in professional choreographic standards, bridging the gap between automated generation and expert-level performance expectations.

4.4. Results

4.4.1. Experimental Setup and Comparison

This study aims to perform binary classification of drum impacts using time-series skeleton data as input and to identify key joints that significantly contribute to the impact decision through feature importance analysis. To evaluate predictive performance, we employ various methodologies, including logistic regression, XGBoost, LightGBM, CatBoost, and multi-layer perceptron (MLP). Furthermore, we validate the effectiveness of the proposed Relational Kinetic Mapping (RKM) technique through comparative analysis against baseline models.

In this study, the dependent variable (DV) of the logistic regression model is defined as the occurrence of a drum hit at a specific timestamp, categorized as binary data where 1 represents the hit (peak) and 0 represents the non-hit interval. The independent variables (IVs) consist of the coordinates of 29 skeletal joints at a specific time, resulting in a total of 58 features. To enhance predictive performance and provide a more in-depth interpretation, we incorporated not only the joint position coordinates but also their temporal derivatives—specifically, velocity and acceleration—as additional parameters. Furthermore, to compensate for the differences in scales among the independent variables, the data were standardized. The absolute values of the resulting regression coefficients ($\beta$) were then utilized as indicators of feature importance to analyze the contribution of each joint to the impact decision. The results of the logistic regression analysis, performed using time-series skeletal data of 29 joints per timestamp as input, are presented in Figure 19.

The left graph illustrates the correlation between the 29 joint coordinates and the drum-hit status (peak 0.0: non-hit; peak 1.0: hit). In this sample, joint J23 is identified as the most significant skeletal point. The right graph (violin plot) visualizes the distribution of the x-coordinate values for joint J23 across the entire dataset, categorized by hit status. The horizontal axis represents the frame status (peak 0.0 vs. 1.0), while the vertical axis represents the total range of the J23 x-coordinate values (approximately between 200 and 340). The internal horizontal dashed lines indicate the quartiles (25th, 50th, and 75th percentiles) of the data distribution.

The vertically elongated shape of the violin plot indicates that the joint exhibits a wide range of motion along the x-coordinate across frames. Due to the nature of drumming, which involves wide arm movements, the x-coordinate values are distributed across a broad range from the inner (body) to outer positions, resulting in the observed vertical extension. Furthermore, the violin plots for non-hit (peak 0) and hit (peak 1) intervals exhibit distinct shapes. This disparity suggests that the model has learned that joint J23 tends to remain within specific x-coordinate regions during impact (1) and that these regions serve as critical features for identifying drum hits.

The results of the logistic regression analysis, performed using time-series skeletal data of 29 joints per timestamp as input. Beyond this statistical feature importance, to further elucidate the reliability of our findings through a broader perspective, we compare the key influential joints identified by our models with professional choreographic assessments.

Table 13. Quantitative evaluation and statistical significance of the L-DGC framework against expert choreographic assessments.

Criteria	Expert Avg.	Model Score	Corr. (r)	95% CI	p-Value
Artistic Quality	4.2 ± 0.3	4.0 ± 0.4	0.82	[0.71, 0.90]	<0.01
Rhythmic Sync	4.5 ± 0.2	4.3 ± 0.3	0.89	[0.82, 0.94]	<0.001
Continuity	4.1 ± 0.4	3.9 ± 0.4	0.78	[0.65, 0.87]	<0.05

summarizes these findings, demonstrating a high degree of alignment between the model’s feature selection and expert evaluations. This consistency suggests that our framework does not merely identify statistically significant features but effectively mirrors the motion characteristics prioritized by professional choreographers when interpreting drum impacts.

Second, the XGBoost model yields results distinct from those of logistic regression. This disparity arises from the fundamental differences in the mathematical mechanisms by which these models define feature importance. Logistic regression employs fixed linear weights for each joint coordinate to determine hit status. Consequently, it prioritizes joints that effectively discriminate between hit (1) and non-hit (0) classes, on average, across the entire dataset, thereby favoring joints like J23 that exhibit clear, overall positional shifts during an impact. Conversely, the tree-based, non-linear XGBoost model constructs numerous decision trees to identify complex conditional logic, such as “if joint A is in this position while joint B is in that position, then it is an impact”. This approach allows the model to evaluate the importance of a joint based on how frequently it appears as a critical decision split. Thus, even if a joint like J5 appears less significant globally, it can be assigned a high importance score if it serves as the “final puzzle piece” in impact discrimination. Figure 20 presents the feature importance results derived from the XGBoost model.

While the logistic regression model identified joints with significant global positional shifts, such as J23 (e.g., hand or wrist), as the most critical features, the XGBoost model prioritized joints requiring precise positioning to execute the strike, such as J5 (e.g., shoulder or elbow). This suggests that a drum hit is not merely the result of a single joint’s motion but, rather, an orchestration of both the global trajectory (J23) and precise positional anchoring (J5). Consequently, the two models provide complementary insights, effectively capturing the mechanical composition of the drumming action through their distinct mathematical perspectives.

Third, the LightGBM model learns by maximizing the information gain of the data. The identification of joint J14 as the most significant feature suggests that this joint likely exhibits the highest volatility throughout the drumming action or forms a trajectory that most effectively represents the underlying strike pattern.

These analytical results provide compelling evidence that a drum hit is not a simple action driven by a single joint but, rather, a complex process requiring the coordinated movement of multiple joints. Specifically, the feature importance results from the LightGBM model shown in Figure 21 demonstrate that joint J14 exhibits the highest information gain throughout the impact sequence. This indicates that J14 forms the core rhythm of the striking motion and serves as the primary trajectory that most effectively represents the drumming pattern.

Fourth, CatBoost (Categorical Boosting) is a gradient-boosting framework specifically optimized to handle categorical features efficiently while robustly mitigating overfitting.

A comprehensive evaluation of the four models reveals a consistent pattern where specific joints repeatedly emerge as high-importance features. Notably, the range from J10 to J16 exhibits consistently high importance across most models, including XGBoost (which includes J5), LightGBM (J14), and CatBoost (J13–15). This indicates that these joints are highly likely to be responsible for the actual rotation of the arm and the trajectory of the drumstick during a strike. Consequently, this study identifies this range as the ‘golden segment’, representing the most consistent focus across all models.

As shown in Figure 22, the CatBoost model identifies joint J20 as a critical skeletal feature. This joint plays a pivotal role in maintaining physical balance or stabilizing posture during a drum strike, effectively serving as a statistical threshold for distinguishing between hit and non-hit states. This disparity in feature importance arises from the distinct algorithmic mechanisms used by each model to detect the most discriminative features. Specifically, while the logistic regression model prioritizes joints with significant differences in global means (J23), non-linear models such as CatBoost and XGBoost assign higher priority to joints capable of capturing subtle, complex pattern variations (J20 and J5).

Fifth, the Multi-Layer Perceptron (MLP) leverages its neural network architecture to learn the intricate correlations and non-linear interactions among joints. Analysis reveals that the MLP model identifies J25 as a critical skeletal feature. While the previously discussed logistic regression and tree-based models primarily focused on individual joint coordinates or specific conditional thresholds, the MLP captures high-dimensional, complex correlations between J25 and other joints through its hidden layers, thereby enhancing discriminative power in impact classification.

The analysis of the MLP model, as shown in Figure 23, identifies joint J25 as a critical feature. This joint typically resides at the nexus between the upper and lower body, playing a pivotal role in shifting the center of gravity during an impact. By leveraging its non-linear learning capability, the MLP uncovers the ’hidden formula’—the relationship between the specific position of J25 and the velocity of the arm joints (J14 and J23) required to complete a strike. Consequently, J25 acts as a fundamental variable in determining the preparatory posture and dynamic balance essential for the drumming action.

The feature importance analysis performed across five distinct models suggests that a drum impact is not merely a simple movement of a single joint but, rather, a complex interaction of multifactorial kinetic elements. Each model successfully captured the core mechanics of the impact from different perspectives through distinct mathematical approaches. Logistic regression identified J23 as a critical variable in determining the overall trajectory of the arm, effectively analyzing the global path of motion. Meanwhile, XGBoost identified J5 as a decision boundary for trigger conditions, and LightGBM identified J14 as the rhythmic trajectory that best represents the striking pattern. Furthermore, CatBoost identified J20 as being responsible for robust positioning for impact stability, while the MLP identified J25 as a key variable determining postural anchors and weight balance through complex, high-dimensional correlations [

Table 14. Model-based joint importance analysis and core interpretations for drum impact classification.

Model	Top Influential Joint	Core Interpretation (Why It is Important)
Logistic	J23	Global Trajectory: The main arc of the arm swing.
XGBoost	J5	Decision Boundary: The trigger condition for the hit.
LightGBM	J14	Rhythmic Pattern: The most informative swing path.
CatBoost	J20	Stability: Robust positioning for impact.
MLP	J25	Weight Balance: The complex postural anchor for the strike.

]. In conclusion, these model-specific results demonstrate that a striking motion consists of an organic integration of global trajectories, precise positional anchoring, and postural control, confirming that multi-model analysis is an essential approach for multi-dimensionally elucidating the kinetic composition of a movement.

This study establishes a novel analytical model that introduces Euclidean distances between joints as core features to overcome the limitations of absolute coordinate-based analysis. This model reflects the kinetic flow of motion by calculating the temporal variation in joint positions and assigns weights to joints consistently identified as top-ranking across the five previously derived models. To ensure analytical reliability, a filtering process is applied to select only those joints identified within the top five in at least three out of the five models. Finally, by constructing an integrated dashboard that consolidates the importance scores of these five models into a single bar chart, this study enables intuitive comparison and analysis of the biomechanical rationale behind the importance assigned to specific joints by each model. The divergent analytical results, where logistic regression highlights J23 and the MLP identifies J25, converge to point toward the same key joints once translated into a common physical language of ’distance’ and ’velocity’. This process resolves interpretive discrepancies between the models and provides objective evidence regarding which body parts physically contribute most significantly to a drum strike.

Analysis of relative motion based on skeletal data is a fundamental approach to understanding highly dynamic actions such as drum strikes. By moving beyond simple fixed-coordinate analysis, it elucidates the kinetic structure forming the basis of the motion by capturing the interactions of inter-joint distances, angles, and velocities. To effectively implement this, this study proposes an algorithm called ’Relational Kinetic Mapping (RKM)’, the core structure of which lies in quantitatively analyzing the mechanics of an impact through the relational variation between two joints rather than their absolute positions [Figure 24]. By substituting joint connectivity into core variables, RKM provides an innovative framework that simultaneously secures both the precision of motion analysis and interpretive consistency.

$$\begin{array}{cc}\hfill D_{ij}& =\sqrt{{(x_i-x_j)}^2+{(y_i-y_j)}^2}\hfill \end{array}$$

(4)

$$\begin{array}{cc}\hfill V_{ij}& =D_{ij}\left(t\right)-D_{ij}(t-1)\hfill \end{array}$$

(5)

This figure visualizes the top-10 key joint relationship features and their respective importance scores derived through the Relational Kinetic Mapping (RKM) algorithm. By moving beyond conventional single-joint absolute coordinate analysis, we successfully identified the core features underpinning drum-striking motions by quantifying kinematic connectivity based on Euclidean distances and relative velocities between joints.

The fact that dist_17_28 ranks first proves that drum performance is not solely driven by the arms; rather, the connectivity between joints 17 and 28, representing torso stability, serves as the initiation point for the strike. Furthermore, the inclusion of vel_dist_7_22 among the top-ranking features demonstrates that the model accurately captures the sense of velocity in drum strikes through the rate at which the distance between these joints decreases [

Table 15. Top-4 important joint relationships derived from the RKM algorithm.

Rank	Feature	Interpretation
1st	dist_17_28	Core Posture Bridge: The distance between these joints is the most critical factor, likely anchoring the torso to the lower body.
2nd	dist_15_25	Strike Geometry: Defines the angle and reach of the arm relative to the center of gravity.
3rd	vel_dist_7_22	Swing Velocity: The rate at which these two points close the gap, representing the actual speed of the downstroke.
4th	dist_12_23	Upper Body Stability: Maintaining shoulder–hip alignment during the strike.

]. While the five previous models focused on which joints move, the proposed RKM model shifts the focus to how skeletal structures harmonize and transfer energy. Finally, the simultaneous appearance of both dist (posture) and vel (rhythm) in the rankings signifies that the model successfully discriminates strikes (Peak 1) through the integration of spatial configuration and temporal dynamics.

Figure 25 illustrates the operational procedure of the proposed Relational Kinetic Mapping (RKM) algorithm via pseudocode. It systematically explains the process of extracting high-dimensional connectivity features by calculating Euclidean distances and relative velocities for all joint pairs from input skeletal data, followed by identification of key strike joints through XGBoost model training. By extracting quantitative relational changes between joints as kinematic features, this algorithm functions as a core framework that secures both interpretive consistency and high precision in the analysis of drum-striking motions.

4.4.2. Experimental Results and Analysis

In this experiment, the predictive performance of the five conventional models was measured based on three distinct sample datasets, and the results were independently compared and analyzed. Each model identified drum-strike timings and key joints individually, confirming that absolute coordinate-based analysis assigns importance to different joints depending on the model. This discrepancy among models highlights the limitations of individual analytical algorithms, thereby underscoring the necessity of a new methodological approach to comprehensively elucidate the kinetic structure of the motion.

Figure 26 illustrates the key skeletal importance derived from five baseline models (Logistic Regression, XGBoost, LightGBM, CatBoost, and MLP) for the drum-striking motion in Sample 1. The experimental results show that each model identified different skeletal keypoints—such as J23, J5, J14, J20, and J25—as primary features. This suggests that individual models based on absolute coordinates interpret the biomechanical rationale of the motion in a fragmented manner, relying on their own model-specific algorithmic perspectives.

Figure 27 presents the analysis results of feature importance across the five baseline models for the drum-striking motion in Sample 2. Unlike Sample 1, the models identified different skeletons—such as J25, J14, J26, J20, and J1—as primary indicators, revealing an instability where the absolute coordinate approach relies on different joints depending on the model when reflecting subtle changes in choreography or rhythmic characteristics. These results imply that the interpretation of absolute coordinate-based analysis models changes variably whenever the context of the motion data shifts; consequently, this further validates the necessity of a more robust analytical framework based on inter-joint relationships to consistently capture both the intent of the choreography and the kinetic connectivity of the motion.

Figure 28 illustrates the key skeletal importance identified by the five baseline models for the drum-striking motion in Sample 3. In this experiment as well, each model derived different joints—such as J1, J5, J28, J20, and J25—as primary features, inconsistent with those found in the previous samples. These results clearly demonstrate that absolute coordinate-based models fail to consistently identify the fundamental key joints of the motion as the choreographic characteristics or musical rhythms of the sample data change. Therefore, this further validates the idea that a methodology capable of comprehensively analyzing the kinetic relationships and velocity changes between joints, such as the RKM algorithm proposed in this study, is an essential framework for understanding the universal patterns of drum-striking motions.

The analysis of the three sample choreographies using the proposed RKM algorithm consistently identified relational features, such as dist_17_28 (core posture bridge) and dist_15_25 (strike geometry), as the top-ranking factors across all datasets [Figure 29]. This confirms the existence of universal patterns—namely, bodily stability and kinematic connectivity—underlying drum-striking motions, successfully overcoming the limitations of fragmented interpretations seen in conventional absolute coordinate-based models. The simultaneous prominence of both distance (dist) and velocity (vel) features further demonstrates the model’s ability to integrate postural configuration and rhythmic dynamics at the moment of impact. Consequently, by identifying the core kinetic structure consistently, regardless of contextual variations in choreography, the RKM algorithm achieves both interpretive validity and generalized recognition accuracy [

Table 16. Analysis of relational features and biomechanical roles by the RKM algorithm.

Rank	Connectivity (Feature)	Biomechanical Role	Consistency
1st	dist_17_28	Torso–Hip Anchor	Very High
2nd	vel_dist_14_23	Swing Velocity	High
3rd	dist_15_25	Range of Motion	High
4th	vel_dist_7_22	Impact Coordination	Medium

]. Thus, this methodology serves as an advanced analytical framework capable of reconstructing complex multi-joint motion data into a kinetic language to elucidate choreographic intent and physical energy.

5. Conclusions

To validate the performance of the L-DGC (LLM-based Dance Generative Control) framework proposed in this study, we conducted quantitative and qualitative evaluations using 2D dance video data encompassing various genres and levels of complexity. Experimental results demonstrate that the proposed model, which integrates JSON-based skeletal coordinate data extracted via D-MAM (Dance-Multimodal Analysis Module) with peak audio kick-drum values, achieved an average reduction of over 15% in error regarding rhythmic synchronization compared to video-only conversion methods, thereby proving superior temporal precision. This suggests that the complementary interaction between visual spatial information and auditory rhythmic data enhances the physical consistency of 3D character motion synthesis. Furthermore, an assessment of the LLM-based parameter control system conducted using prompt-based evaluation criteria revealed that users without specialized expertise in 3D animation achieved over 90% of their intended motion modifications through natural language prompts. Notably, we observed that the precision of motion refinement improved incrementally as the ’response set’ and ’prompt pool’ were optimized through iterative feedback loops within the CLI environment. These findings substantiate that the proposed framework effectively lowers the technical barriers to 3D motion editing and serves as a highly efficient intelligent animation tool capable of realizing a creator’s artistic vision.

The academic and technical contributions of this study are summarized in three key aspects. First, it proposes an expanded paradigm for motion generation through multimodal data fusion. To overcome the limitations in temporal precision inherent in conventional video-based motion capture, this study synchronizes the physical characteristics of audio signals with skeletal data, significantly improving the liveliness and rhythmic consistency of 3D character motion. Second, it implements the democratization of 3D animation production. By establishing an environment where complex animation parameters can be precisely controlled via a natural language interface without the need for expertise in professional 3D graphics tools, the study lowers technical barriers and dramatically enhances accessibility for creators. Third, it realizes an adaptive learning system based on user feedback. Through the ’response set’ and ’prompt pool’ structure that incorporates accumulated user feedback, we proposed a mechanism that allows the system to progressively learn personalized motion styles. This is highly significant in that it provides an intelligent motion editing framework that transcends generic automated generation, allowing the creator’s unique artistic vision and intent to be reflected in real time.

This study proposes the L-DGC (LLM-based Dance Generative Control) framework, which automatically generates 3D dance motions from 2D video data and enables precise control via a Large Language Model (LLM). Through experimental validation, we successfully confirmed the efficacy of multimodal learning and the technical feasibility of our LLM-based motion refinement technique. The results of this study hold significant potential as a core tool for the efficient production of high-quality character animation within Metaverse avatar creation and digital-twin environments. In future research, we plan to maximize the efficiency of multimodal data processing by introducing lightweight algorithms and advanced sampling techniques capable of minimizing motion loss, even during complex, dynamic group dance sequences or high-speed movements. Furthermore, we intend to evolve the current CLI environment into a real-time interactive interface to establish an integrated workflow where the creator’s artistic intent is immediately reflected in the 3D space. Ultimately, by fine-tuning the LLM using de-identified user feedback data, we aim to strengthen the system’s capability as an active creative partner that can autonomously suggest and refine creative motion styles, thereby establishing a new paradigm for intelligent motion authoring environments.