A Comprehensive Review on Music Transcription

Abstract

Music transcription is the process of transforming recorded sound of musical performances into symbolic representations such as sheet music or MIDI files. Extensive research and development have been carried out in the field of music transcription and technology. This comprehensive review paper surveys the diverse methodologies, techniques, and advancements that have shaped the landscape of music transcription. The paper outlines the significance of music transcription in preserving, analyzing, and disseminating musical compositions across various genres and cultures. It also provides a historical perspective by tracing the evolution of music transcription from traditional manual methods to modern automated approaches. It also highlights the challenges in transcription posed by complex singing techniques, variations in instrumentation, ambiguity in pitch, tempo changes, rhythm, and dynamics. The review also categorizes four different types of transcription techniques, frame-level, note-level, stream-level, and notation-level, discussing their strengths and limitations. It also encompasses the various research domains of music transcription from general melody extraction to vocal melody, note-level monophonic to polyphonic vocal transcription, single-instrument to multi-instrument transcription, and multi-pitch estimation. The survey further covers a broad spectrum of music transcription applications in music production and creation. It also reviews state-of-the-art open-source as well as commercial music transcription tools for pitch estimation, onset and offset detection, general melody detection, and vocal melody detection. In addition, it also encompasses the currently available python libraries that can be used for music transcription. Furthermore, the review highlights the various open-source benchmark datasets for different areas of music transcription. It also provides a wide range of references supporting the historical context, theoretical frameworks, and foundational concepts to help readers understand the background of music transcription and the context of our paper.

1. Introduction

The complex components of music have the power to evoke emotions which can communicate ideas that words alone often struggle to convey. Modern technology has opened up new channels for the expression and preservation of music. One central aspect of this evolution is music transcription, which is the process of converting auditory musical signals into symbolic representations that can be studied, analyzed, and shared. The development of music notation itself is connected to the history of music transcription, which dates back to Mesopotamia and Ancient Greece, where rudimentary notational systems were used to record melodies and rhythms [1,2,3,4]. These techniques developed over the years, eventually leading to the complex and standardized music notation that we have today [5,6,7]. Additionally, music exhibits diversity across various genres and cultures [8,9], each with its unique music notation systems. To accommodate these diversities, different transcription methods are required.

The technological advancements in recent decades have assisted in a new era of music transcription. From the pioneering work of Jean-Jacques Rousseau in the 18th century [10] to the advent of computers in the 20th century [11], numerous scholars and inventors have contributed to the development of transcription techniques. Today, with the rise of digital signal processing, machine learning, and artificial intelligence [12,13,14,15,16,17], there is a significant and substantial change occurring in the field of music transcription [18,19,20,21,22].

The aim of this comprehensive review paper is to examine an in-depth exploration of the various methodologies, techniques, and advancements that have influenced the area of music transcription. By synthesizing a wide range of research studies, we seek to offer a holistic understanding of the challenges, opportunities, and future directions in music transcription. To achieve this goal, we have organized the paper as follows.

In Section 2, we provide an overview of music transcription by tracing from frame-level to note-level, then to stream-level, and finally to notation-level transcription. Section 3 delves into the theoretical foundations of vocal melody transcription, highlighting the complexities of monophonic and polyphonic vocals in both frame-level and note-level transcription tasks. In addition to this, we also explore the various insights and advances in music transcription areas like multi-pitch estimation and single-instrument as well as multi-instrument transcription. In Section 4, we list the current challenges of music transcription methodologies into several main factors: singing techniques, instrumentation, pitch ambiguity, tempo changes, and subjectivity. Section 5 discusses various available open-source as well as commercial music transcription tools. Section 6 describes the various python libraries that are useful for music transcription. In Section 7, we explore the applications of music transcription in various fields like music education, music creation, and music production. Evaluating the effectiveness of transcription systems is a critical aspect. So, we review the established benchmark datasets in Section 8 ranging from vocal melody to general melody, from monophonic vocals to polyphonic vocals, and from single-instrumentation to multi-instrumentation. The future directions are explained in Section 9. Finally, the review concludes in Section 10, with a discussion on the broader implications of music transcription.

Throughout this review paper, we draw upon a diverse array of references encompassing music theory, signal processing, machine learning, and human–computer interaction. The contributions of scholars, researchers, and innovators from around the world have collectively shaped the trajectory of music transcription. By synthesizing these contributions, we aspire to offer a comprehensive resource that inspires further advancements and collaborations in this dynamic field.

2. Overview of Music Transcription

This section presents four different levels of music transcription: frame-level transcription, note-level transcription, stream-level transcription, and notation-level transcription. At the most fundamental level, frame-level transcription involves analyzing very short frames of audio and identifying basic sound characteristics of music such as pitch and timbre. In the next step, note-level transcription takes the information gathered from frame-level analysis and assembles it into individual musical notes. This level of transcription can identify the pitch, duration, and timing of each note in the piece of audio. The stream-level transcription looks beyond individual notes to capture the larger musical phrases. This involves recognizing patterns in the sequence of notes, identifying chord progressions, and determining the overall structure of the music like verses and choruses in a song. The highest level of transcription involves creating a formal written representation of the music notation. This includes not only the individual notes and their timing but also various musical symbols such as dynamics, articulations, time signatures, key signatures, and more. These four levels of transcription, frame-level, note-level, stream-level, and notation-level, are illustrated in Figure 1.

2.1. Frame-Level Transcription

Frame-level transcription or multi-pitch estimation (MPE) is the estimation of the pitch of notes that are simultaneously present in each time frame of the audio signal. This transcription task has been performed in each time frame independently. Figure 1a shows an example of frame-level transcription, where every dot in the x-axis represents the discrete time and the y-axis dots represent its corresponding pitch. The visualization for this transcription has been carried out using Tony [18] from the wave file (opera_male5.wav) of the ADC2004 (http://labrosa.ee.columbia.edu/projects/melody/, accessed on 5 October 2023) vocal melody dataset. Various research studies have been developed using frame-level transcription. The work in [23] introduced a method for estimating multiple concurrent piano notes, dealing with overlapping overtones by using smooth autoregressive models. It handles background noise to reduce pitch estimation errors, creating a comprehensive model for piano notes. The effectiveness of this model has been validated with real piano recordings of the MAPS [23] dataset. Similarly, the work in [24] discusses the challenge of training data for estimating multiple pitches in music. It introduces a new approach called combined frequency and periodicity (CFP), which combines two different types of features to improve the accuracy in simultaneous pitches. The result shows that CFP works well for western polyphonic music and remains effective even when dealing with audio distortions like filtering and compression. The two works described above are conventional approaches for frame-level transcription. However, recently, the neural network-based methods [12,25] have garnered increased attention from researchers. The work in [12] introduced a supervised neural network model for a polyphonic transcription task on the MAPS dataset. It consists of an acoustic model and a music language model. The acoustic model, based on neural networks, estimates pitch probabilities in audio frames, while the recurrent neural network-based language model captures pitch correlations over time. The acoustic and music language model predictions are combined using a probabilistic graphical model, with inference performed using the beam search algorithm. The experimental results demonstrate that convolutional neural network-based acoustic models outperform popular unsupervised models. Similarly, the work in [25] analyzed neural network based frame-level piano transcription by comparing four different input representations: spectrograms with linearly spaced bins, spectrograms with logarithmically spaced bins, spectrograms with logarithmically spaced bins and a logarithmically scaled magnitude, as well as the constant-Q transform. This study highlights the significance of choosing the right input representation and fine-tuning the hyper-parameters, especially the learning rate and its schedule, which can improve the accuracy of frame-level transcription. Frame-level transcription has several advantages as well as disadvantages. First, it provides a high temporal resolution which can enable the detailed representations of music at a fine time scale. Second, it is useful for in-depth music analysis where precise timing information is required. However, there are some notable disadvantages in frame-level transcription. First, it requires high computational resources due to the need to analyze audio at a very fine level, which can make it less efficient for real-time applications. Second, the fine-scale high temporal resolution generates a large volume of data, which makes it challenging to manage, store, and process efficiently due to its size and complexity. Lastly, it focuses deeply on the technical details but fails to capture the broader musical context.

2.2. Note-Level Transcription

The note-based transcription approaches directly estimates notes, including pitches and onsets. This note-based transcription is one level higher than frame-level transcription because the objective of frame-level transcription is to estimate only pitch in a particular time period. But note-level transcription needs to estimate the pitch, onset time, and offset time of a specific note in a particular time period. Figure 1b shows an example of note-level transcription. The blue rectangle shows the note events after the post-processing of frame-level transcription on a continuous pitch based on a hidden Markov model. The visualization for this transcription has also been carried out using Tony [18] from the wave file (opera_male5.wav) of the ADC2004 vocal melody dataset. The note offset in this level of transcription is ambiguous, so they are sometimes neglected during the inference time and only evaluate the pitch and onset time of a note. The work in [26,27] estimates the pitches and onsets in a single architecture. One example for this is [28], which jointly estimates the different attributes of notes like the pitch, intensity, onset, and duration. It estimates the properties of notes concurrently using harmonic temporal structured clustering. The note-level transcription can also be achieved after the post-processing in frame-level transcription. First, the fundamental frequency is estimated concurrently, and post-processing is applied to estimate the musical notes in the second step. The methods used during the post-processing steps are median filtering, hidden Markov Models, and Neural networks [24,29,30]. The work in [24] uses a median filter by comparing the estimated pitch in nearby frames for temporal smoothing. The moving median filter is used with 0.25 s, with a hop size of 0.01 s. This post-processing method is reliable for connecting non-continuous pitch and can effectively delete the isolated one. Similarly, the work in [29] converted the output of the support vector machine to a posterior probability. The steps for pitch smoothing were performed for each note class by using the Viterbi search method on the transition matrix of 2 × 2. The note onset and offset were finally gathered from both the posterior probability of the support vector machine and the training data. Note-level transcription can provide the highest level of detail, capturing individual notes’ onset and offset along with pitch. It allows for an in-depth analysis of musical elements like melody, harmony, and rhythm. However, there are some disadvantages of note-level transcription. First, transcribing note by note can be time-consuming, which is not practical for longer pieces of music. Second, to make an accurate transcription, expert musicians are required for the annotation of onset, offset, and pitch. Third, it is challenging and difficult to capture the musical nuances like vibrato and articulation in note level-transcription because it is highly subjective and depends on the interpretation and style of the performer.

2.3. Stream-Level Transcription

Stream-level transcription is also called multi-pitch streaming (MPS), which can group estimated pitches or notes into separate streams. The grouped or separated stream typically corresponds to an individual instrument. Figure 1c shows an example of stream-level transcription which was obtained from a multi-task multitrack music transcription (MT3) model [31]. The estimated pitches and notes for each instrument in this model have been grouped into separate streams using various music transcription datasets. The pitch streams of different instruments in this transcription can be visually distinguished using varying colors. This level of transcription offers a broader perspective, involving the transcription of musical events or attributes across entire segments or streams of music. These segments may span longer durations and encompass multiple notes or musical phrases. Stream-level transcription has the capacity to capture higher-level musical structures, such as chords, sections, or phrases, thereby aiming to provide a comprehensive and holistic representation of the analyzed music. This level of transcription is useful when the goal is the simplified arrangement of a musical piece. It works well for music with a smaller number of notes. However, it reduced some level of detail compared to note-level transcription, which may limit the accuracy of the transcription. Stream-level transcription is less suitable for complex music because it captures larger musical sections, which means it provides a more generalized overview of the music but does not capture the complex compositions such as rapid note sequences or variations in dynamics. In stream-level transcription, recent research efforts [32] have addressed the challenge by considering three key inputs: the original audio mixture, an existing multi-pitch estimation system, and the number of sources within the audio mixture. The approach undertaken in this work formulates the transcription problem as a clustering task, with a focus on maintaining timbral consistency. This is achieved by assuming that sound objects originating from the same source exhibit similar timbral characteristics. Similarly, other studies [33,34] serve as additional examples of stream-level transcription. Both of these works also operate under the assumption that notes belonging to the same source stream demonstrate comparable timbral traits in contrast to notes from different source streams.

2.4. Notation-Level Transcription

Notation-level transcription is the higher level of transcription which is the process of converting audio recordings into a traditional musical notation format such as sheet music or a musical score. Figure 1 illustrates an example of notation-level transcription. The visualization for this transcription has been carried out using AnthemScore [35] from a wave file (opera_male5.wav) of the ADC2004 vocal melody dataset. This transcription maps the audio signals to specific symbols, notes, rhythms, and other notation elements that can be read and performed by musicians. It focuses on capturing the basic structure, melody, and harmony, making it suitable for less complex music. It provides a quick overview of the musical content which can be easily understood by individuals with limited musical training. However, it is not suitable for in-depth musical analysis or the complex composition of the music. Notation-level transcription is often used to preserve and communicate musical compositions and performances in a standardized form. The transcription in this level requires a deeper understanding of musical structures, including harmonic, rhythmic, and stream structures. Transcribers use specialized music notation software to create the symbolic representation of the music. The popular software for notation-level transcription include Finale V27 (https://www.finalemusic.com/, accessed on 5 October 2023), Sibelius for desktop and mobile version 2023.8 (https://www.avid.com/sibelius, accessed on 5 October 2023), MuseScore 4.x (https://musescore.org/, accessed on 5 October 2023), LilyPond 2.24.2 (http://lilypond.org/, accessed on 5 October 2023), AnthemScore version 4 [35], and more. These tools provide a user-friendly interface for inputting and editing musical elements.

3. Transcription Techniques: Insights and Advances

This section explores a range of transcription techniques beginning with general melody to vocal melody extraction for extracting dominant melodies from polyphonic audio. The dominant melody could be either vocal or any other instruments. It then explores multi-pitch estimation that can concurrently estimate the pitch of different components of audio simultaneously. Note-level transcription for solo and ensemble vocals has also been carried out by showing the techniques for capturing the uniqueness of vocal expressions. The section ends up with a discussion of single-instrument and multi-instrument transcriptions, explaining the complex connections between various sound sources and harmonics in musical recordings.

3.1. General Melody Extraction

General melody extraction is the sub-task of frame-level transcription that deals with identifying the pitch information of the dominant melody present in the polyphonic audio. The research work in [36] proposed a new encoder/decoder network for extracting melodies from polyphonic musical audio. The authors offer two key contributions: first, they use the semantic pixel-wise segmentation method to improve melody localization in frequencies with fewer convolutional layers. Second, they propose utilizing the network’s bottleneck layer to estimate the presence of a melody line for each time frame in the audio. In general melody extraction, each frame is analyzed to estimate the fundamental frequency or pitch of the most prominent musical note. The early work presented in [37] extracts the melody played by a solo instrument by generating multiple fundamental frequency (F0) hypotheses at each time frame and employing various knowledge sources, including instrument recognition and temporal information, to select the most likely F0 sequences. Additionally, the work in [38] introduced a fully convolutional neural network trained on a large, semi-automatically generated dataset for fundamental frequency estimation. The model proves the effectiveness in both multi-fundamental frequency (multi-F0) and melody tracking in polyphonic audio, achieving state-of-the-art results across various datasets. The application of general melody extraction is often used in instrumental music analysis and can help in tasks such as automatic transcription, cover song generation, and music recommendation [39,40].

3.2. Vocal Melody Extraction

Vocal melody extraction is the sub-task of frame-level transcription. It is the process of identifying the sung or spoken pitch in a vocal performance [36,41,42]. The encoder/decoder network described in Section 3.1 for general melody extraction [39] is also used for vocal melody transcription. The work in [41] uses a high-resolution network (HRNet) to separate vocals from polyphonic music and an encoder–decoder network to estimate vocal F0 values. The experimental result shows that the HRNet-based singing voice separation method effectively reduces the accompaniment interference, which outperforms other state-of-the-art algorithms in most cases. Similarly, the work in [42] introduces a novel melody extraction system that treats the problem as semantic segmentation on a time-frequency image. It distinguishes melody from non-melody elements by using a deep convolutional neural network (DCNN) with dilated convolution. The utilization of an adaptive progressive neural network which transfers the knowledge from symbolic to audio data demonstratea the system’s accuracy across different datasets. Similar to general melody extraction, vocal melody involves F0 estimation for each frame, but with a focus on capturing the melody carried by the human voice [43,44,45]. Inspired by the object detection model in image processing, the research work in [43] introduced a patch-based CNN model for extracting vocal melodies. This patch-based CNN allows for efficient training with limited labeled data. Similarly, the work in [44] utilized the power of semi-supervised learning using a teacher–student network, which outperforms supervised learning in terms of accuracy. The work in [45] uses multi-task learning to train the classification and detection network for singing voice detection and pitch estimation at the same time. The vocal melody extraction is essential for applications like lyrics-to-audio synchronization, karaoke generation, and vocal synthesis.

3.3. Multi-Pitch Estimation

Multi-pitch estimation or multi-F0 estimation is a signal processing and music analysis technique used to identify and track multiple simultaneous pitches in an audio signal [34,46,47,48,49,50]. Many works have been carried out using this approach. The work in [34] clusters the multi-pitch F0 values into different sources by using probabilistic latent component analysis (PLCA). Similarly, the work in [46] outlines a system developed for the MIREX 2009 contest in Multiple Fundamental Frequency Estimation and Tracking, utilizing frame-by-frame analysis and a tracking mechanism. Additionally, the computational model in [47] involves the human auditory periphery followed by a novel periodicity analysis. This method is computationally efficient and straightforward to implement since the peripheral hearing model needs to be computed only once. Moreover, the proposed algorithm in [48] operates at frame-level for multi-F0 estimation, searching for the set of fundamental frequencies that minimize a spectral distance measure in each audio frame. The spectral distance measure is defined under the assumption that a polyphonic sound can be modeled by a weighted sum of Gaussian spectral models. Similarly, the book and tutorials in [49,50] outline the fundamental concepts, statistical methods, filtering methods, and subspace methods related to pitch estimation, making it a valuable resource for those interested in this field. Multi-pitch estimation is particularly important in scenarios where multiple sound sources, such as different musical instruments or vocalists, are present together in the same recording [51]. Multi-pitch estimation has applications in various fields, including music transcription, source separation, music analysis, and audio effects processing. In music transcription, it can be used to extract the individual notes played by different instruments in a polyphonic musical piece. In speech processing, it can help separate and analyze multiple speakers in an audio recording.

3.4. Solo Vocal Note Transcription

The note-level solo vocal transcription focuses specifically on capturing the sung or spoken notes within a solo vocal performance in a song [22,52,53]. The work in [22] introduces VOCANO for singing note transcription (SVT) using multi-task semi-supervised learning techniques. The work transcribes the vocal notes of singing from polyphonic audio using virtual adversarial training (VAT). Similarly, the probabilistic transcription approach is carried out in [52] to solve the problems of pitch fluctuations in singing voice transcription. It employs a hierarchical Hidden Markov Model (HMM) to manage note transitions and pitch variations, achieving promising results when assessed on a monophonic sung melody dataset. A huge number of labelled data is necessary to train the supervised deep learning model. Keeping this in mind, the work in [53] introduces a novel contrastive learning-based data cleansing model after identifying the incorrect labels in vocal note event annotations. The transcription process for solo vocals is more complex than in instrumental music due to the inherent variability and expressiveness of the human voice [54]. Note-level transcription for vocals involves accurately determining the pitches, timings, and durations of each sung note [18,55]. To correctly identify vocal notes, the work in [18] uses Viterbi decoding as a post-processing technique after following pitch estimation in monophonic audio. To overcome the limited number of labeled training data, the work in [55] constructs a dataset of solo vocals that includes the annotations in frame-level f0, notes, lyrics, and track-level meta-data. To accurately transcribe the notes of solo vocals, the machine learning-based models must consider factors like vibrato, ornamentation, and changes in articulation that contribute to the unique character of the vocal performance. This type of solo vocal note transcription is essential for applications like lyrics synchronization, vocal synthesis, and cover song generation.

3.5. Vocal Ensemble Transcription

Note level transcription in vocal ensembles is a challenging task because vocal lines are simultaneously occurring in an audio recording [56,57,58]. The comprehensive review of choir acoustics provided in [56] is particularly valuable for future researchers engaged in vocal transcription studies. The research in [57] investigates the intonation accuracy in four-part of singing (SATB) by comparing individual singers and collaborative ensembles, examining how different listening conditions affect pitch accuracy. Similarly, the work in [58] introduces a spectrogram factorization method for the automatic transcription of a cappella performances with multiple singers. It employs a sparse dictionary of spectral templates for vowel vocalizations and includes a post-processing step using overtone-based features to reduce false-positive in pitch detections. The process in vocal ensemble requires identifying and separating the various vocal lines, estimating the pitches and timings of each line’s individual notes, and creating a comprehensive representation of the entire vocal ensemble [51,59]. The early work in [59] investigated the preferences of experienced listeners for pitch and formant frequency dispersion in unison choir sounds using synthesized stimuli. The later work in [51] utilizes convolutional neural networks for extracting F0 values in ensemble singing. The method is trained on a dataset containing vocal quartets with F0 annotations and is evaluated across various scenarios, including recordings with reverb. Transcription in vocal ensemble is essential for analyzing and understanding multi-part vocal arrangements, harmonies, and interactions between different voices. It has applications in music analysis, arrangement, and synthesis, enabling the recreation of complex vocal textures from recorded performances. The vocal ensemble transcription can distinguish between different vocal ranges—soprano, alto, tenor, and bass—which requires special analysis to extract the individual contributions of each voice [60]. This process enables the reconstruction of multi-part vocal arrangements and the interpretation of polyphonic vocal music [61].

3.6. Single-Instrument Transcription

Single-instrument transcription is the process of converting an audio recording containing a single musical instrument into a symbolic representation, such as musical notes or sheet music. Most of the previous automatic music transcription focused on single-instrument piano transcription [25,62,63,64,65]. The proposed methods in [25,62,63,65] use deep learning-based approaches for piano transcription. Similarly, the proposed method in [64] uses non-negative matrix factorization by modeling a piano note as a percussive and harmonic decay in a supervised way. The work in [25] is designed only for frame-level transcription, while the work in [62] jointly estimates the onset events and pitch. Similarly, the method proposed in [63] can estimate the onset and offset times with an arbitrary time resolution, which outperforms the accuracy in the MAESTRO dataset [66]. Similarly, the work in [65] automatically predicts the piano note onset events given a video of a person playing a piano. The works described above are all for piano transcription. However, there are some other single-instrument transcriptions like guitar [67,68], violin [69,70], and flute transcriptions [71,72]. The work in [67] is the conventional approach which iteratively processes frequency peaks by marking fundamental frequency, while [68] is the recently published paper based on an attention mechanism and convolutional neural network for guitar transcription. Similarly, the research works in [69,70] discuss the analysis of violin music, but they have different objectives and methods. Additionally, [69] focuses on transcribing the pitch, duration, and playing techniques based on the time and frequency domain properties of the audio signal. In contrast, [70] aims to recover fingerings from violin music to recreate timbre, using the hidden Markov model. Moreover, the work in [71,72] is related to the transcription of Arabian flute music using audio features like Discrete Fourier Transform (DFT) and Fast Fourier Transform (FFT). The work in [71] is designed to detect octave falls and to perform the mapping from intensity to MIDI velocity, while [72] detected the Arabian flute pitches, which improved the accuracy over the occidental flute.

3.7. Multi-Instrument Transcription

Multi-instrument transcription involves transcribing an audio recording containing multiple musical instruments or sound sources. The objective is to separate and extract the individual musical lines of each instrument, capturing their respective pitches, timings, and contributions to the overall musical texture. Recently, deep learning-based multi-instrument transcription [73,74,75,76,77] has been widely used and is crucial for analyzing and understanding the interactions between different instruments in a polyphonic musical piece. A recently published paper [73] jointly considers the instrument recognition module, the transcription module, and the source separation module, which is capable of transcribing, recognizing, and separating multiple musical instruments from the audio signal. Similarly, the work in [74] adapts the concept of computer vision methods like multi-object detection and instance segmentation for multi-instrument note tracking. The multi-instrument multi-pitch estimation (MI-MPE) task, which is a clustering-based method for music transcription, was proposed by [75]. It estimates piano rolls for diverse musical instrument categories. The authors in this article claim that their method became the very first MI-MPE to be published. Similarly, the paper in [76] implements the concept of multi-object semantic segmentation to solve the problem of pitch activity detection and instrument recognition. Moreover, the work presented in [77] focuses on transcribing snare drums, bass drums, and hi-hats using a convolutional neural network and convolutional recurrent neural network after synthesizing a large number of training data. There are some other methods that carry out multi-instrument transcription after source separation. In this type of multi-instrument transcription task, the algorithm first separates each independent instrument using a source separation algorithm and estimates the pitch and note for each source separately [78,79]. Finally, the transcribed notes and musical elements from each instrument are combined to create a comprehensive representation of the polyphonic music, capturing the interplay between different instrumental lines.

4. Challenges for Music Transcription

Polyphonic music is made up of multiple simultaneous sources that include a variety of instruments and vocals with varying pitches, volumes, and timbres, each of which produces one or more musical components. The difficulty of inferring musical qualities from a mixing signal is exceedingly understudied. Moreover, vocal transcription is currently facing a primary challenge in the field of music transcription due to the inherent complexity and diversity of singing techniques employed by vocalists [80,81]. Unlike many musical instruments, the human voice is incredibly versatile and capable of producing a wide range of timbres, dynamics, articulations, and expressive nuances. This inherent variability gives rise to several factors that make vocal transcription more challenging. In addition to this, the harmonic relationship between overlapping sound events in music makes it complex for any type of transcription. The annotation of ground-truth for polyphonic music takes a long time and demands a lot of skill. This creates a limitation for the powerful supervised machine learning techniques. The following points describe the challenges for music transcription in more detail.

4.1. Singing Techniques

Vocalists can produce an extensive array of timbres and tonal qualities, ranging from breathy and soft to powerful and resonant [82,83]. These timbral variations are influenced by factors such as vocal tract shaping, tension, and breath control [84]. Accurately capturing and representing these timbral changes in vocal transcription requires sophisticated analysis techniques. Singers often employ glissandi (continuous pitch slides) and microtonal inflections, which makes vocal transcription challenging in precisely determining discrete pitch values [85,86]. The continuous nature of vocal pitch transitions requires advanced algorithms to track and transcribe these pitch changes accurately.

4.2. Instrumentation

The type of instrument used in a recording can significantly affect the transcription process due to its distinct characteristics, timbres, and playing techniques associated with different instruments [87]. For example, transcribing piano music involves handling multiple simultaneous notes across a wide range [88,89]. Each musical instrument consists of distinctive timbres, which arise from the combination of the physical attributes and material and playing techniques of singers. Transcribing instruments with diverse timbral characteristics requires advanced algorithms to accurately capture and improve the performance of music transcription. In addition to this, instruments have varying pitch ranges which influence the frequency range of the audio signal. The handling for this problem requires more sophisticated transcription algorithms that must be effective in tracking pitch variations across the spectrum.

4.3. Pitch Ambiguity

Some audio segments may have multiple possible interpretations, leading to ambiguity in choosing the correct pitch or note. This phenomenon can occur due to various reasons, such as harmonics, overtones, noise, or the limitations of pitch estimation algorithms [90,91]. Pitch ambiguity poses a significant challenge for accurate music transcription because it can lead to incorrect or uncertain note identifications. Musical sounds often consist of a fundamental frequency and its harmonics. The presence of strong harmonics can lead to ambiguity in pitch estimation, as the algorithm may detect multiple potential fundamental frequencies that align with different harmonics.

4.4. Tempo Changes

Tempo changes, also known as tempo fluctuations or tempo variations, refer to shifts in the speed or pacing of a musical performance. These changes occur when the tempo, or the perceived speed of the music, deviates from a consistent or steady rhythm [92]. Variations in tempo can affect the perceived durations of notes [93,94]. Faster tempi might result in shorter note durations, while slower tempi can lead to longer note durations. Sudden tempo changes can make the rhythm ambiguous, making it challenging to accurately transcribe the timing and alignment of notes. Tempo changes can disrupt the perception of the underlying beat or pulse, making it difficult to identify beats and downbeats properly.

4.5. Subjectivity

Music transcription involves the process of converting an audio recording of music into a symbolic representation, such as sheet music or a score [95,96]. While transcription aims to capture the essential elements of the music, it is important to acknowledge that there is an inherent degree of subjectivity involved in the process. This subjectivity arises from the fact that different transcribers, even with expertise in music theory and transcription techniques, may interpret the same musical performance in slightly different ways [94].

5. Music Transcription Tools

The process of translating audio recordings into some symbolic notation has been greatly facilitated by the scope of specialized software tools. These tools offer various transcription needs by accommodating both monophonic and polyphonic musical contexts. The review of well-known tools in this section highlights their features and contributions to the field of music transcription.

Tony [18] stands as a prominent choice among the tools made to handle monophonic music. It can implement both melody estimation methods and note tracking based on pYIN (probabilistic YIN) [19], which is an extension of the YIN algorithm [20], originally developed for pitch detection. Another popular tool called IntelliScore (http://www.intelliscore.net/, accessed on 5 October 2023) of polyphonic version is particularly known for its capability to transcribe polyphonic audio, which means it can analyze and separate multiple instruments or voices that are playing simultaneously in a recording. It can directly convert audio recordings into an MIDI format that can represent musical notes, rhythms, and other forms of musical elements. The key features of IntelliScore include batch processing for polyphonic transcription, instrument recognition, note detection, and tempo estimation. The users are able to listen to the transcribed MIDI output and make corrections and adjustments based on what they determine to be incorrect. IntelliScore can be used for various purposes including transcribing recorded music, creating MIDI tracks for remixes or arrangements, and extracting melodies or instrumental parts from recordings.

Among the music transcription tools described above, there are many other popular tools available online. Amazing Slow Downer for windows, Mac, and iOS (https://www.ronimusic.com, accessed on 5 October 2023) provides MIDI software such as sweet MIDI player pro, sweet sixteen MIDI sequencer, and sweet MIDI converter. AnthemScore version 4 [35] utilizes artificial intelligence to automatically transcribe complex compositions. AnthemScore can analyze an audio file, such as an MP3 or WAV, and convert it into sheet music or an MIDI file. This can be particularly useful for musicians who want to learn to play a piece of music, composers who want to transcribe their own compositions, or educators who need to notate music for teaching purposes. Sibelius for desktop and mobile version 2023.8 (https://www.avid.com/sibelius, accessed on 5 October 2023), is a powerful notation software for precise musical representation. It provides a comprehensive set of tools for creating and editing music notation. Users can input notes, rests, dynamics, articulations, lyrics, chord symbols, and other musical symbols using a graphical interface. Sibelius supports the MusicXML file format, which allows users to import and export scores between different music notation software. This makes it easier to collaborate with musicians who use different notation software. ScoreCloud version 4.7 [97] employs audio recognition technology for intuitive melody-to-sheet music conversion. Users can sing or play an instrument into a microphone connected to their computer, and ScoreCloud will attempt to notate the music in real-time. Capo version 4 (http://supermegaultragroovy.com/products/capo/, accessed on 5 October 2023) isolates and analyzes audio sections for detailed transcription. It mainly focuses on music transcription, chord detection, and pitch and tempo adjustment. Capo provides visual representations of chords, waveforms, and spectrograms from which the users can better understand the music. Audacity version 3.2 [98] provides basic transcription like beat detection, pitch shifting, and time stretching.

6. Python Libraries for Music Transcription

There are various python libraries available on the internet that can collectively offer a range of capabilities, from comprehensive score analysis to MIDI manipulation, deep learning-based transcription, specialized pianoroll processing, and advanced audio analysis. Their unique strengths provide various aspects of music transcription for researchers, developers, and musicians with diverse tools for musical compositions.

Librosa [99] is a prominent library renowned in music and audio analysis. The primary focus of Librosa lies in audio processing, not in transcription, but it plays a crucial role in transcription-related workflows. It supplies a python build-in function for estimating beat, pitch, and tempo. By extracting this information from audio recordings, Librosa serves as an essential preliminary step for accurate and insightful music transcription. Another popular music transcription tool is Music21 [100]. It is a comprehensive toolkit designed to facilitate tasks in computer-aided musicology. With its rich feature set, it enables the parsing, manipulation, and analysis of musical scores. Music21 supports multiple music notation formats, including MIDI and MusicXML, making it a versatile choice for transcribing and processing music data. Researchers and musicians alike benefit from its capabilities for extracting musical features and converting between different notation representations. The next python library for music transcription is Midiutil 1.2.1 (https://github.com/MarkCWirt/MIDIUtil, accessed on 5 October 2023). It is a library specifically designed for creating and manipulating MIDI files. While it does not provide direct music transcription capabilities, it is useful for generating MIDI files from scratch or modifying existing ones. This can be handy for tasks such as converting transcribed music data into an MIDI format. Similarly, the transcription library called Pypianoroll [21] was designed especially to handle multitrack pianorolls data, which is a compact representation of musical scores often used in electronic music. While not strictly for music transcription, it can be useful for tasks involving pianoroll-based music analysis, such as generating and modifying MIDI-like sequences. Moreover, another popular python library called madmom [101] focuses on research and experimentation in the field of music information retrieval. It is suitable for both novice users who want to perform basic transcription tasks and researchers who want to develop and test new algorithms for more advanced applications. It provides a wide range of tools and functions for analyzing audio signals and extracting useful information from them. Music transcription in madmom involves converting audio recordings of music into a symbolic representation, often in the form of musical notes, chords, or other musical elements. Some other popular python libraries related to transcription include Essentia 2.0.1, (https://github.com/MTG/essentia, accessed on 5 October 2023) [102], Aubio 0.4.9, (https://github.com/aubio/aubio, accessed on 5 October 2023) [103], and pydub 0.25.1 (https://github.com/jiaaro/pydub, accessed on 5 October 2023).

7. Applications of Music Transcription

A successful automatic music transcription (AMT) system would allow for a wide range of interactions between people and music. It is an enabling technology with clear potential for both economic and societal impacts. The most well-known application of automatic music transcription is enabling musicians to capture the notes of an improvisational performance so that they can replicate it afterwards. AMT is also useful for generating musical styles in the absence of a score. Due to the numerous applications associated with the area, such as the automatic search and annotation of musical information, interactive music systems (e.g., computer participation in live human performances, score following, and rhythm tracking), and musicological analysis, the problem of automatic music transcription has attracted considerable research interest in recent years. The various applications for AMT are highlighted below:

Music education: It can be used to make a system that can help for the automatic tutoring of various types of instruments [104,105].

Music creation: It can be used to improve and create the music by automatically identifying the musical notes and accompaniment [106,107].

Music production: It can be used for music content visualization and intelligent content-based editing [108].

Music search: It can be used for indexing the various genres of music such as pop, rock, metal, jazz, etc. It can also be used for recommendation of various musical lines such as the melody, chord progression, bass, and rhythm [109].

Musicology: It can be used for analyzing jazz improvisations and unannotated music [110].

Vocal synthesis: Transcribing vocals is crucial for vocal synthesis and harmonization technologies. It allows for the creation of realistic vocal performances or harmonies through software and synthesizers [111,112].

Music recommendation system: The data transcribed by an AMT system can be used for music recommendation systems as tools for suggesting relevant music to the users by analyzing musical trends and listening habits [113].

Karaoke generation: Music transcription contributes to karaoke systems allowing users to sing along accurately to their favorite songs [114,115].

Music research: The transcribed examples using AMT can support research scholars in analyzing the different topics of music including melody structures, rhythmic patterns, and chord and harmonic progressions [116].

Among the applications described above, AMT systems can also be used for cover song creation and detection, music archiving and restoration, music copyright and licensing, music arrangement and orchestration, and many more music analysis-related tasks.

8. Music Transcription Datasets

Music transcription datasets play a crucial role in transcription research. So, in this section, we will provide a comprehensive overview of these datasets. The datasets we will describe encompass various aspects, including vocal melodies, general melodies, polyphonic vocals, multi-pitch estimation, single-instrumentation, and multi-instrumentation. A detailed description of these datasets is shown in

Table 1. The description of currently available music transcription datasets.

Dataset Name	Propose	Description
MIR-1K [117]	Vocal Melody	The MIR-1K dataset offers a diverse collection of music excerpts spanning various genres, with detailed annotations of pitch contours in semitones, indices, and types of unvoiced frames, lyrics, and vocal/non-vocal segments, making it an ideal resource for vocal transcription research. It contains 1000 song clips in which the musical accompaniment and singing voice are recorded at the left and right channels.
VOCADITO [54]	Vocal Melody	VOCADITO offers four different types of annotation: frame-level f0, notes, lyrics, and track-level meta-data. It consists of 40 short segments of monophonic singing in seven different languages. First, the frame-level and note-level annotations were constructed using the Tony [13] algorithm. It also allows the user to manually correct mistakes made by the algorithm.
Medley-DB [118]	Vocal/General Melody/Multi-instrument	Medley-DB, originally designed for multi-instrumental analysis, also includes isolated vocal stems that enable researchers to isolate and transcribe vocal melodies accurately. It can not only be used for a multi-instrumental context but also allows users to transcribe general melodies. There are two versions of the MedleyDB dataset. The first version of MedleyDB consists of 122 multitracks of mix, processed stems, raw audio, and metadata, while the second version of MedleyDB 2.0 consists of 74 new tracks, making a total of 196 multitracks.
MIREX05 (https://nema.lis.illinois.edu/nema_out/mirex2013/results/ame/mrx05/, accessed on 5 October 2023)	Vocal Melody	The MIREX05 dataset, utilized in the Music Information Retrieval Evaluation eXchange, features pop songs with annotations for vocals, providing a valuable platform for vocal melody extraction. There are a total of 13 wav files along with their corresponding pitch in the MIREX05 dataset.
ADC2004	Vocal Melody	The ADC2004 dataset, from the Audio Description Contest 2004, offers audio tracks with descriptions of different musical aspects, including vocals, facilitating research into vocal transcription techniques. There are a total of 21 wav files along with their corresponding pitch annotation.
Bach10 [119]	Polyphonic vocal/multi-pitch estimation	The Bach10 dataset features a collection of compositions by J.S. Bach, encompassing both vocal and instrumental melodies. The dataset comprises audio recordings for each individual part and the complete ensemble of ten four-part chorales composed by J.S. Bach. Additionally, it includes MIDI scores, precise alignment data between the audio and scores, as well as accurate pitch values for each part and note information for each composition. The audio recording of the Soprano, Alto, Tenor, and Bass of each piece is performed by a violin, clarinet, saxophone, and bassoon, respectively.
TONAS [120]	Polyphonic vocal	TONAS offers a unique collection of 72 Flamenco songs annotated in semitones, presenting a diverse range of vocal styles and melodic structures. It consists of three cappella singing styles, Deblas, and two variants of Martinete.
Bach chorales [121]	Polyphonic vocal	The Bach chorales dataset, comprising harmonized vocal compositions, is widely used for studying both melodic and harmonic elements in polyphonic vocal music.
MAPS [23]	Single-instrument (Piano)	The MIDI aligned piano sounds (MAPS) dataset centers on solo piano recordings, making it an excellent resource for piano transcription tasks. It is composed of isolated notes, random-pitch chords, and usual western chords. The corrected version of the MAPS dataset has been created by screening the error using an algorithmic strategy [122]. MAPS contains 40GB of music, which represents about 65 h of recordings.
MAESTRO [66]	Single-instrument (Piano)	MAESTRO offers a collection of classical piano performances, enriching the dataset’s diversity and enabling researchers to explore nuances in solo piano transcription techniques. The dataset is composed of about 200 h of paired audio and MIDI recordings. The audio and MIDI files align with an approximate 3 millisecond range. There are three versions of MAESTRO: V1.0.0, V2.0.0, and V3.0.0, which are of the sizes 87 GB, 103 GB, and 101 GB, respectively.
Slakh2100 [123]	Multi-instrument	The Synthesized Lakh (Slakh) Dataset comprises multi-track audio and synchronized MIDI data intended for music source separation and the automated transcription of multiple instruments. Utilizing high-quality sample-based virtual instruments, individual MIDI tracks are generated from the Lakh MIDI Dataset v0.1. These MIDI tracks are then combined to create musical mixtures, resulting in 2100 tracks and corresponding aligned MIDI files in this release, known as Slakh2100. These tracks are synthesized from 187 patches categorized into 34 classes.
MusicNet [124]	Multi-instrument	MusicNet is a publicly available dataset for music feature representation learning, containing 330 classical music recordings with aligned labels. These labels encompass 1,299,329 individual segments within 34 h of recordings and are based on 513 distinct instrument/note combinations. The dataset is skewed towards Beethoven’s compositions and Solo Piano due to their popularity, but researchers can augment it to enhance the coverage of less-represented instruments like Flute and Oboe.
URMP [125]	Multi-instrument	URMP, the University of Rochester Multimodal Music Performance dataset, offers recordings of 44 chamber music performances, serving as a valuable resource for understanding and transcribing interactions between multiple instruments in ensemble contexts.

below.

9. Future Directions

In the rapidly evolving landscape of music transcription, the future directions of this review paper hold promising avenues for advancing the field. Several key areas emerge as potential focal points for further investigation and development.

First, the integration of advanced machine learning techniques, such as deep learning and neural networks, offers the potential to enhance the accuracy and robustness of music transcription algorithms. The power of the transformer model is currently being used in many different fields of artificial intelligence including automatic music transcription. Inspired by the successful sequence-to-sequence transfer learning in natural language processing, one of the recent works demonstrates the effectiveness of a general-purpose transformer model in transcribing various combinations of instruments across multiple datasets [31]. Additionally, another study takes the transformer model into account for the purpose of piano transcription. Through studies on the MAESTRO dataset and cross-dataset evaluations on the MAPS dataset, the work validates the effectiveness of the transformer model in velocity detection tasks, resulting in performance gains in both frame-level and note-level measures [126]. Exploring novel architectures, data augmentation strategies, and transfer learning approaches can contribute to more effective models capable of handling complex musical structures, including tempo changes, expressive techniques, and harmonies. Second, the seeking of real-time and interactive transcription systems holds great significance for practical applications. The design and optimization of algorithms that enable instantaneous music transcription for live performances, interactive music software, and real-time feedback systems are areas that need to be explored. Addressing challenges, developing efficient algorithms, and considering user-centered design principles will be essential in shaping the future of real-time music transcription. Moreover, the cross-disciplinary collaboration between musicologists, signal processing experts, and human–computer interaction researchers holds promise for advancing transcription methodologies. Integrating musicological insights, cognitive models of human perception, and interactive user interfaces can result in more accurate and context-aware transcription systems that align with human musical understanding and creativity. Furthermore, as music is diverse across genres, cultures, and styles, the adaptation of transcription techniques to accommodate this diversity becomes a vital endeavor. Exploring cross-cultural music transcription, genre-specific challenges, and the incorporation of non-Western musical elements can lead to more inclusive and versatile transcription tools.

10. Conclusions

This comprehensive review paper has traversed through the complex domain of music transcription by highlighting its diverse significance and challenges across various dimensions. We have explored the fundamental methodologies in four levels of transcription: frame-level transcription, note-level transcription, stream-level transcription, and notation-level transcription, emphasizing their distinct roles in musical compositions. We have explored insights and advancements in transcription techniques including general melody extraction, vocal melody extraction, multi-pitch estimation, solo vocals, ensemble vocals, as well as single-instrument and multi-instrument transcription. Similarly, we listed singing techniques, instrumentation, pitch ambiguity, tempo changes, and subjectivity as the major challenges in music transcription. Furthermore, we have explored several software tools and python libraries that can be used for different levels of transcription. Additionally, we have described the currently available music transcription datasets and their proposed uses.

This review paper serves as a guide for researchers, practitioners, and enthusiasts for the evolving landscape of music transcription. The future calls for utilizing the potential of machine learning, real-time applications, and interdisciplinary collaborations as technology innovation continues to amplify our analytical capacities. The smooth integration of computational capability and musical sensitivity holds the potential to close the gap between the auditory and the symbolic, bringing up new opportunities for music synthesis, analysis, and innovation.