An Efficient Hidden Markov Model with Periodic Recurrent Neural Network Observer for Music Beat Tracking

Abstract

In music information retrieval (MIR), beat tracking is one of the most fundamental tasks. To obtain this critical component from rhythmic music signals, a previous beat tracking system of hidden Markov model (HMM) with a recurrent neural network (RNN) observer was developed. Although the frequency of music beat is quite stable, existing HMM based methods do not take this feature into account. Accordingly, most of hidden states in these HMM-based methods are redundant, which is a disadvantage for time efficiency. In this paper, we proposed an efficient HMM using hidden states by exploiting the frequency contents of the neural network’s observation with Fourier transform, which extremely reduces the computational complexity. Observers that previous works used, such as bi-directional recurrent neural network (Bi-RNN) and temporal convolutional network (TCN), cannot perceive the frequency of music beat. To obtain more reliable frequencies from music, a periodic recurrent neural network (PRNN) based on attention mechanism is proposed as well, which is used as the observer in HMM. Experimental results on open source music datasets, such as GTZAN, Hainsworth, SMC, and Ballroom, show that our efficient HMM with PRNN is competitive to the state-of-the-art methods and has lower computational cost.

1. Introduction

Digital music is becoming increasingly ubiquitous in today’s multimedia world. Different from other audio signals, there are patterns occurring recurrently in the flow of music, which is called rhythm. Rhythm is an indispensable element and the most basic aspect of music [1]. Automatic beat tracking becomes a fundamental part of music information retrieval (MIR) and has a wide range of applications, such as music tagging [2,3,4], genre classification [5,6,7], emotion recognition [8,9], and music transcription [10,11,12].

To recognize the periodic pulse-like beat in music signal, machine learning techniques usually extract features, firstly from spectrogram, which is a time-frequency representation of raw music, such as short-term Fourier transform (STFT) [13,14] and Mel-spectrogram [15,16,17]. Then, the tracking system predicts beat positions based on these features. In the stage of feature extraction, deep learning-based methods [13,18,19,20,21] have been adopted instead of the hand-engineered onset detection functions [22,23]. Böck [18] uses bidirectional recurrent neural network to automatically extract features from logarithmic Mel-spectrograms. Later, temporal convolutional networks [24,25] is developed for beat tracking task as well. These deep learning models are trained using ground-truth beat annotations, which is called supervised learning. In the post-processing stage, the period and phase of rhythmic patterns (tempo and beat positions) in musical signal are estimated based on extracted features. Autocorrelation [26,27], comb filters [15,16,28], tempograms [29], and also some sophisticated machine learning methods, such as hidden Markov model (HMM) [18,24,30,31], Gaussian mixture model [32], and support vector machine [33] are widely used.

As one of the most popular post-processing methods among the state-of-the-art beat tracking systems, HMM performs quite well [30,34,35,36,37] but with a high space and time complexity. Hidden states of the above models are usually defined as a two-dimensional space of tempo and beat position. Even the space has been optimized by several articles [30,36], but the HMM still has a high computational cost when it performs inference. In order to reduce the number of redundant hidden states of HMM-based beat tracking systems, in this paper, we propose a periodic recurrent neural network (PRNN) that obtains a more reliable tempo prediction and decreases the complexity of hidden space in HMM several times. In PRNN, a module that is able to globally aware the periodic features among the outputs in RNN. As a consequence, the activations of our PRNN result in a more stable and reliable frequency, which precisely estimates the beat period or the tempo in the post-processing stage. To the best of our knowledge, we are the first to leverage the characteristic of rhythmic music signal in end-to-end RNN training in the field of beat tracking. Experimental results on music datasets GTZAN, Hainsworth, SMC, and Ballroom show that our model consists of PRNN and HMM is competitive to the state-of-the-art algorithms in prediction accuracy and has several times lower computational complexity in the post-processing stage, which much reduce the inference time of beat tracking.

To summarize, the main contributions of this paper are as follows:

• We propose an efficient deep neural network and HMM based method to take advantage of the characteristic of period music beat;
• We propose a new module based on attention mechanism called PRNN, which can capture partial period states in RNN;
• We combine the PRNN module with HMM algorithm for beat tracking task and experimental results show the inference time is much reduced.

The rest of the paper is structured as follows. The proposed model is described in Section 3. The related works are introduced in Section 2. The experimental settings are given in Section 4. The results and discussion of experiments are presented in Section 5. The conclusions of this paper are summarized in Section 6.

2. Related Works

In the beat tracking system, music feature extraction is a core part for upcoming learning and beat decoding parts. Feature extraction reduces the size of raw music data. More importantly, it makes machine learning algorithms easier to process. In this section, we review the feature extraction methods including traditional or hand-crafted feature extraction and deep learning-based feature extraction. We review the neural network-based period patter recognition methods, as well as those that are related to our proposed module.

2.1. Traditional Feature Extraction

In music beat tracking algorithms, commonly used traditional music features are harmony, timbre, bass content, rhythmic pattern, and melody. These feature extraction methods are often implemented by time-frequency transform methods in digital signal processing. Harmony refers to the pitches, tones, notes or chords that occur simultaneously in music. To extract harmony, STFT with Hann window, constant-Q transform (CQT), and averaging operations are adopted to the raw music data. Another way to obtain the harmony feature is by using the chroma feature [38] at a constant frame rate, then synchronizing the features to the beat. Furthermore, the similarity of harmonic patterns is related to beat positions and is a commonly used feature. Timbre distinguishes different musical instruments and voices. It is determined by the physical characteristics of sound, including frequency spectrum and envelope. The timbre feature is often represented by Mel-frequency cepstral coefficients (MFCC) and the feature of timbre similarity is computed based on it. As a low-frequency feature, the bass content contains drum kicking and bass instruments that are highly related to the music beat. Low-frequency spectrogram is calculated to obtain this kind of feature. Rhythmic patterns frequently repeat including the sound and silences in music and are often used to represent the bar boundaries. Melodic constant-Q transform is adopted in [39,40] for feature extraction, and melody consists of a linear succession of musical tones.

2.2. Deep Learning-Based Feature Extraction

The traditional feature extraction methods mentioned above work well when dataset is small. As digital music has grown rapidly in recent years, the complexity and diversity of music data has also improved. The performance of beat tracking system with hand-crafted features becomes unsatisfactory. On demand of the automatic feature recognition in music data, deep neural network based learning methods are utilized in MIR. In [18], bi-directional RNN extracts the features in two directions forming a reversible context for beat decoding processing. Convolutional recurrent neural network (CRNN) was investigated in [19] for the downbeat tracking task. A multi-model of RNNs approach [20] is used for beat tracking. The temporal convolutional network (TCN) was taken for tracking beat [24,25], and in [21] the authors only used visual information. Tempo-invariant convolutional neural networks was proposed in [41]. The model learns rhythmic patterns at the tempi presented in training dataset, which brings a better tempo generalization and network efficiency. A variant of transformer, named spectral-temporal transformer in transformer (SpecTNT), is proposed in [42] for a beat tracking task. The self-supervised training strategy and fully convolutional networks were used in [43] for small and unbalance datasets, and the model improved cross-dataset generalization at no extra annotation cost.

2.3. Periodic Pattern Recognition

Periodic pattern recognition is extended from frequent item mining because the periodic behaviors reveal more important information. For instance, analyzing and identifying the periodic movements of stock market, improving the performance of online recommender system by extracting customer’s periodic shopping habits, and finding genes that frequently correlated in DNA sequences, these tasks all heavily rely on cyclic features. Recently, several neural network-based periodic pattern mining methods have been proposed. Pyramidal convolutional recurrent model with periodic representations are fused in [44] for crowd density prediction. In [45], a hop residual recurrent graph neural network are designed to improve the traffic prediction that can find the periodic patterns among the time series information. For learning long-term correlations and periodic relationships from historical traffic data, spatio-temporal convolutional neural network [46] is proposed. The CPU workload of loud virtual machine exhibits both periodic and non-periodic patterns with the sudden peak of load, and Karim et al. [47] propose a hybrid RNN to tackle this problem. In the field of software development, a spatial-temporal graph neural network [48] is utilized for bug triaging process of dynamic developer collaboration network that includes hourly-periodic, daily-periodic, and weekly-periodic components. Attention based encoder–decoder neural network [49] is proposed to predict freeway traffic speed, and the attention weights are extracted which highlights the contribution of daily and weekly periodic input towards speed prediction.

3. Proposed Model

Most of deep learning-based beat tracking system are modeled as a sequence-to-sequence problem with spectrogram input and post-processing for beat correction. In a popular framework proposed in [18], the neural network extracts features from the spectrogram to predict the probability of each frame being a beat. Specifically, normalized activations of bidirectional long-short term memory (LSTM) [50] are used. HMM decodes the beat time using Viterbi algorithm based on observations of previous activations. The HMM and the decoding Viterbi algorithm are explained in Appendix A. In this paper, we improve this framework as shown in Figure 1 by introducing an attention based period mechanism into RNN, called PRNN. Then hidden space in the HMM part is much reduced by leveraging the benefits of PRNN. Next, we describe the PRNN part and its role in framework in detail.

3.1. Periodic Recurrent Neural Network

In conventional RNNs, such as LSTM, gated recurrent unit (GRU) [51] mainly solve the problem of gradient vanishing. Hidden states of current time step are based on the states of previous time step and the input of current time step. In music information systems, a large amount of similar features occurs periodically, which is highly related to music beats. Existing RNN methods lack a mechanism to deal with periodic features. In this paper, a variation of attention mechanism is proposed to tackle this problem.

Attention mechanism is widely used in sequence-to-sequence tasks which performs a soft-selection over features in neural network and obtain variable context for different step or time. Formally, given a query vector q and a source sequence $x=[x_1,x_2,\cdots ,x_n]$, attention uses an annotation function $\theta (x_i,q)$ that calculates the alignment score between q and $x_i$. The result of $\theta (x_i,q)$ represents the attention of q to $x_i$. Next, a softmax function is as follows

$$p(z=i|x,q)=\frac{e^{\left(\theta (x_i,q)\right)}}{{\sum }_{i=1}^n{e}^{\left(\theta (x_i,q)\right)}}$$

(1)

normalizes the scores ${\left[\theta (x_i,q)\right]}_{i=1}^n$ over the whole source sequence to a probability distribution $z\sim p\left(z\right|x,q)$. The attention mechanism described above results in a weighted sum of the input x. Function $\theta$ is usually defined as a neural network and we adopt scaled dot-product as the following Equation (2).

$$\theta (x_i,q)=\frac{x_i^{T}q}{\sqrt{n}}$$

(2)

The score is calculated by dot-product of query q and source sequence $x_i$. A scaling factor of $\sqrt{n}$ is introduced for preventing the case that the softmax function has an extremely small gradient which leads to inefficient learning when the input is large.

In order to make the attention mechanism being capable of perceiving or capturing period features in RNN, we set a matrix $S\in {\mathbb{R}}^{w\times n}$ for storing historical hidden states, and a hyperparameter w of S, given a fix-length window for retrieving potential similar period patterns. As shown in Algorithm 1, when time step t is less than window size w, each hidden state $h_t$ has been stored in matrix S. When the length of S equals the window size, scaled dot-product attention described in Equation (2) compares the current hidden state to previous hidden states in S, which captures the similar parts among historical states and stores the result in vector $p\in {\mathbb{R}}^n$. Then the algorithm fuses the current hidden state with the vector p that mainly contains the similar parts of previous hidden states using a learnable parameter $W\in {\mathbb{R}}^{n\times n}$. After the first update of hidden state $h_t$, we do this update when t is in multiples of w until the maximum time step T. When t is not in multiples of w, hidden state $h_t$ is stored into matrix S as a queue operation shown in Algorithm 1.

Algorithm 1 Update of period states in PRNN
Require:$1\le t\le T,(t,T)\in \mathbb{Z},S\in {\mathbb{R}}^{w\times n},h_t\in {\mathbb{R}}^n$
▹t the time step, T the total time step, S the matrix saves the past hidden states of window size w and $h_t$ is the hidden state of time step t
Ensure:$h_t$	▹ updated $h_t$ with period states
for $1\le t<w$do	▹ Initialization
$S_{t,n}\leftarrow h_t$
end for
for$w\le t\le T$do	▹ Update period states vector $p\in {\mathbb{R}}^n$
if $tmodw=0$then	▹ Window size w
$a\leftarrow softmax(h_t{S}^T/\sqrt{n})$	▹ Scaled dot-product attention
$p\leftarrow aS$	▹ Attention scores $a\in {\mathbb{R}}^w$
$h_t\leftarrow h_t+Wp$	▹$W\in {\mathbb{R}}^{n\times n}$ is a learnable parameter
end if
$S_{w-1,n}\leftarrow S_{w,n}$	▹ Dequeue the first state vector
$S_{w,n}\leftarrow h_t$	▹ Enqueue the current hidden state $h_t$
end for

3.2. Periodic LSTM in HMM

LSTM [50] is introduced to overcome gradients vanishing in RNN. In this work, we adopt LSTM embedded with the period states updating module, called PLSTM, in the beat-tracking system. The PLSTM is formalized as following.

$$\begin{gathered} \left(\begin{array}{c}{i}_t\hfill \\ f_t\hfill \\ o_t\hfill \\ g_t\hfill \end{array}\right)=\left(\begin{array}{c}\sigma (W_i[x_t,h_{t-1}]+b_i)\hfill \\ \sigma (W_f[x_t,h_{t-1}]+b_f)\hfill \\ \sigma (W_o[x_t,h_{t-1}]+b_o)\hfill \\ f(W_g[x_t,h_{t-1}]+b_g)\hfill \end{array}\right) \\ {c}_t=f_t\ast c_{t-1}+i_t\ast g_t \\ {\overline{h}}_t=o_t\ast f\left(c_t\right) \\ {h}_t=period\_states\_update\left(\overline{h_t}\right) \end{gathered}$$

(3)

As shown in Figure 2, the PLSTM part is used in the last layer in multi-layer bi-directional LSTM network. In the stage of HMM decoding, frequency component analysis of the neural activations is processed by discrete Fourier transformation. Then the potential tempi obtained from first k frequency components are used for reducing the number of hidden states in HMM. Importantly, with the proposed PLSTM the neural network is able to capture the similar features in the hidden states, which contributes to the analysis of frequency components and makes a prediction of the potential tempi more precisely. We remove the redundant hidden states in HMM decoding phase using the prior knowledge of the potential tempi. According to [30], the time complexity of the inference part is $T\times F$, where F is the total frames of the music signal and T is transitions at each time step determined by the transition model. In our model, by using the potential beat tempo, the intervals of state space change from 1 to a set of integers $\mathbb{I}$ with larger value than 1 and the length of the set is the hyperparameter k representing the first k-th potential beat tempo. The time complexity of our post-processing part is roughly $T\times F/min\left(\mathbb{I}\right)$. By using Viterbi algorithm and the observation of PLSTM in HMM (Appendix A), the final music beats are decoded.

4. Experimental Settings

In experiments, we evaluate the performances of accuracy and inference time of the proposed model with other state-of-the-art models on these datasets as shown in

Table 1. Comparison table between the proposed method and other state-of-the-art models.

Models	Preprocessing	Neural Network Type	Post Processing
Bock [25]	STFT	Bi-directional LSTM	DBN
Davies [24]	STFT	Temporal Convolutional Network	DBN
Elowsson [52]	HP separation MF estimation	Cepstroid Neural Network	CQT
Ours	STFT	Period LSTM	DBN

. Our model is compared with three models, two of them have the same preprocessing method called log magnitude spectrograms using STFT and post processing method of dynamic Bayesian network (DBN) called a bar pointer model [30]. The other model is proposed in [52], the authors process the audio with harmonic/percussive source separation (HP-separation) and multiple fundamental frequency (MF0) estimation. Then use these features are input in a cepstroid invariant neural network and post-processed mainly using CQT. The settings of hyperparameter k representing the number of potential beat frequencies are discussed. The relationships between accuracy and inference time of different k settings are tested as well.

4.1. Datasets

For training and evaluating our model, the following datasets with available beat annotations are used.

• GTZAN
  The GTZAN (http://marsyas.info/downloads/datasets.html, accessed on 11 January 2022) dataset contains 1000 excerpts of 10 genres evenly. Each clip lasts 30 s and the total length is 8 h 20 min. This dataset was originally used for music genre classification and its annotations were extended for beat tracking in [53].
• Ballroom
  The Ballroom (http://mtg.upf.edu/ismir2004/contest/tempoContest/node5.html, accessed on 11 January 2022) dataset has 698 instances of ballroom dancing music with different genres, such as Samba, Tango, Waltz, etc. The duration of each excerpt is 30 s and total length is 5 h 50 min.
• Hainsworth
  In total, 222 excerpts taken direct from CD recordings are in Hainsworth (http://www.marsyas.info/tempo, accessed on 11 January 2022) [54] dataset. The examples have been categorized into different genres, such as dance, pop, and choral. This dataset consists of 3 h 20 min of audio, therefore giving a length of about 60 s for each clip.
• SMC_MIRUM
  The SMC_MIRUM (http://smc.inescporto.pt/data/SMC_MIREX.zip, accessed on 11 January 2022) [55,56] dataset was formed with a purpose of challenging beat tracking systems. Several properties including changes in tempo, wide dynamic range, pauses, poor audio quality, etc., make this set more difficult. In total, 217 valid beat annotations are in this dataset. Each sample lasts 40 s and the total length of audio recordings in this dataset is 2 h and 25 min.

4.2. Preprocessing and Training Settings

We use the same preprocessing and training settings in the four datasets. Then, 8-fold cross validation is used and final test results are the average of all the validations. A Mel-spectrogram with 512 samples of window-size, 256 samples of hop-size, and 96 Mel-bins is applied to each music sample uniformly. In the training process, cross-entropy loss is employed as loss function. The network is trained with Adam [57] optimizer and the momentum is 0.9. We set the learning as 0.01 at the beginning of the training and is reduced by exponential decay function and the factor is 0.1. In total, 50 epochs are trained and we select the model whose validation loss is the minimum value as the best model. In the decoding part, we set the first k frequency components as {3, 5, 10, 15, 20} to explore a balanced value for improving efficiency while keeping accuracy. We also half the length of intervals 2 times when initialising the space of hidden states to create more options for Viterbi decoding. The experiments are implemented on the platform of Intel Core i7-10870H CPU, Nvidia GeForce RTX 3070 GPU, CUDA 11.1, and Pytorch 1.8 framework.

5. Results and Discussion

Table 2. Evaluation Results on GTZAN, Ballroom, Hainsworth, and SMC datasets for beat tacking. Evaluation metrics of F-measure, CMLt, and AMLt are shown.

	F-Measure	CMLt	AMLt
	GTZAN
Bock [25]	0.864	0.768	0.927
Davies [24]	0.843	0.715	0.914
Ours	0.860	0.754	0.925
	Ballroom
Bock [25]	0.938	0.892	0.953
Elowsson [52]	0.925	0.903	0.932
Davies [24]	0.933	0.881	0.929
Ours	0.941	0.902	0.959
	Hainsworth
Bock [25]	0.884	0.808	0.916
Elowsson [52]	0.742	0.676	0.792
Davies [24]	0.874	0.795	0.930
Ours	0.892	0.796	0.928
	SMC
Bock [25]	0.529	0.428	0.567
Elowsson [52]	0.375	0.225	0.332
Davies [24]	0.543	0.432	0.632
Ours	0.563	0.450	0.685

shows the evaluation results on the four music datasets. In each dataset, the first result of Bock is the baseline of our experiments. The F1 scores upgrade on the three datasets except for the GTZAN dataset for music beat tracking. For the CMLt, the proposed model performs the best only on the SMC dataset and performs slightly worse than the other models on GTZAN, Ballroom, and Hainsworth datasets. The last column is the result of AMLt, and shows our model performs better than the others on Ballroom and SMC datasets. On GTZAN dataset, 0.004, 0.014, and 0.002 of F-measure, CMLt, and AMLt values are less than the Bock’s model, respectively, but 66 times faster inference process has been obtained when $k=5$. The similar performances are observed on Ballroom, Hainsworth, and SMC datasets and the accuracies of the proposed model improved due to the reliable frequency of music beat determined by PRNN. Figure 3 displays the beat prediction result on a test music clip with the ground truth values and the neural activations of the proposed PRNN. The green waves represent the activation values that are the outputs of PRNN, and the pulses suggest the potential beat frequency of the music clip. The predicted beats that are represented by the green dots are close to the ground-truth that is represented by the red dash lines.

Table 3. “Average inference time | Average F-measure” on different settings of first k frequency components on evaluation datasets.

k	GTZAN	Ballroom	Hainsworth	SMC
3	0.0052 0.795	0.0047 | 0.855	0.0082 | 0.771	0.0065 | 0.482
5	0.0116 | 0.860	0.0106 | 0.941	0.0185 | 0.892	0.0117 | 0.563
10	0.0585 | 0.863	0.0515 | 0.935	0.0311 | 0.881	0.0619 | 0.557
15	0.2509 | 0.861	0.2112 | 0.910	0.3672 | 0.885	0.2781 | 0.560
20	0.4115 | 0.863	0.3932 | 0.925	0.4691 | 0.890	0.4203 | 0.558

and Figure 4 describes the detailed prediction accuracy and inference efficiency on different settings of k in frequency component analysis with the input of neural activations. As shown, $k=5$ is a appropriate choice of this hyperparameter and the average F-measure on evaluation datasets reaches a high range.

Table 4. The average state number and inference time of the proposed model compared with baseline model. The upper two rows are the results of the proposed model with $k=5$, and the lower two rows the baseline model.

	GTZAN	Ballroom	Hainsworth	SMC
Avg. State Number	400	344	342	414
Avg. Inference Time	0.0116	0.0106	0.0185	0.0117
State Number	5617
Avg. Inference Time	0.775	0.778	1.282	0.776

indicates that our model has more efficient inference time and needs lower computational resources as well. The average hidden states number in the proposed model is one-tenth less than the baseline model when set $k=5$ in frequency component analysis. Consequently, the inference time is 70 times faster on these four evaluation datasets. The PLSTM outputs better neural activations than original LSTM for frequency component analysis because of the additional periodic pattern recognition module.

6. Conclusions

In this paper, we propose an efficient music beat tracking model. Firstly, PRNN that can perceive the period features of input information is presented. In the beat tracking framework of RNN and HMM, we utilized the frequencies of neural activation values. The PLSTM can capture the periodic patters in the hidden layers and output neural activations with more accurate frequency components for potential tempo estimation. Then the number of hidden space in HMM part is much reduced, which leads shorter inference time and lower computational complexity. The performances of our model on different datasets are close to the state-of-the-art beat tracking models and are much faster in the meanwhile. The main limitation of our method is that the period attention module needs re-training when used in a new dataset. We will explore the applicability and migration feature of the PRNN in the future work.