Leveraging MFCC and Mel-Spectrogram Representations for Deep Learning-Based Speech Recognition ^†

Abstract

This work propose an audio feature pipeline to support machine learning tasks through the extraction of the Mel Frequency Cepstral Coefficients and Mel-spectrogram which is then used as the input of an Convolutional Neural Network which is trained to make the classification tasks. This approach enables de creation of a rich-feature dataset and an end-to-end pipeline that reduces the gap between the audio and Machine Learning ready models with application in sound classification, speech recognition and spatial audio analysis.

Speech is one of the most natural and efficient forms of human communication. On the artificial intelligence field the fact of making the machines able to understand the spoken language has been a long-standing goal. Among the speech technology, the Mel-frequency cepstral coefficients (MFCC) [1] have become one of the most widely adopted representations. MFCC are inspired into the human auditory system, mapping frequencies onto the Mel scale for a better perception of the human pitch. It also provides a compact set of features that capture both the spectral envelope and timbral properties of speech.

The present work explores the design of a speech recognition system based on MFCC feature extraction and a convolutional neural network CNN [2] as the classifier by leveraging the perceptual strengths of MFCCs and the learning capabilities of the model as shown in Figure 1.

This study makes use of the Common Voice dataset Version “Common Voice Delta Segment 10.0” [3] which has 25 recorded hours and 4 validated hours. Although it has a high quality of the audio data its also divided on different subsets like ’reported.tsv’,’dev.tsv’, ’validated.tsv’. In this project, data from the ’validated.tsv’ file were used. In particular, the last five clients (IDs 118 to 122) contributed approximately 112 s, 82 s, 206 s, 125 s, and 308 s of audio, respectively.

When conducting Fourier analysis on finite-length signals, the Hamming window [4] minimizes spectral leakage, which makes it crucial in signal processing. It reduces sudden discontinuities that allow energy from one frequency to diffuse into another by gently tapering the signal at its edges. As a result, the spectrum has reduced side lobes, which lessen leakage, and the main lobe is somewhat widened, which only marginally lowers frequency resolution. This window is represented by Equation (1).

$$w\left(n\right)=0.54-0.46cos\left({\displaystyle \frac{2\pi n}{N-1}}\right),0\le n\le N-1$$

(1)

In order to convert from the time-domain to the frequency-domain it is important to use the FFT, that’s because it allows us to extract the perceptually meaningful features using the Mel scale. This transform is presented in Equation (2) [5].

$$S_i\left(k\right)=\sum _{n=1}^N{S}_i\left(n\right)w\left(n\right)e^{{\displaystyle \frac{-2j\pi kn}{N}}}$$

(2)

MFCCs were extracted from each input speech signal and were segmented into frames of 20 ms, 30 ms and 40 ms, to capture different temporal resolutions of the speech features. Furthermore, for each segmentation, an overlap of 25%, 30%, and 35% was integrated to reduce the loss of information at the frame boundaries. This combination of both frames and overlap allows us a better comprehension of the speech signal’s in both spectral and temporal characteristics, improving the accuracy and the analysis of the features. The specific parameters for the MFCCs extraction are summarized in

Table 1. MFCC extraction parameters.

Parameter	Value
Sampling rate	32,000 Hz
FFT window size	512
Hop length	128
Number of Mel bands	20
Number of MFCC coefficient	20
Feature normalization	z-score (mean and std)

.

In order to prepare the CNN for training, the parameters shown in

Table 2. CNN training parameters.

Parameter	Value
Convolutional layers	32 and 64 filters, kernel size (3,3)
Pooling layers	MaxPooling2D with pool size (2,2) after each conv layer
Dense layers	128 neurons (ReLU)
Optimizer	Adam
Loss function	Sparse categorical cross-entropy
Batch size	32
Epochs	26
Training split	80%
Test split	20%
Random state	42

are considered, resulting in the outcomes displayed in

Table 3. Performance metrics for different Separation times and Overlaps.

Separation_TIME (ms)	Overlap (%)	Epoch	Precision	Recall	F1-Score
20	25	26	94.45	94.50	94.40
	30	26	93.72	93.70	93.58
	35	26	94.94	95.01	94.93
30	25	26	95.37	95.44	95.34
	30	26	93.38	93.50	93.30
	35	26	93.65	93.74	93.60
40	25	26	93.01	92.99	92.91
	30	26	94.55	94.57	94.51
	35	26	95.08	95.10	94.98

Metrics are averaged over multiple training runs.

, which demonstrate that both segmentation length and overlap ratio significantly influence the recognition metrics of Precision, Recall, and F1-Score. This configuration appears to provide an optimal trade-off between temporal resolution and contextual information, enabling the model to capture discriminative speakers features more effectively. The training curves as well as the normalized confusion matrix is shown in Figure 2. Where each label on the confusion matrix (0 to 4) represent a speaker.