Advanced Identification of Prosodic Boundaries, Speakers, and Accents Through Multi-Task Audio Pre-Processing and Speech Language Models

Abstract

In recent years, the advances in deep neural networks (DNNs) and large language models (LLMs) have led to major breakthroughs and new levels of performance in Natural Language Processing (NLP), including tasks related to speech processing. Based on these new trends, new models such as Whisper and Wav2Vec 2.0 achieve robust performance in speech processing tasks, even in speech-to-text translation and end-to-end speech translation, far exceeding all previous results. Although these models have shown excellent results in real-time speech processing, they still have some accuracy issues for some tasks and high latency problems when working with large amounts of audio data. In addition, many of them need audio to be segmented and labelled for speech synthesis and annotation tasks. Speaker diarisation, background noise detection, prosodic boundary detection and accent classification are some of the pre-processing tasks required in these cases. In this study, we will fine-tune a small Wav2Vec 2.0 base model for multi-task classification and audio segmentation. A corpus of spoken American English will be used for the experiments. We intend to explore this new approach and, more specifically, the performance of the model with regard to prosodic boundaries detection for audio segmentation, and advanced accent identification.

1. Introduction

Speech is the basic form of communication that humans use every day. Because of this, the possibility of communicating with machines through speech has been of interest for many decades. Over the past few years, language and speech models have acquired significant importance in society, mainly due to the recent advances in Natural Language Processing (NLP) and deep neural networks (DNNs). Current models, like Whisper [1] and Wav2Vec 2.0 [2], allow users to have fluent communication with conversational bots, either over text or speech, in real time. The improvements in speech processing have also been reflected in many other system features, for instance, the automatic captioning of speech generation for artificial text readers.

Despite these improvements, there are still many scenarios where speech processing needs improvement [3,4]. Automatic Speech Recognition (ASR) has achieved high scores in performance for common languages, but for low-resource languages, the performance of ASR models is still far from acceptable. In the case of the Arabic language, the word error rate is around 25% or even higher in some cases [5,6]. As demonstrated in previous studies, to improve their performance, ASR models can be trained applying pre-processing techniques such as segmentation, word modelling, or the detection of different environmental or speaker variations [7,8]. These tasks are modelled as speech classification tasks and can be easy to address when sufficient labelled data are available. However, for many languages, it is hard to find enough transcribed and labelled data to train a robust ASR model [9].

In this work, we propose a pre-processing speech model that can perform three tasks at once: prosodic boundary detection (PBD), speaker change detection (SCD), and accent classification (AC). Our hypothesis is two-fold: (i) that a Wav2Vec 2.0 base model, as a feature extractor block, can be fine-tuned to extract the features from the audio that are needed to perform the three classification tasks correctly, and that (ii) this transformer block can generate context representations that share all these features. The most distinctive feature of our model is, precisely, that it is trained only on audio data. We assume that this can help to improve speech recognition performance in terms of precision and resource optimisation.

The rest of the text is organised as follows. Section 2 presents a brief overview about past and recent methods for speech processing. Section 3 describes the methodology and materials used in our experiments. Section 4 reports on the experimental results, which are then discussed in Section 5. Finally, Section 6 provides the main conclusions and directions for future work.

2. A Brief Overview

In computer science, speech processing is defined as the study of speech signals and their processing methods. It encompasses a range of tasks, including classification, recognition, transformation, and the generation of speech data, among others. Speech processing tasks have traditionally been based on signal processing [10,11] and linguistic features [12,13,14] until the growth of Deep Learning (DL) [15,16,17,18]. In this section, we focus on DL-based techniques applied to three tasks which are particularly relevant to our research: phrase boundary detection (PBD), speaker change detection (SCD), and accent classification (AC). As a convenient theoretical framework, a short description of the techniques applied to these tasks before and after the popularity of DNNs will be presented in order to locate this study in context.

Before Deep Learning, Hidden Markov Models (HMM) [11,19] were being applied for speech recognition (ASR) and became the dominant paradigm until DNNs started to outperform them [20,21]. Other speech tasks, such as voice activity detection (VAD), speaker recognition and diarisation, prosody detection and segmentation, among others, play an important role as speech pre-processing tasks to ensure optimal performance in ASR and speech generation (text-to-speech, TTS). Regarding these tasks, numerous alternative methods have been employed prior to the emergence of DNNs. Signal processing algorithms and linguistic feature methods have demonstrated a notable performance for VAD [10], speaker diarisation [22], classification problems [23], and prosody-related tasks [13].

Even though some of the former methods and algorithms are still in use [24,25], DNNs are currently the most popular method for speech pre-processing, as well as for ASR, to the point where most of the new methods for these tasks are based on or derived from DNNs (see [8] for a comprehensive review).

2.1. Speech Processing Methods and Techniques

As DNNs became the standard for ASR tasks, the encoder-decoder architecture obtained the most promising performance. In this approach, the encoder extracts features from the input speech, while the decoder transforms those features into the desired output, namely a transcription [26]. In this manner, the encoder learns to generate representations or embeddings that represent the extracted features. Different methods of representation are employed in the literature for speech processing tasks. For instance, i-vectors [27], d-vectors [28], and x-vectors [29] are embeddings based on convolutional NNs (CNNs) and recurrent NNs (RNNs) that have resulted in improved performance in tasks such as speaker diarisation or VAD. However, transformer NNs [30] have surpassed RNNs and CNNs in the context of speech processing tasks. The utilisation of self- and cross-attention transformer NNs allows parallelisation and performance improvements that explain the success of them in Natural Language Processing (NLP) tasks [31].

2.2. Automatic Speech Segmentation

Despite the mentioned computational progress, automatic speech segmentation is still one of the most difficult challenges in speech processing. The variation in speaker characteristics, speaking style, and environment make the learning process of prosodic boundaries hard for machines [32]. However, detecting and processing prosodic phrasing is relevant for many speech processing tasks as it can help to improve naturalness in speech synthesis and to enhance audio data for training [33]. In the case of speech synthesis, the common scenario for the detection of prosodic boundaries is performed by relying solely on text [34], while other systems rely solely on acoustic information [35], or on a combination of both lexical and acoustic information [24]. For the purpose of this study, we opted for the detection of prosodic boundaries based solely on acoustic information (cf. Section 3. ‘Materials and Methods’).

Speaker change detection (SCD) is another speech segmentation task used to determine the boundaries between speakers in a conversation. In simpler words, SCD is the action to detect the moments when a speaker stops talking and another speaker starts to talk, regardless of the identity of the speakers. SCD is part of speaker diarisation systems (who talks and when) [36], but it is also useful for speaker tracking in video data [37] and for transcribing audio with multiple speakers [38]. There are many approaches to performing this task based on DNNs, such as the previously mentioned x-vectors [39] or methods based on self-attention mechanisms [40,41]. Overlapped segments (i.e., several speakers talking at the same time) are a common, recurrent problem in SCD. Many of the existing methods opt for the omission of the overlapped segments. However, similar DNN-based approaches are being applied specifically to overlapped speech detection [42,43,44]. In addition, adding extra information during training through multi-tasking techniques has shown that multi-task models can achieve better results than models trained for only one task [45].

2.3. Accent Identification

Before DNNs, studies in accent identification focused on leveraging linguistic features in statistical approaches. For instance, some of the features that have been combined with statistical analysis are prosodic parameters, syllable structure, and speaker characteristics [23,46,47]. Recent studies are centred on DNNs for spoken language identification [48], the generation of representations around features related to accent identification [49,50,51] or other DL approaches for accent classification [52,53,54]. Other recent approaches have explored the possibilities of combining accent identification and ASR as a viable way to improve the performance of ASR models [55]. It is notable to mention that some methods, like the one presented by Ghorbani and Hansen [56], have been able to reach better scores than human-perceived scores.

2.4. Joint and Multi-Task Learning in Speech Processing

Joint learning and multi-task learning are both approaches used in machine learning to leverage shared information across tasks. In speech processing, this approach was demonstrated to be beneficial and to achieve better performance in many tasks [57]. The method presented in [58] shows that joint training could help the encoder of the NN to learn transferable, robust, and problem-agnostic features that carry relevant information from the speech signal, which contributes to discovering general representations. For speech segmentation, joint and multi-task frameworks have been applied to prosodic boundary detection and SCD with successful results [35,45,59]. With regard to joint learning methods involving accent classification, recent studies have shown high accent classification performance by adding this task to speech recognition models [60,61,62,63].

3. Material and Methods

The main goal of the research presented in this paper is to develop an innovative multi-task training approach for PBD, SCD, and accent classification by relying only on acoustic information. To address this multi-task problem, we fine-tuned a Wav2Vec 2.0 model for audio frame classification with three classification layers on top of the model, one layer for each task. For this experiment, we used an American English oral corpus as a base dataset for training and validation of the models. This corpus was pre-processed using different methods for each task, which led to the generation of new datasets adapted for the purpose of the experiment. The following sections provide detailed information on the base dataset selected and the transformations applied to generate the final datasets, as well as the methods applied for the training, evaluation, and validation of the models.

3.1. Speech Corpus Used in the Experiment

The selected corpus for the experiment is the Santa Barbara Corpus of Spoken American English (SBCSAE) (the SBCSAE corpus can be accessed and downloaded here: https://www.linguistics.ucsb.edu/research/santa-barbara-corpus (accessed on 15 January 2025)) [64]. This is a prosodically annotated speech corpus (also called a spoken corpus), which encompasses a total of 60 transcribed and annotated conversations, including the timestamps at the level of individual intonation units. The corpus is based on a large body of recordings of naturally occurring spoken interaction from all over the United States, representing a wide variety of people of different regional origins, ages, occupations, genders, and ethnic and social backgrounds. The predominant form of language use represented is face-to-face conversation, but the corpus also documents many other ways people use language in their everyday lives: telephone conversations, card games, food preparation, on-the-job talk, classroom lectures, sermons, storytelling, town hall meetings, tour-guide spiels, and more. The transcriptions are formatted in one utterance per line, together with the initial and final timestamps, the name of the speaker, and the intonation information of the utterance. In addition, the metadata of each conversation includes a short description and the location where it was recorded. All this information makes the SBCSAE a useful dataset for many supervised learning tasks in speech processing. Given the purpose of this work, we are specifically interested in the timestamps for SCD and PB, and the location information for the accent detection.

3.2. Data Preparation

This section covers all the automatic pre-processing and annotation tasks applied to the dataset used for the experiments. The first two subsections explain the design of the annotation process for each task, while the last one describes how all the processes were applied automatically.

3.2.1. Audio Frame Labels

Audio segmentation can be seen as the act of slicing the audio into pieces following a specific criterion. Our approach aims at detecting the specific frames in which the speaker finishes an utterance or ends their intervention. For this reason, we needed to define the frames and assign specific labels for each frame. We followed the methodology presented in [65], where the authors trained a prosodic boundary detection (PBD) model for the Czech language. In their work, the authors assigned a label every 20 ms of audio, giving a 1 if the frame contains a boundary, or a 0 if not. In addition, to avoid delays from the manual annotation, the frames around each prosodic boundary (PB) were given a label between 0 or 1, thus creating a linearly increasing and decreasing interval. For the PBD task, we followed this exact same method applied to the SBCSAE.

The SBCSAE includes the name of the speaker for each utterance in the transcript file. We used this information to create the reference data for the SCD task in a similar manner to previously mentioned PB references: we gave the label ‘1’ to the frames when a new speaker starts to talk and ‘0’ for later utterances of the same speaker. This method for label setting has one problem: it does not consider whether two or more speakers are talking at the same time. To address this issue, the SBCSAE corpus includes the labels “BOTH” and “MANY” when more than one person is speaking. This annotation was adapted to our labelling method by assigning the label ‘2’ to the frames that correspond to these utterances. This is also known as Overlapping Speech Detection (OSD). In addition, the corpus also includes specific labels for timestamps where a sound or environmental noise occurs. Following our labelling process, we assigned the label ‘3’ to these timestamps. In this way, the label ‘3’ indicates frames without voice activity. Although our dataset is focused on SCD, these annotations enable the model to also perform OSD and VAD.

3.2.2. Accent Classification Annotation

The labelling process for accent detection differs considerably from the process applied to SCD and PBD (see above). The SBCSAE includes the city and state from the United States corresponding to each of the conversations as part of the metadata of the audio. For this reason, it was necessary to label the whole audio with its accent rather than splitting the audio into frames. This poses another challenge, that is, to merge the audio classification task with two audio frame classification tasks. To solve this, we assigned the corresponding accent label to every frame of the audio and then reduced the set of frame labels to only one for the whole audio.

To establish the number of classes for this task, instead of assigning the labels based on the states, we applied the classification of the major regional dialects of American English as demarcated by Labov et al. [66]. This classification divides the map of the United States into six dialect regions.

Table 1. States included in the SBCSAE per dialect region, plus the label assigned to each region.

Regional Dialects	States	Label
Inland North	Illinois, Wisconsin, Michigan	0
New England	Massachusetts, Vermont	1
Mid-Atlantic	Delaware, Pennsylvania,	2
South	Alabama, Texas, Kentucky, Louisiana	3
Midland	Indiana, Kansas	4
West	Montana, California, New Mexico, Arizona, Oregon, Washington, Nevada, Idaho, Colorado	5

shows the six dialects, the states that belong to each region and the labels that we assigned for each dialect to prepare the data for this task.

3.2.3. Automatic Annotation Process

Once the labelling process is completed, the annotation files created from the SBCSAE are converted to the desired format (see Section 3.2.1. ‘Audio Frame Labels’ and Section 3.2.2. ‘Accent Classification Annotation’). In order to automate the process, various Python (v3.8.12) scripts were designed to streamline a variety of sequential tasks. The first task consists of splitting the audios into chunks of 30 s with a 15 s overlap between one audio and the next, to avoid context missing. Then, reference labels for PBD and SCD tasks are generated from the timestamps in the original annotations, and the speaker labels. The result is a set of 1500 labels for each task. Figure 1 shows example representations of the set of labels for PBD and SCD. It depicts the graphical representation of the reference labels for 30 s audio. Given that one label is assigned every 20 ms, the Y axis represents the number of the frame (1500 frames for the 30 s audio) while the X axis shows the value assigned to each frame. The last script assigns the labels corresponding to the dialects according to the name of the state which appears in the metadata. The part of the scripts that perform the extraction of the key information were coded using regular expressions. The extracted data were saved as JSON files.

Once all the conversion and annotation processes described had been performed, the target dataset was created through the Datasets package from HuggingFace (https://huggingface.co/docs/datasets/index (accessed on 18 January 2025)). This package allows the organisation of the dataset in three splits: train, test, and validation. The entries inside each one of the splits contains the attributes “path” and “label”. The “path” is the absolute path to the audio file and the “label” includes the three sets of reference labels, one set for each task. After the generation of the dataset format, the entries are randomly divided into the three splits as follows: 80% of the dataset for training, 10% for testing, and another 10% for validation.

3.3. Model Architecture and Experimental Design

Wav2Vec 2.0 (hereafter referred to as “wav2vec2”) is a pretrained speech model whose architecture is mainly based on transformers [2]. The model has a structure of three main parts: the feature encoder, the context network, and the quantisation module. The encoder consists of several blocks of temporal convolutions that normalise the speech input and outline the number of time-steps of the speech which, in turn, serve as input to the context network. The context network is a transformer NN which learns contextualised representations from the output of the encoder. The quantisation module discretises the output of the feature encoder via product quantisation for self-supervised training. The wav2vec model is able to learn representations from speech audio in the same way language models learn linguistic representations from text, achieving better results in ASR than previous state-of-the-art models [2].

In this study, we leveraged the language representations generated by wav2vec2 to perform a supervised learning experiment. This experiment consisted of fine-tuning and validating a wav2vec2 model for multi-task audio frame classification. To implement the desired architecture of the model, we used the transformers (https://huggingface.co/docs/transformers/index (accessed on 18 January 2025)) package by HuggingFace, which already includes an implementation of an audio frame classification architecture using a wav2vec2 encoder and a classification head on top. Given that we need three classification heads instead of only one, we built our own architecture for audio frame classification based on the existing implementation in the transformers package. Each one of the classification layers performs its own task: one of them for PBD, another for SCD, and the last one for accent classification. During training, the context representations generated by the wav2vec2 block are given to each one of the classification heads and they produce their own results. This process is illustrated in Figure 2.

Our architecture is composed of a wav2vec2 layer [2] as a feature extractor and three classification heads, one for each task. During training, the loss function is computed through a two steps process: first, the loss for each classification layer is calculated separately, and then given to a combined loss function as weighted parameters. This process is further detailed in the next section (cf. Section 3.4).

3.4. Training, Testing, and Validation Processes

Following previous studies (cf. Section 2.4 ´Joint and Multi-Task Learning in Speech Processing´), we focused on multi-task training approaches as a key part of the training methodology. To train our model, we used the wav2vec2 base model which can be obtained from the HuggingFace models repository (https://huggingface.co/facebook/wav2vec2-base (accessed on 20 January 2025)). This is the wav2vec2 block of our architecture. As shown in Figure 2, only the transformer layer of this block is fine-tuned during training. On the other hand, the three classification layers are randomly initiated and fully trained. The training is a supervised learning process, as the dataset includes the audio as inputs and the reference labels to adjust the loss function. For each step of the training, the context representations generated by the wav2vec2 block are given to the classification layers as the hidden states, and each one of the classification layers produces their corresponding label outputs. Following the starting hypothesis (cf. Section 1. ‘Introduction’), the Transformer layer of the wav2vec2 block was fine-tuned to extract the necessary features to perform the three classification tasks and generate the context embeddings that represent those features.

To achieve our goal, we designed a composed loss function which is the combination of the loss from each of the classification layers. For PBD and SCD we applied mean squared error (MSE) as the loss function, and cross-entropy (CE) was the loss function applied for accent classification (AC). The reason for this difference in loss calculation is that for AC the labels are the same for every frame; therefore, the output of the layer is converted into a probability distribution instead of a set of labels. The composed loss is calculated as the weighted average of the three individual functions as follows:

$$\mathrm{loss}={\mathrm{loss}}_{\mathrm{pbd}}{\mathrm{w}}_{\mathrm{pbd}}+{\mathrm{loss}}_{\mathrm{scd}}{\mathrm{w}}_{\mathrm{scd}}+{\mathrm{loss}}_{\mathrm{ad}}{\mathrm{w}}_{\mathrm{ad}}$$

(1)

The challenge of this loss function was to determine the optimal weights (${\mathrm{w}}_{\mathrm{pbd}},{\mathrm{w}}_{\mathrm{scd}},{\mathrm{w}}_{\mathrm{ad}}$) applied to every task. To address this challenge, we ran a set of training experiments, using different parameters and weights. This process helped to obtain the best context representations by fine-tuning the transformer layer. The post-training results are presented below (cf. Section 4. ‘Results’).

Next, the fine-tuned models were evaluated through test and validation; for this task, the corresponding subsets from the dataset were used. Testing was carried out during fine-tuning, after each epoch. To this end, a composed metric was used, similarly to what we carried out in the case of the loss function. However, we used accuracy as a metric for accent classification instead of CE, and the MSE values from PBD and SCD were inverted (1—MSE). After training, the best checkpoint was selected based on the results of the test phases.

The best weights for the loss function were established on the basis of the results of the validation phase, starting with the same value for each task (0.33). With this value, the AC task was the one with the highest accuracy, while SCD reported the lowest accuracy. Finally, the selected weights were obtained from the training in which the penalisation in accuracy for the AC task was not found to be significant, while the accuracy rates for SCD and PBD were the highest.

4. Results

Once the model was fine-tuned, the validation was performed for each task separately by applying post-processing and different metrics. Post-processing consists of the reverse conversion of the label set of SCD and PBD classifiers by turning the labels into timestamps. After the reverse conversion, the F1 metric is applied to measure the performance of SCD and PBD tasks. In the case of the accent classifier, its performance is measured using the accuracy metric (ACC).

4.1. Post-Processing Approach and Validation Measures

To convert the output labels given by the model back to timestamps, we applied different criteria to the PBD outputs and the SCD outputs. For the first task, the frames labelled with ‘1’ are the PBs detected, although the model is not entirely accurate when assigning the labels. For that reason, we established a threshold of 0.5. Every label bigger than this threshold is considered as a PB. In the case of SCD, the approach is similar, every frame with a label above 0.5 is considered an SC timestamp. We also established thresholds for the labels corresponding to more than one person talking (1.5) and the labels for environmental sound or noises (2.5). For AC, no post-processing was needed.

After the post-processing was applied, we obtained two lists of detected timestamps for PBD and SCD, respectively, and the corresponding reference timestamps. To evaluate the performance of each task, we applied several metrics, namely Precision, Recall, and F1. In the case of the AC output, we measured the performance using ACC by comparing the reference set of accent labels and the output given by the accent classifier.

4.2. Parameter Evaluation and Model Validation

The results discussed in the section are intended to fulfil a two-fold purpose. One is to validate the methodology and the trained models. The other is to find the optimal weights for the loss function.

Table 2. Validation results (%) for the PBD task. The values in bold represent the highest result of the corresponding metric.

Models	$\mathbf{Applied}\mathbf{Weights}({\mathbf{w}}_{\mathbf{pbd}},{\mathbf{w}}_{\mathbf{scd}}\mathbf{and}{\mathbf{w}}_{\mathbf{ad}}$)	Epochs	Precision	Recall	F1
1	0.38, 0.6, 0.02	10	63.49	50.43	56.21
2	0.33, 0.33, 0.33	20	67.19	44.24	53.35
3	0.4, 0.4, 0.2	20	66.66	60.67	57.58
4	0.4, 0.5, 0.1	20	68.83	49.82	57.8
5	0.38, 0.6, 0.02	20	69.56	58.3	64.44
6	0.39, 0.6, 0.01	50	75.29	75.9	75.59
7	0.399, 0.6, 0.001	50	78.73	75.01	76.78
8	0.299, 0.7, 0.001	50	76.02	71.21	73.54

,

Table 3. Validation results (%) for the SCD task. The values in bold represent the highest result of the corresponding metric.

Model ID	$\mathbf{Applied}\mathbf{Weights}({\mathbf{w}}_{\mathbf{pbd}},{\mathbf{w}}_{\mathbf{scd}}\mathbf{and}{\mathbf{w}}_{\mathbf{ad}}$)	Epochs	Precision	Recall	F1
1	0.38, 0.6, 0.02	10	63.2	42.8	51.04
2	0.33, 0.33, 0.33	20	61.43	33.6	43.45
3	0.4, 0.4, 0.2	20	62.43	38.16	47.37
4	0.4, 0.5, 0.1	20	63.07	43.88	51.76
5	0.38, 0.6, 0.02	20	62.18	57.49	59.74
6	0.39, 0.6, 0.01	50	70.89	72.72	71.79
7	0.399, 0.6, 0.001	50	69.74	75.39	72.45
8	0.299, 0.7, 0.001	50	74.78	74.46	74.12

and

Table 4. Validation results (%) for the accent classification task. The values in bold represent the highest result of the corresponding metric.

Model ID	$\mathbf{Applied}\mathbf{Weights}({\mathbf{w}}_{\mathbf{pbd}},{\mathbf{w}}_{\mathbf{scd}}\mathbf{and}{\mathbf{w}}_{\mathbf{ad}})$	Epochs	Accuracy
1	0.38, 0.6, 0.02	10	99.46
2	0.33, 0.33, 0.33	20	99.46
3	0.4, 0.4, 0.2	20	99.28
4	0.4, 0.5, 0.1	20	99.28
5	0.38, 0.6, 0.02	20	99.82
6	0.39, 0.6, 0.01	50	99.82
7	0.399, 0.6, 0.001	50	83.27
8	0.299, 0.7, 0.001	50	92.63

show the key results of each training, and their corresponding parameters, in terms of the applied loss weights and the number of epochs. For PBD and SCD task results, Table 2 (PBD) and Table 3 (SCD) contain the Precision, Recall, and F1 values. In addition, Table 3 contains the accuracy results for the accent classification task.

In the three tables, each entry corresponds to the results from the model trained using the indicated weights for the loss function and the number of epochs. Entries are ordered by the number of epochs, since we observed that this parameter has a higher impact on the results. This can be seen in the results from models 1 and 5. Model 5 achieved a higher punctuation in all three tasks, even though both models were trained with the same weights. No improvements can be observed in learning for more than 50 epochs, which justifies the maximum number of epochs (50) presented in the tables.

The results obtained during the fine-tuning of the applied weights indicate that the task that presents the most problems for the models to predict is SCD. This is not unexpected, as the references from the SCD task present higher variations at labels per frame than the other two (see Section 3.2 ´Data Preparation´). On the contrary, the high accuracy levels achieved at the accent classification task show that this is the task that presents fewer difficulties. Since the model only needs to predict one label for the AC task, the high difference in the results between this task and the other two is not surprising. This explains why the weight for accent classification showed the highest decrease, whereas the weight for SCD increased in each training by the highest factor (see Section 3.2 ´Data preparation´).

Finally, our findings suggest that model 8 is the one that exhibits the best performance. Even if the results for accent classification and PBD are affected negatively by the weights of this model, it shows an appropriate balance in its performance for the three tasks.

5. Discussion

From the perspective of previous studies, this is not the first time that wav2vec2 has been leveraged to perform PBD in a single task environment or leveraged to perform multi-task speech processing. In [65], the authors present a methodology to fine-tune a wav2vec2 model to detect PB on Czech speech data, relying solely on acoustic information. An extended approach based on the previous one was also applied for multi-task speech processing on English data in [59], where the trained model performed SCD, OSD, and VAD as different tasks. Regarding other NN architectures, a few studies have introduced multi-task or joint learning approaches for SCD, PBD, and AC [35,45,60]. In [32], fine-tuned Whisper models for PBD on the SBCSAE presented satisfactory results, although their acoustic model performances were worse than those of their lexical models. Referring to accent classification, the authors of [67] present a particular Multi-Kernel Extreme Learning Machine (MKELM) architecture which achieved similar scores to other DNNs architectures.

Given the possibilities of transformer models for speech processing, the objective of this work was to fine-tune a wav2vec2 model with three classification layers to perform PBD, SCD, and AC by relying only on acoustic information. We tested our method on spoken English by using the SBCSAE corpus, like most of the models considered in this study. To the best of our knowledge, this is the first experiment that applies one single model to perform the three target tasks at the same time. The results obtained during the fine-tuning experiments show that our model can achieve a performance score close to previous methods or even higher.

A comparison between our model and the most relevant previously proposed models is highlighted in

Table 5. Summary of results from previous studies and our results.

Models	PBD Score (% F1)	SCD Score (% F1)	AC Score (% Accuracy)
[65]	82.73	-	-
[59]	-	90.79	-
[35]	91	-	-
[45]	-	88.32	-
[60]	-	-	75.2
[32]	73	-	-
[67]	-	-	84.72
Our model	73.54	74.12	92.63

. This comparison shows that our model achieved slightly lower performance scores for SCD and PBD than previous methods, but its performance on AC is the best. The results of the proposed model are highlighted in bold (Table 5).

Our model exhibits lower performance for PBD than the models presented in [35,65], but higher than the model presented in [32]. In [65], the described method is applied to the Czech language, and the model presented in [35] is trained on text information, not only on acoustics/audio data. This explains the difference in performance compared to our model. On the other hand, the approach in ref. [32] is the only one that used the same corpus (SBCSAE). The performance of this model, which relies solely on acoustic information, is slightly lower than our model’s performance. Regarding the SCD task, our model has the lowest performance score on the table, although the three models were trained on the English corpora following multi-task approaches. However, the architecture and the tasks of our model are more complex than the previous models, which can explain the lower results. Finally, for AC, our model shows better performance than previous works. The main differences in this case are that the model in [67] is not a transformer model and that the approach applied in [60] combines ASR and AC.

Model sizes and architectures could be major factors responsible for the differences between our results and those from other previous studies. For example, refs. [59,65] similarly use a wav2vec2 model, but it is not the same base model applied in our experiments. The other studies applied other models and architectures, which could be instrumental for the lower performances of PBD and SCD. The evaluation data can equally impact the performance. The differences in dataset size, language, and possible biases could also explain the differences in performance for the three tasks.

In addition to the mentioned differences between our work and previous studies, it is important to note that, except for the model described in [59], this is the only model that performs three speech processing tasks. In terms of model size, adding a new classification layer does not have a significant impact, keeping the model size almost the same. This means that our model requires one-third of the memory compared to single-task models that perform the same three tasks, allowing for the more efficient and cost-effective use of resources.

6. Conclusions and Future Work

Following our initial hypothesis, we assume that speech representations from the wav2vec2 framework could be leveraged in multi-task speech processing. In this context, the main aim of this work was to demonstrate that a single wav2vec2 acoustic model can be fine-tuned, using joint learning techniques, for PBD, SCD, and accent classification. Our findings confirm our hypothesis: our model (trained and validated on the SBCSAE) is the best performing one, as it achieved an F1 score of 0.74 for PBD and SCD and a 0.93 Accuracy score for accent classification.

To the best of our knowledge, ours is the first fine-tuned speech model which is able to perform all three tasks, namely PBD, SCD, and accent classification, by relying solely on acoustic information. This can be particularly useful in situations where no transcriptions or other language data are available. These tasks can be applied to speech pre-processing and segmentation, which can help to improve speech recognition performance in terms of precision and resource optimisation. This is the main contribution of this study. Another contribution is the data preparation process applied to the SBCSAE, which was fully automated. This makes the process reproducible, scalable, and transferable to other timestamp-annotated corpora (with some minor adjustments, if needed). Finally, the proposed model architecture further advances the state-of-the-art fine-tuning methods, as the classifier layers can be trained to perform any audio frame classification task, which means that our model can be easily adapted to fine-tune models on other corpora.

The scope of this study was limited in terms of metrics and the relatively small data sample. It would be worth considering further complementary metrics, such as coverage and measure training, as well as the inference times of the models. But the most important limitation is related to the data used for training and validation. Our findings suggest that better results could be achieved by expanding the training and validation dataset with more speech corpora. Despite the significant contributions of our study, the issue of data size is an intriguing one which could be usefully explored in further research. To this end, we intend to repeat the experiments using more English datasets, in addition to the SBCSAE, to train and validate our model. Another future solution to address data limitations could be the application of different transfer learning techniques. One of the advantages of our model is that the pre-trained base model can be altered without major changes. Given this, we intend to test transfer learning from larger wav2vec2 or other models and larger datasets. In addition, several issues related to the limitations of our study still remain unsolved. For instance, we must apply more metrics to expand the evaluation of the methods analysed. Finally, a natural progression for this work would be to test our model on other languages and on multilingual data (e.g., parallel corpora). In addition, our methodology could be applied to test other speech models and compare the results. Another possible study could involve applying our model to cascaded or multi-model speech systems, for instance, in a complete speech recognition system. This would be a fruitful area for further work.