An Ensemble of Convolutional Neural Networks for Sound Event Detection

Abstract

Sound event detection tasks are rapidly advancing in the field of pattern recognition, and deep learning methods are particularly well suited for such tasks. One of the important directions in this field is to detect the sounds of emotional events around residential buildings in smart cities and quickly assess the situation for security purposes. This research presents a comprehensive study of an ensemble convolutional recurrent neural network (CRNN) model designed for sound event detection (SED) in residential and public safety contexts. The work focuses on extracting meaningful features from audio signals using image-based representation, such as Discrete Cosine Transform (DCT) spectrograms, Cocheagrams, and Mel spectrograms, to enhance robustness against noise and improve feature extraction. In collaboration with police officers, a two-hour dataset consisting of 112 clips related to four classes of emotional sounds, such as harassment, quarrels, screams, and breaking sounds, was prepared. In addition to the crowdsourced dataset, publicly available datasets were used to broaden the study’s applicability. Our dataset contains 5055 audio files of different lengths totaling 14.14 h and strongly labeled data. The dataset consists of 13 separate sound categories. The proposed CRNN model integrates spatial and temporal feature extraction by processing these spectrograms through convolution and bi-directional gated recurrent unit (GRU) layers. An ensemble approach combines predictions from three models, achieving F1 scores of 71.5% for segment-based metrics and 46% for event-based metrics. The results demonstrate the model’s effectiveness in detecting sound events under noisy conditions, even with a small, unbalanced dataset. This research highlights the potential of the model for real-time audio surveillance systems using mini-computers, offering cost-effective and accurate solutions for maintaining public order.

1. Introduction

1.1. Research Context and Motivation

It is evident that advancements in machine learning and deep learning technologies have facilitated the creation of highly accurate systems for image processing [1], audio data analysis, speech recognition [2], and various other related applications. Concurrently, these technologies have given rise to new tasks that can find application in the fields of security, early detection, and the prevention of emergency situations. One of the most urgent problems today is the development of low-cost hardware and software that can recognize emotional sounds in emergency situations in the environment, make decisions, and promptly provide information to the situation center, especially in smart cities, to ensure safety. Cities are the result of constant evolutionary processes and are ideal places to test and implement new digital technologies in order to improve the living conditions of the population. Among the many challenges that are typically prioritized in smart cities are those related to public safety. These involve the use of low-cost sensor-based surveillance capabilities that are strategically placed in specific areas (crowded or uncrowded) to create a level of virtual monitoring depending on the situation. Such remote surveillance systems can be implemented in the conventional way using video cameras. However, in order to cover all the designated surveillance zones, several video cameras are needed to continuously record the surrounding environment.

The implementation of the tasks associated with this approach include the cost of hardware and infrastructure, complexity of maintenance, high power consumption, large number of different internal customizations, high need for processing power, and dependence on large storage devices. In addition to the above, there is the need to rely on security personnel to closely monitor the most complex images.

A more convenient and cheaper option to solve such problems is to use environmental sounds. Instead of video, audio from the observed environment is used. Rather than use human beings who are constantly listening to the environment, it would be easier to create a system that automatically detects potential or ongoing incidents that pose a risk to public safety and makes timely decisions. Detecting or recognizing emotional sounds in environmental events is more difficult than the intelligent processing of other speech or music signals because, since the sounds are taken from the environment, their characteristics are subject to strong non-stationary noise and interference. This is why new methods need to be introduced at each stage of sound processing in the classification of sounds.

1.2. Research Aims and Contributions

The detection of sound events in the environment aims to digitally process a continuous acoustic signal and convert it into a specific description of the relevant sound events present in the listening scene. The field of research that studies this process is called digital auditory scene analysis [3].

Analysis systems can be separated into two categories depending on whether they extract temporal information from the sounds being analyzed. Performing detection refers to a system that outputs information on the temporal activity of target classes. Different time scales are utilized depending on the requirements of the application, and detection can be performed for one or multiple sounds. If temporal information is not output, the system performs classification, which can output only one of the possible classes for the analyzed object, or labeling, which can output multiple classes. In machine learning terminology, this is commonly referred to as multi-label classification (Figure 1).

Automatic sound event detection (SED) systems have been employed in a number of applications including the contextual indexing and searching of multimedia databases [4,5], unobtrusive monitoring in healthcare [6], surveillance [7], and military applications [8]. Some information regarding acoustic phenomena can be used in other research areas, for instance, in audio context recognition [9,10], automatic labeling [11], and audio segmentation tasks [12].

This research includes a comprehensive study of an ensemble CRNN model and the use of image-based methods to extract important features from audio signals in order to quickly estimate sounds in situations related to maintaining public order in residential buildings and various public safety problems. Image-based data representation provides a deep and informative description of acoustic signals. This representation allows you to effectively extract features important for recognizing sound events and suppress the influence of noise. Ensemble models combine predictions from multiple base models, resulting in reduced variance, lower bias, and improved generalization compared to stand-alone models. This aggregation of diverse methodologies mitigates individual model errors, enhances robustness against overfitting, and typically yields more accurate and stable predictions across varied datasets.

The main contribution of this research includes the development of a novel dataset containing audio recordings of various disruptive and hazardous events that may occur at the entrances of residential buildings, such as harassment, quarrels, screams induced by alcohol or other substances, and the breaking of windows and doors. Additionally, we propose an image-based feature extraction method utilizing Discrete Cosine Transform (DCT) spectrograms, which enhances robustness against external noise and interference. Furthermore, we explore the effectiveness of Cochleagram and Mel spectrogram representations for audio data visualization in combination with DCT spectrograms to improve feature extraction. Finally, we introduce a CRNN model capable of extracting reliable acoustic features from each frequency–time spectrogram in a time series and integrating them into a unified ensemble model for SED.

1.3. Structure of the Paper

The structure of this article is laid out as follows: Section 2 presents an overview of the currently popular approaches. In Section 3, we inquire into the specific materials and approach used for image-based sound event detection. Section 4 concentrates on the data preparation and training process using deep learning techniques. The paper concludes with Section 5, which summarizes the main results and essential aspects of our discussion.

2. Related Work

Over the years, numerous researchers have conducted research on the classification of sound events using various methods and approaches. These range from approaches based on parametric signal processing [11] to methods for developing automatic speech recognition systems [13], typically using Mel-kepstral coefficients (MFCC) and other similar spectral methods [14], as well as signal parameters in the frequency–time domain [15] as basic parameters. In the work presented by Guodong Guo and Stan Z. Li [16], perceptual features composed of total power, sub-band power, brightness, bandwidth, pitch, and MFCCs were used as signal parameters to recognize voice signals, and support vector machines (SVMs) were used as classifiers. With the combination of perceptual features and eight MFCCs, the classification results achieved an error rate of 11.0%.

In the work of Giambattista Parascandolo et al. [17], Bi-directional Long Short-Term Memory (BLSTM) recurrent neural network (RNN) architectures were used to solve the classification problem of polyphonic audio events consisting of 61 classes based on 40 MFCCs from a signal with a frame size of 50 ms and an overlap of 50%. As a result, the classification system achieved an accuracy of 65.5% on the F1 score in 1 s blocks and an accuracy of 64.7% on single frames. In another research work on sound classification, Victor Bisot et al. [18] used an image-based approach (spectrogram images) for acoustic scene classification. They achieved an accuracy of 92.6% using a Sinkhorn kernel classifier by extracting sub-band power distribution (SPD) and histogram of gradients (HOG) features from the spectrogram image. A similar approach in the work of Alain Rakotomamonjy and Gilles Gasso [19] is based on converting the audio signal into a frequency–time representation and then extracting relevant shape features and changes in the frequency–time structure. These features were based on a gradient histogram and then fed into SVMs for classification, achieving an average recognition accuracy of 96%. Emre Cakır et al. [20] introduced a hybrid approach, CRNN, that integrates the advantages of CNNs and RNNs. This CRNN model was applied to polyphonic SED and demonstrated significant performance improvements over standalone CNNs, RNNs, and other conventional methods across four datasets of common sound events. In another approach to address the problem of audio event detection based on deep learning, Miquel Espi et al. [21] considered two approaches that emphasize the importance of extracting features from audio. The first approach combines results from several high-resolution spectrogram models at different spectral resolutions, and its superiority over a one-dimensional model was proven by practical results. In the second approach, a CNN was used to model the local characteristics of acoustic events, which gave superior results. Although the second approach has been shown to be effective, the first approach is promising and emphasizes the need to improve the combination schemes.

McLaughlin et al. [22] proposed a sound event classification system that compares the external features of an auditory image with external features based on spectrogram images using SVMs and deep learning classifiers. The performance of the system was evaluated on a standard reliable classification task at different noise levels and several improvements to the system and compared against that of state-of-the-art classification methods. Parallel research has focused on leveraging image-based representations of audio signals. Sharan and Moir [23] showed that incorporating Cochleagram features into CNN architectures enhanced robustness in noisy conditions, reporting an improvement in classification accuracy of approximately 8% over conventional spectrogram methods, with some configurations achieving accuracies exceeding 90% under controlled conditions. Similarity, Dennis et al. [24] conducted an analysis of various spectrogram image methods and found that log-Mel spectrograms often outperformed other representations, yielding classification accuracies around 85% on standard SED tasks.

The applicability of these approaches in real-world scenarios has also been demonstrated in smart city contexts. Spadini et al. [25] reported detection rates reaching about 75% in urban surveillance applications, while Ciaburro and Iannace [26] highlighted that deep neural network (DNN)-based sound event detection methods could improve safety by enhancing detection accuracy by up to 15% compared to traditional algorithms. In efforts to optimize SED for resource-constrained environments, Ranmal et al. [27] utilized hardware-aware neural architecture search to achieve environment sound classification accuracies of 88% on benchmark datasets, thereby enabling efficient edge-device deployment.

Further improvements have been observed through advanced CNN architectures. Zhang et al. [28] demonstrated that robust CNN models could attain recognition accuracies close to 88%, and Kwak and Chung [29] enhanced performance by incorporating derivative features into their networks, achieving overall F1 scores near 78%. Finally, ensemble methods have also proven effective. Nanni et al. [30] reported that an ensemble of CNN could boost classification performance by 5–7% with some systems approaching overall accuracies of 90%. Additionally, Xiong et al. [31] extended these techniques to construction activity monitoring, achieving detection accuracies of around 82% using deep learning models. In [32], multi-label neural networks were proposed to detect temporally overlapping audio events in real environments, and the frame-wise spectral characteristics of the signal were taken as the input parameters of the network for multi-label classification. The model was tested on audio recordings from real environments and achieved an accuracy of 63.8%.

Toni Heittola et al. [33] addressed the issue of the context-dependent detection of sound events, and the approach proposed by the authors consists of two stages: automatic context recognition and sound event detection stage. Contexts are modeled using a Gaussian mixture model and sound events are modeled using three-dimensional hidden Markov models.

Recent advancements in sound event detection (SED) have leveraged deep learning techniques to address various application domains. Zheng et al. [34] proposed a convolutional recurrent neural network (CRNN) system for detecting gastrointestinal sounds using data collected from wearable auscultation devices, demonstrating the potential of CRNNs in medical audio analysis. Similarly, Lim et al. [35] explored SED in domestic environments, employing an ensemble of convolutional and recurrent neural networks to enhance detection accuracy in noisy settings. In the context of audio surveillance, Arslan and Canbolat [36] investigated the performance of deep neural networks, highlighting their effectiveness in identifying sound events under controlled conditions. Furthermore, Hu et al. [37] introduced a track-wise ensemble event-independent network for polyphonic sound event localization and detection, achieving robust performance in complex acoustic scenes. These studies collectively underscore the versatility of deep learning architectures in SED across medical, domestic, surveillance, and polyphonic environments.

3. Materials and Methods

3.1. Workflow

Our methodology was scrupulously designed to attain our objectives. The project was divided into six segments: data acquisition, pre-processing, data annotation, feature extraction, data augmentation, and acoustic modeling. For the data acquisition, the required sounds were collected from crowdsourced and public open datasets. In the pre-processing phase, the audio signals were resampled to achieve a uniform sampling rate. Subsequently, spectral techniques were applied to mitigate the impact of various artifacts and noise, thereby ensuring data consistency and enhancing overall signal quality for further analysis.

In the next step, the data labeling process was carried out, and three different types of spectrogram images, DCT, Cochleagram, and Mel, were obtained during the feature extraction phase. Augmentation techniques were then applied to these spectrogram images, in the final step of developing an acoustic model based on an ensemble of CRNNs. The process of the proposed methodology is shown in Figure 2.

3.2. Challenges in Sound Event Classification (SED)

The development of automated systems for SEDs is hampered by a number of issues: the dependence of detectable sounds on their source in the natural environment; that sounds have different acoustic characteristics (e.g., some sounds are relatively short while others are longer in time and can be harmonic); external factors such as the distance of the incident sound from the microphone and data collection as well as annotation procedures complicate the development of SED systems. In addition, the natural environment is polyphonic, meaning that several sounds are active simultaneously during the analysis process, sound events demonstrate harmonic relationships with each other, and fundamental frequencies form small-integer relationships, further complicating the separation of sounds.

There are no predefined rules for how sounds can co-occur, therefore modeling them requires the use of data-derived statistics such as direct counts and the degree of overlap between different classes in data.

The number of possible classes of sound phenomena is probably very large, since any object or creature can produce sound as a natural phenomenon [38]. In practice, each SED application is customized for different classes of sounds and is used in a variety of environments. Given this variability, no datasets or models are universally relevant. This is why each application requires data collection and system architecture to satisfy its unique requirements.

This is in direct opposition to other areas of sound event processing, such as speech recognition, where language has a set of specified classes (phonemes), and language models help acoustic models perform recognition tasks.

3.3. Data Acquisition

Depending on the purpose of the SED, audio signals may vary depending on the acoustic environment (e.g., outside or inside a building, type of reflective surfaces), the relative location of the sound source and microphone, the capture device used, and the source of interfering noise.

The purpose of our study is to detect the sounds of various events that disturb the peace of the population in and around the entrances of multi-story residential buildings: glass breaking, various knocks (door slam, door knock), harassment, fights, shouting caused by alcohol or other substances, various sounds of construction tools, opening and closing of the front door, loud music, car horns, and the sounds of everyday events near the entrance of the building, including sounds of children playing, footsteps, and conversations of ordinary people.

Our data collection project on the sounds mentioned above is called UzSoundEvent. The data collection for this project was conducted jointly with researchers at the “Machine learning and Intelligence” laboratory from the Samarkand branch of the Tashkent University of Information Technologies, named after Muhammad Al-Khwarizmi, and police inspectors in the region. The dataset was collected by two means: crowdsourcing and open source sound event datasets.

3.3.1. Crowdsourcing

The crowdsourcing approach consists of three major stages—namely, choosing places, recording sounds, and audio checking—which will be thoroughly described in the following sections.

Choosing places. To collect sounds disturbing public order and various emotional events (e.g., harassment, fights, shouting, yelling caused by alcohol or other substances, breaking of windows and doors), we, together with police, selected entrances of residential buildings in neighborhoods with unfavorable social conditions and poor lighting.

Recording sounds. A cloud-based approach was chosen to record and collect the sounds. The recording process included a Raspberry pi4 with a Wi-Fi module, mini-computer, and omnidirectional electret condenser microphone. The audio file was recorded in 16 kHz, 16 bit, and mono format. The “omnidirectional” microphone had a sensitivity of −44 dB and frequency characteristics of 20–20 KHz. You can connect it to a breadboard or perfboard or solder wires to the small wires hanging out of the rear. All recorded audio was saved in the Google Drive cloud. Figure 3 below shows the functional diagram of an IoT module for recording and storing event sounds.

The recording of the sounds of these events is assumed to take place at a distance of 20 m from the entrance to the building. The process of sound recording and information gathering was conducted throughout the year in the evenings from 19:00 p.m. to 07:00 a.m. in 26 entrances of 13 residential buildings.

Audio checking. To ensure the high quality of the data gathered, the recorded sounds were reviewed by project participants. In order to quickly extract the event sounds needed for our study from the large amount of audio data, audio data from moments of rule violations recorded by the police during the study period were checked, and event sounds were extracted. There was a total of 6 incidents of violence (harassment), 36 fights and quarrels, 33 screams under the influence of alcohol or other intoxicants, and 37 cases of breaking windows and doors.

3.3.2. Open Source Datasets

Currently, there are ready-made open source datasets such as UrbanSound8K [39], AudioSet Ontology [40], FSD50K [41], TUT Sound Events 2017 [42], and ESC-50 [43]. In our study, we used other relevant event sounds (different types of construction tool sounds, doors opening and closing, loud music sounds, car horns, footsteps, children crying, guns shooting, sirens) from this dataset.

3.4. Data Annotation

The problem of classifying sound events is multi-label classification, and therefore, there are rules for labeling data [44]. To customize the task of multi-label classification in short time segments, the audio annotation should also contain information about audio events in short time segments. There are special requirements for this type of annotation, which are called strong labels. That is, the annotation includes temporal information for each sound sample, including its start time and offset [45]. The strong-label process is performed as shown in Figure 4.

Manually annotating large amounts of audio data is a difficult and time-consuming task. Manually annotation starts and offsets in polyphonic mixes is also difficult, slowing down the annotation process when strong labels are needed. To facilitate the annotation process, many web-based audio annotation tools have been analyzed for use [46,47,48,49,50].

Among the tools analyzed, we used online audio annotation tools like [50] SuperAnnotate [51].

3.5. Pre-Processing

Pre-processing is usually implemented before the feature extraction stage. The primary goal of this process is to improve the extraction of important features of the incoming signal to maximize the efficiency of audio analysis in the subsequent stages of the analysis system. This can be realized by reducing the noise effect or by enhancing the prominence of the target sounds within the signal.

The input signal in the time domain is of the form y(n) = x(n) + s(n), containing only the events of interest x(n) and the additive noise s(n). In this material, the input acoustic signal y(n) is resampled with a sampling rate of fs = 16,000 Hz, 16-bit resolution, and single-channel resampling. The signal was framed at 32 ms and 50% overlap, and we applied a Hamming window to smooth the signal at the boundaries. After that, we used a short-time Fourier transform (STFT).

The noise reduction process involves two steps: noise power spectral density (PSD) estimation [52] and spectral enhancement. PSD noise estimation is critical for accurate noise reduction. The Minimum Statistics (MS) method is used in cases of slow time-varying noise. The noise spectrum computed from the decision-driven approach (DDA) was dynamically updated using a weighted combination of the previous noise estimate and the instantaneous spectral power minimum.

The recursive framework of the approach facilitates smoother estimation of both noise and signal components, thereby minimizing the introduction of artifacts. Finally, the time-domain signal x(n) is reconstructed by applying the inverse short-time Fourier Transform (ISTFT). An overall view of the noise reduction procedure is illustrated in Figure 5.

3.6. Feature Extraction

Audio feature extraction can be categorized into time-domain, frequency-domain, and time–frequency-domain feature representation, as statistical and psychoacoustical. The feature extraction pipeline utilized in various acoustic feature analyses involves several key steps, including framing, windowing, calculating spectral coefficients, and subsequent feature analysis [52]. When analyzing audio signals, the large amount of data available involves the use of relative energy distribution over the frequency, frequency-domain characteristics, or frequency–time-domain characteristics. The most common transform used for audio signals is the discrete Fourier transform (DFT) [46]. Other transform algorithms for audio signals include the constant-square transform (CQT) [53], MFCCs [54], and the discrete wavelet transform (DWT) [55,56].

Audio Representations as Images

In this research work, we generated spectrogram images from an audio signal using a Discrete Cosine Transform (DCT), MFCCs, and Cochleagram:

• DCT-based spectrogram: The incoming audio signal has a duration of 2 s (sampling rate, 16 kHz); after going through a framing procedure (frame size = 32 ms, overlap = 50%). the DCT spectral coefficients are calculated for each frame. The formula for the DCT of a 1D signal x[n] is given by

$$X\left[k,r\right]=\sum _{n=0}^{N-1}x\left[n\right]\times cos cos \left[{\displaystyle \frac{\pi }{N}}\left(n+{\displaystyle \frac{1}{2}}\right)\times k\right] ,k=\mathrm{0,1},..,N-1$$

(1)

where N is the length of the window, x[n] is the time-domain signal, $X\left[k,r\right]$ is the kth harmonic corresponding to the frequency f(k) = kFs/N for the rth frame, and Fs is the sampling frequency. The spectrogram values are obtained from the log of the magnitude of the DCT values as

$$S(k,r)=loglog \left|X\left(k,r\right)\right|$$

(2)

To obtain the same time–frequency image resolution for all signals, each sound event signal was divided according to the framing rule [57] into 128 frames with 50% overlap between frames. A Hamming window was then applied to the frames, and a DCT was applied using 512 points so that the final image had dimensions of 256 $\times$ 128.

After obtaining the spectral coefficients for each frame, the spectrogram image was constructed in grayscale, where each value is scaled in the range from 0 to 255. The sequence of spectrogram image formation is shown in Figure 6. Figure 7 shows the time, frequency, and frequency–time (spectrogram) regions of the loaded signal used to generate spectrogram images.

2.

MFCC spectrograms [58]: These spectrograms are derived by computing coefficients that correspond to compositional frequency components, utilizing the short-time Fourier transform (STFT). This extraction process involves applying each frame of the frequency-domain representation to a Mel filter bank. The intensity values of the Mel spectrogram image are calculated using the energy of the filter bank, similar to the MFCCs [12] but without using DCT. The Mel filter bank output of the mth filter can be determined as

$$C\left(m,r\right)=\sum _{k=0}^{{\displaystyle \frac{N}{2}}-1}F\left(m,k\right)\left|X\left(k,r\right)\right|, m=\mathrm{1,2},..,M$$

(3)

where $C\left(m,r\right)$ is the filtr bank energy of the mth filter in the rth frame, $F\left(m,k\right)$ is the normalized response of the triangular filters uniformly spaced on the k-Mel scale, and $m$ is the total number of Mel filters. The signal processing parameters for generating a Mel spectrogram are the same as for generating a DCT spectrogram. In total, 128 values of Mel coefficients were used from each frame, and an image of size 128 × 128 was generated. The log values are then calculated as in Equation (2).

3.

Cochleagram: This mapping models the frequency selectivity property of the human cochlea [23,30]. To extract cochlear images from a signal, the incoming signal is first filtered using a gammatone filter bank. The gammatone filter bank is a comb of gammatone filters, each of which is associated with a certain characteristic frequency. The impulse characteristic of a gammatone filter with center frequency fc is described by the expression

$$g\left(t\right)=t^{\left(l-1\right)}{exp}^{{-2\pi bERB\left(f_c\right)}^t}cos\left(2\pi f_{c}t\right),t>0,$$

(4)

where t—time; l—filter order; b—parameter that adjusts the filter bandwidth; fc—filter center frequency; and ERB(fc)—equivalent rectangular bandwidth of an auditory filter. In practice, the parameter values l = 4 and b = 1.019 [59] are often used. Typically, the central frequencies f_c of a filter bank are distributed uniformly relative to the ERB (equivalent rectangular bandwidth) scale. The ERB scale is similar to the scale of critical bands of human hearing. Knowing the frequency f (in Hz), one can reference the ERB scale using expression (3).

$$ERB\left(f_c\right)=\mathrm{24,673}\left(0.004368f+1\right)$$

(5)

After the signal is filtered, a spectrogram-like image can be generated by adding energy to the windowed signal for each frequency channel. The process of converting the image can be found in [60].

$$C\left(m,t\right)=\sum _{n=0}^{N-1}\left|\widehat{x}\left(m,n\right)\right|w\left(n\right), m=1,\dots , M$$

(6)

where $\widehat{x}\left(m,n\right)$—filtered signal from the gammatone; and $C\left(m,t\right)$—is the mth harmonic corresponding to f_cm for the tth frame.

With the Cochleagram representation, we set the number of gammatone filters to 256. The filtered signal was divided into 128 frames with 50% overlap between frames. The result was an image of size 256 × 128. The signal processing parameters for generating Cochleagram images were the same as for generating a DCT spectrogram.

Figure 8 shows examples of grayscale images of a DCT spectrogram, Cochleagram, and Mel spectrogram.

4. Data Preparation and Training

4.1. Data Augmentation Techniques

Data augmentation is an increasingly popular method that also helps to reduce selection bias. This method involves altering the training data during training to introduce more variance into the sample. As a result of augmentation, the model works with a larger and more diverse training sample, which in turn can better describe the decision boundaries between classes. Perturbations can range from simple effects such as additive background noise, time dilation, and pitch changes to more complex domain-specific deformations (e.g., changes in speech path length or variable rate perturbations) [60,61,62,63]. The underlying concept is that training on these augmented examples enables the model to develop invariance to the specific deformations applied, thereby enhancing its robustness and generalization capabilities.

In our research, augmentation methods were only applied to the group of voices collected through crowdsourcing, because the volume of voices collected over a year is not sufficient compared to other voices. We used Time Masking and Frequency Masking techniques to simulate temporary signal loss and frequency dropouts by randomly masking parts of the time and frequency axis spectrograms; to simulate speed changes, the spectrogram was deformed along the time axis using Time Warping, and additive noise was used to add random Gaussian noise to simulate background noise. In addition, we applied Time Shifting to shift the spectrogram along the time axis to simulate delayed or leading signals and Dynamic Range Compression to compress the amplitude range to simulate effects such as sound normalization.

4.2. Dataset

This study employed both crowdsourcing data and publicly available open datasets to cover a wide range of sound events and situations. The datasets are summarized here.

4.2.1. Crowdsourced Dataset

The UzSoundEvent dataset was developed through a crowdsourcing campaign. In

Table 1. Information of crowdsourced dataset (UzSoundEvent).

	Harassment	Quarrel	Screams	Sounds of Breaking
Number of clips	6	36	33	37
Duration (minutes)	35	77	81	52

, it is shown to comprise 112 audio pieces that last approximately 4 h. The dataset contains four separate types of sound events, with an average clip time of two minutes and a sample rate of 16 kHz. The recordings were made in a wide range of real-world contexts, including urban, rural, indoor, and outdoor settings. Pre-processing methods, including spectral denoising and cropping, were used to improve the audio quality. Annotation was performed using a multi-label approach, with an average of 2.5 labels per clip.

4.2.2. Open Datasets

In addition to the crowdsourced dataset, publicly available datasets were used to broaden this study’s applicability. The datasets, sound classifications, number of clips, average duration, and sample rates are all shown in

Table 2. Information about sounds collected from open datasets.

Sound Class	Datasets	Number of Clips	Average Duration (Seconds)	Sample Rate
Construction Tools	UrbanSound8K	~394	~4.0	44.1 kHz
Door Opening/Closing	AudioSet Ontology	~500+	~10
Loud Music	UrbanSound8K	~1000	~3.5
Car Horn	AudioSet Ontology	~1000+	Varies (5–30)
Footsteps	FSD50K	~500+	Varies (5–30)
Crying Baby	ESC-50	~100+	~5.0
Gun Shooting	UrbanSound8K	374	~2.5
Street Sound	TUT Sound Events 2017	~200	~5.0	48 kHz
Siren	UrbanSound8K	~929	~3.7	44.1 kHz

.

The dataset statistics are reported in

Table 3. The dataset specifications.

Sound Category	Total (Minutes)	Train (80%)	Validation (10%)	Test (10%)	Source
Construction Tools	26.4	21.12	2.64	2.64	Open source
Door Opening/Closing	83.4	66.72	8.34	8.34
Loud Music	66	52.80	6.60	6.60
Car Horn	300	240	30	30
Footsteps	150	120	15	15
Crying Baby	10.2	8.16	1.02	1.02
Gun Shooting	15.6	12.48	1.56	1.56
Street Sound	16.8	13.44	1.68	1.68
Siren	60	48.00	6.00	6.00
Harassment	35	28.0	3.50	3.50	Crowdsourcing
Quarrel	77	61.60	7.70	7.70
Screams	81	64.80	8.10	8.10
Sounds of Breaking	52	41.60	5.20	5.20

. Our dataset contains 5055 audio files of different lengths totaling 14.14 h, and the dataset contains strongly labeled data. The dataset consists of 13 separate sound categories with their corresponding durations and sources. The total duration of each category (in minutes) was divided into three subsets: 80% for training, 10% for validation, and 10% for testing. The number of clips and duration distributions are shown in Figure 9.

4.3. Development of Ensemble CRNN-Based Model for Sound Event Detection

Neural networks are well known for outperforming traditional machine learning approaches in a variety of pattern recognition tasks, making them the most common method, particularly for image and audio recognition. The following section focuses on an ensemble of CRNNs (GRUs), which are used in our research.

Convolutional Recurrent Neural Network

In the proposed ensemble CRNN-based model for SED, we first consider a time–frequency representation $X\in R^{T\times F}$, where T and F, respectively, denote the number of time frames and frequency bins (e.g., DCT coefficients). A series of L_c ∈ N convolutional layers are then applied to $X$ to extract local time–frequency features. Each layer l performs a two-dimensional convolution using a bank of filters $W^{(l,m)}$ of size $K_t^{\left(l\right)}\times K_f^{\left(l\right)}$, producing outputs $X^{\left(l\right)}$ for $m\in \left\{1,\dots ,C^{\left(l\right)}\right\}$. Specifically, the feature map of the m-th filter at layer l is given by

$$X_{t,f,m}^{\left(l\right)}=\sigma \left({\sum }_{i=0}^{K_t^{\left(l\right)}-1}{\sum }_{j=0}^{K_f^{\left(l\right)}-1}{\sum }_{c=1}^{C^{(l-1)}}{W}_{i,j,c}^{(l,m)}{X}_{t+i,f+j,c}^{(l-1)}+b^{(l,m)}\right),$$

(7)

where σ(⋅) is the nonlinear activation (e.g., ReLU), and $X^{(l-1)}$ is the output of the previous layer (with $X^{\left(0\right)}=X$). A non-overlapping pooling operation is then performed over the frequency axis to reduce dimensionality, typically captured by

$$P_{t,f,m}^l={max}_{0\le j<P_f}{X}_{t,P_f*f+j,m}^{\left(l\right)}$$

The final pooled features are stacked along the frequency axis to form a sequence ${\left\{Z_t\right\}}_{t=1}^T$ of dimension $D=\left({\displaystyle \frac{F}{P_f}}\right)\times C^{\left(L_c\right)}$, which serves as the input to $L_r$ stacked recurrent layers (GRU). In a GRU, the hidden state $h_t$ and cell state at time t are updated using the following set of equations:

$$\begin{gathered} z_t=\sigma \left(W_z{Z}_t+U_z{h}_{t-1}+b_z\right), \\ {r}_t=\sigma \left(W_r{Z}_t+U_r{h}_{t-1}+b_r\right), \\ {n}_t=tanh\left(W_n{Z}_t+U_n\left(r_t⨀h_{t-1}\right)+b_n\right), \\ { h}_t=\left(1-z_t\right)⨀n_t+z_t⨀h_{t-1}, \end{gathered}$$

(8)

where $z, r_t, \mathrm{a}\mathrm{n}\mathrm{d} n_t$ are the update, reset, and hidden candidate states, ⊙ signifies elementwise multiplication, $\sigma \left(·\right)$ is the elementwise sigmoid function, and $tanh\left(·\right)$ is the hyperbolic tangent (Figure 10).

The last hidden states, $h_t^{\left(L_r\right)}$, is transmitted to a fully connected feedforward layer, which generates frame-level class probabilities $y_t=\sigma \left(W_{ff}{h}_t^{\left(L_r\right)}+b_{ff}\right)$, where $y_t=R^C$ and σ(⋅) is the sigmoid activation for each C sound event class.

These probabilities are then binarized to obtain $\widehat{y_t}={\left\{\mathrm{0,1}\right\}}^C$ via the thresholding function, where ${\widehat{y}}_{t,c}=1$ if $y_{t,c}\ge 0$ and 0 otherwise, thereby yielding per-frame event activity decisions.

Finally, to exploit multiple representations such as DCT, Mel spectrograms, and Cochleagram, the above CRNN pipeline is replicated for each representation $k\in \left\{1,\dots .,K\right\}$, generating predictions $y_t^{\left(k\right)}$ that are then combined—commonly by averaging to form an ensemble output, Equation (9).

$$y_t^{\left(ens\right)}={\displaystyle \frac{1}{K}}\sum _{k=1}^K{y}_t^{\left(k\right)}$$

(9)

The final ensemble prediction is subsequently thresholded to yield improved, robust sound event detection by leveraging complementary information from distinct time–frequency domains. Figure 11 shows an illustration of the system structure.

4.4. Regularization

In order to address overfitting in our proposed model, we incorporate a variety of regularization techniques proven effective in earlier work. Following the original dropout concepts introduced in the seminal work by Srivastava et al. [64], we applied dropout not only in the convolution stages but also within the recurrent layers, consistent with the theoretically grounded approach for RNNs proposed by Y.Gal [65]. We also utilized batch normalization to stabilize and accelerate training by normalizing intermediate activations [66].

Beyond these layer-level methods, we find that ensembling multiple spectrogram-specific CRNN branches provides an additional model-averaging effect, yielding robustness against variance in the training data [67,68]. By combining these mechanisms—dropout, batch normalization, and ensemble model averaging—we obtain substantial improvements in mitigating overfitting and enhancing the overall performance of our multi-label sound event detection system.

4.5. Training

To train the proposed model on the annotated dataset, the training set was separated into training, validation, and test sets in a ratio of 80/10/10 (Table 3). Because the dataset is imbalanced and has multiple labels, this study used the iterative stratification method [69] and data augmentation methods outlined in Section 4.1 to achieve a balanced distribution of training, validation, and test sets.

The proposed ensemble CRNN-based model architecture incorporates three types of input spectrograms (DCT, Cochleagrams, and Mel), all processed through a CRNN. The architecture begins with two convolutional layers, each followed by max-pooling operations to reduce spatial dimensions while retaining essential features. After convolutional processing, the feature maps are reshaped into sequences suitable for two bi-directional GRU layers, each with 64 hidden units. Dropout regularization (0.3 for DCT and Cochleagrams, 0.2 for Mel spectrograms) is applied to both convolutional and GRU layers to mitigate overfitting problem. The last dense layer uses a sigmoid activation to predict numerous sound events in a multi-label setup using binary cross-entropy as the loss function. Training is optimized using the Adam optimizer with a learning rate of $1\times {10}^{-4}$, a batch size of 32, and 100 training epochs. Batch normalization is employed, and an ensemble method is applied to aggregate model outputs via voting. This architecture effectively combines convolutional and recurrent layers to extract spatial and temporal features for sound event detection. Complete information on the hyperparameters of the proposed model architecture is presented in

Table 4. Hyperparameters of the proposed model architecture.

Layers	Hyperparameters
	DCT and Cochleagram Spectrogram	Mel spectrogram
Input	$256\times 128$	$128\times 128$
Conv1	32 filters, 5 $\times$ 5, stride = 1, pad = 2, ReLU	32 filters, 5 $\times$ 5, stride = 1, pad = 2, ReLU
Pool1	MaxPool 2 $\times$ 2	MaxPool 2 $\times$ 2
Conv2	64 filters, 3 $\times$ 3, stride = 1, pad = 1, ReLU	64 filters, 3 $\times$ 3, stride = 1, pad = 1, ReLU
Pool2	MaxPool 2 $\times$ 2	MaxPool 2 $\times$ 2
Reshape	(B, 64, 64, 32)$\to$ (B, 32, 64 $\times$ 64 = 4096)	(B, Time = 32, 64 $\times$ 32 = 2048)
Bi-GRU1	64 hidden units	64 hidden units
Bi-GRU2	64 hidden units	64 hidden units
Dropout (Conv +RNN)	0.3 in Conv blocks and GRU layers	0.2 in Conv blocks and GRU layers
Dense	Number of sound events, sigmoid	Number of sound events, sigmoid
Loss Function	Binary cross-entropy (multi-label)
Optimizer	Adam
Learning rate	$1\times {10}^{-4}$
Batch normalization	32
Number of training	50
Ensemble Method	Vote over model outputs

. For feature extraction, the Python (version 3.8.0) library Librosa and our python function creating DCT spectrogram images were used in this work. For classifier implementations, the deep learning package Keras (version 3.8.0) is used with TensorFlow as the backend. The networks were trained on an NVIDIA Tesla P100 single GPU.

4.6. Metrics

We used two evaluation approaches proposed by Mesaros et al. [70], segment-based and event-based metrics, as evaluation metrics in this research study. In segment-based evolution, both the ground truth and the system’s predictions are divided into fixed-length time segments (commonly used: one second), and the performance is assessed per segment. For each segment, true positives (TP), false positives (FP), and false negatives (FN) are computed, and the segment F1 score is defined as

$$F_1^{segment}={\displaystyle \frac{2TP}{2TP+FP+FN}}$$

(10)

Complementary to the F1 score, the segment-based error rate (ER) is also calculated. Here, the error rate considers substitution errors (S), deletion errors (D), and insertion errors (I) with respect to the total number of events in the reference (N). It is given by

$${ER}^{segment}={\displaystyle \frac{S+D+I}{N}}.$$

(11)

In contrast, the event-based evaluation treats events as temporal intervals with specific onsets and offsets. An event is regarded as correctly detected if its onset, and optionally its offset, falls within a predefined tolerance around the reference annotation. With TP, FP, and FN now representing correctly detected, spurious, and missed events, respectively, the event-based F1 score is computed as

$$F_1^{event}={\displaystyle \frac{2 TP}{2 TP+FP+FN}}.$$

(12)

Similarly, the event-based error rate is defined as

$${ER}^{event}={\displaystyle \frac{S+D+I}{N}},$$

(13)

where the definitions of substitutions, deletions, and insertions are analogous to those in the segment-based evaluation but applied to entire events rather than fixed segments. The calculation process of segment-based and event-based metrics is illustrated in Figure 12 and Figure 13 [70].

Together, these metrics of the F1 score and error rates in both segment-based and event-based contexts offer complementary insights: while the F1 score emphasizes the balance between precision and recall, the error rate provides a direct measure of the frequency and types of errors obtained by the system. This dual-metric approach facilitates a more nuanced evaluation of system performance in complex acoustic scenes.

5. Results

Performance Evaluation

In this section, we evaluate the performance of the proposed model based on segment and event metrics for three different spectrogram images using CRNN model architectures and their combined ensemble model. The dataset selected for testing was composed of audio recordings from crowdsourced and public datasets. The total size of the test dataset was 1.37 h for all sound events. In segment-based metrics, the segment length was set to 1 s, whereas in event-based metrics, the tolerance for estimating the start/end time was T collar = 200 ms and the percentage tolerance for the end was Offset (length) = 50%. We conducted test experiments on the ensemble and, separately, its three neural network models (DCT + CRNN spectrogram, Cochelogram + CRNN spectrogram, Mel + CRNN spectrogram). Based on the final ensemble model, we obtained the following results, namely, the F1 segment score was 71.5% (precision: 84,2%; recallL 62.13%) and F1 event score was 46% (precision: 53.82%; recall: 40.13%), respectively.

Table 5. The F1 score and error rate values.

	F1-Score(%)		Error Rate
Methods	Segment-Based	Event-Based	Segment-Based	Event-Based
DCT spectrogram + CRNN	64.7	40.8	0.74	0.84
Cochelogram spectrogram + CRNN	53.5	37.8	1	1
Mel spectrogram + CRNN	50.4	32.8	0.97	1.2
Ensemble	71.5	46.0	0.92	1.1

details the F1 score and error rate values for segment-based and event-based metrics for the model methods. The detail information is shown in Table 5, and Figure 14, Figure 15 and Figure 16.

As can be seen in Figure 13, the “Sounds of breaking” sound received the highest F1 score across segments at 96%, followed by “Gun Shot”, “Door Opening/Closing”, and “Construction Tools” sounds with 86.1%, 81.4%, and 81.2% accuracy, respectively. The lowest score was 41.8% for the “Harassment” cases. The main reasons for this are the small size of the dataset, the relatively distant location of the event from the recording device, and the similarity of the sounds of the “quarrel” and “screams” events. In addition, noise mixing resulted in false detections.

Figure 14 is a graphical representation of the model results presented in Table 5. Figure 16 shows the event F1 score, precision, and recall of the event sounds. In this case, we can see a relatively different graph in Figure 15. As we know, event-based metrics require the model to correctly identify and classify each overlapping event independently. This is a difficult task, especially if the events have similar acoustic properties or the duration of the event sound is short.

When evaluated using the event-based metric, the F1 scores of the sounds of the events “Gun shots”, “Construction tools”, and “Footsteps” are 37%, 42.3%, and 34%, respectively. We can see that these are low scores compared to those of the other sounds. One of the main reasons for this is the sensitivity of event metrics to rare or short-lived events. Short sounds (e.g., gunshots, screams) are particularly difficult to detect because they occupy fewer frames, making them more susceptible to complete omission, and small errors in temporal localization can lead to the complete exclusion of an event from the evaluation.

The analysis of the results shows that the variation in the detection accuracy of sound events is due to a combination of factors related to the characteristic acoustic features, temporal characteristics, recording quality, and the amount of training data. Events with unique and easily distinguishable spectral features, such as “sound of breaking” sounds, show high detection accuracy because their distinct acoustic features allow the model to efficiently extract and memorize characteristic features. At the same time, events with less distinct or similar features, such as “harassment” and ”scream” incidents, make correct classification difficult due to acoustic features overlapping with other sounds, resulting in lower accuracy.

The temporal aspects also play a significant role in determining accuracy. Prolonged events provide more information to the model to extract stable features, which contributes to more accurate segmentation and classification under segmental estimation. However, short-lived and transient events, such as “gunshots” or “footsteps”, occupy fewer temporal frames, and even small errors in localization can result in missing events entirely. Strict temporal accuracy criteria in estimation, especially when using event-based metrics, further exacerbate this problem.

Furthermore, the quality of the audio recording has a considerable impact on the results. Events recorded at a high signal-to-noise ratio or close to the microphone are characterized by clear and distinct acoustic signals, which makes them easier to detect. On the contrary, sounds recorded at a distance or with high background noise lose their distinctive features, resulting in poorer recognition and higher error rates. Also, the lack of training data and class imbalance hinder effective model training, as the limited number of examples does not fully capture all variations in sound events. Thus, high accuracy in SED is achieved by having unique and distinct acoustic features, sufficient duration of the signals, favorable recording conditions, and an extensive training sample. Low accuracy, on the other hand, is due to the similarity of acoustic features, short duration of events, low signal-to-noise ratio, and limited data, resulting in difficulties in accurately localizing and classifying sounds in complex acoustic environments.

Our ensemble CRNN approach for SED excels across 13 classes, exploiting DCT spectrograms, Cochleagrams, and Mel spectrograms to achieve a segment-based F1 score of 71.5% with a small, unbalanced 14.14 h dataset. Outperforming the models of Çakır et al. (63.8%, 16 classes), Sang-Iek Kang et al. (53%, 12 classes), and Wei et al. (51.4%, 5 classes), it demonstrates robust handling of moderate class diversity. The multi-spectrogram strategy enhances noise robustness and generalization compared to single-spectrogram methods (e.g., Zheng et al., ~85%, 6 classes), while its ensemble design ensures high performance in urban safety contexts, surpassing niche methods (Wuyue Xiong et al., 82% 5 classes) in applicability. Cost-effective for low-cost hardware, it balances scalability and simplicity, though its event-based F1 score suggests limitations in temporal precision across these classes (

Table 6. Comparative experiments with other existing sound event detection methods.

Authors	Model	Dataset	Classes	Key Features	F1 Score (Segment/Event)	Application
Yüksel Arslan et al. (2018) [36]	DNN	Custom	2	MFCC	F-score 75.4%	Urban safety
Xue Zheng et al. (2020) [34]	CRNN (LSTM)	GI Sound set	6	MFCC	81.06%/-	Medical monitoring
Emre Çakır et al. (2017) [20]	Single CRNN (GRU)	TUT-SED Synthetic 2016	16	MFCC	F1frame = 66.4%, F1sec = 68.7%	General polyphonic SED
Wuyue Xiong et al. (2022) [31]	CRNN (GRU)	Custom	5	Mel spectrogram	82% (accuracy)/-	Construction monitoring
Sang-Ick Kang et al. (2022) [37]	CRNN ensemble	DCASE 2022 SELD Synthetic dataset	12	Mel spectrogram	53%/-	Polyphonic SED + location
Wei Wei et al. (2020) [71]	Adapted CRNN	DCASE	5	MFCC	51.4%/-	Domestic SED
Wootaek Lim et al. (2019) [35]	CRNN (Bi-GRU)	DCASE 2019	10	Mel spectrogram	66.17%/40.89%	Domestic SED
Ours (2025)	CRNN (LSTM) ensemble	Crowdsourced and Open source	13	DCT, Mel, Cochleagram spectrograms	71.5%/46%	Urban safety

).

6. Conclusions

This research has developed a deep learning model for detecting sounds of emotional or disruptive events occurring around residential buildings. The audio data were converted into three types of images (DCT spectrograms, Cochelograms, and Mel spectrograms) to ensure that the developed model performs effectively under different noise conditions. We used CRNN models that extract reliable acoustic features from each frequency–time spectrogram in the time series and used them as a single ensemble model for SED. This model can detect sound events and perform temporal signal localization tasks. Important features of this model are that it can identify and distinguish each individual sound event from mixed acoustic sources that include strong background noise, as has been reported in real-world environments.

To validate the proposed model, real data were collected from residential building entrances using an audio recorder for a group of sounds (harassment, quarrel, screams, and sound of breaking), as well as extracted sounds of events that may occur around residential buildings from open datasets. Two metrics were used to evaluate the performance of the model. For the 13 sound event types, the ensemble architecture achieved an F1 score of 71.5% for the segment metric and an F1 score of 46% for the event metric.

The results show that the proposed model gives good results even on small datasets that are not class-balanced. This means that the methods used in each stage of model training, such as pre-processing, feature extraction, data annotation, and the acoustic model architecture, are properly chosen.

The experiments conducted confirm the universality of the proposed approach and its applicability in a wider range and under different conditions in detecting incidents occurring near residential buildings. Considering the above observations, it can be concluded that the developed model offers an effective solution to detect different types of situations disturbing the peace of citizens around residential buildings in real time, accurately and promptly. Therefore, its new features such as less need for training sets and easy data labeling can effectively reduce the training and application costs.

The developed model can be installed on mini-computers (for example, Raspberry Pi5, Jetson Nano) for audio surveillance. This is particularly useful in potentially dangerous events, which can be detected early in heavily cluttered spaces where visual events are likely to be hidden. Further research could focus on collecting a larger and more diverse dataset covering the types of incidents that may occur near residential buildings. This facilitates the development of more general models and improves performance in real-world applications. Additionally, audio data can be integrated with video streams to create multimodal surveillance systems capable of cross-validating sound events with visual cues, thereby improving detection accuracy and contextual understanding as well as developing attention mechanisms that prioritize either audio or visual modalities based on environmental conditions.

Despite the promising results achieved in this study, several limitations should be acknowledged and addressed in future work. To address the shortcomings in event-based detection and elevate the F1 score from 46%, several research directions can be pursued.

First, enhancing model architectures with advanced temporal modeling techniques, such as transformer-based models or CRNNs with attention mechanisms, could improve the capture of long-term dependencies essential for precise event localization.

Second, integrating hybrid feature representations—such as combining spectrograms with temporal embeddings and derivative features—may enhance feature extraction and better represent event dynamics.

Third, tackling data-related challenges, such as class imbalance, through techniques like oversampling rare events or employing weighted loss functions, could improve model generalization across diverse sound events.

Additionally, our research focuses on detecting environmental sounds from hazardous events to ensure public safety. Building a dataset of hazardous sounds (harassment, screams, quarrels, etc.) is a very time-consuming process, making it difficult to develop high-accuracy recognition models. In future work, the training dataset will be expanded to increase the accuracy of the model. Finally, building on this study’s use of ensemble methods, which contributed to the current 46% event-based F1 score, further optimization could be achieved by diversifying the ensemble’s constituent models and refining post-processing approaches, such as temporal smoothing, to reduce prediction errors.