Effectively Obtaining Acoustic, Visual, and Textual Data from Videos

Featured Application

The proposed method for generating large-scale audio–image–text datasets from videos addresses the critical scarcity of high-quality multimodal data, enabling advancements in audio-conditioned image-to-image generation and related tasks. By extracting semantically aligned audio–image pairs and augmenting them with descriptive texts, this work facilitates the training of more robust multimodal models, such as those for enhancing low-resolution recordings, creating dynamic video content like music videos or virtual assistant interactions, and developing augmented reality systems that incorporate real-time environmental audio for immersive user experiences. Ultimately, it promotes the democratization of AI by providing accessible, diverse datasets that support transfer learning and reduce reliance on modality conversions, paving the way for innovative applications in fields such as creative media production, remote sensing, and deep audiovisual learning.

Abstract

The increasing use of machine learning models has amplified the demand for high-quality, large-scale multimodal datasets. However, the availability of such datasets, especially those combining acoustic, visual, and textual data, remains limited. This paper addresses this gap by proposing a method of extracting related audio–image–text observations from videos. We detail the process of selecting suitable videos, extracting relevant data pairs, and generating descriptive texts using image-to-text models. Our approach ensures a robust semantic connection between modalities, enhancing the utility of the created datasets for various applications. We also explore the obtained data, discuss the challenges encountered, and propose solutions to improve data quality. The resulting datasets, which are publicly available, aim to support and advance research in multimodal data analysis and machine learning.

1. Introduction

In recent years, there has been an unprecedented development in the world of machine learning [1]. Several models have begun to excel in creative activities (previously considered exclusive to human minds by many) [2,3], even using non-specialized hardware [4]. In this scenario, models that can generate text associated with an image have emerged [5,6,7]; similarly, others that, based on texts/prompts, are capable of generating images that can fairly faithfully represent said texts have appeared [2,8,9,10]. An example of this can be seen in Figure 1.

From this last task, usually referred to as text-to-image, several others emerge, such as inpainting [11], outpainting [12], and image-to-image [13,14]. Commonly, the conditioning of all the aforementioned tasks tends to be text-based, and there are a few popular datasets for training such models [15,16,17,18]. This is a simple example of multimodal data being used today.

Figure 1. Text -to-image generation example. Text-to-image is a technique that generates images from textual descriptions, allowing users to create visual content based on their written prompts. Some popular models that perform this task are Stable Diffusion [8,19,20,21], DALL·E [22,23,24], Imagen [9,25], and FLUX [10].

Figure 1. Text -to-image generation example. Text-to-image is a technique that generates images from textual descriptions, allowing users to create visual content based on their written prompts. Some popular models that perform this task are Stable Diffusion [8,19,20,21], DALL·E [22,23,24], Imagen [9,25], and FLUX [10].

However, it is not unheard of to find undesirable entries in any third-party dataset [26] or a lack of datasets for specific tasks (e.g., medical image analysis [27]). Similar inconveniences can also be found when dealing with datasets that include audio [28]. In particular, it has been mentioned that, relative to image datasets, audiovisual datasets are few and far between [29]. Currently, this, in turn, can be explained by the apparent low motivation for exploring fields such as audio-conditioned image-to-image [30,31,32], in contrast with text-conditioned image-to-image [2,8,13,19]. While there are numerous image-to-image works that condition the input image using text, there are not many that do so with audio (whether with or without added text involved), nor are there common guidelines helping researchers form these datasets on their own. Additionally, as we will explain below, the option of adapting textual/visual models to work with acoustic inputs has significant drawbacks that discourage it, rather than directly training an acoustic model for the given task.

It goes without saying that efforts in this topic could have an impact on the following: multimodal data analysis, correction of low-quality/low-resolution recordings, video generation for various purposes (virtual assistants, music videos, video transitions, etc.), democratization of artificial intelligence, augmented reality that incorporates the user’s environmental audio, and transfer learning with multimodal models, among others [33,34,35,36].

In light of the above and with the desire to work with a specific type of acoustic–visual data, we formalized a method to generate audio–image–text observations based on videos (including the textual modality, in order to expand the utility of our datasets), and we employed it to generate the data that we desired for our future research. This paper delves into all of that.

In summary, in this paper, we address the need for high-quality, large-scale multimodal datasets that combine acoustic, visual, and textual data (which are currently limited). Keeping in mind the importance of maintaining a strict semantic connection between audio and visual data to improve dataset quality, as well as the ideal of minimizing data modality conversions to preserve data integrity and quality, we propose a coherent and systematic approach to extracting audio–image–text observations from videos. We discuss our results, as we generated more than 2,000,000 audio–image pairs from over 280,000 videos, together with the transformation that we utilized to obtain the respective texts and some pending challenges that we encountered along the process.

2. Related Work

Our literature review provided clear evidence on the existence of relationships between audio and text that represent the same situation, as well as between audio and images, that should be further exploited by research and modern models (for a small summary on generative tasks that involve said modality combinations, consult

Table 1. A summary of the most common generative audio–text and audio–image tasks.

Task	Description	Nuances
Image-to-audio	Based on an image, an audio is generated that conveys the same semantic information as the input image.	Advances have been made in the generation of audios that mimic the possible soundscape for a given image [31,37]. In a similar fashion, audios can also be generated from videos, which are nothing more than an ordered collection of images [35,36].
Text-to-audio	Based on a text, an audio is generated that conveys the same semantic information as the input text.	Some models are able to resemble a human voice reading the text given as input (subtask usually referred to as text-to-speech [34,36,38,39]). Moreover, some even make music [40] and generate lyrics based on text input [41], or they generate sounds that accommodate a given description [31,42,43,44].
Audio-to-image	Based on an audio, an image is generated that conveys the same semantic information as the input audio.	Voice recordings can be used to condition the modification of human faces so their mouths adapt to the corresponding sounds (i.e., lip sync [34,45]), and even the whole face can be created from scratch with the aforementioned recordings [36]. In addition, some models are capable of representing scenarios where a specific audio is produced [31,35].
Audio-to-text	Based on an audio, a text is generated that conveys the same semantic information as the input audio.	The most popular subtask here is probably speech transcription (or recognition) [34,46,47,48]. However, models that remarkably generate text description (or captions) from audios in general have begun to arise in recent years [31,49,50,51].

).

Exploring the cases most relevant to image-to-image conditioned by audio, there are some examples of image generation based on audio and text [52,53], and there are even cases of image-to-image generation assisted only by audio, but for mainly specific cases such as face changes (which replace a person’s features with another’s while maintaining consistency with the original voice recording) or lip synchronizations (where, for an image of a person, a video is generated while simulating mouth movement according to a voice recording) [34,36], which could be labeled more as a case of inpainting than image-to-image. Finally, advances in other similar areas can also be highlighted (such as text-to-video, appreciable with models such as Sora [54,55], Veo [56], Gen-3 [57], and Movie Gen [58]), and more information on some of these developments can be found at [59,60].

Currently, image-to-image generation conditioned by audio is a little-explored area of high interest in the community. To the best of our knowledge, one of the best models to date for this task is the recent CoDi model [31]. This is a model that can take any combination of audio, image, text, and video inputs and create material of any of those types (a task they called any-to-any). Additionally, a new version (CoDi-2) has also been published, and it is more flexible and adapted to conversations [30]. Another similar option is NExT-GPT, which also allows for a conversational creative process, and it works as well with audio, image, text, and video inputs [61]. Despite their promising results for future iterations, they have not yet reached a quality that could be considered ideal. Probably, the best open-source model for this task is BindDiffusion [32]. This model is based on both the image generation model Stable Diffusion [19] and the multimodal encoder ImageBind, which incorporates six modalities, including, predictably, audio and images [62]. Notwithstanding its apparently higher quality than CoDi or NExT-GPT, it also has room for improvement, and it is not evident that it is always advisable to include the largest possible number of data modalities in these models (as seems to have been attempted in all of these cases).

The datasets involved in the training of the three previous models also shed some light on the lack of and demand for more multimodal datasets. For instance, CoDi needs to leverage different datasets (namely, LAION-400M [63], AudioSet [64], AudioCaps [65], Freesound 500K, BBC Sound Effect, SoundNet [66], WebVid-10M [67], and HD-VILA-100M [68]), with none of them combining all the required modalities. Similarly, ImageBind also makes use of multiple datasets (namely, AudioSet, SUN RGB-D [69], LLVIP [70], Ego4D [71], and private “large-scale web datasets with (image, text) pairings”), presumably due to a lack of simultaneous modalities and/or a small number of observations in each dataset. Lastly, the NExT-GPT team curated their own public dataset (called MosIT), with all the modalities they were interested in, although it only encompasses 5000 observations. We later compare the available datasets of those just mentioned with the one we generated.

3. State of the Art

In the last decade, image generation has experienced enormous growth, driven by significant advances in fields such as artificial intelligence, machine learning, and computer vision [72,73]. This progress has led to the creation of increasingly realistic and stylized images [74]. While, thanks to advances in the quality of computer-generated images (with recent examples such as Stable Diffusion XL [8] or 3 [20], DALL·E 3 [23,24], Imagen 3 [9], or FLUX [10]), the level of these images has reached a degree that makes it difficult to differentiate them from human-generated images; there is still much work to be done in terms of improving quality consistency, reducing bias, lowering computational costs, and facilitating user control over the generations (i.e., generating what the user actually expects/wants) [2].

To address this last challenge, one of the strategies that has been adopted is to increase the number of data modalities that the models receive (i.e., the types of data that are taken as input; e.g., text, image, audio, etc.) [31,46,60,75]. It is pertinent to comment that this increase in the number of modalities not only allows for greater control on the respective tasks but also opens a way to perform new ones (for example, a detailed analysis can be seen in [76], where the capabilities of GPT-4V, a colossal multimodal model of text and images, are particularly studied). In order to better illustrate the concept of data modalities, and inspired by the classification of data types explained in [45], in Figure 2, we present a conceptual map of the types of data modalities that can be used, along with examples for each (for the sake of brevity, in our conceptual map, we just include the most popular examples). An example of the use of multiple data modalities tends to be seen in image-to-image generation, where an image is taken as a reference to generate a new image, since the input image is usually accompanied by a text or a label to better condition/guide the final result [13]. In contrast, as seen in Section 2, audio-conditioned image-to-image generation has not been explored as much as text-conditioned image-to-image generation. The latter may be because working with audio is not as intuitive as working with text [77,78], but that does not invalidate the potential benefits that could be obtained by using audio in certain scenarios (such as those mentioned in Section 1).

Despite what we just wrote, we could still come up with ways to adapt the use of existing models to work with different data modalities from those that were originally intended. For instance, given the mentioned advancements in image-to-image models that are conditioned on textual inputs, it could be worth considering a new approach for scenarios where the objective is to perform image-to-image generations using audio instead of text. A logical strategy for this goal could be to transcribe the audios into the corresponding textual representations/descriptions, which could then be utilized within existing text–image models. This method should leverage the strengths of well-established text–image models, potentially validating their use with audio–image data or data of another kind that are unlike the originally intended text–image data. However, it is crucial to acknowledge that, in addition to the fact that fields such as audio-to-text conversion are still evolving and have not received as much attention as their visual counterparts [35,47,89,90], such approach presents several challenges that should be kept in mind. Let us review the main ones:

A

Word limit in current models: currently, the problem of increasing the token window (i.e., words and characters) of text-to-image and audio-to-text models is open. For example, Stable Diffusion (an open-source neural network model that generates images based on text and/or image [19]) has a context window of 75 tokens [91].

B

Compatibility between text–image and audio–text models: Even if a capacity of hundreds of thousands of tokens is reached to describe any audio (as can be seen analogously in certain current text generation models [92,93,94,95]), the syntax of the text obtained with such an audio–text model must match that used by the respective text–image model with which it is to be combined in order to maximize communication between the two [19,76,90,96].

C

Noise incorporation (see [97] for a brief classical exploration of the definition of the term): In addition to the above, it has repeatedly been shown that transforming one modality to another is prone to incorporating noise or failing (to some extent) due to the noise that the data contain beforehand [47,98,99,100]. As a result, the more transformations we make, the more noise we risk adding in the process.

D

Incorporation of biases: Finally, it is pertinent to highlight that, influenced both by the data and their training architectures and configurations, models tend to prioritize and specialize in certain types of audio and have their own preferences for describing them [101,102,103,104]. For example, typical cases of this can be seen in the underestimation/distortion of the order of events [90,96] or in the omission of details considered irrelevant [96,105].

It is due to these reasons that, even if, in some cases, audios could/can be converted into texts and images conditioned with the generated texts, this is a significantly more problematic approach than just using audios and images. For this reason, in this research, we claim that, when working with a given set of modalities, it is convenient to perform the lowest number of data modality conversions possible. Furthermore, we believe that more audio–image research is needed to better address the respective tasks instead of just attempting to get by with what is already available.

Complementarily, it is relevant to point out that, as alluded to in [37,49,50,51], there are not many public datasets with audio–text pairs. In our opinion, and despite the issues enumerated previously with modality transformation approaches, the best that can be done in this scenario is to leverage a model such as CLIP [5] or BLIP [6], which, for a frame/image of a video, could return a descriptive text. Said text could, in turn, be paired with a section of audio from the video that coincides with the time interval from which the frame was extracted.

The generation of audio–image–text observations could easily be automated, so the biggest challenge would lie in finding relevant and varied videos (as well as those free from copyright conflicts). In any case, the videos collected in other research could be leveraged, within which there are recordings of musical instruments [106,107], as well as various objects, animals [108,109], and even different everyday environments [66,110].

Regarding the kind of data collected, while it would be interesting to include relationships according to the lexical meaning of spoken words, as done in [111] by relating spoken numbers and drawings of them, it would probably be better to focus on strictly non-abstract and non-artificial relationships (i.e., sounds only related to the recordings of when they were generated). This would restrict the training of the any model with these data (simplifying the range of relationships they must incorporate), facilitating convergence, and it could even make its generations more intuitive.

In summary, we have noted a valuable opportunity to explore audio–image and audio–text tasks. This demands a great volume of data, for which there are not as many datasets as one would hope for or common guidelines on how to collect the data. Due to this, in this research, we precisely propose a method of obtaining related audio–image–text observations from videos, and we describe the datasets that we created with it.

4. Materials and Methods

In the rapidly evolving landscape of multimodal data research, the integration of different kinds of data has become increasingly relevant. In this this section, we will outline a systematic approach to generate audio–image–text observations from videos. By leveraging high-quality video content, we aim to extract meaningful audio–image pairs and generate descriptive texts that enhance the utility of the resulting datasets. None of the steps that we explain below is truly novel by itself, but the combination of all of them is. We chose to unify these techniques, as they have been validated by previous works, although many did not received a proper explanation or even a brief mention. This method should be helpful to face the current scarcity of comprehensive multimodal datasets.

We will describe a multimodal data collection and processing method (specifically, for generating audio–image–text observations based on videos). It involves three key phases, which, for order’s sake, we will explain in subsections of their own: 1. Initially, suitable videos are selected, prioritizing high-quality and continuous recordings, with synchronized audio and frames (see Section 4.1). 2. The next phase involves extracting audio–image pairs from these videos, ensuring that the audio is closely tied to the visual content and minimizing file sizes without significant information loss (see Section 4.1). 3. Finally, the extracted pairs are used to generate descriptive texts using an image-to-text model, creating a comprehensive dataset for further use (see Section 4.3). All of this is also illustrated in Figure 3.

4.1. Video Selection

First of all, it is essential to talk about the videos that we would want to work with. In addition to obviously avoiding copyrighted material and favoring HD videos with Hi-Fi audio, ideally we would hope to mainly use continuous recordings (i.e., without cuts or mixes), where each audio recording is strictly associated with the footage (i.e., without sounds that are not actually being produced in the images). Ensuring that the audio is strictly associated with the corresponding images/frames will allow for a consistent and accurate semantic connection between them, regardless of the task for which the data is being used. Additionally, using continuous recordings increases the likelihood of finding suitable video fragments to convert, especially when seeking longer audio segments.

Once again, as stated in Section 3, one can leverage public material from other research, such as that from [112]. Keep in mind that if we are not certain of the alignment between the frames and audios of our videos, we could perform an acoustic–visual alignment preprocessing step to fix this issue [113,114]. However, as we will see in the next subsection, we do not require much precision and, therefore, consider such preprocessing futile in our method. After we have collected our videos, we can start extracting pairs that consist of an image and its corresponding audio.

4.2. Audio–Image Pair Extraction

Let us define the properties of the images and audios with which we will work; they are designed to minimize their size as much as possible while preserving their core contents. Based on what has been seen in other works that generated good results [34,53,111,115], we would generally advise for the images to have a resolution of 512 × 512 pixels, in .jpg or .png format (.jpg is probably the best option, as it usually uses less space) and in RGB24 (a standard color model, consisting of a red channel, a green channel, and a blue channel, with values ranging from 0 to 255); for the audios, we would suggest a duration of 1 s, with 16,000 Hz and 16 bit depth, and they should be in .wav format and monophonic (i.e., with a single channel). In fact, these are the properties we chose for the datasets that we will show in Section 5.

Regarding the strategy for extracting audio–image pairs, the following procedure is proposed (which is also summarized in Figure 4).

• Inspect each video, evaluating if it has black borders. If so, these must be cropped to only consider relevant information in the final images. This can be accomplished by taking a frame in the middle and verifying if each first and last column/row does not have a pixel with value higher than a certain threshold in any of its channels (we use a threshold of 15 for this). In that situation, that column/row should be deleted, and the step is repeated until all of the black borders have been erased (similar to what is done in [116]).
• To ensure that no drastic/unnatural changes are present in any audio, go through each video frame by frame. If an abrupt change is detected (for example, if the average of the squared differences of pixels between two consecutive frames is greater than a threshold of 90), then proceed to divide the footage into two and, for all purposes, treat them as distinct videos going forward. This is known as shot boundary detection [117], particularly of the pixel-based approach kind (there is no well-extended threshold for this, so it is up to the architect of the respective dataset to find an appropriate value; we empirically found that the aforementioned 90 for the average of the squared differences of pixels between two consecutive frames works for our interests, but it may be different in other cases). It should be remarked that the videos with fade transitions could present some problem with this approach and, to compensate it, more future frames could be used in the comparison.
• For each resulting video fragment, extract consecutive audios of one second, along with the frame that is approximately in the middle of that time interval to form the respective pairs. Please note that this is known as middle frame extraction, and it is a well-extended heuristic to select a representative frame of a video fragment, which should have better odds to properly match semantically with the respective audio [118,119,120,121]. If the final part does not reach one second, it must be ignored.
• Discard pairs whose audio has at least a given amount of continuous silence, as they will probably not contain enough information to be useful (we looked for continuous intervals of 0.5 s where none of their samples had an absolute value higher than 100) (keep in mind that we are considering samples of 16 bits, implying that the values they take go from −32,768 up to 32,767; once again, there is no well-extended threshold for this task, and the value we chose was found empirically). This, in turn, can be combined with a discount of pairs where the mean of all pixels in the image does not surpass a given threshold (we suggest a threshold of 10). The latter should further ensure that no frames that are too dark are included.
• To increase diversity in the data (and thus not skew the research), also consider skipping a given number of pairs from each video fragment (we just kept one pair from every three).
• To preserve the dimensions, crop each frame according to the smaller dimension and around the center of the image, and rescale to 512 × 512 pixels. In addition, make sure to use the correct configuration for both saved files.

An example of steps 3, 4, and 5 of this process, where the extraction and filtering of audio–image pairs is used on an isolated video fragment, is shown in Figure 5. The blue fragment is discarded, as it is the last one and it does not reach a duration of 1 s. The red fragment is also not considered due to it having at least 0.5 s of continous silence. Finally, just one of every three pairs is considered (denoted by their green color) in order to increase the diversity in the data.

4.3. Text Generation

After we generated our audio–image pairs, we are ready to create the respective texts for each one of them. As suggested in Section 3, this will be accomplished by taking each image from every pair and, based on it, generating an associated text with the image-to-text model BLIP [6] (for the sake of speed, we used 2 beams, with a minimum length of 10 tokens and a maximum of 20); the audio will be represented as a vector of 16,000 elements, with signed 16 bit integers. The motivation for using an image-to-text model lies in the fact that the manual writing of textual descriptions for each audio–image pair is a time-consuming process that makes it impractical for a large number of observations. BLIP is also a well-tested model with a relatively fast inference time under the appropriate configuration, and it has a validated performance [122,123]. It is worth commenting that this modern possibility of leveraging image-to-text models is not something particularly novel and has also been validated in similar research [124,125,126]. A reasonable alternative would be to employ an audio-to-text model instead (like the ones mentioned in Table 1), although such models still need more development before being used reliably in tasks like this. In the end, both the text and the vector will then be added as a new observation/entity in a data table so that they can be manipulated more easily. For better results, it should also be pointed out that if the reader has access to a more advanced computer, then a more sophisticated image-to-text model (such as BLIP-2 [7]) should be leveraged instead of BLIP.

An illustration of how this preprocessing would look is shown in Figure 6. It is relevant to note that the use of a table to save the resulting data is an optional step, and the data can be stored in any form that best suits the user.

An interesting nuance to highlight is that, in the world of audio processing, there has been a tendency to prefer converting audios into spectrograms, moving from the temporal space to the temporal–spectral space, in order to facilitate pattern extraction with classical methods. This is still seen with more modern techniques [111,115,127], but, in this paper, we are only interested in creating datasets with common audio. Therefore, such a conversion is omitted in our case.

It is also worth mentioning that the minimum number of observations (or samples) “necessary” to train machine learning models is a topic open to debate (even for LLMs [128]). While a popular rule of thumb is to employ at least ten times the number of parameters of the respective model, more formal and older estimations determined that twenty times the number of parameters would be reasonable [129]. This aspect should be kept in mind when generating any dataset to train machine learning models.

Lastly, we can comment that one could even incorporate complementary data created by any-to-any models [30,31,62]. While this could be enticing at a first glance, it is a must to always remember the concerns presented in Section 3 about generating or converting data with third-party models (whether publicly validated or not). For most cases, we strongly advise prioritizing non-artificial data.

5. Results and Discussion

This section presents the datasets that we created using the method described in Section 4.

To create our audio–image pairs, we utilized videos from the public MUSIC [106,107], AudioSetZSL [108,109], and SoundNet datasets [66,110]. The videos that we employed from MUSIC dataset only contain solo performances of twenty-one different kinds of instruments, while the other two are much more diverse, ranging from musical instruments to various objects and animals, as well as different everyday environments. This diversity is relevant, as it brings substantial versatility to our final dataset. Given that AudioSetZSL is intended solely for research purposes, our datasets will also be made available for research use only, ensuring that they contribute to the advancement of multimodal data analysis while adhering to ethical standards in data sharing.

We applied the method outlined in Section 4 to process 282,081 videos, resulting in the generation of 2,240,231 audio–image pairs. From these pairs, $\mathrm{63,849}$ come from MUSIC dataset, $\mathrm{546,254}$ from AudioSetZSL, and $\mathrm{1,630,128}$ from SoundNet (proportions that can be further appreciated in Figure 7). This predominance of observations coming from SoundNet implies no issues, as this one is the most diverse dataset of the ones we chose, and we were precisely interested in favoring it to maximize the variety in our final dataset. To exemplify the kinds of images obtained by each of these datasets, in Figure 8, we exhibit a small random sample of images, highlighting the dataset from which they derived. These audio–image pairs have been named with numbers, in a rising manner, and they are organized into 639 separate .zip files, which are publicly accessible on Kaggle, categorized into the following three distinct datasets: $Part 1/3$: https://www.kaggle.com/datasets/jorvan/image-audio-pairs-1-of-3, $Part 2/3$: https://www.kaggle.com/datasets/jorvan/image-audio-pairs-2-of-3, and $Part 3/3$: https://www.kaggle.com/datasets/jorvan/image-audio-pairs-3-of-3 (accessed on 23 November 2025). This structured approach not only facilitates easy access for researchers but also promotes further exploration and utilization of the datasets in various multimodal applications.

It is relevant to point out that some possibly problematic frames for certain uses were deemed acceptable by our filters. Namely, we noted that frames with text, with blurry images, and/or with mainly a plain color were included (see images #817922, #817723 and #1323895, respectively, from row b in Figure 9). This reinforces the value of corroborating that the original videos selected for the audio–image pair extraction process align with our interests, meaning that it would be ideal to make sure that no video with flaws that we cannot fix should be considered in the first place. Of course, curating lists of hundreds of thousands of videos is unfeasible for many researchers, which implies that the heavy work must focus on harnessing videos collected by other individuals, as well as applying the respective filters to assure the desired properties. As seen with our results, the specific filters we employed seem to still have room for improvement. The impact of these questionable frames on subsequent model training should not be underestimated, as they introduce noise into the visual modality, which can degrade model performance. Future work could improve dataset quality by adding post-processing filters for blurriness (e.g., using a Laplacian operator [130]) or for detecting and removing frames with heavy text overlays. In any case, these cases we mentioned are a minority in our data (for instance, in our aforementioned random sample of observations, we only found six possibly problematic frames, which translates to ∼10%). Nevertheless, it is important to keep this in mind and we think that the removal of these pairs could also lead to some interesting research.

In a subsequent step, we generated descriptive captions for all of the images using the BLIP model. We paired these texts with the respective audios and stored them in 893 .csv tables. These, in turn, were saved in 263 .zip files, along with the associated images, and the numeric names were preserved. The final audio–image–text data can be found in the following 4 public datasets that we uploaded to Kaggle: $Part 1/4$: https://www.kaggle.com/datasets/jorvan/text-audio-pairs-1-of-3, $Part 2/4$: https://www.kaggle.com/datasets/jorvan/text-audio-pairs-2-of-3, $Part 3/4$: https://www.kaggle.com/datasets/jorvan/text-audio-pairs-2-of-4, and $Part 4/4$: https://www.kaggle.com/datasets/jorvan/text-audio-pairs-4-of-4 (accessed on 23 November 2025). In addition, as we cannot properly share audio through this document, we have prepared a public page, where we share 60 random samples of our final datasets to give a more solid idea of our results [131].

Going into greater detail regarding the final data, we conducted a small statistical study across all the texts. We confirmed that all descriptions have a length from 1 to 16 words, where the mean is $7.37$ and the standard deviation is $1.74$, approximately. We regard these values as appropriate to avoid redundancies from the image-to-text model. For comparison, we can comment that the well-known acoustic–textual dataset AudioCaps ended up with an average of $9.03$ words per description [65], which does not stray too far from our result. We also counted the number of different words in the texts and found out that there were 8824 different words in use. From the list of different words, we discarded prepositions, pronouns, conjunctions, and determiners, ending up with a new total of 8652 different words. Finally, we went through the latter preprocessed list, counting the number of times that each word appeared in an observation (counting just once per observation). We show the top 60 words with the biggest percentages of presence across all observations in

Table 2. The top 60 words that appear in most observations of our final datasets (#X means the word is the Xth most common one). Prepositions, pronouns, conjunctions, and determiners are not considered, and percentages in parentheses show the proportion of observations that include them (keep in mind that observations can contain multiple or none of these words, so these percentages are not meant to add up to a hundred).

#1 people:	15.67%	#16 white:	3.84%	#31 train:	2.55%	#46 red:	1.56%
#2 man:	15.58%	#17 road:	3.68%	#32 sky:	2.48%	#47 beach:	1.55%
#3 person:	9.31%	#18 words:	3.44%	#33 building:	2.39%	#48 tree:	1.51%
#4 group:	9.14%	#19 two:	3.35%	#34 floor:	2.32%	#49 bird:	1.4%
#5 car:	8.93%	#20 driving:	3.21%	#35 dog:	2.24%	#50 guitar:	1.38%
#6 playing:	7.66%	#21 table:	3.18%	#36 middle:	2.01%	#51 shirt:	1.36%
#7 sitting:	7.65%	#22 crowd:	3.14%	#37 cars:	1.94%	#52 boat:	1.35%
#8 room:	7.25%	#23 suit:	3.13%	#38 holding:	1.77%	#53 parking:	1.35%
#9 street:	6.83%	#24 field:	2.99%	#39 child:	1.7%	#54 little:	1.27%
#10 background:	6.6%	#25 water:	2.97%	#40 trees:	1.69%	#55 band:	1.27%
#11 down:	5.8%	#26 front:	2.96%	#41 riding:	1.67%	#56 girl:	1.25%
#12 standing:	4.68%	#27 city:	2.93%	#42 cat:	1.67%	#57 truck:	1.25%
#13 woman:	4.58%	#28 tie:	2.87%	#43 laying:	1.66%	#58 bed:	1.23%
#14 walking:	4.44%	#29 parked:	2.73%	#44 black:	1.61%	#59 chair:	1.19%
#15 baby:	3.93%	#30 stage:	2.66%	#45 night:	1.56%	#60 wall:	1.16%

. From this, we can confirm that, despite a significant amount of observations containing situations featuring people, these are not the majority according to the text descriptions. Moreover, as planned, there is a nice range of diversity, given the varied collection of words that can be seen in Table 2.

To attest to the usefulness of our data, we also conducted two additional tests to inspect both biases and diversity in our audios. On one hand, for the biases, we created a $\mathrm{65,536} \times \mathrm{16,000}$-matrix (coinciding with our chosen bit depth and total samples per audio, respectively), filled with zeros, and we proceeded to add to each element the count of times where the corresponding instantaneous amplitude was present in the given timestamps across all audios. We then plotted the resulting matrix (assigning to zero the white color and to the maximum count of the matrix a the black color, with all the counts in between a gray color that linearly denotes its closeness to each extreme) and obtained the result, as shown on the right side of Figure 10, which also illustrates the whole procedure. The observable Gaussian distributions in all timestamps coincide neatly with the theory in [132,133], and thus, this shows that no evident biases are present. Despite this, we also analyzed the skew and kurtosis of each timestamp. We found that the skews are approximately equal to $-0.04\pm 0.02$, while the kurtoses are $9.93\pm 0.09$, which means that, due to the high kurtoses, these are not real normal distributions [134]. Regardless, situations like this tend to be seen when working with large samples of values (as this makes the kurtosis more sensitive to outliers [135]), and, more importantly, this fact has no major implications that we may be concerned about. As an added point, we can also consider that such a leptokurtic (or super-Gaussian ) distribution is a well-documented characteristic of speech and audio signals, which have a sharp peak at zero (representing the noise floor), with fat tails (representing sparse high-amplitude events) [136]. On the other hand, for the diversity, we calculated the corresponding Acoustic Diversity Index (ADI) [137,138,139,140]. This is a popular metric based on the Shannon index [141], and it has high popularity for audio diversity measurements (especially in fields related to biology, although it can be employed with any kind of audio dataset). We summarize its calculation in Figure 11, and we also need to point out that, in our case, this metric may take values between 0 and ∼3.4657 (with bigger values conveying a greater diversity). Dividing this interval into three equally distributed ones, we end up with the following: $[0, 1.1552]$ for low diversity values, $(1.1552, 2.3105)$ for medium diversity values, and $[2.3105, 3.4657]$ for high diversity values. Our resulting ADI was ∼3.0525, which serves as further evidence of the diversity in our dataset.

Complementarily, the contribution of our dataset becomes clear in

Table 3. A comparison table between many multimodal datasets and ours. A means that the observations include Audios, I means the same for Images, T for Texts, and V for Videos. ✔ means that the data modality is present in the respective dataset. K stands for thousands, and M stands for millions.

Name of the Dataset	A	I	T	V	# of Samples	Contents
AudioCaps [65]	✔		✔		>45.5K	Alarms, various objects and animals, natural phenomena, and different everyday environments.
AudioSet [64]	✔		✔		>2.0M	632 audio event classes, including musical instruments, various objects and animals, and different everyday environments.
CMU-MOSEI [144]	✔	✔	✔		>3.2K	People speaking directly to a camera in monolog form, intended for sentiment analysis.
Ego4D [71]	✔			✔	>5.8K	Egocentric video footage of different everyday situations, with portions of the videos accompanied by audio and/or 3D meshes of the environment.
Flickr30k Entities [145]		✔	✔		>31.7K	Diverse environments, objects, animals, and activities, with the addition to bounding boxes to the image–text pairs.
Freesound 500K [31]	✔		✔		$500.0$K	Diverse situations, sampled from the Freesound website and accompanied by tags and descriptions.
HD-VILA-100M [68]			✔	✔	>100.0M	A wide range of categories, including tutorials, vlogs, sports, events, animals, and films.
InternVid [146]			✔	✔	>233.0M	Diverse environments, objects, animals, activities, and everyday situations.
LAION-400M [63]		✔	✔		$400.0$M	Everyday scenes, animals, activities, art, scientific imagery, and various objects.
LLVIP [70]		✔			>16.8K	Street environments, where each visible-light image is paired with an infrared one of the same scene.
MMIS [147]	✔	✔	✔		>150.0K	A wide range of interior spaces, capturing various styles, layouts, and furnishings.
MosIT [61]	✔	✔	✔	✔	$5.0$K	Diverse environments, objects, animals, artistic elements, activities, and conversations.
SUN RGB-D [69]		✔			>10.0K	Everyday environments, where each image has the depth information of the various objects in it.
SoundNet [66]	✔			✔	>2.1M	Videos without professional edition, depicting natural environments, everyday situations, and various objects and animals.
WebVid-10M [67]			✔	✔	>10.0M	Natural environments, everyday situations, and various objects and animals.
AVT Multimodal Dataset (Ours)	✔	✔	✔		>2.2M	Musical instruments, various objects and animals, and different everyday environments.

, where we compare the scope of our dataset with those of the datasets mentioned in Section 2 (leveraged by models that include acoustic, textual, and visual modalities), as well as some additional datasets that could also be employed in similar audio–image–text tasks. As we can see, most datasets do not even contain one million observations, which is a real handicap, given that modern models deal with millions of parameters and, therefore, require larger datasets to be properly trained. Currently, researchers need to arduously search for multiple datasets and artfully come up with ways to utilize them in audio–image–text tasks, as they not only be too small but also do not contain all the modalities needed and/or their contents are too specialized (not to mention the extra preprocessing steps one must add when the data are not homogeneous). All of this hinders the potential research that could be done in the field, and thus, we expect both our dataset and our detailed method contribute to easing this struggle, especially when noticing the large supply of audiovisual datasets.

Once again, there are two shortcomings that we must highlight. The first one is that, naturally, there may be a semantic drift in which a generated text accurately describes the respective image but is irrelevant to the concurrent audio. This potential disconnection between modalities may bring some issues, so it cements the need to perfect our method with future tools (possibly, adding an audio-to-text model into the pipeline). The second one is that, despite the relatively long time taken to create the text descriptions, the BLIP configuration used was fairly basic in maximizing speed. This means that the text quality is not nearly as high as one would wish for in some instances. To illustrate the latter, let us look at Figure 9. Contrasting with the appropriate descriptions that we get in cases such as row a, row b presents various kinds of errors. Image #645922 misidentifies a person in a costume as a dog, #817723 has an imprecise caption due to the poor resolution of the image, #817922 may also be negatively affected by the presence of texts, #1077163 imagines a text that does not exist, and #1323895 hallucinates the presence of a man when it is clearly just a color gradient. Again, the quality of the images we work with has a fundamental influence on the quality of our final texts, but so does the model we use. For other researchers, we strongly recommend the use of better hardware, as well as a better image-to-text model than we used (to run BLIP, we only had a MacBook Pro available (with a, M1 chip, 8 cores, and 8 Gb of unified memory)). As a final proposal, the use of multiple image-to-text models could be considered, possibly even including audio-to-text ones. The outputs of these models could be fed into a large language model, which can generate a new text that averages and encompasses the semantic meaning all descriptions, improving the chances of ending up with an appropriate caption.

6. Conclusions

In this study, we tackled the significant challenge of generating high-quality multimodal datasets, specifically focusing on audio–image–text observations derived from videos. Our motivation stemmed from the increasing demand for diverse and large-scale datasets in the machine learning community, particularly for multimodal data that include audio, which are often scarce [29,37].

We proposed a method of generating these datasets by leveraging continuous video recordings, ensuring a strict semantic connection between acoustic and visual data (i.e., both audio and image in each pair are extracted from and related to the same situation). This approach addresses the common issue of undesirable entries in third-party datasets [26] and the lack of datasets for specific tasks, such as medical image analysis [27], reinforcement learning [148], and audio–text in general [37,49,50,51].

Our method involved three key steps: collecting suitable videos, extracting audio–image pairs, and generating textual descriptions for each pair using the BLIP model [6]. This process resulted in the creation of over 2 million audio–image pairs, which were further extended to include textual descriptions, forming a comprehensive multimodal dataset. Despite some limitations, such as the inclusion of frames with text or blurry images, as well as the basic configuration used for text generation, our dataset represents an advancement in the availability of multimodal data for research purposes.

The literature review highlighted the potential of exploiting relationships between audio, image, and text data [33,35,45,60]. These relationships can enhance various applications, including multimodal data analysis, correction of low-quality recordings, video generation, augmented reality, and transfer learning with multimodal models.

Our research underscores the importance of minimizing data modality conversions to preserve data quality. We also emphasized the need for more research on audio–image and audio–text tasks given the current lack of high-quality data and guidelines for new researchers.

Future work could focus on refining the filtering process to exclude undesirable frames more effectively, ensuring temporal alignment by incorporating more recent techniques in the pipeline (such as those seen in [113,149,150]), employing more advanced image-to-text models to improve the quality of textual descriptions, and even leveraging future audio-to-text models, which could complement the aforementioned descriptions. Additionally, exploring the potential of incorporating complementary data from any-to-any models while being mindful of the concerns related to data modality conversions could further enhance the utility of these datasets.

Overall, our contributions provide a valuable resource for the research community and highlight the importance of multimodal data in advancing machine learning models. The impact of this method and our resulting dataset extends beyond audio-conditioned image generation. We believe that this resource can significantly benefit other multimodal tasks, such as cross-modal retrieval (e.g., searching for images using audio queries), acoustic–visual source separation, and generative video content creation. We hope that our work will inspire further research and development in this area, ultimately leading to more robust and versatile AI systems.