Sound-to-Image Translation Through Direct Cross-Modal Connection Using a Convolutional–Attention Generative Model

Abstract

Sound plays a fundamental role in human perception, conveying information about events, objects, and spatial dynamics that may not be visually accessible. However, current technologies such as Acoustic Event Detection typically reduce complex soundscapes to textual labels, often failing to preserve their semantic richness. This limitation motivates the exploration of sound-to-image (S2I) translation as an alternative connection between audio and visual modalities. Unlike multimodal approaches guided by intermediary constraints during the learning process, we investigate S2I translation without class supervision, cluster-based alignment, or textual mediation, a paradigm we refer to as direct S2I translation. To the best of our knowledge, apart from our previous work, no prior study addresses S2I translation under this fully direct setting. We propose a convolutional–attention generative framework composed of an audio encoder and a densely connected GAN integrating self-attention and cross-attention mechanisms. The attention-based model is systematically compared with a purely convolutional baseline. Results show that introducing attention at early stages of the generator significantly improves translation performance, increasing the likelihood of producing interpretable and semantically coherent visual representations of sound. These findings indicate that attention strengthens semantic correspondence between audio and vision while preserving the fully direct nature of the translation process.

1. Introduction

Sound plays a fundamental role in human perception, contributing continuously to our understanding of events, objects, and spatial dynamics in the surrounding environment. Auditory perception provides information beyond the field of view, enhancing situational awareness by enabling the detection of events that might otherwise go unnoticed. Despite its expressive capacity, most computational approaches to environmental audio analysis simplify this modality considerably. Technologies such as Acoustic Event Detection predominantly frame the problem in a classification setting, mapping complex soundscapes to discrete textual labels. While effective for machine-driven monitoring tasks in restricted contexts, such representations often fail to capture the richness inherent in real-world acoustic scenes from a human perspective. This limitation becomes especially relevant when sound must be conveyed across modalities, as in multimodal human–computer interaction systems, including assistive applications. In these contexts, reducing auditory information to text may not adequately reflect the semantic and contextual depth present in the original acoustic signal.

Prior research on assistive technologies for deaf and hard-of-hearing individuals indicates that visual feedback is an effective and often preferred modality for representing environmental sounds and supporting situational awareness [1,2,3]. This observation has encouraged the exploration of alternative approaches capable of providing more expressive and intuitive representations of acoustic events. Despite the challenges involved, sound-to-image (S2I) translation emerges as a promising direction. Rather than reducing sound to discrete labels, S2I systems aim to generate images that are interpretable and semantically coherent with the corresponding acoustic events. By leveraging advances in generative modeling, such approaches have the potential to convey richer contextual information about acoustic environments and provide more meaningful cross-modal representations.

Visual generative models have achieved great impact since the advent of generative adversarial networks (GANs) [4], especially the conditional ones [5] that enabled to prompt the network for a desired output. The efficacy of these networks even provoked the first debates on the ethics of producing synthetic content that looked so real. Simultaneously, attention mechanisms gained force on text translation tasks with the arrival of the transformer architecture [6]. Then, GANs again took a step over within multimodal and cross-modal approaches, some of them including audio-visual processes. In this scenario, diffusion models emerge beating GANs in terms of merging different concepts in a synthetic image, despite the increased computational cost during inference. Suddenly, image generation has been partly pushed into the background when large language models (LLMs) appeared taking the transformers to a higher level of textual generation. These models could be prompted with natural language and outputs began to achieve human-level writing quality. Due to the availability of massive data online and inheriting semantic organization, text became the natural reference for aligning different modalities. Also, the increasing offer of pre-trained LLMs increased the use of text as reference, since many approaches use these available models for encoding text queries. In this scenario, Multimodal-to-Multimodal (M2M) models then begin to appear, where translation can be performed among all addressed modalities, usually with text as the bridging modality. Regarding image generation, such approaches achieved high visual quality, whether using diffusion models or GANs for decoding.

The present study focuses specifically on S2I translation and, as in our previous work [7], investigates the problem under a fully direct setting without supervision. In the proposed translator, the generation of interpretable and semantically coherent images relies entirely on establishing a direct connection between audio and visual modalities, since no prior information about the input sound’s class or the target image’s class is provided at any stage of the process. This differentiates our research from previous works that address S2I translation depending on supervised learning or cluster-based alignment to relate acoustic and visual data. In such approaches, audio and visual information are constrained to predefined semantic categories or clusters, thereby limiting the expressive capacity of the translation. As a result, it becomes difficult to determine whether the model has learned a genuine cross-modal connection or merely generates images that match the visual characteristics associated with a particular class or cluster. Furthermore, our translator does not use text as an intermediary modality to bridge audio and vision. Although text may provide a convenient semantic anchor for aligning heterogeneous modalities, text-mediated approaches [8,9] introduce several limitations in the S2I translation process. First, in the sound-to-text stage, translating acoustic signals into textual descriptions inevitably compresses the rich temporal and spectral information of environmental sounds. Therefore, valuable semantic content that could otherwise be directly aligned with visual representations may be lost during the process. Second, in the text-to-image (T2I) stage, textual descriptions themselves may be ambiguous or incomplete with respect to the acoustic events they describe, yet fail to report this uncertainty. Even if explicitly reported in the intermediary text, the uncertainty is rarely reflected in the generated images. Consequently, text-mediated S2I translators, relying on vision–language models in the T2I stage, may even produce visually detailed and plausible outputs that nonetheless fail to correspond accurately to the original sound. These inconsistencies may remain difficult to detect, particularly when the generated images appear realistic. This issue becomes especially critical in S2I applications intended to support deaf and hard-of-hearing users, where visually convincing but semantically incorrect representations of sound events could lead to misleading interpretations of the acoustic environment.

Throughout this text, images generated by S2I translation that are interpretable and semantically coherent with their corresponding acoustic events are referred to as ‘informative’, aligning with Neubert and Shreve’s [10] concept of informativity in text translation. In this context, informativity measures how well information is transmitted from source to target language. Similarly, in S2I translation, informativity refers to the amount of information conveyed from the aural to the visual modality, capturing sound-related events, sound-emitting sources, as well as picturable surrounding elements that can be inferred from the audio, including landscapes, environments, people, backgrounds and objects.

Building on prior studies on hierarchical representation learning [11], we hypothesize that higher levels of abstraction in deep neural networks provide a shared semantic space in which audio and visual modalities can converge. At these stages, modality-specific details are suppressed in favor of abstract concepts related to events, objects, and scenes, enabling the extraction of common semantics that facilitate cross-modal alignment. This property is particularly relevant for S2I translation, which, like other cross-modal tasks, relies on uncovering meaningful semantic correlations between heterogeneous modalities. Prior works discuss this behavior in audio [12] and in vision modality [13,14,15]. The S2I translation system must link acoustic events in audio to corresponding visual elements, a process that is intuitive for humans but difficult for machines due to the disparity between the audio waveform and image RGB domains. For example, while a barking dog in an audio scene and a prominently positioned dog in an image share a common semantic entity, their vastly different representations across modalities make S2I translation challenging. Moreover, S2I translation faces the problem of the lack of appropriate audio-visual data. There is a lot of video data available online, but the quality of the records is usually low. In addition, there are few audio-visual datasets that provide cross-modal semantic correspondence, i.e., that the video frames picture the sound-emitting source that can be heard in the audio. And even in these curated datasets, frequently, the video frames present other elements that are not related to the addressed sound.

Aware of these challenges from our previous study [7], we investigate whether architectural refinements can further strengthen the cross-modal connection between audio and vision. In particular, we examine the role of attention mechanisms in enhancing semantic alignment within a fully direct S2I framework. We aim to use attention in order to guide the generative network to focus on the parts of the image more related to the acoustic event in the audio stream.

To the best of our knowledge, apart from our previous study [7], no other work has addressed sound-to-image translation without relying on textual mediation, class supervision, or cluster-based alignment, a paradigm we refer to as direct S2I translation. The primary contribution of this work lies in advancing direct S2I translation by strengthening cross-modal alignment within this fully direct setting. We extend the previously proposed convolutional S2I translator by integrating attention mechanisms and systematically evaluating their impact on cross-modal correspondence. Unlike approaches that rely on text as a bridging modality, our framework preserves a direct connection between acoustic and visual representations, allowing the model to learn semantic correspondence from perceptual signals alone. Among other potential approaches to improve S2I translation, such as bags of acoustic events [16] and/or visual elements [17], sound separation [18] and image segmentation [19], attention was the only one that did not impose any structuring on the data, as its mechanisms do not require supervision, aligning with our proposal.

We present an end-to-end convolutional–attention S2I translator composed of an audio encoder and a densely connected generative adversarial network (GAN) for image synthesis (Figure 1). The generator can incorporate one to six attention modules at different stages, enabling controlled investigation of their influence on translation performance. Inspired by the original transformer decoder architecture [6], each attention module includes both self-attention and cross-attention mechanisms. Using a pre-trained audio encoder, we evaluate nine architectural configurations of the attention-enhanced generator and compare them against the purely convolutional baseline previously proposed [7].

Our results show that integrating attention at early stages of the generator significantly improves translation performance, increasing informativity from 17.9% to 20.7%, which corresponds to a relative improvement of about 16%. These findings demonstrate that attention can strengthen semantic correspondence between audio and vision modalities while preserving the fully direct nature of the translation process.

2. Related Work

In this section, we review key developments in S2I translation, highlighting methods employing diffusion models and GAN-based approaches for image generation. We further discuss Multimodal-to-Image methods, where images are synthesized from different input modalities, with audio included. Also, Multimodal-to-Multimodal processes are discussed, covering generalized multimodal generation that addresses or can potentially perform S2I tasks. We conclude the section with a discussion on Multimodal Large Language Models (MLLMs), examining their relevance and limitations with respect to S2I translation.

2.1. Sound-to-Image

In [8], in an effort to overcome the low availability of semantically aligned audio-visual pairs, the authors produce a dataset through cross-modal retrieval to sonify image data. Then, they leverage adapted pre-trained T2I models for performing S2I generation. In the first process, they use pre-trained vision language models (VLMs) like LLaVA [20] and CogVLM [21] to extract textual descriptions of acoustic events present in images. From those descriptions, they perform audio retrieval based on the similarity between the audio and text embeddings using the CLAP [22] latent space. With the audio-visual dataset in hand, they then adapt a pre-trained T2I diffusion transformer [23]. With this method they were able to generate plausible images from unknown sounds. Their work is based on the assumption that ground-truth audio-visual correspondence is not necessary for training S2I generative models. However, these datasets built through image sonification are not suitable for our approach since we propose to find a direct cross-modal connection. Also, the T2I transformer they use is initialized with a class-condition model’s parameters. Furthermore, their method relies on aligning sound and visual modalities through textual descriptions, which, as discussed in the Introduction, introduces additional issues related to the indirect nature of the cross-modal connection and the uncertainty of the resulting translation.

SonicDiffusion [24] introduces an audio-driven image generation and editing framework by incorporating sound as a conditioning input into pre-trained diffusion models. The method includes sound-image cross-attention layers as adapters to tune the Stable Diffusion denoising process. An audio projector module maps audio embeddings into a latent space, which are then injected into the cross-attention layers. In a first stage, the audio projector is trained using a contrastive loss employing InfoNCE [25] and Mean Squared Error (MSE) loss to align audio representations with visual semantics. In the second stage, applying DDPM loss [26], the cross-attention layers that receive the audio tokens are trained while the original diffusion UNet [27] layers are frozen. The framework supports both sound-only and hybrid text–audio conditioning, enhancing control over image synthesis. For audio-guided image editing, it utilizes DDIM inversion, enabling sound-guided modifications to existing images. The approach is evaluated on three datasets representing natural scenes, material properties, and human expressions, demonstrating improvements in audio–image alignment and image realism compared to previous multimodal methods. Despite the verified results, the framework relies on a supervision, since the audio projector is trained employing class-based losses. As previously mentioned, supervised approaches imply a rigid mapping process when attempting to align audio and visual modalities.

Sound2Scene [28] propose a S2I framework utilizing visual discrete representation learning. Their method employs self-supervised training for both a visual encoder and an image generator, which operates within a GAN architecture, conditioned on representations extracted from the visual encoder. To align audio embeddings with the visual latent space, they train an audio encoder using contrastive loss. The model is trained and evaluated on the VGGSound [29] and VEGAS [30] datasets, achieving over 83% accuracy in generating images that correctly depict the sound-emitting source. Unlike our approach, their self-supervised training strategy is susceptible to class bias, as previously discussed.

2.2. Multimodal-to-Image

GlueGen [31] is a framework designed to extend existing diffusion-based T2I models by allowing the integration of single-modal or multi-modal encoders, including audio-based inputs. It serves as an alignment module, mapping feature representations from a new encoder (e.g., AudioCLIP [32]) into the latent space of a pre-trained T2I model. This enables sound-to-image translation without requiring retraining of the image generator. The model is trained using parallel corpora, leveraging element-wise and distribution-wise losses to align embeddings across modalities and reconstruction loss in the decoder network. Experiments on the UrbanSound8K dataset show improved sound-to-image translation quality compared to a baseline that combines AudioCLIP-based sound-to-text inference with T2I diffusion models. While offering a computationally efficient way to integrate new modalities into pre-trained diffusion-based models, both the CLIP [33] text encoder originally used jointly with the T2I model and the aligned AudioCLIP encoder were trained with supervision, potentially biasing the connection between audio and visual modalities.

2.3. Multimodal-to-Multimodal (M2M)

In ImageBind [34], the authors present a multimodal representation framework that aligns embeddings of six modalities: images, text, audio, depth, thermal and Inertial Measurement Unit (IMU) data. The images play a central role as the bind for this joint embedding space. As they show, this is enough for binding modalities that are not directly paired. The experiment conducted includes a S2I generation using a pre-trained text-to-image (T2I) DALLE-2 model [35], replacing the text prompt embeddings by audio embeddings. For its part, the DALLE-2 decoder is conditioned on CLIP [33] image representations. Using these combinations of models, they were able to generate plausible images using distinct sounds. Since CLIP is based on natural language supervision, the S2I performed by the ImageBind is restricted to the textual description capacity, which can harm the semantic connection between audio and visual modalities. Also, the image and audio encoders use ViT [36], which is a supervised model designed for image recognition.

Composable Diffusion (CoDi) [37] is a generative model designed for flexible multimodal synthesis, allowing input and output combinations across text, image, video, and audio. The model is built on latent diffusion models (LDMs), each specialized in a single modality but trained to align in a shared multimodal space. To facilitate multimodal integration, CoDi employs Bridging Alignment, which uses text as an intermediary to unify representations across different modalities. Additionally, Latent Alignment enables diffusion models to synchronize their outputs, ensuring coherence in jointly generated modalities, such as video with aligned audio. CoDi supports M2M generation without requiring explicit training on all possible modality pairings, addressing the challenge posed by the scarcity of paired multimodal datasets. The model achieves state-of-the-art performance in several unimodal tasks, including text-to-audio and audio captioning. However, their approach also relies on inherited supervision to guarantee image generation quality. Specifically, its image diffusion model is initialized using the weights of Stable Diffusion 1.5 [38], which is trained employing a perceptual loss [39] based on features extracted from VGG [40], a supervised model. Furthermore, the reliance on a text-aligned shared representation may prevent the model from establishing a direct semantic link between generated images and corresponding audio.

2.4. Multimodal Large Language Models (MLLMs)

Rather than being designed to translate between specific modalities, multimodal large language models (MLLMs) are typically intended to interact with users in a chatbot-style setting. In fact, most MLLMs are focused on Multimodal Understanding (MU), limited to Multimodal-to-Text tasks. Although, recently, M2M tasks have been increasingly addressed by MLLMs. As reported in the survey by Zhang et al., MLLMs fit in a general architecture that includes five components: Modality Encoder, Input Projector, Large Language Model (LLM) Backbone, Output Projector and Modality Generator. In the case of MU tasks, only the first three components are included. Following the mentioned general design, NextGPT [41] adopts an LLM-centered architecture, accepting text, image, video and audio as input and output modalities. Another common approach in MLLMs is modality-interleaved processing. In the decoding stage, the system uses ‘modality signal’ tokens to respond using the modality or modalities it infers as most appropriate for maintaining the flow of the conversation. The study does not provide any example of S2I translation, and it is not clear how NextGPT would answer to a non-speech audio input, apart from performing audio captioning. In GroundingGPT [42] and BuboGPT [43], considering prompts that include audio, the provided tasks are limited to audio captioning and sound source localization. ModaVerse [44] presents an example of audio-and-text–to-video generation; however, it does not provide a quantitative evaluation of this task. Moreover, since the model relies on explicit language instructions to guide the generation process, it tends to produce random or semantically irrelevant outputs when only audio inputs are provided and no textual hint is available. CoDi-2 [45] is a MLLM that enables CoDi [37] to process modality-interleaved prompts and instructions. Although the model can be instructed to perform S2I translation, the referenced paper neither provides illustrative examples nor presents an evaluation of this task. Despite that MLLMs are intended to bridge the semantic gap between different modalities, the core of the system remains aligned to the textual representation space, since its reasoning engine employs a frozen LLM.

3. Convolutional–Attention Sound-to-Image Translator

Our work introduces an end-to-end attention-guided S2I translator, consisting of a convolutional audio encoder and a conditioned, densely connected GAN, equipped with attention modules followed by dense convolutional blocks. Since our goal is to compare the proposed attention-guided model with the exclusively convolutional architecture from our prior study [7], we integrate the attention modules while keeping the remainder of the architecture unchanged. Training algorithms and model regularization methods were also inherited from our previous approach.

3.1. Overview

The training process of the S2I translator is illustrated in Figure 1. It begins with utilizing a pre-trained audio encoder, inherited from our previous study, which was trained on log-mel spectrograms derived from 1 s audio segments. This pre-trained encoder is frozen and used to extract audio embeddings, which are subsequently passed to the generator.

During the generator’s training phase, it is optimized to deceive the discriminator, which is updated once for every five updates of the generator. At this stage, the discriminator processes balanced batches of real and synthetic images, paired with their respective target scores, to learn visual feature extraction. To ensure aural–visual coherence, the source audio embedding is concatenated with the input to the discriminator’s final layer. This integration effectively merges the aural and visual modalities, allowing the discriminator to jointly evaluate both the realness of the generated images and their semantic coherence with the corresponding audio. The output of this evaluation is referred to as the realness-and-coherence score (RC-score).

3.2. Network Architecture

The audio encoder is composed of 13 convolutional layers. It was originally trained in our previous study [7] as part of a symmetric autoencoder architecture comprising 26 convolutional layers in total, each followed by batch normalization (BN). Rectified linear units (ReLU) are used in the internal layers of both the encoder and decoder, while the final output layer employs a hyperbolic tangent (TanH) activation. In order to prevent overfitting, dropout regularization is applied to the inner layers of the decoder.

With respect to the generator, it consists of a deep dense 25-layer architecture borrowed from our previous study, in which attention modules can be inserted before each dense block. All convolutional layers in the generator are followed by BN and a ReLU activation, except for the output layer, which employs a TanH function. Dropout regularization is inserted between the dense blocks to improve model generalization and to enable stochastic inference when applied also during testing, following the approach of Gal and Ghahramani [46] and Isola et al. [47]. Figure 2 provides a scheme of the audio encoder and the generator networks. Notice that since the generator is intended to increase the feature maps’ resolution while decreasing the channels’ depth, the convolutional layers as well as the attention modules present different shapes along the six iterations.

The discriminator architecture shares a similar structure with the audio encoder, differing primarily in input and feature map shape. Its final convolutional layer is conditioned with the embeddings extracted from the audio encoder, and outputs a scalar corresponding to the RC-score. Dropout regularization is applied after each inner layer.

3.3. Attention Module

Consisting of self-attention, cross-attention, a feed forward network, and residual connections, an attention module (Figure 3) can be inserted before each dense block of the generator, depending on the desired configuration. The generator embeddings used for attention are extracted from the feature maps of the previous transition layer, while the encoder embeddings used for cross-attention are obtained from the audio encoder.

To generate the embeddings for attention, we apply a linear projection to adjust the channel dimensionality of the audio encoder output and the audio-visual features extracted from the generator. The projection output is then activated using tanh. Also, positional encodings are added to the embeddings. While convolutions capture positional cues due to their spatial inductive bias, they do so only locally and do not inherently encode absolute global positions, which are necessary for image reconstruction. The attention module follows the method of the original Transformer decoder [6]. The module output is then projected back to the channel dimension of the next convolutional dense block and a ReLU activation is applied.

3.4. Formulation of Network Transformations and Loss Functions

We define a set of spectrogram–embedding–image data triples $\{S_i, x_i,Y_i\}$, where $S_i\in {\mathbb{R}}^{h \times w}$ are spectrograms, $x_i \in {\mathbb{R}}^f$ are audio feature vectors, and $Y_i\in {\mathbb{R}}^{h \times w \times c}$ are real-color images. Each triple represents the same $i^{th}$ acoustic event. Pixel values from both the spectrogram and the images are normalized to $[-1, 1]$. Audio embeddings and the realness-and-coherence score (RC-score) $r\in \mathbb{R}$, both activated by TanH, also lie within the same interval. Concerning dimensional notation, $h$ and $w$ indicate the height and width of spectrograms or images, $c$ denotes the number of channels in color images, and $f$ corresponds to the dimensionality of the audio feature space.

Transformations:

Audio encoder: $A_E\left(S\right) :{\mathbb{R}}^{h \times w}\to {\mathbb{R}}^f$

Audio decoder: $A_D\left(x\right) :{\mathbb{R}}^f\to {\mathbb{R}}^{h \times w}$

Generator: $G\left(x\right) :{\mathbb{R}}^f\to {\mathbb{R}}^{h \times w \times c}$

Discriminator: $D\left(Y\vee G\left(x\right),x\right) :{\mathbb{R}}^{h \times w \times c + f}\to \mathbb{R}$, where the real image $Y$ alternates with the synthetic image $G(x)$ as the first argument of $D$.

The training of the proposed models relies on mean-squared error (MSE) losses computed in pixel space, providing an unbiased and scalable optimization strategy for both the audio autoencoder and the S2I generator. This choice enabled extensive experimentation and empirical analysis, which informed architectural improvements and parameter tuning.

The audio autoencoder networks $A_E$ and $A_D$ are optimized by minimizing the pixel loss $L_A(S, \widehat{S})$ defined in Equation (1), computed as the batch-averaged MSE between the target spectrogram $S$ and the reconstructed spectrogram $\widehat{S}\leftarrow A_D(A_E\left(S\right))$, where $b$ denotes the batch size and $h$ and $w$ are the spectrogram dimensions:

$$L_A(S, \widehat{S})={\displaystyle \frac{1}{bhw}}\sum _{i=1}^b\sum _{j=1}^h\sum _{k=1}^w{\left(S_{ijk}-{\widehat{S}}_{ijk}\right)}^2$$

(1)

The discriminator $D$ is optimized by minimizing the score loss $L_D(r, \widehat{r})$ defined in Equation (2), computed as the batch-averaged MSE between the predicted RC-score $\widehat{r}\leftarrow D(Y\vee G\left(x\right), x)$ and the target RC-score $r$. The target score takes the maximum value of $1$ for real images and the minimum value of $-1$ for synthetic ones:

$$L_D(r, \widehat{r})={\displaystyle \frac{1}{b}}\sum _{i=1}^b{\left(r-{\widehat{r}}_i\right)}^2$$

(2)

The generator $G$ is trained using two complementary objectives. The first is a pixel reconstruction loss $L_G(Y,\widehat{Y})$ defined in Equation (3), computed as the MSE between the target image $Y$ and the generated image $\widehat{Y}\leftarrow G(x)$, where $c$ denotes the number of image color channels:

$$L_G(Y, \widehat{Y})={\displaystyle \frac{1}{bchw}}\sum _{i=1}^b\sum _{j=1}^c\sum _{k=1}^h\sum _{l=1}^w{\left(Y_{ijkl}-{\widehat{Y}}_{ijkl}\right)}^2$$

(3)

The second objective is an adversarial loss [4,5] based on the RC-score obtained from the discriminator $D$. To mitigate GAN training instability, a moving-average adversarial loss $L_G^{ma}(r_{max}, \widehat{r})$ is introduced, as defined in Equations (4) and (5). This loss computes an average of past adversarial losses up to the current epoch $t$, where $r_{max}$ is the maximum RC-score, ${\overline{L}}_{Gi}$ is the averaged adversarial loss at epoch $i$, and $k$ is the number of averaged data points:

$$L_G(r_{max}, \widehat{r})={\displaystyle \frac{1}{b}}\sum _{i=1}^b{\left(r_{max}-{\widehat{r}}_i\right)}^2$$

(4)

$$L_G^{ma}(r_{max}, \widehat{r})={\displaystyle \frac{L_G(r_{max}, \widehat{r})+{\sum }_{i=t-k+1}^{t-1}{\overline{L}}_{Gi}}{k}}$$

(5)

Finally, the overall generator loss is given by Equation (6), where the adversarial term is scaled by a factor $\lambda$ to balance its contribution with the pixel loss:

$$L_G=L_G\left(Y,\widehat{Y}\right)+\lambda L_G^{ma}\left(r_{max},\widehat{r}\right)$$

(6)

This combined loss formulation stabilizes GAN training while maintaining reconstruction fidelity.

4. Experiments

In this section, we outline the guiding principles of our approach, describing the training strategies and the datasets utilized in the experiments. Also, we present the configuration of the employed models varying the number of attention modules and/or the number of attention layers. Finally, we conclude the section by providing both quantitative evaluation and qualitative analysis of the S2I translation. For further information regarding the experiments and the code implemented, please refer to https://purl.org/s2i (accessed on 19 January 2026).

4.1. Dataset

The Audio Set dataset, introduced by Gemmeke et al. [48], is a large-scale audio-visual collection of manually annotated acoustic events. It features a hierarchical ontology of 632 audio classes, curated through literature and manual labeling, to classify 10 s YouTube video segments. With over 2 million videos, the dataset provides diverse audio-visual content, distinguishing it from prior studies. The present study focuses on five sound classes, extracted from more than 8000 distinct scenes, ensuring high inter- and intra-class diversity. For training and testing, the VEGAS subset, introduced by Zhou et al. [30], is used. VEGAS offers cleaner annotations, precisely marking event start and end points, with non-matching segments removed to ensure alignment between visual and aural elements. The dataset contains 28,109 videos in 10 sound classes, of which five are used: Baby crying, Dog, Rail transport, Fireworks, and Water flowing. To prevent class bias, an equal number of segments per class is used.

With respect to audio segmentation, 1 s excerpts are used, a duration chosen to capture the temporal characteristics of typical environmental sound events while maintaining a manageable input size for the neural network. Then, audio data is processed into log-Mel spectrograms by splitting the signal into 25 ms frames (15 ms overlap), applying a Hamming window, computing the Short-Time Fourier Transform (STFT), and integrating squared magnitudes into 128 sub-bands based on the Mel scale. The resulting 1 s transformed segments form 100 × 128 matrices. For visual data, the central frame of each corresponding 1 s video segment is extracted to represent the scene associated with the audio event. Frames are square-cropped around the center to increase the likelihood of including the sound-emitting source and resized to 96 × 96 pixels. This spatial resolution was selected as a compromise between preserving sufficient visual structure to depict the source of the sound and maintaining computational efficiency during training. The final dataset extracted from the video segments comprises 48,945 sound–image pairs for training, 5575 for validation, and 6825 for testing, evenly distributed across the five sound categories.

Since we are committed to finding a direct connection between audio and visual modalities, we do not employ any data augmentation technique. The changes on the audio signal or the target image would break this connection.

4.2. Training Scheme

For all networks, we applied the Xavier initialization method, also referred to as ‘normalized initialization’, as described by Glorot and Bengio [49]. Accordingly, the weights $W$ for each network layer are sampled from a random uniform distribution, as defined in Equation (7), where $n_i$ represents the number of input connections to the layer $i$, while $n_{i+1}$ denotes the number of output connections from that layer.

$$W\sim U \left(-{\displaystyle \frac{\sqrt{6}}{\sqrt{n_i+n_{i+1} }}} ,{\displaystyle \frac{\sqrt{6}}{\sqrt{n_i+n_{i+1}}}} \right)$$

(7)

The generator network was trained with an initial learning rate ranging from 0.1 (for smaller models) to 0.01 (for larger models), while the discriminator used always an initial learning rate of 0.1. Both networks were trained with a momentum value of 0.5. The adversarial loss scale factor for the generator (referencing Equation (6)) was set to 0.1, while the discriminator was updated once for every five generator updates. Training was performed using mini-batch gradient descent optimization, with a batch size of 64 applied to each network update. The entire system was developed using Python 3.5.2, with machine learning functionalities implemented in PyTorch 1.1.0. All models were trained on a Supermicro^TM SYS-7048GR-TR server (Super Micro Computer, Inc., San Jose, CA, USA), equipped with 160 GB of RAM and two Intel^TM Xeon E5-2670 processors operating at 2.30 GHz (Intel Corporation, Santa Clara, CA, USA). Computational performance was enhanced using a GeForce^TM Titan Xp GPU (NVIDIA Corporation, Santa Clara, CA, USA), accessed via the CUDA^TM platform, version 9.0.176.

4.3. Embeddings Dimensionality

As presented in

Table 1. Resume of channel and spatial dimension processed from the generator feature maps until the attention modules input. In columns 5 and 8 we present respectively the final embedding dimension and sequence length for each attention module.

Attention Module	Patch Size	Channel Dimension (No. of Feature Maps)	Channel Dimension × Patch Flat Size	Final Embeddings Dimension (After Projection)	Spatial Dimension (Feature Maps Shape)	Spatial Dimension After Patching	Final Sequence Length (Flattened Spatial Dimension)
A1	1 × 1	512	512	320	2 × 2	2 × 2	4
A2	1 × 1	512	512	320	4 × 4	4 × 4	16
A3	1 × 1	352	352	320	6 × 6	6 × 6	36
A4	1 × 1	224	224	320	12 × 12	12 × 12	144
A5	2 × 2	160	640	320	24 × 24	12 × 12	144
A6	2 × 2	112	448	320	48 × 48	24 × 24	576

, we defined the embeddings dimension on 320 since it keeps the bottleneck for most attention modules, except A4, which presents channel dimension of 224. Lowering the representations’ dimensionality even more to adjust it to this module would excessively force the bottleneck of the audio embeddings with dimensions of 512, and the A1, A2 and A5 modules, since their dimensions before linear projection are, respectively, 512, 512 and 640. This way, we reduce the risk of discarding valuable features from modules with higher dimensionality as well as the audio embedding, and at the same time, we avoid losing too much the bottleneck of modules with lower dimensionality, maintaining good generalization.

4.4. Model Variants

Table 2. Configuration and number of parameters of the evaluated S2I model variants.

Model [No. of Layers for Each Attention Module]	Params
S2I (baseline)	39 M
S2I_A[3,3,3,1,1,1]	55 M
S2I_A[3,3,3,3,3,3]	65 M
S2I_A[9,9,9,0,0,0]	69 M
S2I_A[9,9,9,9,9,9]	114 M
S2I_A[18,0,0,0,0,0]	69 M
S2I_A[0,18,0,0,0,0]	69 M
S2I_A[0,0,18,0,0,0]	69 M
S2I_A[3,0,0,0,0,0]	44 M
S2I_A[9,0,0,0,0,0]	54 M

summarizes the different model configurations evaluated in the experiments. Starting from the baseline S2I translator without attention, we consider nine attentional model variants (denoted as S2I_A) that differ in the number of attention modules and/or the number of attention layers. The models are listed from top to bottom following the chronological order in which they were explored during our experimental study. For all configurations, the attention modules use embedding projections of dimension 320, an MLP hidden size of 1280 in the feed forward block, and 8 attention heads.

4.5. Informativity Classifiers

Evaluating the S2I translator is inherently challenging, as the synthesized images often differ in visual structure from real images associated with the input sounds. Even when semantic coherence is preserved, discrepancies in visual appearance make pixel-based loss measures ineffective. To address these limitations, we focus on two key quality criteria: interpretability and semantic coherence, which together define the notion of informativity. Accordingly, to assess this aspect of S2I translation, we adopt the same evaluation strategy as in our previous study [7], employing informativity classifiers to quantitatively determine whether the generated images convey recognizable and meaningful visual information about the sound source. Rather than relying on standard image classification accuracy, we adopt a different evaluation strategy. Classification accuracy proved misleading due to the high proportion of non-informative images and the domain gap between real and synthetic visuals. Therefore, we frame the evaluation as a binary informativity problem. An image is considered informative if it exceeds a minimum threshold of interpretability and semantic coherence, i.e., if the primary sound-emitting source can be visually identified. This approach allows us to report a more reliable quantitative measure of S2I translation performance.

To this end, we train five informativity classifiers, one for each sound class, as generic models were found to perform poorly and would not serve the objective of accurately measuring translation quality. Balanced datasets are constructed using outputs from multiple S2I generator models, ensuring uniform representation across models to avoid bias from generator-specific artifacts. It should be emphasized that two different visual data sectioning are considered here. In the first, images are grouped according to the five sound classes, independently of whether they are real or synthetically generated. In the second, images are labeled as informative or non-informative, a categorization introduced exclusively for evaluating the performance of the S2I translator. Based on these data perspectives, we constructed five balanced datasets comprising a total of 9000 synthetic images drawn from the outputs of 37 previously trained S2I translation models, employing both attention and non-attentional architectures. Each class contains 1800 images, with 1440 used for training and 360 for testing. The dataset maintains equal proportions of informative and non-informative samples, as well as attentional and non-attentional outputs.

Importantly, these classifiers are used exclusively for evaluation purposes and do not provide any supervision during S2I translators’ training. The classifiers adopt a CNN architecture comprising five convolutional layers dedicated to visual feature extraction and two fully connected layers for classification. Batch normalization, ReLU activations, and dropout are applied after each layer during training, except for the final fully connected layer, where the softmax output provides the class log-probabilities. Training is performed using an initial learning rate of 0.001, a momentum of 0.9, and a weight decay of 5 × 10⁻⁵. The resulting models achieve accuracies of 80.8% (Baby cry), 80.3% (Dog), 82.8% (Rail transport), 80.3% (Fireworks), and 81.9% (Water flowing).

4.6. S2I Translation Results

This section presents the results of the conducted S2I translation experiments and is organized into two complementary subsections. First, a quantitative evaluation is provided, focusing on an objective comparison of multiple S2I translator models using informativity-based metrics. This is followed by a qualitative analysis, which visually illustrates the translation capabilities of the best-performing model through representative generated samples.

4.6.1. Quantitative Evaluation

To quantitatively assess the performance of the proposed S2I translators, we employ a set of five informativity classifiers, one per sound class, as introduced in Section 4.5. These classifiers are used to compare the translation quality of nine attention-based S2I models against a baseline S2I model that does not incorporate attention mechanisms. The underlying generator architecture shared by all models has been previously described in Section 3.2 and Section 3.3.

The evaluated attention-based models differ in the position of attention modules and/or the number of attention layers for each module integrated within the generator network. This design allows us to analyze how the placement and depth of attention mechanisms influence the ability of the model to produce informative images from audio inputs.

Due to fluctuations in informativity observed during adversarial training, we evaluate the translator’s performance using two averaging metrics. The first reports the overall average informativity computed across epochs 500 to 4000, while the second corresponds to the value of a 50-epoch moving average, computed over the five sound classes. Since the informativity classifiers were not available during the training of the translator models, performance assessment is limited to the saved checkpoints, corresponding to models stored every 25 training epochs.

Among all evaluated configurations, only those employing attention exclusively at the first module A1 (Figure 4e,h,i) surpass the baseline, with the model denoted as S2I_A[18,0,0,0,0,0] achieving the best overall translation performance (Figure 4e). This model incorporates a single attention module A1 in the first generator block, composed of 18 attention layers. Using this configuration, the translator reaches a maximum 50-epoch moving average informativity of approximately 20.7% across the five sound classes, outperforming the baseline model without attention, which achieves an informativity of 17.9% (

Table 3. Maximum 50-epoch moving average informativity across all model variants. The best result is highlighted in bold.

Model [No. of Layers for Each Attention Module]	Maximum Average Informativity
S2I (baseline)	17.9%
S2I_A[3,3,3,1,1,1]	15.2%
S2I_A[3,3,3,3,3,3]	15%
S2I_A[9,9,9,0,0,0]	17.2%
S2I_A[9,9,9,9,9,9]	15.6%
S2I_A[18,0,0,0,0,0]	20.7%
S2I_A[0,18,0,0,0,0]	16.1%
S2I_A[0,0,18,0,0,0]	13.4%
S2I_A[3,0,0,0,0,0]	18.3%
S2I_A[9,0,0,0,0,0]	19.2%

). This corresponds to a relative improvement of about 16%. The non-attentional baseline model is the same reported in our previous study [7], where an informativity score of approximately 14.4% was obtained. The higher informativity observed in the present work is primarily attributable to the use of updated informativity classifiers trained on a substantially larger dataset, which is nearly twice the size of that used in the earlier study.

Overall, the quantitative results demonstrate that incorporating attention mechanisms into the S2I generator can lead to measurable improvements in translation quality, provided that attention is applied at appropriate stages of the network. Although we anticipated that early generator layers would be more suitable for attention integration, we initially expected that distributing attention modules throughout the network could further enhance performance. To test this, we conducted experiments placing an attention module before each dense block (Figure 4a,b,d). However, this configuration produced a clear performance degradation. We then progressively removed attention from deeper stages of the generator (Figure 4c), and also tested integrating attention only in the second or third module (Figure 4f,g). We observed that improvements were preserved only when attention was restricted to the first module. In particular, the superior performance of the S2I_A[18,0,0,0,0,0] configuration highlights the importance of early generator blocks for cross-modal feature alignment. These layers appear to provide a representational space in which audio and visual information share higher-level semantic abstractions, making them especially suitable for attention-driven integration. Conversely, incorporating attention at later stages degrades translation quality. These observations offer useful guidance for the design of attention-based S2I architectures.

4.6.2. Qualitative Analysis

In addition to the quantitative evaluation, we provide a qualitative analysis to visually assess the effectiveness of the S2I translation process. Figure 5 presents a selection of translated images from the five sound classes. All generated images presented in this subsection are produced by a single translator model employing the S2I_A[18,0,0,0,0,0] architecture, which achieved the best performance in the quantitative evaluation. These examples complement the informativity-based metrics by offering visual insight into the perceptual quality and interpretability of the synthesized images.

Overall, the qualitative results obtained with attention mechanisms are largely consistent with those observed in our previous study employing a non-attentional model [7]. When informative images are successfully generated, their visual quality exhibits similar characteristics, suggesting that attention primarily improves the frequency of informative outputs rather than altering their intrinsic perceptual properties. Representative outputs from the baseline model are shown in Figure 6, allowing qualitative comparison with the attention-based results presented in Figure 5.

All images discussed here are generated from sounds extracted from unseen videos, demonstrating the generalization capability of the proposed model. Across all sound classes, the synthesized images often present recognizable shapes, coherent color distributions, and plausible scene structures that are broadly consistent with real-world environments. As in the non-attentional setting, many images display a generally blurred appearance, which we discuss further below. Nevertheless, identifiable contours, interpretable silhouettes, and occasional sharp details can be observed, contributing to the overall interpretability of the results.

Differences across sound classes remain evident. Fireworks translations continue to produce the most visually convincing images, followed by Water flowing. In the case of Fireworks, images are characterized by higher coherence and sharper details. This behavior is likely related to the relatively low visual diversity of original fireworks scenes and their abstract nature, which may facilitate cross-modal mapping. In contrast, translations conditioned on dog sounds remain the most challenging, often yielding abstract or poorly defined outputs. This limitation is likely due to the short and impulsive nature of dog sounds, which may not provide sufficient temporal context for accurate audio-visual correspondence, even when attention mechanisms are employed.

The lack of sharpness observed in many images remains comparable to that of the baseline model. This effect is likely influenced by the use of pixel-wise reconstruction losses and regularization strategies during training, which tend to favor smoother outputs. At the same time, the adversarial component helps discourage unrealistic textures. In some cases, blurred regions may also be interpreted as an implicit indication of uncertainty, where the model lacks confidence about specific visual elements inferred from the audio input.

As in the previous experiments [7], the model is capable of producing diverse outputs for the same sound category, although some loss of diversity persists when compared to real images associated with the sounds. While most translations show a visual decoupling from their corresponding ground-truth scenes, occasional examples exhibit partial or strong alignment in structure, pose, or color composition. These instances suggest that the model may capture audio-visual correlations that enable it to also infer lower-level contextual information about the original sound.

Finally, qualitative inspection also reveals the presence of various non-informative image types, such as defective, incomplete, artifactual, implausible, and surreal images. These images appear with similar frequency and characteristics as in the non-attentional setting, indicating that attention mechanisms do not fundamentally change the nature of translation errors. As mentioned above, their primary benefit lies in increasing the overall likelihood of producing informative images, as confirmed by the quantitative evaluation.

5. Conclusions and Future Work

In this work, we advanced direct sound-to-image (S2I) translation by investigating how attention mechanisms influence cross-modal alignment between audio and visual representations. To the best of our knowledge, apart from our previous study [7], no other work has addressed S2I translation under a fully direct setting without textual mediation, class supervision, or cluster-based alignment. Building upon that prior convolutional framework, we introduced attention mechanisms and systematically evaluated their impact on translation performance.

The results demonstrate that integrating attention at early stages of the generator increases informativity, raising performance from 17.9% in the purely convolutional baseline to 20.7%. This gain is achieved within the direct sound-to-image translation setting. The findings suggest that early generator layers provide a representation space where audio and visual semantics can be more effectively aligned, and that attention strengthens this alignment. However, attention does not produce a perceptible enhancement in the visual realism of the generated images. Qualitative analysis indicates that while informative outputs become more frequent, the intrinsic visual limitations of the generative process remain largely unchanged.

Despite these improvements, several limitations remain. First, the quality and alignment of available audio-visual datasets impose constraints on the achievable performance of S2I systems. Environmental recordings often contain background noise, partial visual correspondence, or visually ambiguous scenes that make cross-modal translation inherently uncertain. Second, the present study focuses on scenarios dominated by a single salient sound event. In more diverse acoustic environments where multiple sound sources occur simultaneously, the correspondence between audio events and visual entities becomes more complex. Addressing such multi-source conditions was outside the scope of this work, but represents an important direction for future research. Techniques such as sound source separation or audio event disentanglement could help isolate individual acoustic elements prior to translation, potentially improving the robustness of S2I translation in complex environments.

Future work may also explore alternative generative architectures, such as audio-visual-centered transformers incorporating both encoder and decoder components, uncertainty-aware translation strategies, and improved audio-visual data curation to enhance both informativity and visual quality of generated images. In the longer term, advances in direct S2I translation may contribute to the development of assistive systems capable of conveying acoustic information visually, potentially supporting enhanced situational awareness for deaf and hard-of-hearing individuals.