Deep Hybrid Synesthesia Model for Audio-Image Transfer

Abstract

Most artistic expressions are conveyed through images (e.g., painting) and audio (e.g., music), and deep learning has been successfully applied to neural style transfer within each of these modalities. However, there is still a lack of deep models that explicitly learn to transfer style between images and audio. Motivated by synesthesia, which reflects intrinsic connections between vision and hearing in the human brain, we propose a deep hybrid synesthesia model for audio–image style transfer. Our framework consists of two main components: (1) a component conversion module that learns cross-modal mappings between audio rhythm/spectrum and image color/shape in a continuous valence–arousal (VA) emotion space; and (2) a style conversion module that transfers high-level artistic styles between Eastern (ink-wash, shui-mo) and Western painting and their corresponding musical counterparts. We first learn emotion-aware feature networks that align low-level audio and visual components based on shared affective representations, and then model long-term stylistic structures for cross-modal style transfer. Experiments include “seeing the sound” (audio-to-image generation with controllable components) and full audio–image style transformations. Both objective analyses and subjective evaluations suggest that our model can produce cross-modal artworks whose perceived style and emotional content are consistent with human synesthetic impressions.

1. Introduction

Deep learning models have been successfully applied to neural style transfer in both images and audio. For images, pixel arrays can be decomposed into content and style representations, where “style” is often characterized by color distributions and localized texture patterns. Starting from the seminal work of Gatys et al. [1], convolutional neural networks (CNNs) have been used to extract neural representations that disentangle content from style, enabling high-quality transfer of artistic styles between arbitrary images. Subsequent works have improved the efficiency and flexibility of image style transfer using feed-forward networks [2,3] and adaptive normalization techniques for arbitrary style transfer [4], and extensive surveys summarize the rapid progress in this field [5].

For audio, the notion of style is typically related to spectral envelope, pitch contour, and rhythmic structure. Spectrogram-based CNN models have been used to manipulate timbre and other spectral features for music and audio style transfer [6], while recurrent neural networks (RNNs) and generative adversarial networks (GANs) have been introduced to better capture temporal dependencies such as rhythm and phrasing across multiple timescales [7,8]. Recent generative models further demonstrate high-fidelity music synthesis and transformation [9,10], and position papers review the challenges of music style transfer in more detail [11]. Despite these advances, most existing approaches operate within a single modality—either image-to-image or audio-to-audio.

In contrast to “true” cross-modal painting style transfer—where one would expect a direct, semantically meaningful mapping from an input audio clip to a specific artistic painting style—our goal in this paper is more modest and explicitly affect-focused. Throughout this work, we formulate the problem as valence–arousal (VA)–consistent affective transfer between audio and images under weak alignment, rather than strict semantic style transfer from audio to a curated painting corpus. All our models are therefore designed to preserve and transform cross-modal affective properties in a shared VA space, rather than to recover detailed artistic content or composition in a particular artwork.

To date, there is still no deep learning framework that explicitly focuses on mapping and transferring styles between images and audio. Yet, images and sounds are among the most fundamental artistic media, and many outstanding artists create works that evoke cross-modal impressions: a painting that seems to contain an audible atmosphere, or music that spontaneously gives rise to vivid visual imagery. This phenomenon is closely related to synesthesia. At the same time, a growing body of work in cross-modal representation learning and audiovisual generation explores how visual and auditory signals can be jointly modeled [12,13,14,15,16], but these methods mainly focus on recognition or sound prediction from videos rather than explicit cross-modal style transfer.

Physiological and anatomical studies suggest that sensory modalities are highly interconnected and integrated in regions such as the prefrontal cortex [17]. Based on such neural organization, Lewkowicz [18] examined the hypothesis that young infants respond to audiovisual events in a unified fashion rather than strictly separating light and sound. Meltzoff [19] demonstrated intermodal matching abilities in human neonates, indicating early cross-modal correspondences. Cytowic [17] argued that synesthesia is a latent capacity of the human brain, although only a minority of individuals experience it vividly enough to create synesthetic art. Identifying common structures between visual attributes (e.g., color, shape) and auditory attributes (e.g., pitch, timbre, rhythm) offers a promising way to bridge scientific models and artistic practice.

In this work, we propose a deep hybrid synesthesia model that learns shared representations across audio and images and applies them to cross-modal style transfer. As illustrated in Figure 1, we distinguish between Eastern style features (Chinese ink paintings and eastern music) and Western style features (color water paintings and western music). For the component transfer part, we design a component conversion network that maps image color and shape features to audio spectral and pitch features, and vice versa. Crucially, this mapping is constrained by a shared continuous VA emotion space: instead of attempting arbitrary style conversion, our model is trained to preserve and reflect valence–arousal similarity across modalities, so that the generated image and sound fragments are affectively consistent even when their semantic content is not aligned. This mapping is learned in a continuous valence–arousal (VA) emotion space [20,21,22,23], leveraging emotional reactions as an intermediate representation that underlies synesthetic processes.

For the style transfer part, we focus on transferring high-level artistic styles between Eastern and Western arts. In the visual domain, this includes ink-wash (shui-mo) landscape painting versus Western oil painting; in the audio domain, we consider traditional Oriental instruments versus Western symphonic music. Our style conversion module learns to associate these cultural and stylistic differences across modalities, enabling transformations such as generating an ink-wash style image conditioned on a segment of traditional music, or synthesizing a Western-style visual scene from symphonic audio. Throughout the paper, we therefore use the term “audio–image style transfer” in this affective, VA-grounded sense, and we explicitly distinguish our problem setting from exact semantic style replication onto specific paintings.

We conduct experiments in two main settings. First, in the “see the sound” scenario, we generate images from audio by controlling specific components (e.g., rhythm, spectral brightness) and examine whether the resulting images exhibit consistent color and shape patterns. Second, we perform full cross-modal style transfer combining component and style features, and evaluate the outputs through subjective preference tests, where participants are asked to choose the images that best match given sounds in terms of style and emotion. Because our datasets are only weakly aligned via VA labels rather than paired audio–image instances, all evaluations are phrased in terms of perceived affective and stylistic correspondence, not one-to-one content fidelity. The results indicate that our deep hybrid synesthesia model can produce audio–image transformations whose perceived cross-modal correspondences are consistent with human synesthetic tendencies [17,18,19].

The main contributions of this work are three-fold:

• We introduce a deep hybrid framework for audio–image style transfer grounded in synesthesia theory, going beyond unimodal style transfer.
• We propose emotion-aware component conversion networks that learn cross-modal mappings between audio rhythm/spectrum and image color/shape in a continuous VA space.
• We demonstrate cross-cultural style conversion between Eastern and Western visual and musical arts, and validate the effectiveness of our model through both qualitative examples and human evaluation.

2. Background

Since our work focuses on transferring style between audio and images, we first discuss the fundamental differences between image and audio features, and then introduce representative image and audio style transfer models that we build upon in our hybrid framework.

2.1. Differences Between Image and Audio Features

There is a long line of work on image representation for computer vision and multimedia applications [24,25]. In traditional hand-crafted frameworks, RGB color space or decorrelated color spaces (e.g., YCbCr, HSV) and PCA over RGB pixel values are commonly used as low-level descriptors [26]. In the MPEG-7 standard [27], color descriptors include histogram-based features such as the Dominant Color Descriptor (DCD), Color Layout Descriptor (CLD), and Scalable Color Descriptor (SCD), while texture descriptors include the Homogeneous Texture Descriptor (HTD), Texture Browsing Descriptor, and Edge Histogram Descriptor. These features are designed to capture spatially localized patterns of color and texture that are meaningful for visual perception and image retrieval.

In contrast, audio signals are inherently one-dimensional temporal waveforms sampled at high temporal resolutions, and learning to synthesize or transform audio requires capturing structure across a wide range of timescales, from millisecond-level periodicity to long-term rhythm and form [8,9]. Modern deep learning models typically operate on time–frequency representations such as the short-time Fourier transform (STFT) or Mel spectrograms, where frequencies are converted to the Mel scale to better reflect human pitch perception [28]. For practical training, spectrograms are often normalized and segmented along the time axis into fixed-length windows of L frames, enabling mini-batch optimization on variable-length audio inputs.

Figure 2 illustrates the first eight principal components (PCs) computed from patches of natural images and slices of speech signals, respectively [29,30]. For images, the leading PCs typically capture intensity, gradient, and edge-like structures, reflecting dominant spatial variations. For audio, however, the leading PCs form approximately periodic basis functions that decompose the signal into constituent frequency bands over time. This difference highlights an important property: natural audio signals are much more likely to exhibit strong periodicity and harmonic structure than natural images, whose variability is more strongly tied to spatial edges and textures.

These fundamental representational differences imply that a cross-modal style transfer model must bridge spatial, largely non-periodic structures in images with temporal, often periodic structures in audio. Our proposed hybrid synesthesia model explicitly takes this into account by separating short-term component features (e.g., color vs. spectral envelope) from long-term style features (e.g., composition vs. rhythm), as detailed in Section 3.

2.2. Style Transfer Models

There are many neural style transfer models proposed in recent years for both images and audio [5,6,11]. In this subsection, we briefly review representative approaches that inspire our design.

Image style transfer models: Most image style transfer methods are built upon convolutional neural networks (CNNs). Gatys et al. [1] first demonstrated that feature activations of a pretrained CNN (e.g., VGG-19 trained on ImageNet) can be used to separately represent image content and style. Given a content image $x^{\mathrm{cont}}$ and a style image $x^{\mathrm{style}}$, they define a content representation as the activation maps $F^l\left(x\right)$ at certain higher layers l of the CNN, and a style representation based on the Gram matrices $G^l\left(x\right)$ of feature correlations within each layer. Style transfer is formulated as an optimization problem that finds an output image $\widehat{x}$ minimizing a weighted sum of content and style losses:

$${\mathcal{L}}_{\mathrm{total}}\left(\widehat{x}\right)=\alpha {\mathcal{L}}_{\mathrm{content}}(\widehat{x},x^{\mathrm{cont}})+\beta {\mathcal{L}}_{\mathrm{style}}(\widehat{x},x^{\mathrm{style}}),$$

where each gradient descent step back-propagates from the selected feature activations all the way to the image pixels.

Subsequent works accelerate this optimization by training feed-forward networks to perform style transfer in a single forward pass [2,3], and extend to arbitrary style transfer via adaptive instance normalization and related techniques [4]. These methods demonstrate that CNN features provide a powerful, flexible representation of image content and style, which we leverage on the visual side of our hybrid model.

Audio style transfer models: Audio style transfer has been explored by adapting image-style-transfer ideas to time–frequency representations. Verma and Smith [6] applied a Gatys-like optimization directly on log-Mel spectrograms, using a CNN to define content and style features, and reconstructing waveforms via phase estimation. Other works use encoder–decoder architectures, WaveNet-like autoencoders, and adversarial training to capture timbre, pitch contour, and rhythmic structure across scales [7,8,9,10]. Compared to images, audio style transfer must deal with longer temporal dependencies and phase-sensitive reconstruction, which motivates the use of recurrent networks or temporal convolutions in addition to CNNs.

In our hybrid framework, we adopt spectrogram-based CNN encoders to extract audio content and style features compatible with the emotional and structural representations used on the image side, and use sequence models to better handle temporal dynamics.

Cross-modal style transfer: Recent work has addressed text-to-image style transfer by conditioning image generators on textual descriptions while preserving style attributes [31,32]. In music, performance generation and expressive rendering can be viewed as a special case of text-to-music or score-to-audio style transfer, where the goal is to add expressive timing and dynamics to a neutral performance while preserving the underlying score content [8]. Beyond machine learning, art studies have long discussed how artists map visual ideas into musical structures and vice versa [33,34]. These efforts have given rise to synesthetic art forms and tools such as color music [35], Lumia art [36], and music visualization techniques widely used in modern multimedia systems [12,37]. More recently, multimodal contrastive learning and CLIP-style architectures have achieved impressive progress in learning joint audio–image or audio–text representations by maximizing agreement between paired samples in a shared embedding space. Representative examples include large-scale contrastive pretraining for audio–image retrieval and cross-modal recognition, as well as updated CLIP variants that operate on video and audio streams. These methods typically rely on strongly paired or co-occurring multimodal data and focus on discriminative objectives, whereas our framework uses a continuous VA space as an affective bridge and is trained for generative audio–image style transfer under weak alignment.

Our work differs from these approaches in that we aim to explicitly learn a shared, emotion-aware feature space for both visual and auditory modalities, and use it to perform bi-directional audio–image style transfer rather than unidirectional text-to-image or score-to-audio mappings.

2.3. Datasets

To enable a model to learn relationships between the styles of visual art and music, we require datasets that provide both images and music with aligned or comparable semantic and emotional annotations. In particular, we focus on datasets annotated in a continuous valence–arousal (VA) space, which supports fine-grained modeling of affective content across modalities [20,22,23].

Image datasets: We combine three affective image datasets, IAPS, NAPS, and EMOTIC as our image corpus. The International Affective Picture System (IAPS) [38] is a widely used emotion-evoking image set in psychology, containing 1182 documentary-style natural color images, each annotated with 9-point ratings on valence, arousal, and dominance (VAD) by about 100 college students. The Nencki Affective Picture System (NAPS) [39] consists of 1356 realistic, high-quality photographs rated by 204 mostly European participants using 9-point bipolar semantic sliding scales on valence, arousal, and approach–avoidance dimensions. EMOTIC [40] contains 23,082 images of people in unconstrained environments, annotated via Amazon Mechanical Turk with continuous 10-point ratings on VAD dimensions. It is important to note that all three corpora consist primarily of affective photographs collected for psychology and affective computing research, rather than carefully curated artistic painting datasets. Consequently, our experiments evaluate cross-modal affective transfer between music and affective photos, not between music and specific paintings.

Music dataset: For music, we adopt the DEAM dataset [41] as our corpus. DEAM contains 1802 music excerpts and full songs annotated with VA values (from $-1$ to $+1$) both continuously (per second) and as global song-level ratings. For annotation stability, we use segments starting from the 15th second of each track, following the original protocol. All audio signals are sampled at 44,100 Hz. Most songs (1723 in total) are 45 s long, with the remaining tracks varying in length, up to more than 600 s.

Preprocessing and splits: Since image and music datasets use different rating scales, we first normalize all VA values to the range $[0, 1]$ independently for images and music, based on their respective minima and ranges. After normalization, we randomly split both image and music data into 80% for training, 5% for validation, and 15% for testing, as summarized in

Table 1. Statistics of the image and music datasets used in our experiments. We report the number of samples in each split after normalization of VA labels.

Dataset	Train	Val	Test
IAPS [38]	945	59	178
NAPS [39]	1085	68	203
EMOTIC [40]	18,465	1154	3463
DEAM [41]	1442	90	270

. These aligned VA labels across modalities form the basis for learning our emotion-aware synesthetic feature space. Because the alignment is performed only in the shared VA space and there is no one-to-one pairing between specific images and audio tracks, our cross-modal mapping should be understood as weakly aligned affective transfer. The VA embeddings act as a pivot space that constrains the model to generate outputs with consistent affective properties, but do not guarantee semantic or instance-level correspondences. We explicitly acknowledge this limitation and view the extension of our framework to strongly aligned, curated painting–music datasets as an important direction for future work.

VA-Based Alignment of 0.5 s Audio Segments and Static Images

In an early pilot experiment, we explored art videos where image frames were temporally aligned with the soundtrack at approximately 0.5 s resolution. The current paper, however, does not use such video data in the main experiments. Instead, all reported results are based on static affective image datasets (IAPS, NAPS, EMOTIC) and the DEAM music corpus. For audio, we segment each track into non-overlapping 0.5 s windows and assign each window the corresponding local VA label, obtained either from the continuous annotations (DEAM) or from the global VA rating replicated across segments. For images, we use their provided VA annotations (IAPS/NAPS VAD ratings, EMOTIC VAD scores) and normalize them to the same $[0, 1]$ range. During training, “alignment” between a 0.5 s audio segment and a static image means that their VA labels fall into the same VA bin or neighborhood in the 2D VA space, not that they are time-synchronized audio–video frames. We have removed or rewritten all ambiguous references to “art videos” in the method description to avoid confusion.

3. Our Approach

We propose a novel deep hybrid synesthesia model for audio–image style transfer. The key motivation comes from the observation that emotion provides a natural bridge between music and painting: synesthetic experiences are often described in terms of coherent affective impressions across modalities. By learning emotion-aware shared representations for audio and images, our model can discover common synesthetic features and use them to perform cross-modal style transfer.

Existing neural style transfer models are typically unimodal, operating either on 2D images or on 1D/2D audio representations, and therefore cannot jointly handle 2D spatial image features and time-series audio features in a unified framework. Our method addresses this limitation by: (1) decomposing each modality into short-term component features and long-term style features; (2) mapping component features across modalities through an emotion-guided feature space; and (3) aligning long-term style codes between audio and images.

In the following, we first provide an overview of the training manner and pipeline, using an Oriental music–to–ink-and-color landscape painting task as an illustrative example (Section 3.1). We then describe the component transfer module and the style transfer module in detail (Section 3.2). Finally, we present the training objectives and loss functions (Section 3.2.1).

3.1. Overview

3.1.1. Training Manner

In Figure 3 we illustrate the overall training scheme. We start from paired or weakly aligned audio a and image i sampled from the audio dataset and image dataset. Two encoders, $E_a$ for audio and $E_i$ for images, map the inputs into content and style codes:

$$(C_a,S_a)=E_a\left(a\right),(C_i,S_i)=E_i\left(i\right),$$

where $C_a$ and $C_i$ denote content codes, and $S_a$ and $S_i$ denote style codes in their respective domains. Two decoders, $D_a$ and $D_i$, reconstruct signals from these codes:

$$\widehat{a}=D_a(C_a,S_a),\widehat{i}=D_i(C_i,S_i).$$

For audio, we further decompose the content code into a spectral component and a prosodic component:

$$C_a=(C_{\mathrm{spec}},C_{f0}),$$

where $C_{\mathrm{spec}}$ encodes timbre/spectral envelope and $C_{f0}$ encodes pitch and prosody. For images, we decompose the content code into shape and color:

$$C_i=(C_{\mathrm{shape}},C_{\mathrm{color}}).$$

The component transfer module learns cross-modal mappings between these codes. Specifically, we learn:

$$C_{a\to i}=T_{a\to i}(C_{\mathrm{spec}},C_{f0}),C_{i\to a}=T_{i\to a}(C_{\mathrm{shape}},C_{\mathrm{color}}),$$

where $C_{a\to i}$ is an image-like content code derived from audio, and $C_{i\to a}$ is an audio-like content code derived from images.

Within each domain, combining content and style codes allows reconstruction:

$$a2a:\widehat{a}=D_a(C_a,S_a),i2i:\widehat{i}=D_i(C_i,S_i).$$

For cross-domain translation, we sample style codes

$$Z_i\sim \mathcal{N}({\mu }_{S_i},{\sigma }_{S_i}),Z_a\sim \mathcal{N}({\mu }_{S_a},{\sigma }_{S_a}),$$

where $({\mu }_{S_i},{\sigma }_{S_i})$ and $({\mu }_{S_a},{\sigma }_{S_a})$ are estimated statistics of image and audio style codes, respectively. Assuming $Z_i$ lies in the image style space, we obtain audio-to-image translation:

$$a2i:\tilde{i}=D_i(C_{a\to i},Z_i),$$

indicated by the red arrows in Figure 3. Similarly, assuming $Z_a$ lies in the audio style space, we obtain image-to-audio translation:

$$i2a:\tilde{a}=D_a(C_{i\to a},Z_a).$$

Training alternates between within-domain reconstruction ($a2a$, $i2i$) and cross-domain translation ($a2i$, $i2a$), with shared encoders/decoders and shared emotion-guided constraints. For simplicity, we primarily describe the audio-to-image direction, but the same framework applies to image-to-audio and unimodal style transfer (e.g., between different painting styles).

3.1.2. Pipeline

In Figure 3, we detail the end-to-end pipeline for audio-to-image style transfer. Given an input audio signal a:

Audio analysis: we compute its time–frequency representation (e.g., log-Mel spectrogram) and extract short-term content features $C_{\mathrm{spec}}$ and $C_{f0}$ (spectrum and F0) and long-term style code $S_a$ (capturing global rhythm, instrumentation, and expressive characteristics).

Component transfer: using the component transfer model $T_{a\to i}$, we map $(C_{\mathrm{spec}},C_{f0})$ to image-side content features $(C_{\mathrm{shape}},C_{\mathrm{color}})$, based on our synesthetic mapping theory (details in Section 3.2). This yields $C_{a\to i}$, which is compatible with the image decoder $D_i$.

Style conversion: a pretrained style transfer model maps the audio style code $S_a$ to a corresponding image style code $Z_i$ in the visual style space. This module captures, for example, the correspondence between traditional Oriental music and ink-and-color landscape style versus Western symphonic music and oil painting style.

Image synthesis and discrimination: the translated image

$$\tilde{i}=D_i(C_{a\to i},Z_i)$$

is fed into an image discriminator $D_{\mathrm{img}}$, which evaluates both: component-level realism (color/shape plausibility) and style-level realism (consistency with target artistic style). We employ a fusion hinge loss that combines component-level and style-level adversarial hinge loss terms to jointly drive the learning of $T_{a\to i}$, $E_i$, and $D_i$.

This unified pipeline allows us to explicitly disentangle and recombine content and style across modalities, under the guidance of a shared emotional representation.

3.2. Component Transfer

Auditory–visual synesthesia is a special form of cross-modal integration in which sounds evoke involuntary visual experiences. Prior studies show that synesthetic color and shape perceptions are often associated with timbre and prosodic patterns in audio. Motivated by this, our audio–image component transfer focuses on learning structured correlations between audio components (spectrum and F0) and image components (color and shape), and then performing efficient, temporally-aware cross-modal mapping.

3.2.1. Signal Analysis

An audio signal is commonly decomposed into timbre and pitch components, which are closely related to the spectrum and F0, respectively. Timbre (spectrum) characterizes the tone color or unique quality of a sound, enabling us to distinguish different speakers and instruments. Pitch (F0), generated at the vocal folds (or analogous excitation mechanism in instruments) and modulated by tension and length, conveys melodic contour and prosodic emphasis, and is strongly linked to perceived emotion.

As discussed in Section 2, principal component analysis (PCA) on natural image patches and speech slices (see Figure 2) shows that: image PCs mainly capture intensity, gradient, and edge-like patterns in 2D spatial structures, while audio PCs tend to form periodic basis functions over time, decomposing the signal into quasi-harmonic bands.

Moreover, in spectrograms, the two axes represent fundamentally different quantities (time vs. frequency), in contrast to the purely spatial dimensions in images. Therefore, naively mapping spectral and F0 features to 2D image features without accounting for temporal structure can lead to suboptimal audio–image component transfer.

To address this, we impose a common temporal unit for both modalities. Specifically:

• We segment audio spectrograms into fixed-length windows of size $32\times 256$ (time × frequency), corresponding to approximately 0.5 s of audio. Each segment is encoded into $(C_{\mathrm{spec}},C_{f0})$.
• We select video clips of around 1 s at 25 fps, and align or average their frame-level features into the same temporal resolution (e.g., 0.5 s units), so that each audio segment corresponds to one or two image frames. From these frames, we extract RGB-based features and higher-level representations, which are then decomposed into $(C_{\mathrm{shape}},C_{\mathrm{color}})$.

In the final experimental setup of this paper, we do not rely on true audio–video pairs. Instead, audio and images are related only via their VA labels. To obtain a common temporal unit on the audio side, we segment spectrograms into fixed-length windows of approximately 0.5 s (32 frames in our configuration) and encode each window into $(C_{\mathrm{spec}},C_{f0})$. On the visual side, we work with static affective images rather than video frames; each image is treated as a single “time-agnostic” unit with associated VA label. When we speak of “alignment” between 0.5 s audio segments and images in the following, this always refers to matching in the VA space, not frame-level synchronization.

3.2.2. Feature Mapping

As illustrated in Figure 4, we first extract spectral features and F0 trajectories from audio segments, and RGB and structural features from image frames derived from art videos.

To transfer audio features $(C_{\mathrm{spec}},C_{f0})$ to image features $(C_{\mathrm{shape}},C_{\mathrm{color}})$ effectively, we introduce an intermediate, modality-agnostic component space:

$$H={\mathrm{\Phi }}_{\mathrm{comp}}(C_{\mathrm{spec}},C_{f0})={\mathrm{\Psi }}_{\mathrm{comp}}(C_{\mathrm{shape}},C_{\mathrm{color}}),$$

where ${\mathrm{\Phi }}_{\mathrm{comp}}$ and ${\mathrm{\Psi }}_{\mathrm{comp}}$ are learnable mappings from audio components and image components, respectively, into the shared component space H. This space is: temporally aligned (each element corresponds to ~$0.5$ s units), emotionally informed (supervised or regularized by VA labels), and structured such that similar emotional and structural patterns cluster together regardless of modality.

The cross-modal component mappings are then defined as:

$$\begin{array}{cc}\hfill C_{a\to i}& ={\mathrm{\Psi }}_{\mathrm{comp}}^{-1}\left({\mathrm{\Phi }}_{\mathrm{comp}}(C_{\mathrm{spec}},C_{f0})\right),\hfill \\ \hfill C_{i\to a}& ={\mathrm{\Phi }}_{\mathrm{comp}}^{-1}\left({\mathrm{\Psi }}_{\mathrm{comp}}(C_{\mathrm{shape}},C_{\mathrm{color}})\right),\hfill \end{array}$$

(1)

where implemented via neural networks trained with reconstruction and consistency losses. In practice, we parameterize these mappings with temporal convolutional networks or lightweight sequence models, ensuring that local temporal dynamics are preserved during transfer.

3.2.3. Transfer Model

The component transfer model is realized as a conditional encoder–decoder with adversarial regularization: An audio component encoder $E_a^{\mathrm{comp}}$ maps $(C_{\mathrm{spec}},C_{f0})$ into the shared space H; An image component encoder $E_i^{\mathrm{comp}}$ maps $(C_{\mathrm{shape}},C_{\mathrm{color}})$ into H; Two decoders, $D_i^{\mathrm{comp}}$ and $D_a^{\mathrm{comp}}$, map from H back to image and audio component codes, respectively:

$$\begin{array}{cc}\hfill C_{a\to i}& =D_i^{\mathrm{comp}}\left(E_a^{\mathrm{comp}}(C_{\mathrm{spec}},C_{f0})\right),\hfill \\ \hfill C_{i\to a}& =D_a^{\mathrm{comp}}\left(E_i^{\mathrm{comp}}(C_{\mathrm{shape}},C_{\mathrm{color}})\right).\hfill \end{array}$$

(2)

Both $E_a^{\mathrm{comp}}$ and $E_i^{\mathrm{comp}}$ are implemented as temporal convolutional stacks with residual connections, so that they can model local temporal patterns (e.g., rhythmic pulses) in audio and temporal evolution of visual elements in video frames, and their receptive fields cover the 0.5 s segments while maintaining sufficient resolution.

To encourage realistic component distributions after transfer, we use auxiliary discriminators at the component level (one in the audio space, one in the image space), which judge whether a component sequence is real or transferred. These discriminators are trained with hinge losses, and their gradients are back-propagated to $E_a^{\mathrm{comp}}$, $E_i^{\mathrm{comp}}$, $D_a^{\mathrm{comp}}$, and $D_i^{\mathrm{comp}}$.

3.3. Style Transfer

While the component transfer module focuses on short-term correlations (spectrum–color, F0–shape, etc.), the style transfer module handles long-term, high-level artistic style. For audio, the style code $S_a$ captures global attributes such as tempo, instrument ensemble, articulation patterns, and overall emotional trajectory. For images, the style code $S_i$ encodes global brushwork, composition, texture statistics, and color palette typical of specific art movements (e.g., ink-wash vs. oil painting).

We learn two style encoders, $E_a^{\mathrm{style}}$ and $E_i^{\mathrm{style}}$, and a shared style space $\mathcal{Z}$ where:

$$S_a=E_a^{\mathrm{style}}\left(a\right),S_i=E_i^{\mathrm{style}}\left(i\right),Z_a,Z_i\in \mathcal{Z}.$$

A style alignment network $T_{\mathrm{style}}$ is trained to map between audio and image style codes:

$$Z_i=T_{\mathrm{style}}\left(S_a\right),Z_a=T_{\mathrm{style}}^{-1}\left(S_i\right),$$

subject to:

• distribution alignment: the mapped codes follow the empirical style distributions in each modality;
• emotion consistency: mapped styles preserve the VA trajectory associated with the source signal;
• cycle consistency: mapping audio style to image style and back recovers the original style up to small perturbations.

During translation, $Z_i$ is combined with $C_{a\to i}$ and fed into the image decoder $D_i$ to produce an image that is consistent both in local components and global style with the source audio.

3.4. Loss Functions

The model is trained end-to-end with a combination of reconstruction, adversarial, and regularization losses:

• Within-domain reconstruction losses:

  $${\mathcal{L}}_{\mathrm{rec}}^a={\parallel a-D_a(C_a,S_a)\parallel }_1,{\mathcal{L}}_{\mathrm{rec}}^i={\parallel i-D_i(C_i,S_i)\parallel }_1,$$

  encouraging faithful encoding–decoding in each modality.
• Component consistency losses:

  $${\mathcal{L}}_{\mathrm{comp}}^{a\to i}={\parallel H_a-H_{a\to i}\parallel }_1,{\mathcal{L}}_{\mathrm{comp}}^{i\to a}={\parallel H_i-H_{i\to a}\parallel }_1,$$

  where $H_a$ and $H_i$ are shared representations from $E_a^{\mathrm{comp}}$ and $E_i^{\mathrm{comp}}$, and $H_{a\to i}$, $H_{i\to a}$ are their cross-modal reconstructions.
• Adversarial hinge losses at component and style levels:

  $${\mathcal{L}}_{\mathrm{adv}}^{\mathrm{comp}}={\mathcal{L}}_{\mathrm{adv}}^{\mathrm{comp-a}}+{\mathcal{L}}_{\mathrm{adv}}^{\mathrm{comp-i}},{\mathcal{L}}_{\mathrm{adv}}^{\mathrm{style}}={\mathcal{L}}_{\mathrm{adv}}^{\mathrm{style-a}}+{\mathcal{L}}_{\mathrm{adv}}^{\mathrm{style-i}},$$

  with hinge formulations for the discriminators to ensure realistic component distributions and style codes.
• Emotion consistency loss:

  $${\mathcal{L}}_{\mathrm{emo}}={\parallel f_{\mathrm{VA}}\left(a\right)-f_{\mathrm{VA}}\left(\tilde{i}\right)\parallel }_2^2,$$

  where $f_{\mathrm{VA}}(·)$ is a pretrained or jointly learned emotion regressor that predicts VA coordinates from audio or images. This term enforces that the translated image preserves the emotional content of the source audio.
• Cycle consistency loss for style codes:

  $${\mathcal{L}}_{\mathrm{cycle}}^{\mathrm{style}}={\parallel S_a-T_{\mathrm{style}}^{-1}(T_{\mathrm{style}}\left(S_a\right))\parallel }_1+{\parallel S_i-T_{\mathrm{style}}(T_{\mathrm{style}}^{-1}\left(S_i\right))\parallel }_1,$$

  which stabilizes the bidirectional mapping in the style space.

The overall objective combines these terms:

$$\begin{array}{cc}\hfill \mathcal{L} =& {\lambda }_{\mathrm{rec}}\left({\mathcal{L}}_{\mathrm{rec}}^a+{\mathcal{L}}_{\mathrm{rec}}^i\right)+{\lambda }_{\mathrm{comp}}{\mathcal{L}}_{\mathrm{comp}}\hfill \\ \hfill & +{\lambda }_{\mathrm{adv}}\left({\mathcal{L}}_{\mathrm{adv}}^{\mathrm{comp}}+{\mathcal{L}}_{\mathrm{adv}}^{\mathrm{style}}\right)+{\lambda }_{\mathrm{emo}}{\mathcal{L}}_{\mathrm{emo}}+{\lambda }_{\mathrm{cycle}}{\mathcal{L}}_{\mathrm{cycle}}^{\mathrm{style}},\hfill \end{array}$$

(3)

where ${\lambda }_{·}$ are weighting coefficients. By jointly optimizing this objective, our model learns to generate images from audio (and vice versa) that are consistent at the component level, stylistically coherent, and emotionally aligned across modalities.

To assess the robustness of the loss weighting coefficients in Equation (3), we conducted a small sensitivity analysis by varying one coefficient at a time while keeping the others fixed. Specifically, we scaled each of ${\lambda }_{\mathrm{rec}}$, ${\lambda }_{\mathrm{comp}}$, ${\lambda }_{\mathrm{adv}}$, and ${\lambda }_{\mathrm{emo}}$ by factors of 0.5 and 2.0 and measured the resulting FID and VA-MSE on the validation set. We observed that moderate changes (within this range) lead to changes of less than 3.5 in FID and 0.004 in VA-MSE, indicating that the model is not overly sensitive to the exact choice of these hyperparameters. Stronger deviations (e.g., reducing ${\lambda }_{\mathrm{emo}}$ by an order of magnitude) noticeably degrade VA consistency while having smaller impact on FID, which is consistent with the role of the emotion loss in our objective.

4. Experiments

In this section, we evaluate the proposed hybrid synesthesia model on several audio–image style transfer tasks. We first describe the experimental setup, including datasets, preprocessing, and implementation details. We then present quantitative results using objective metrics and ablation studies, followed by qualitative visualizations and user studies.

4.1. Experimental Setup

4.1.1. Tasks

We consider three families of tasks:

Audio-to-image (A2I) component transfer: given an audio clip, generate an image whose local components (color, shape) correspond to audio components (spectrum, F0) under the learned synesthetic mapping, while preserving the overall emotional content.

Audio-to-image (A2I) full style transfer: given an audio clip, generate an image that not only matches local components but also exhibits a coherent visual artistic style (e.g., ink-wash vs. oil painting) corresponding to the audio style (e.g., Oriental vs. Western music).

Image-to-audio (I2A) transfer (optional): symmetric experiments where an input image is translated into an audio segment with matching emotional content and style.

Unless otherwise stated, we focus on the A2I direction as our primary evaluation, and report I2A results qualitatively. In all reported experiments, we restrict ourselves to monophonic or moderately polyphonic audio excerpts in which a dominant F0 trajectory is meaningful. For highly complex, multi-instrumental mixtures where no single F0 contour can adequately summarize the pitch structure, our current F0-based component representation becomes less representative. Handling such material would require extending the model with multi-F0 estimation and/or richer timbre embeddings, which we regard as an important direction for future work.

4.1.2. Datasets and Splits

We use the affective image and music datasets described in Section 2:

Images: IAPS [38], NAPS [39], and EMOTIC [40], all annotated with continuous or discretized valence–arousal (VA) labels.

Music: DEAM [41], containing music excerpts with continuous and global VA annotations.

We normalize all VA values into $[0, 1]$ within each modality and randomly split both image and music data into 80% training, 5% validation, and 15% testing, as summarized in Table 1. For cross-modal experiments, we do not require exact pairs of image and audio; instead, samples are weakly aligned via their VA labels (e.g., in the same VA bin or neighborhood), which we use for sampling during training mini-batches.

4.1.3. Preprocessing

Audio signals are resampled to 44.1 kHz. We compute log-Mel spectrograms with a 1024-point FFT, a hop size of 256 samples, and 128 Mel bands. F0 contours are extracted using the PYIN algorithm, configured with a frame hop of 256 samples, a minimum F0 of 50 Hz and a maximum F0 of 1000 Hz, and voicing thresholds tuned on the DEAM validation set. The resulting F0 trajectories are encoded as normalized continuous values.

For image corpora, we resize images to $256\times 256$ and normalize RGB channels. For art videos, we sample clips of about 1 s at 25 fps and either use a central frame or aggregate frame features into temporally aligned representations (0.5 s resolution) to match audio segments. Image content and style features are extracted using pretrained CNN backbones (e.g., VGG-like architectures) and then fed into our encoders $E_i$, $E_i^{\mathrm{comp}}$, and $E_i^{\mathrm{style}}$.

4.1.4. Implementation Details

Feature Dimensions

For audio, we compute log-Mel spectrograms with $F=128$ Mel bands and segment them into $T=32$ time frames per 0.5 s window, resulting in $128\times 32$ inputs to the audio content encoder $E_a$. The encoder outputs a 256-dimensional content code $C_a$ (split into a 192-dimensional spectral code $C_{\mathrm{spec}}$ and a 64-dimensional F0/prosodic code $C_{f0}$) and a 128-dimensional style code $S_a$. For images, we resize each image to $256\times 256$ and extract a 512-dimensional content embedding $C_i$ and a 128-dimensional style embedding $S_i$ using a CNN backbone followed by global average pooling. These dimensions are chosen to balance expressiveness and computational cost.

In the component transfer module, both audio and image component encoders map their inputs into a shared component space H of dimension 256. Concretely, $E_a^{\mathrm{comp}}$ and $E_i^{\mathrm{comp}}$ output 256-dimensional vectors per 0.5 s segment, and the decoders $D_a^{\mathrm{comp}}$ and $D_i^{\mathrm{comp}}$ map from this 256D space back to their respective content codes $(C_{\mathrm{spec}},C_{f0})$ and $(C_{\mathrm{shape}},C_{\mathrm{color}})$. In the style transfer module, both $E_a^{\mathrm{style}}$ and $E_i^{\mathrm{style}}$ produce 128-dimensional style codes that live in a shared style space $\mathcal{Z}$; the style alignment network $T_{\mathrm{style}}$ therefore learns mappings between 128D audio and image style vectors.

Table 2. Key feature dimensionalities in the proposed model. $C_{\mathrm{spec}}$ and $C_{f0}$ denote audio spectral and F0 content codes; $C_{\mathrm{shape}}$ and $C_{\mathrm{color}}$ denote image shape and color content codes. H is the shared component space, and $\mathcal{Z}$ is the shared style space.

Quantity	Symbol	Dim.
Audio content code	$C_a$	256
Spectral component	$C_{\mathrm{spec}}$	192
F0/prosodic comp	$C_{f0}$	64
Image content code	$C_i$	512
Shape component	$C_{\mathrm{shape}}$	256
Color component	$C_{\mathrm{color}}$	256
Shared component space	H	256
Audio style code	$S_a$	128
Image style code	$S_i$	128
Shared style space	$\mathcal{Z}$	128

summarizes all the main feature dimensionalities.

Inverse Mappings and VA Regressors

The inverse mappings from the shared VA space back to audio/image latent spaces are implemented as small multilayer perceptrons (MLPs) with two hidden layers of size 256 and ReLU activations, mapping 2D VA coordinates to 128D style codes in the audio and image domains, respectively. The VA regressors $f_{\mathrm{VA}}^{\mathrm{audio}}$ and $f_{\mathrm{VA}}^{\mathrm{image}}$ are trained separately on DEAM and the combined affective image corpus using the mean-squared-error loss on the normalized VA labels, an Adam optimizer with learning rate $1\times {10}^{-4}$, and early stopping on a held-out validation set. We then fix these regressors when training the main model and use them only for computing the emotion consistency loss ${\mathcal{L}}_{\mathrm{emo}}$.

Style Label Estimation and Normalization

For DEAM, we use the provided continuous VA annotations and derive global VA labels by averaging over time for each song. For IAPS and NAPS, we convert the 9-point valence and arousal ratings to the $[0, 1]$ range via linear scaling; for EMOTIC, we normalize the 10-point VAD scores in the same manner. When global labels are available, each 0.5 s audio segment or each static image inherits the corresponding global VA label. All VA values across datasets are then linearly rescaled to $[0, 1]$ per modality to construct the shared VA space used by the component and style transfer modules.

4.2. Evaluation Metrics

We evaluate the model from three complementary perspectives.

4.2.1. Visual Quality and Realism

For generated images, we use:

• Fréchet Inception Distance (FID) between generated images and real images in the corresponding style domain, computed on Inception features.
• Style classification accuracy: a pretrained style classifier (e.g., ink-wash vs. oil painting) predicts the style of generated images; we report the fraction of images classified into the intended target style.

4.2.2. Emotion and Component Consistency

To measure emotional alignment across modalities, we use:

• Cross-modal VA consistency: a pretrained emotion regressor $f_{\mathrm{VA}}$ is applied to source audio and generated images; we compute the mean squared error (MSE) between predicted VA coordinates:

  $${\mathrm{MSE}}_{\mathrm{VA}}=\mathbb{E}\left[{\parallel f_{\mathrm{VA}}\left(a\right)-f_{\mathrm{VA}}\left(\tilde{i}\right)\parallel }_2^2\right].$$
• Component correlation: we compute correlations between low-level descriptors (e.g., spectral centroid vs. color saturation, F0 variance vs. shape complexity) across matched audio–image pairs, before and after transfer, to assess whether the component mapping increases cross-modal consistency.

4.2.3. Ablation Metrics

For ablation studies, we monitor: Reconstruction SNR / PSNR in within-domain tasks ($a2a$, $i2i$); FID and VA consistency when disabling specific modules (component transfer, style transfer, emotion loss).

4.3. Quantitative Results

4.3.1. Audio-to-Image Style Transfer

Table 3. Audio-to-image (A2I) style transfer performance on the test set. Lower FID and VA-MSE are better; higher style accuracy is better.

Method	FID	Style Acc. (%)	VA-MSE
Baseline-1: CNN U-Net	68.4	61.2	0.042
Baseline-2: Emo-cGAN	55.7	73.5	0.031
Ours (full model)	41.3	85.9	0.019

reports FID, style accuracy, and VA consistency for A2I full style transfer. We compare:

• Baseline-1 (CNN U-Net): a model that directly maps Mel spectrograms to images using a U-Net without explicit component or style disentanglement.
• Baseline-2 (Emotion-conditioned GAN): a conditional GAN that takes spectrograms and VA labels as input and generates images conditioned only on emotion, without component transfer.
• Ours (full): the proposed hybrid synesthesia model with component and style modules and emotion loss.

Our full model achieves the lowest FID (best visual quality), highest style accuracy, and significantly lower VA MSE than both baselines. Including component transfer and emotion consistency loss improves the correlation between audio and image VA predictions, indicating better cross-modal emotional alignment.

4.3.2. Ablation Study

To understand the contribution of each component, we evaluate several ablated variants:

• w/o component: remove the component transfer network, directly feeding audio content features into the image decoder.
• w/o style: remove the style alignment network, using only image style priors or random style codes.
• w/o emotion: remove the emotion consistency loss ${\mathcal{L}}_{\mathrm{emo}}$.

Removing the component module significantly degrades FID and component correlation metrics, indicating that explicit component mapping is crucial for coherent low-level cross-modal structure. Removing the style module reduces style classification accuracy and leads to more ambiguous art styles in generated images. Removing emotion loss deteriorates VA consistency while having less impact on FID, confirming that emotional alignment is specifically enforced by ${\mathcal{L}}_{\mathrm{emo}}$.

4.4. Qualitative Analysis

Figure 4 shows representative examples of audio-to-image transfers. For Oriental music clips, the model tends to produce images with ink-like textures, soft edges, and limited color palettes, consistent with traditional ink-and-color landscape styles. For Western symphonic excerpts, the generated images often exhibit richer color palettes, more pronounced brush strokes, and structured compositions reminiscent of Western oil paintings.

We also inspect the effect of manipulating specific audio components. When we artificially increase spectral brightness or F0 variability while keeping VA constant, the generated images shift toward higher color saturation or more complex shapes, while their overall emotional tone (e.g., calm vs. tense) remains similar. This suggests that the model has learned meaningful component-level correspondences between audio and visual features.

4.5. User Study

We conducted a perceptual user study with $N=24$ participants (age 20–32 years, mean = 24.7; 10 female, 14 male). All participants were non-experts in visual arts and music production but reported normal hearing and normal or corrected-to-normal vision. None of them had prior exposure to our model or stimuli.

Each participant completed two tasks in a quiet lab environment using headphones and a calibrated display. In Task 1 (emotion matching), each trial presented a 6 s audio clip together with three candidate images generated by Baseline-1, Baseline-2, and our method (order randomized per trial and per participant). Participants were asked: “Which image best matches the emotional impression of the audio?” and responded by clicking one of the three images. Each participant completed 30 trials in Task 1, with audio clips sampled without replacement from the test set; clip and condition order were fully randomized.

In Task 2 (synesthetic plausibility), the procedure was similar, but the question was: “If this music were a painting, which image would it most likely look like?” Again, each participant completed 30 trials with randomized ordering. We discarded trials with response times below 0.5 s as inattentive (on average <2% of trials) and recomputed preference percentages after exclusion.

We report mean preference percentages across participants together with 95% confidence intervals (CIs) estimated using the t-distribution. For each comparison between our method and a baseline, we perform a binomial test over aggregated choices; all reported improvements are statistically significant at $p<0.01$ unless otherwise noted.

5. Limitations and Discussion

While Table 3 and

Table 4. Ablation study on the A2I task. We report FID, style accuracy, and VA-MSE. “w/o component”: without component transfer module; “w/o style”: without style alignment; “w/o emotion”: without emotion consistency loss.

Variant	FID	Style Acc. (%)	VA-MSE
Ours (full)	41.3	85.9	0.019
w/o component	49.8	82.1	0.024
w/o style	45.6	69.3	0.021
w/o emotion	42.7	84.1	0.028

show clear improvements over baselines in terms of FID, style accuracy, and VA consistency, these gains should be interpreted in light of several limitations. First, all results are obtained under weak VA alignment: audio and images are only matched in the 2D VA space, not paired at the instance level. When VA estimates are noisy or biased, the quality of cross-modal transfer degrades and the emotional alignment metrics become less reliable. Second, the visual side of our model is trained on affective photo datasets rather than curated painting corpora, which limits the diversity and fidelity of the generated artistic styles. Third, our current architecture focuses on global VA and high-level styles; capturing more fine-grained, temporally evolving emotions and complex multimodal narratives remains an open challenge. We regard addressing these issues—especially training on strongly aligned painting–music datasets and incorporating richer affective representations—as important directions for future work.

6. Conclusions

This paper presents a novel deep hybrid synesthesia model for cross-modal artistic style transfer between audio and images, grounded in neuroscientific principles of sensory-emotional integration. By learning emotion-aware mappings in a continuous valence–arousal (VA) space, our framework successfully bridges low-level components (audio rhythm/spectrum ↔ image color/shape) and high-level styles (Eastern shui-mo ink-wash ↔ Western painting, traditional Oriental ↔ symphonic music). Extensive experiments demonstrate that the model generates perceptually coherent cross-modal artworks: audio-to-image synthesis (“seeing the sound”) achieves controllability over visual components (e.g., spectral brightness → color saturation), while full style transfer preserves emotional content (54% lower VA-MSE than baselines) and stylistic authenticity (85.9% style accuracy). Human evaluations confirm strong synesthetic plausibility, with 63% preference for emotion/style consistency. This work establishes emotion as a unifying substrate for multimodal creativity, enabling machines to reinterpret artistic expression beyond unimodal constraints. To facilitate further research on cross-modal affective transfer, we plan to publicly release the core implementation of our model together with the VA-binned alignment metadata (subject to the licensing constraints of the underlying DEAM, IAPS, NAPS, and EMOTIC datasets), so that other researchers can reproduce and extend our experiments.