Perspectives on Generative Sound Design: A Generative Soundscapes Showcase

Abstract

Recent advancements in generative neural networks, particularly transformer-based models, have introduced novel possibilities for sound design. This study explores the use of generative pre-trained transformers (GPT) to create complex, multilayered soundscapes from textual and visual prompts. A custom pipeline is proposed, featuring modules for converting the source input into structured sound descriptions and subsequently generating cohesive auditory outputs. As a complementary solution, a granular synthesizer prototype was developed to enhance the usability of generative audio samples by enabling their recombination into seamless and non-repetitive soundscapes. The integration of GPT models with granular synthesis demonstrates significant potential for innovative audio production, paving the way for advancements in professional sound-design workflows and immersive audio applications.

1. Introduction

1.1. Transformers in Media Generation

The presence of media created by artificial neural networks, often labeled as “Artificial Intelligence” (AI), is becoming more prevalent in many areas of daily life. It has reached a level where it can both deceive and captivate human audiences. This is particularly evident in creative domains, where AI’s ability to mimic human artistry continues to blur the lines between machine- and human-generated content. A recent study in Nature revealed that AI-generated poetry is becoming increasingly indistinguishable from human work (Porter and Machery 2024). However, AI has the potential to expand existing media by introducing new, contextually dependent material, enriching both the content and the experience it offers (Samson 2024).

1.2. Historical Context

1.2.1. The Beginnings of Generative Music

In the history of music, the composition has involved meticulous preplanning and crafting of fixed sequences of notes, with classical composers like Bach and Beethoven establishing structured frameworks that combined careful planning with space for interpretation (Taruskin and Gibbs 2013). Bach’s basso continuo notation, for instance, provided a harmonic foundation while allowing performers to interpret and elaborate on the given material, leading to subtle variations between performances. Similarly, dynamics, phrasing, and ornamentation were shaped by a performer’s individual style, introducing an element of “human randomness” into otherwise structured compositions.

An early example of deliberately incorporating stochastic processes into music can be found in Mozart’s Musikalisches Würfelspiel (Musical Dice Game). This compositional tool uses dice rolls to assemble measures of music from predefined options, creating a unique piece with each play. While Mozart provided the building blocks, the element of chance determined the final outcome, blending deterministic composition with probabilistic execution evoking inspiration in modern composers (Wayne 2023).

1.2.2. Stochastic Soundscapes Origins

Outside the musical tradition, generative soundscapes have a rich historical precedent. Listening to natural phenomena, such as wind, waves, or rain, inspires a form of sonic awareness rooted in change and irregularity. In many cultures, wind chimes or water-driven instruments were created to channel these forces into sound-producing structures. These devices exemplify an early form of procedural sound design, where their creators set physical parameters, but the output was determined by environmental variability, such as wind speed or water flow (Toop 1995; Schafer 1993).

One notable example is the aeolian harp, a stringed instrument played by the wind, which dates back to ancient Greece and was later popularized in the Romantic era (Blesser and Salter 2007). Its ethereal tones, shaped by airflow rather than human agency, represent the early fusion of natural indeterminacy with intentional sonic construction. Similarly, sea organs, such as the modern Sea Organ in Zadar, Croatia, transform the irregular force of ocean waves into evolving harmonic textures by channeling air through tuned pipes (Stamać 2005).

1.2.3. Analogue Electronics for Generative Sound

The advent of analogue technology in the 20th century expanded the boundaries of sound creation. Composers such as Edgard Varèse and Karlheinz Stockhausen began experimenting with electronic sound generators, tape manipulation, and feedback systems (Griffiths 2010). These innovations introduced the concept of semi-automated sound creation, wherein the composer defined a system or process but left room for variability within its execution.

This era marked a philosophical shift, as composers like John Cage embraced indeterminacy, surrendering control over certain musical parameters to allow systems or performers to introduce randomness. Iannis Xenakis expanded this approach by applying mathematical models, such as probability theory and Markov chains, to shape musical structures (Xenakis and Kanach 1992).

However, this creative explosion would not have been possible without the parallel innovations of electronic engineers. The development of modular synthesizers, pioneered by figures such as Robert Moog and Don Buchla, brought voltage-controlled oscillators, filters, envelope generators, and operational amplifiers into the hands of artists, giving them even more space for creative exploration, parametrization, and randomization (Pinch and Trocco 2009).

1.2.4. Digital Electronics for Generative Sound

With the transition to digital technology in the late 20th century, the potential for computationally driven sound design has expanded exponentially. Digital synthesizers, samplers, and music software provide unprecedented precision and flexibility. At the same time, composers began exploring stochastic and algorithmic composition methods, using probability and randomness as creative tools. Visual programming environments, like Max/MSP and Pure Data, allow composers to build custom generative audio systems (Holmes 2020).

1.2.5. Transformers for Generative Sound Design

The current era of generative audio is driven by machine learning advancements. The use of generative neural networks for sound design is a dynamically developing field of research (Ji et al. 2020; Koutini et al. 2021). Owing to advances in Generative Pre-trained Transformer (GPT) networks (Huang et al. 2023b; Kreuk et al. 2023; Le et al. 2023; Verma and Berger 2021; Verma and Chafe 2021; Vyas et al. 2023), it has become possible to create complex soundscapes based on textual and visual prompts in an automated manner. As the potential of new generative technologies grows, there is a need to understand how sound designers can effectively use these new technologies and evaluate their capabilities (Ashvala and Lerch 2022; Oh et al. 2023).

The aim of this article is to explore how generative transformers can be used to create multilayered generative soundscapes, showcasing their potential in the field of sound design.

2. Related Work

2.1. A Definition of Soundscape

A soundscape refers to the acoustic environment as perceived or experienced by an individual, encompassing all audible components, both natural and man-made. For example, a forest soundscape may include the rustling of leaves, chirping of birds, and distant running water, while an urban soundscape may consist of traffic noise, human chatter, and construction sounds. Coined by composer and researcher R. Murray Schafer, the concept plays a central role in his theory of Acoustic Ecology, which studies the relationship between humans and their sonic environments (Schafer 1993). Schafer introduced key distinctions within soundscapes, such as keynotes (background sounds), signals (foreground sounds demanding attention), and sound marks (unique sounds with cultural or geographical significance). These elements highlight how soundscapes shape our perception of the world and influence our interactions with our environment.

In Acoustic Ecology, the quality of a soundscape is also evaluated, with an emphasis on preserving “hi-fi” environments—characterized by clarity and low ambient noise, such as in rural countryside—over “lo-fi” environments, where excessive noise masks sonic details. The soundscape concept extends beyond ecological concerns, influencing contemporary sound design and composition, where it is used to create immersive auditory experiences that reflect, interpret, and transform real-world sonic environments.

2.2. Generative Soundscapes

Research on generative soundscapes has a history, with previous studies exploring diverse technologies and methodologies for creating dynamic auditory environments. A generative soundscape creation study (Birchfield et al. 2005) introduced a framework for real-time soundscape generation rooted in the principles of Acoustic Ecology. This approach utilized keynotes and signals from various locations as foundational elements, dynamically managed through a probabilistic system implemented in Max/MSP that was adapted to user behavior and interaction. A later study (Eigenfeldt and Pasquier 2011) explored the use of autonomous artificial agents in soundscape composition. These agents analyzed the spectral content and metadata from a large sound database to create soundscapes with minimal spectral overlap, showcasing the potential of metadata-driven autonomous systems.

Further advancements in soundscape modeling and simulation (Koutsomichalis and Valle 2014) introduced a system capable of modeling three-dimensional soundscapes using generative algorithms. This system supports real-time user interaction and complex spatial configurations, thereby expanding the possibilities for dynamic and interactive soundscapes. A more recent study (Zhuang et al. 2024) demonstrated the integration of soundscapes with visual elements using generative AI by translating semantic audio vectors into street-view images.

2.3. Generative Audio Transformers

Several leading companies have proposed solutions for generating sound using transformer models. Many solutions focus on speech synthesis (Chen et al. 2021; Chen et al. 2022; Le et al. 2023; Rubenstein et al. 2023; Zhang et al. 2023) or generating music-like content (Agostinelli et al. 2023; Huang et al. 2023a; Rouard et al. 2024), with prominent examples including Meta’s MusicGen (Copet et al. 2023) and consumer-grade platforms Udio and Suno (Yu et al. 2024).

Models with broader capabilities, such as Meta’s Audiobox (Vyas et al. 2023), are also being developed and published, particularly in academic research (Evans et al. 2024; Huang et al. 2023b; Liang et al. 2024; Liu et al. 2023a, 2023b; van den Oord et al. 2016). However, these models often lack significant market visibility despite their technical sophistication. Recent studies have also explored how to apply these transformer models in various domains, such as games (Marrinan et al. 2024), as creative assistants (Liu et al. 2023b; Tur 2024), or for music understanding (Li et al. 2024), and description (Qingqing et al. 2022).

Eleven Labs stands out as one of the leading companies in the generative audio field, developing transformers fine-tuned for different audio synthesis applications, including speech, music, and sound effects.

2.4. Generative Audio for Video

Eleven Labs provides prototype GitHub repositories featuring their models, including a tool for generating soundscapes for video clips (Eleven Labs 2024a). The referred version corresponds to commit 78c1c9d, dated 4 November 2024. This web-based application, hosted by the user via Node.js, communicates with the Eleven Labs API to generate cohesive audio tracks for the video segments. The generated soundscapes are contextually relevant to the video frames; however, the system exhibits some latency, extracting four frames per second. Additionally, it does not support the generation of separate audio stems for multilayered soundscapes.

2.5. Ethical Discourse

An important consideration in the application of artificial neural network technologies is the ethical implications of their use (Barnett 2023). The fair use of audio samples and the responsibility of model creators to source training data ethically and legally are significant issues in the discourse on the development and application of such tools.

More ethical challenges emerge in the domain of generative sound. For instance, when AI models replicate the stylistic elements or voices of real composers, musicians, or sound designers, the boundaries of artistic authorship and identity become blurred. Similarly, the use of AI-generated audio in contexts such as media, advertising, and video games may obscure the origin and authorship of content, leading to reduced accountability, as seen in other media domains (Ray 2023).

Another concern is bias in the training data, which may result in unrepresentative outputs or the perpetuation of cultural stereotypes. Deepfake audio technology further complicates ethical considerations because it enables the synthesis of realistic yet deceptive content, potentially leading to misinformation or the erosion of trust in recorded media.

While this article focuses on the technical and creative potential of generative transformers for sound design, further research is needed to establish ethical guidelines and verification frameworks for sound-generation systems that ensure fairness, consent, and accountability throughout their lifecycle.

3. Methodology

3.1. Main Assumptions

This study employs a methodology centered on designing a pipeline to generate multilayered soundtracks based on the prompting medium. Textual or visual information is processed into layers of sound messages that are directly or indirectly related to the semantic content of the source material. These layers were then evaluated through empirical research to assess the effectiveness and coherence of the generated soundscapes.

Two sonification paradigms guided this approach:

• Direct Translation: A sound is directly mapped to a concept, where its meaning aligns literally with the source (e.g., forest sounds generated from text or images depicting a forest).
• Free Interpretation: Sounds represent a concept indirectly using illustrative elements (e.g., fatigue expressed through rhythmic breathing, yawning, or the ticking of a clock).

3.2. GPT Feedback

An essential component of this pipeline is the use of a Generative Pre-trained Transformer (GPT), particularly a large language model (LLM), to generate and refine prompts that serve as inputs for another GPT model. This iterative process uses the capabilities of one GPT to enhance the effectiveness of the prompts used by the subsequent GPT, resulting in higher-quality and more contextually accurate soundscapes.

This methodology draws on the principles of prompt engineering, emphasizing the creation of targeted inputs that optimize the performance of the generative models. The approach resembles a “collaborative network”, where multiple models cooperate in synergy to enhance results. It also shares conceptual similarities with Generative Adversarial Networks (GANs), in which networks are designed to compete, but here they collaborate to achieve refined results.

4. Pipeline Implementation

4.1. The Pipeline Concept

The pipeline, illustrated in Figure 1, consists of complementary modules designed to transform the input content into sound descriptions and, ultimately, into soundscapes. The process can begin with either textual or visual input. A source-to-sound-prompt module processes the text or visual input into sound descriptions, which are then passed to a sound-prompt-to-sound module. This module converts descriptions into actual soundscapes.

4.2. Applied Technologies

The pipeline was implemented in Python (version 3.10) and integrated external APIs, incorporating widely used consumer-grade GPT models for its core functionality.

• Eleven Labs API: A sound effects generator that enables users to create sound by providing descriptive prompts. This API uses natural language understanding to interpret user input and produce high-quality, context-specific sound effects (Eleven Labs 2024b).
• OpenAI ChatGPT: A multimodal large language model (LLM) capable of handling both text and image inputs. The implementation utilized the gpt-4o-mini (18 July 2024) version, with additional testing conducted using gpt-4o (6 August 2024). (OpenAI 2024).

4.3. The Source-to-Sound-Prompt Module

The source-to-sound-prompt module processes text or image input (provided via a URL for images) using the ChatGPT model. The model was instructed to act as a sound designer assistant and generate detailed sound prompts.

Initially, ensuring consistency and reliability in the generated outputs posed challenges due to the generative and probabilistic nature of GPT models. Unlike explicitly defined algorithms with deterministic outputs, pre-trained transformer models produce unique and context-dependent responses, limited only by token and context window constraints.

To address these challenges and achieve consistency, a JSON schema was implemented to standardize the output formats. Each sound layer is defined by the following attributes:

• Prompt: A descriptive text specifying the auditory characteristics of the sound layer.
• Category: A folder structure categorizing sound layers based on general characteristics and specific concepts (e.g., “General_Tag\Concept_Name”).
• Filename: A descriptive identifier for each sound layer formatted as “snake_case”.
• Duration: Length of the audio segment for each sound layer.
• Num_generations: Number of versions of the sound layer to be generated.
• Prompt_influence: A value that determines the extent to which the descriptive prompt influences the generated sound. This allows for fine-tuning the balance between creative and literal adherence to the prompt.

4.4. Custom Instruction

A tailored custom instruction was provided to the GPT model to optimize its role as a sound design assistant. The instruction tasks the model to generate a JSON file that represents sound layers matching the auditory characteristics of a given concept. This process involves:

• Decomposing the concept into distinct sound elements (e.g., natural sounds, ambiance, and human activity).
• Generating 5–8 sound layers, each accompanied by a detailed description.
• Populating metadata for each sound layer

4.5. Data Supplementary Materials and Code

The complete pipeline implementation, including the source code, JSON schema samples, and generated audio examples, is available on the Open Science Framework repository (Open Science Framework 2024a).

5. Preliminary Testing and Empirical Research

Initial testing and empirical research yielded preliminary conclusions and observations regarding the proposed solution.

5.1. Highlights

The proposed pipeline effectively facilitates the generation of multilayered soundscapes by combining atmospheric backgrounds with distinct point sounds. This approach has demonstrated substantial potential for creating complex auditory representations of specific locations or conceptual ideas, offering valuable tools for sound designers.

An example of the generated output is shown in Figure 2, which presents a sample generative soundscape pack featuring forest sounds visualized as waveforms with the corresponding nametags. The figure illustrates the arrangement of atmospheric and point sound layers in a 10-s long soundscape.

5.2. Limitations

While the solution shows significant promise, several limitations were identified during the testing process.

5.2.1. Overlapping Frequencies

The model often generates soundscapes with layers that span the entire frequency spectrum, thereby resulting in overlapping atmospheric layers. This overlap can cause excessive noise, particularly in the higher frequency ranges, and reduce the clarity of the combined soundscape. Ideally, each atmospheric layer should occupy a specific segment of the frequency spectrum to avoid masking effects and to maintain a balanced auditory experience.

5.2.2. Abrupt Endings Without Proper Attenuation

Some generated sound samples ended abruptly without fade-outs or attenuation, creating a jarring effect that disrupted the natural flow of the soundscape. This lack of smooth transitions reduces immersion and highlights the need for automated fade-in and fade-out mechanisms to ensure professional quality outputs.

5.2.3. Unintended Sound Artifacts

The model also exhibited issues with adequate representation of the intended sounds and generated unwanted artifacts at the end of the samples. During the initial tests, Eleven Labs demonstrated better performance when prototype instructions generated by gpt-4o-2024-08-06 were used. This suggests that more sophisticated models or refined instructions may improve the fidelity of the generated outputs. When the process was automated using the gpt-4o-mini-2024-07-18 version, certain distortions were observed in some cases. Prominent examples of these distortions include unwanted signals in the output sound files, which are further discussed.

5.2.4. Misinterpreting Prompt Focusing on Key Words

A problem was identified with the test file “footsteps_on_cobblestones_01.mp3”, which was generated using the following prompt: “The sound of soft footfalls on cobblestone paths can be heard as visitors explore the grounds. The subtle crunch of footsteps adds to the immersive experience of the castle surroundings”.

The generated sound unexpectedly featured a human voice delivered in a radio-style commentary, as though narrating a sporting event. This issue may be related to the transformer misinterpreting “footfalls” as “footballs”, leading to a representation resembling a sports-related context.

Transformers are capable of recognizing letter patterns even in misspelled words, allowing them to detect meaning despite errors. However, this tendency can have negative consequences when a correctly spelled word is misinterpreted as a similar but incorrect term, leading to misinformation, which is one of the universal challenges of using GPT models (Bubeck et al. 2023; Ray 2023).

5.2.5. Generating Random Musical Elements

The issue with the test file “horse_and_carriage_01.mp3” was generated using the following prompt: “Occasionally, the distant sound of a horse and carriage rolling along a nearby lane fades in, evoking images of the past and adding to the overall historical charm of the castle setting”.

The generated sound included unexpected musical elements, seemingly designed to evoke an emotional or illustrative effect rather than accurately representing the sound of horses and carriages. This indicates that the model, likely trained on a large dataset containing musical content, occasionally defaults to illustrative reasoning, producing music-like content intended to evoke the scene described rather than faithfully replicating the literal auditory input. Keywords such as “fades in”, “evoking images”, and “charm”, which lack literal sonification equivalents, may have been interpreted as descriptors in a musical context, guiding the model toward a more musically oriented approach.

5.3. Possible Improvements

To address these challenges, more control over the model output is essential. For instance, incorporating parameters for exclusion directives (e.g., “no” or “do not include”) could allow users to specify elements to omit, thereby improving the precision and relevance of the results. Similar features have been implemented in tools like Midjourney, where users can explicitly control unwanted elements in the generated images (Midjourney 2024).

5.4. The Necessity of Human Verification and Refinement

Human oversight is one of the main strategies to ensure the production of high-quality materials in automated systems incorporating transformer models (Edwards et al. 2020; Ouyang et al. 2022). While the pipeline successfully produces complex, multilayered soundscapes, its probabilistic nature can lead to errors or inconsistencies. Manual refinement ensures alignment with the intended concept, resolves deviations and enhances the overall quality of the final soundscape. This intervention is essential for professional and high-quality auditory production.

6. The Development of a Complementary Solution

6.1. Limitations of Generative Samples

Generatively created samples play a valuable role in soundscape creation despite their constraints, particularly in terms of sample length. The maximum length is influenced by factors such as computational complexity, temporal coherence, API-imposed constraints (e.g., 20 s for the Eleven Labs model), and query costs. These restrictions make it challenging to generate long, cohesive soundscapes in a single pass, leading to noticeable repetitions in extended audio environments.

6.2. The Granular Synthesizer as a Solution

To address these limitations, a granular synthesizer prototype was developed. This synthesizer processes short generative samples and enables looping and recombinations over time. Using a nondeterministic approach, it rearranges samples based on user-defined parameters, ensuring varied soundscapes that avoid predictable patterns and repetition.

6.3. Prototype Design and Implementation

The developed prototype, shown in Figure 3, is a granular synthesizer capable of generating dynamic and random soundscapes from a folder of attached audio samples. The full implementation is publicly available on the Open Science Framework platform (Open Science Framework 2024b) for further exploration and usage.

The system supports output through single or multiple channels, including stereo (left-right) and more complex multichannel configurations, such as 5.1 surround sound, with each channel independently populated by distinct sound grains. The synthesizer extracts grains from the stored audio files and reassembles them into nondeterministic combinations. Each grain was randomly selected and processed according to user-defined parameters to create a dynamic and immersive auditory experience.

6.3.1. User-Adjustable Parameters

The granular synthesizer features a Graphical User Interface (GUI) that allows users to control key parameters influencing the generation, combination, and presentation of sound grains:

• Grain Size defines the duration of each grain extracted from the audio files.
• Overlap determines the degree of overlap between consecutive grains.
• Fade Duration sets the fade-in and fade-out times for each grain, ensuring smooth transitions.
• Simultaneous Grains specifies the number of grains played simultaneously.
• Duration Deviation introduces variability in the duration of each grain to prevent repetitive patterns.

6.3.2. Support for Multiple Instances

The prototype synthesizer can run in multiple instances, allowing users to categorize sounds into distinct types, such as atmospheric/ambient layers and one-shot effects. This categorization enables users to fine-tune the presentation and interaction of sound types, thereby creating a tailored and immersive experience.

Suggested use of instances:

• Atmospheric/ambient layers: Parameters can be adjusted to favor longer grain sizes, increased overlap, and softer fade durations to create smooth, continuous soundscapes (Schafer’s keynotes).
• One-shot effects: Parameters can be configured for shorter grain sizes, minimal overlap, and precise fade settings, focusing on the clarity and impact of isolated sound elements (Schafer signals).

7. Conclusions

7.1. Overview

This study investigates the application of generative transformers for creating multilayered soundscapes, uncovering both their potential and limitations. The proposed pipeline successfully demonstrates the feasibility of combining Generative Pre-trained Transformers to produce rich auditory environments using textual and visual prompts for sound design. Testing revealed the system’s ability to generate diverse soundscapes, including ambient layers and distinct point sounds, providing a powerful tool for sound designers seeking innovative approaches to creating sounds.

7.2. Challenges

Unintended artifacts, issues in prompt interpretation, and other challenges may underscore the need for refined control mechanisms. These limitations highlight the importance of implementing features like exclusion directives and parameter tuning, as well as the necessity of human verification and refinement to ensure quality and accuracy. Addressing these challenges would improve the precision of the outputs and their suitability for professional applications.

7.3. Complementary Solution

The development of a granular synthesizer prototype demonstrates a viable solution for extending the usability of generative samples by enabling dynamic and non-repetitive recombination. This tool, with its adjustable parameters and multichannel capabilities, provides sound designers with greater flexibility in creating immersive and adaptable soundscapes tailored to specific contexts.

7.4. Future Research

Future work should focus on enhancing the reliability and coherence of generative outputs, further refining prompt-engineering strategies, and exploring the integration of exclusion parameters. Additionally, expanding the capabilities of granular synthesis to incorporate real-time feedback and adaptive layering could further enhance its utility in both artistic and commercial sound-design contexts. By addressing these areas, generative transformer-based soundscapes could achieve broader adoption and significantly impact audio production in the future.