StyleForge: Enhancing Text-to-Image Synthesis for Any Artistic Styles with Dual Binding

Abstract

Recent advancements in text-to-image models, such as Stable Diffusion, have showcased their ability to create visual images from natural language prompts. However, existing methods like DreamBooth struggle with capturing arbitrary art styles due to the abstract and multifaceted nature of stylistic attributes. We introduce Single-StyleForge, a novel approach for personalized text-to-image synthesis across diverse artistic styles. Using approximately 15 to 20 images of the target style, Single-StyleForge establishes a foundational binding of a unique token identifier with a broad range of attributes of the target style. Additionally, auxiliary images are incorporated for dual binding that guides the consistent representation of crucial elements such as people within the target style. Furthermore, we present Multi-StyleForge, which enhances image quality and text alignment by binding multiple tokens to partial style attributes. Experimental evaluations across six distinct artistic styles demonstrate significant improvements in image quality and perceptual fidelity, as measured by FID, KID, and CLIP scores.

1. Introduction

Recent breakthroughs in text-to-image diffusion models—including Stable Diffusion [1,2], DALL-E [3,4], and Imagen [5], SEDD [6], CoMat [7]—have enabled users to generate high-quality, diverse images simply by describing them in natural language prompts. However, these models typically generate outputs in a limited set of visual styles learned from large-scale training datasets, and adapting them to produce images in specific or user-provided artistic styles remains a significant challenge.

In creative applications such as digital illustration, personalized avatars, and visual storytelling [8,9], users often wish to generate content in a consistent, stylized aesthetic (e.g., anime, romanticism, or pixel-art), using only a handful of style reference images. However, existing diffusion models do not provide a reliable way to make this adaptation, especially for users without access to large datasets or advanced fine-tuning skills.

Beyond these creative domains, controllable text-to-image generation is increasingly relevant to smart city services, where visual communication and immersive content play a critical role [10]. Applications [11,12] such as urban simulation dashboards, digital signage for public health and safety, cultural-heritage promotion, and citizen engagement platforms require stylized yet coherent imagery tailored to local identities. Manual design of such visuals is costly and inflexible, whereas few-shot style-adaptive generation can provide scalable solutions [13] for producing culturally adaptive and context-aware media in smart city environments. Therefore, a personalization framework that reliably learns a city- or service-specific visual idiom from a small number of examples is of practical importance.

Prior methods such as DreamBooth [14], Textual Inversion [15], SOD-T2I [16], and LoRA [17] enable subject-specific fine-tuning of pre-trained text-to-image models with a small number of reference images. Despite significant progress in synthesizing images that mimic the artistic styles of renowned painters and artistic icons, such as Picasso or Impressionism, these methods are designed for object- or identity-level personalization and often fail to generalize when applied to abstract artistic styles. The concept of “artistic styles” involves complex visual elements across subjects, backgrounds, and textures, e.g., “an Asian girl on a London street in the style of Van Gogh”. Moreover, they are prone to overfitting, language drift [14], and limited compositional control, especially under low-data settings.

To address these limitations, we propose StyleForge, a user-adaptive personalization framework that transforms a general-purpose text-to-image diffusion model into a style-specialized generator using only a few reference images. StyleForge introduces a dual-binding mechanism, in which the model learns to associate the following: (i) a style token (e.g., “[V] style”) with the overall visual characteristics of the specific artistic style, which we call the target style, and (ii) an auxiliary token (e.g., “style”) with curated human-centric images from semantically related styles to guide the rendering of people within the style domain. This auxiliary-guided token binding helps preserve style consistency across both people and backgrounds, while preventing overfitting to the limited reference set. Furthermore, we extend our framework to Multi-StyleForge, which separates different style components (e.g., person and background) into multiple tokens (e.g., “[V] style” and “[W] style”), enabling more fine-grained control and compositional alignment between prompt and image.

StyleForge can be applied to any pre-trained diffusion model (e.g., Stable Diffusion v1.5) without modifying the architecture, making it suitable for real-world scenarios where users want to adapt generation models to their desired style quickly and with minimal effort. We evaluate StyleForge across six domains and show that it significantly outperforms existing personalization baselines in both quantitative metrics (FID, KID, CLIP scores) and qualitative fidelity. To summarize, our contributions are as follows:

• Dual-binding personalization framework: we introduce StyleForge, a novel approach that binds a target style to both a unique style token and an auxiliary token, enabling controllable and robust style adaptation from limited data.
• Auxiliary-guided human rendering: we leverage carefully curated auxiliary images containing human-centric style elements to guide the rendering of people, enhancing generalization and mitigating language drift.
• Multi-token decomposition for fine-grained control: we extend our framework, called Single-StyleForge, to Multi-StyleForge, which disentangles person and background characteristics into separate tokens to improve the compositional alignment between the prompt and image.
• Plug-and play adaptation for any artistic style: StyleForge requires only 15–20 reference images, imposes no architectural changes, and can be easily integrated with existing text-to-image diffusion models for user-friendly personalization.
• Extensive evaluation: We conduct experiments across six distinct art styles and demonstrate superior style fidelity, text–image alignment, and robustness compared to state-of-the-art methods. To summarize, our main result is illustrated in Figure 1.

2. Related Work

2.1. Text-to-Image Synthesis

Text-to-image generation has rapidly evolved with the development of powerful generative models that translate natural language prompts into high-resolution images by learning joint distributions over text and image modalities. Imagen [5] uses pixel-space diffusion with a pyramid structure, while Stable Diffusion [1] performs diffusion in latent space for improved efficiency. DALL-E [3,4] adopts transformer-based autoregressive and two-stage priors for text-conditioned image generation. Other notable models include Muse [18], which employs a masked generative transformer, and Parti [19], which combines ViT-VQGAN tokenization with autoregressive decoding. ConsiStory [20] enhances consistency across image batches using shared attention and feature correspondence mechanisms.

2.2. Style Transfer

Neural style transfer modifies the visual appearance of a content image by applying the stylistic characteristics of a reference image while preserving the content. Early methods leveraged CNN features—such as VGG-based correlations [21]—or GANs like StyleGAN [22], SDP-GAN [23], and PGGAN [24] to generate high-resolution stylized outputs. Domain-specific variants, such as anime-oriented architectures [25], further tailored these methods. Later approaches enhanced content–style disentanglement and control through mechanisms like AdaIn [26], AdAttN [27], and RAST [28], which introduced attention and multi-loss objectives. StyleCLIP [29] enabled intuitive language-based style control. Recent methods based on diffusion models [30,31,32,33,34] offer improved flexibility and fidelity. StyleDiffusion [35] and PatchMatch [36] apply statistical and patch-based transformations, while inversion-based methods [37] and DreamStyler [38] leverage text prompts and language-image models (e.g., BLIP-2 [39]) for text-to-style binding during generation.

2.3. Personalizing and Controlling Diffusion Models

Recent efforts in text-to-image synthesis have focused on personalizing diffusion models [16,40,41] to reflect user-specific content or style. Textual Inversion [15] and NeTI [42] learn new text token embeddings that capture the semantics of a subject or style from limited examples. DreamBooth [14] fine-tunes pre-trained models on a few subject images using a prior preservation loss to maintain semantic consistency, while StyleBoost [43] reinforces style representation by anchoring to a meta-class image. DreamArtist [44] and SpecialistDiffusion [45] explore embedding learning and data augmentation to enhance stylistic fidelity. Beyond diffusion-based models, StyleDrop [46] adapts Muse [18], a generative vision transformer, to render diverse styles via supervised token tuning.

Encoder-based approaches, such as HyperDreamBooth [47], E4T [48], and domain-agnostic tuning encoders [49], inject external priors into diffusion models from a single reference image for rapid personalization. However, these methods struggle with disentangling style from content, often reproducing subjects too closely tied to the reference image, i.e., overfitting. Several works aim to extent personalization to multiple subjects or compositional layouts. CustomDiffusion [50], SVDiff [51], and HyperDreamBooth [47] offer parameter-efficient multi-subject adaptation. SubjectDiffusion [52] and Perfusion [53] control attention via attention maps and rank-one updates, while Break-A-Scene [54] and ControlNet [55] decompose scenes into multiple concepts and inject layout guidance through segmentation-aware control mechanisms. Finally, DAAM [56] and I2AM [57] contribute to understanding and controlling the operation of diffusion models by interpreting their generation processes, with further comparisons available in the Supplementary Materials.

2.4. Toward Stylized Personalization

A few recent works attempt to bridge the gap between style and subject adaptation. StyleDrop [46] uses vision transformers and learns a style-specific token to transfer appearance, but it is limited to known styles and requires architecture-specific modifications. SpecialistDiffusion [45] and DreamArtist [44] improve prompt tuning and sample efficiency but lack generalization to unseen artistic domains. StyleBoost [43] attempts to personalize DreamBooth to stylized outputs by reinforcing latent similarity, but does not disentangle background/person contributions or support compositional control. In contrast, StyleForge introduces a dual-token binding mechanism that explicitly separates the person-oriented and background-oriented components of a target style.

3. Preliminaries

3.1. Diffusion Models

Diffusion models [58,59,60] are probabilistic generative models that learn data distributions by reversing a noise addition process. Starting from random noise, they progressively denoise to generate data samples by learning a series of denoising autoencoders ${\left\{{\mathit{ϵ}}_{\mathit{\theta }}({\mathbf{x}}_t,t)\right\}}_{t=1\dots T}$ that predict cleaner versions of noisy inputs ${\mathbf{x}}_t$ at each timestep t. The training objective minimizes

$$\begin{array}{c}\hfill {\mathcal{L}}_{\mathrm{DM}}={\mathbb{E}}_{\mathbf{x},\mathit{ϵ},t}\left[\parallel \mathit{ϵ}-{\mathit{ϵ}}_{\mathit{\theta }}({\mathbf{x}}_t,t){\parallel }_2^2\right],\end{array}$$

(1)

where $\mathit{ϵ}\sim \mathcal{N}(0,1)$ is Gaussian noise added to the data.

Latent Diffusion Models (LDMs) extend this by operating in a compressed latent space. An encoder $\mathcal{E}$ maps images $\mathbf{x}$ to latent representations $\mathbf{z}=\mathcal{E}\left(\mathbf{x}\right)$, where denoising occurs on ${\mathbf{z}}_t={\alpha }_t\mathbf{z}+{\sigma }_t\mathit{ϵ}$ with time-dependent scaling factors ${\alpha }_t,{\sigma }_t$. For text-conditional generation, LDMs minimize

$$\begin{array}{c}\hfill {\mathcal{L}}_{\mathrm{LDM}}={\mathbb{E}}_{\mathbf{z},\mathbf{c},\mathit{ϵ},t}\left[\parallel \mathit{ϵ}-{\mathit{ϵ}}_{\mathit{\theta }}({\mathbf{z}}_t,t,\mathbf{c}){\parallel }_2^2\right],\end{array}$$

(2)

where $\mathbf{c}={\Gamma }_{\mathit{\varphi }}\left(\mathbf{p}\right)$ encodes text prompt $\mathbf{p}$ via a text encoder ${\Gamma }_{\mathit{\varphi }}$.

3.2. DreamBooth

DreamBooth [14] personalizes text-to-image diffusion models using 4–6 images of a specific subject. It binds a unique identifier token “[V]” to the subject (e.g., “a [V] dog” for a specific dog) while preserving the meta-class semantics (“dog”). The key innovation is the prior preservation loss that prevents overfitting. During training, meta-class images ${\mathbf{x}}^{\mathrm{pr}}$ are generated from the frozen model using the prompt “dog”; then, both the instance and prior images are used to fine-tune the model:

$$\begin{array}{c}\hfill {\mathcal{L}}_{\mathrm{DB}}={\mathbb{E}}_{\mathbf{z},\mathbf{c},\mathit{ϵ},{\mathit{ϵ}}^{\prime },t}\left[\parallel \mathit{ϵ}-{\mathit{ϵ}}_{\mathit{\theta }}({\mathbf{z}}_t,t,\mathbf{c}){\parallel }_2^2+\lambda {\parallel {\mathit{ϵ}}^{\prime }-{\mathit{ϵ}}_{\mathit{\theta }}({\mathbf{z}}_t^{\mathrm{pr}},t,{\mathbf{c}}^{\mathrm{pr}})\parallel }_2^2\right],\end{array}$$

(3)

where $\lambda$ balances instance-specific learning with meta-class preservation, and ${\mathbf{z}}_t$ and ${\mathbf{z}}_t^{\mathrm{pr}}$ are noisy latents of instance and prior images, respectively.

4. Method: StyleForge

We tackle the challenge of generating high-quality images in various artistic styles or simply styles using text prompts as guidance. Unlike DreamBooth [14], which relies on synthetic priors with irrelevant semantics, we use curated auxiliary images. Unlike StyleBoost [43], which reinforces latent similarity without disentangling the style and content, we perform explicit semantic decomposition. Unlike SpecialistDiffusion [45] and DreamArtist [44], which rely on augmentation or prompt-only tuning, we separate style components via multi-token binding. Parameter-efficient methods such as LoRA [17], Textual Inversion [15], and Custom Diffusion [50] optimize limited parameters and often fail on out-of-distribution styles. These limitations are amplified for abstract styles different to renowned painters.

4.1. Single-StyleForge: Overall Architecture

To address the above issues, we first propose Single-StyleForge, which reliably generates style variations guided by prompts through comprehensive fine-tuning and auxiliary binding strategies. The architecture of Single-StyleForge is based on the framework of DreamBooth [14], focusing on synthesizing images of a specific style, which we call a target style, rather than a particular object. As illustrated in Figure 2, Single-StyleForge fine-tunes pre-trained text-to-image diffusion models ${\widehat{\mathbf{x}}}_{\theta }$ using a few reference images $\mathbf{x}$ of the target style, called StyleRef images, and a set of auxiliary images ${\mathbf{x}}^{\mathrm{aux}}$, called Aux images. The loss function for Single-StyleForge is defined as follows:

$$\begin{array}{c}\hfill {\mathcal{L}}_{\mathrm{SSF}}={\mathbb{E}}_{\mathbf{z},\mathbf{c},\mathit{ϵ},{\mathit{ϵ}}^{\prime },t}\left[\parallel \mathit{ϵ}-{\mathit{ϵ}}_{\mathit{\theta }}({\mathbf{z}}_t,t,\mathbf{c}){\parallel }_2^2+\lambda {\parallel {\mathit{ϵ}}^{\prime }-{\mathit{ϵ}}_{\mathit{\theta }}({\mathbf{z}}_t^{\mathrm{aux}},t,{\mathbf{c}}^{\mathrm{aux}})\parallel }_2^2\right],\end{array}$$

(4)

where ${\mathbf{z}}_t^{\mathrm{aux}}:={\alpha }_t\mathcal{E}\left({\mathbf{x}}^{\mathrm{aux}}\right)+{\sigma }_t{\mathit{ϵ}}^{\prime }$ and ${\mathbf{c}}^{\mathrm{aux}}$ are used instead of meta-class prior components in (3). In particular, the second term in (4) acts as an auxiliary term that guides the model with information similar to the human perception of the target style. Finally, $\lambda$ controls the strength of the second term.

For given target style, we use 15–20 StyleRef images $\mathbf{x}$ that showcase the key characteristics of the style. These images include a mix of landscapes, objects, and people to provide a comprehensive representation of the target style. The corresponding StyleRef prompts $\mathbf{p}$ are crafted to encapsulate the essence of the style using a unique token identifier (e.g., “[V] style”). Unlike the meta-class prior images ${\mathbf{x}}^{\mathrm{pr}}$ in DreamBooth, the auxiliary images ${\mathbf{x}}^{\mathrm{aux}}$ are collected to supplement the StyleRef images. These images are chosen to enhance the model’s understanding of additional context in the target style, capturing generic features for challenging elements like human faces and poses. The auxiliary prompts ${\mathbf{p}}^{\mathrm{aux}}$ use a general token (e.g., “style”) to ensure they provide broad guidance without introducing bias. Finally, the dataset $\mathcal{D}$ for training comprises pairs of StyleRef images with their prompts $(\mathbf{x},\mathbf{p})$ and Aux images with their prompts $({\mathbf{x}}^{\mathrm{aux}},{\mathbf{p}}^{\mathrm{aux}})$.

The training process of Single-StyleForge involves the following steps. First, load the pre-trained weights for the encoder $\mathcal{E}$, text encoder ${\Gamma }_{\mathit{\varphi }}$, and U-Net ${\mathit{ϵ}}_{\mathit{\theta }}$ (line 1), and sample a pair of StyleRef and Aux images and prompts from the dataset $\mathcal{D}$ (line 3); then, encode the StyleRef and Aux prompts $\mathbf{p},{\mathbf{p}}^{\mathrm{aux}}$ into latent codes $\mathbf{c},{\mathbf{c}}^{\mathrm{aux}}$ using the text encoder (line 4). Next, the forward diffusion process is performed to generate noisy latent codes ${\mathbf{z}}_t,{\mathbf{z}}^{\mathrm{aux}}$ from sampled noises $\mathit{ϵ},{\mathit{ϵ}}^{\prime }$ (lines 5–7); finally, optimize the model by minimizing the loss (4), taking a gradient descent step to update the model parameters $\mathbf{\omega }$. The detailed training process is outlined in Algorithm 1.

Algorithm 1 Single-StyleForge.

Require: dataset $\mathcal{D}=\{(\mathbf{x},\mathbf{p}),({\mathbf{x}}^{\mathrm{aux}},{\mathbf{p}}^{\mathrm{aux}})\}$, encoder $\mathcal{E}$, text encoder ${\Gamma }_{\mathit{\varphi }}$, U-Net ${\mathit{ϵ}}_{\mathit{\theta }}$, hyper-parameters ${\{{\sigma }_t,{\alpha }_t\}}_{t=1,\dots ,T}$ and control parameter $\lambda$ Ensure: trained model ${\Gamma }_{\mathit{\varphi }},{\mathit{ϵ}}_{\mathit{\theta }}$ with learnable weights $\mathit{\omega }=\{\mathit{\theta },\mathit{\varphi }\}$ 1: Initialize: load pre-trained weights for $\mathcal{E},{\Gamma }_{\mathit{\varphi }},{\mathit{ϵ}}_{\mathit{\theta }}$ 2: repeat 3:     sample a data $(\mathbf{x},\mathbf{p}),({\mathbf{x}}^{\mathrm{aux}},{\mathbf{p}}^{\mathrm{aux}})\sim \mathcal{D}$ 4:     obtain conditioning vectors $\mathbf{c}={\Gamma }_{\mathit{\varphi }}\left(\mathbf{p}\right)$,  ${\mathbf{c}}^{\mathrm{aux}}={\Gamma }_{\mathit{\varphi }}\left({\mathbf{p}}^{\mathrm{aux}}\right)$ 5:     sample time $t\sim \mathrm{Uniform}(1,\dots ,T)$ 6:     sample noise vectors $\mathit{ϵ},{\mathit{ϵ}}^{\prime }\sim \mathcal{N}(0,I)$ 7:     ${\mathbf{z}}_t:={\alpha }_t\mathcal{E}\left(\mathbf{x}\right)+{\sigma }_t\mathit{ϵ}$,  ${\mathbf{z}}_t^{\mathrm{aux}}:={\alpha }_t\mathcal{E}\left({\mathbf{x}}^{\mathrm{aux}}\right)+{\sigma }_t{\mathit{ϵ}}^{\prime }$    ▹forward diffusion processes 8:     $\mathit{\omega }\leftarrow \mathit{\omega }-\mathrm{Optimizer}\left(\parallel \mathit{ϵ}-{\mathit{ϵ}}_{\theta }({\mathbf{z}}_t,t,\mathbf{c}){\parallel }_2^2+\lambda {\parallel {\mathit{ϵ}}^{\prime }-{\mathit{ϵ}}_{\theta }({\mathbf{z}}_t^{\mathrm{aux}},t,{\mathbf{c}}^{\mathrm{aux}})\parallel }_2^2\right)$    ▹gradient descent step 9: until converged

4.2. Rationale Behind Auxiliary Images

The use of auxiliary image ${\mathbf{x}}^{\mathrm{aux}}$ is crucial in the training process of Single-StyleForge, and its two main roles are discussed here.

4.2.1. Aiding in the Binding of the Target Style

While binding a unique token to an object is relatively straightforward, as in DreamBooth [14], capturing the diverse features of a target style and combining them with the identifier pose a challenge. Due to significant variations in artistic styles, learning features such as vibrant colors, exaggerated facial expressions, and dynamic movements in styles like `anime’ becomes difficult with only a few StyleRef images $\mathbf{x}$. Moreover, we observe that a pre-trained text-to-image model (e.g., Stable Diffusion v1.5) embeds a wide range of characteristics associated with the word “style”, mostly including fashion styles, fabric patterns, and often art styles, as shown in Figure A1. Instead of retaining unnecessary meanings of the word “style”, we propose utilizing Aux images ${\mathbf{x}}^{\mathrm{aux}}$ to allow the token “style” to encapsulate some concepts essential for expressing artwork features. As a result, while the StyleRef images $\mathbf{x}$ and prompt $\mathbf{p}$ (e.g., “[V] style”) capture overall information about the target style, the Aux images ${\mathbf{x}}^{\mathrm{aux}}$ and prompt (e.g., “style”) provide more detailed information about specific aspects, such as how to represent a person in that style. This adjustment redirects the embedding of the word “style” from unrelated meanings, such as fashion styles, to general artwork styles, alleviating overfitting and thereby enhancing overall learning performance.

4.2.2. Improving Text-to-Image Performance

In [14], it is demonstrated that using a set of meta-class prior images generated by pre-trained diffusion models, a meta-class prompt can enhance personalization capability. However, in our case, we found that using a set of images generated by pre-trained diffusion models using a prompt “style” resulted in unnecessary context such as fashion styles, making it difficult to incorporate essential information during the training process. Additionally, we noticed that providing detailed descriptions of a person (e.g., hands, legs, facial features, and full-body poses) remains crucial in qualitative evaluations, while drawing landscapes or animals has less impact. Therefore, in Single-StyleForge, Aux images ${\mathbf{x}}^{\mathrm{aux}}$ primarily consist of portraits and/or people collected from high-resolution images on the Internet, with the aim of enhancing the overall text-to-image synthesis performance, particularly for generating high-quality images related to people.

4.2.3. Mitigating Language Drift

As pointed out in [14], personalization of text-to-image models commonly leads to several issues: (i) overfitting to a small set of input images (i.e., StyleRef images), resulting in images that are specific to a particular context and subject appearance, such as images with the same background, as well as a lack of alignment between text and image, and (ii) language drift, which causes the model to associate the prompt with a limited set of input images and lose diverse meanings of the meta-class name. However, when personalizing models to fit a style rather than a specific subject, it is observed that language drift becomes less of a concern, as “style” is an abstract concept that does not require strict adherence to the meaning and diversity of the word. We expect that if the desired style concept is encoded in the “style” token, then language drift is not a significant issue.

4.3. Multi-StyleForge

While Single-StyleForge focuses on learning a comprehensive representation of a target style, Multi-StyleForge enhances this capability by separating the stylistic attributes into multiple specific components. This approach aims to improve the alignment between the text prompts and the generated images, particularly for styles that involve complex compositions of backgrounds and persons.

Multi-StyleForge builds on the foundation of Single-StyleForge by dividing the components of the target style and mapping each to a unique identifier for training by adopting the method in [50]. Since Single-StyleForge maps StyleRef images to a single prompt (e.g., “[V] style”), thus, during the inference, using a prompt without person-related descriptions, it often produces an image that includes a person. To address this issue, Multi-StyleForge uses two StyleRef prompts (e.g., “[V] style, [W] style”), one for persons and another for backgrounds, to train the model more effectively. By separating these elements explicitly, we address the ambiguity that can arise when a single prompt is used to capture both. We also note that it can be extended to separate into more than two components.

4.3.1. Multi-StyleRef Prompts Configuration

The StyleRef images consist of two parts: elements of people and backgrounds in the target style, following the structure from Single-StyleForge. Each component is then associated with its specific prompt (e.g., “[V] style” for persons and “[W] style” for backgrounds). The Aux images and prompt ${\mathbf{x}}^{\mathrm{aux}},{\mathbf{p}}^{\mathrm{aux}}$ are kept unified as “style” to ensure they provide general guidance applicable to both components. As a result, Multi-StyleForge trains the model to differentiate stylistic features (people and backgrounds) and obtain separate embeddings.

4.3.2. Training of Multi-StyleForge

Our Multi-StyleForge is built on a custom-diffusion [50] framework that personalizes multiple subjects, but the difference is learned by full-tuning. The training process of Multi-StyleForge involves simultaneous or parallel learning of the two StyleRef prompts. When personalizing each StyleRef prompt sequentially, there is a risk of losing information about previously learned StyleRef prompts in the subsequent process. Therefore, Multi-StyleForge takes the simultaneous learning of multiple StyleRef prompts. The Multi-StyleForge operation typically involves two text–image pairs ${\mathcal{D}}_1$ and ${\mathcal{D}}_2$, as illustrated in Algorithm 2. In particular, each component is selected with a probability proportional to the number of data samples (line 3); then, the model is trained using the selected StyleRef data, similar to the Single-StyleForge, to capture the associated stylistic components (lines 4–8).

Algorithm 2 Multi-StyleForge.

Require: Data ${\mathcal{D}}_1=\{(\mathbf{x},\mathbf{p}),({\mathbf{x}}^{\mathrm{aux}},{\mathbf{p}}^{\mathrm{aux}})\}$, ${\mathcal{D}}_2=\{(\mathbf{x},\mathbf{p}),({\mathbf{x}}^{\mathrm{aux}},{\mathbf{p}}^{\mathrm{aux}})\}$, encoder $\mathcal{E}$, text encoder ${\Gamma }_{\mathit{\varphi }}$, U-Net ${\mathit{ϵ}}_{\mathit{\theta }}$, hyper-parameters ${\{{\sigma }_t,{\alpha }_t\}}_{t=1,\dots ,T},\lambda$ Ensure: Trained models ${\Gamma }_{\mathit{\varphi }},{\mathit{ϵ}}_{\mathit{\theta }}$ with learnable weights $\mathit{\omega }=\{\mathit{\theta },\mathit{\varphi }\}$ 1: Initialize: $q={\displaystyle \frac{|{\mathcal{D}}_1|}{|{\mathcal{D}}_1|+|{\mathcal{D}}_2|}}$, load pre-trained weights for $\mathcal{E},{\Gamma }_{\mathit{\varphi }},{\mathit{ϵ}}_{\mathit{\theta }}$ 2: repeat 3:     select a dataset $\mathcal{D}={\mathcal{D}}_1$ if $Q\sim \mathrm{Uniform}\left(\right[0,1\left]\right)<q$ else ${\mathcal{D}}_2$ 4:     sample a data $(\mathbf{x},\mathbf{p}),({\mathbf{x}}^{\mathrm{aux}},{\mathbf{p}}^{\mathrm{aux}})\sim \mathcal{D}$ 5:     $\mathbf{c}={\Gamma }_{\mathit{\varphi }}\left(\mathbf{p}\right)$, ${\mathbf{c}}^{\mathrm{aux}}={\Gamma }_{\mathit{\varphi }}\left({\mathbf{p}}^{\mathrm{aux}}\right)$, sample time $t\sim \mathrm{Uniform}(1,\dots ,T)$ 6:     sample a noise $\mathit{ϵ},{\mathit{ϵ}}^{\prime }\sim \mathcal{N}(0,I)$ 7:     ${\mathbf{z}}_t:={\alpha }_t\mathcal{E}\left(\mathbf{x}\right)+{\sigma }_t\mathit{ϵ}$, ${\mathbf{z}}_t^{\mathrm{aux}}:={\alpha }_t\mathcal{E}\left({\mathbf{x}}^{\mathrm{aux}}\right)+{\sigma }_t{\mathit{ϵ}}^{\prime }$    ▹forward diffusion processes 8:     $\mathit{\omega }\leftarrow \mathit{\omega }-\mathrm{Optimizer}\left(\parallel \mathit{ϵ}-{\mathit{ϵ}}_{\theta }({\mathbf{z}}_t,t,\mathbf{c}){\parallel }_2^2+\lambda {\parallel {\mathit{ϵ}}^{\prime }-{\mathit{ϵ}}_{\theta }({\mathbf{z}}_t^{\mathrm{aux}},t,{\mathbf{c}}^{\mathrm{aux}})\parallel }_2^2\right)$ 9: until converged

Multi-StyleForge improves the alignment between text and images by reducing ambiguity through the use of multiple specific tokens. During inference, these tokens guide the generation process to ensure that images align more accurately with the corresponding components and prompts. For instance, using tokens “[V]” for persons and “[W]” for backgrounds helps the model generate images with clear distinctions between these two elements. Experiments conducted with Multi-StyleForge show high fidelity performance and better text-image alignment, which will be discussed in Section 5.

5. Experimental Results

In this section, we assess the performance of Single/Multi-StyleForge in personalizing text-to-image generation across different artistic styles. Through experiments, we investigate how well our methods generate high-quality images that faithfully reflect the target styles.

5.1. Experimental Setup

We conducted experiments on six common artistic styles: realism, midjourney, anime, romanticism, cubism, and pixel art. The characteristics of each style are summarized as follows:

• Realism focuses on an accurate and detailed representation of subjects.
• Midjourney is characterized by detailed rendering and dramatic imaginative expressions, reflecting the distinctive style of the MidJourney model [61].
• Anime refers to a Japanese animation style characterized by vibrant colors, exaggerated facial expressions, and dynamic movement.
• Romanticism prioritizes emotional expression, imagination, and the sublime, often portraying fantastical and emotional subjects with a focus on rich dark tones and extensive canvases.
• Cubism emphasizes representing visual experiences by depicting objects from multiple angles simultaneously, often in polygonal or fragmented forms.
• Pixel art involves creating images by breaking them down into small square pixels, adjusting their size and arrangement to form the overall image.

Example images for each style are provided in the top row of Figure 1, demonstrating various visual attributes of the target styles. The StyleRef images for training and evaluation are collected by using pre-trained diffusion models or from the Web. For the target styles of realism, midjourney and anime, we used pre-trained diffusion models from Hugging Face [62] to generate 18,764 images. For the target styles of romanticism and cubism, we collected 3600 images from WikiArt [63], and we obtained 1000 images from Kaggle [64] for the target style of pixel art. In addition, to enhance the ability to generate images of people, we carefully gathered auxiliary images from different auxiliary styles found on the Web. The details of the Aux images are discussed in Section 5.4.

We assessed the quality of the generated images using various metrics, including FID [65], KID [66], and CLIP [67] scores. FID and KID measure the similarity between real and generated images, where lower scores indicate better image quality. In contrast, the CLIP score evaluates the correspondence between images and text, with higher scores reflecting better alignment. To generate diverse images from the trained model, we used 1562 prompts from the Parti Prompts dataset [19], covering 12 categories such as people, animals, and artwork. Examples of these prompts include “the Eiffel Tower”, “a cat drinking a pint of beer”, and “a scientist”. For each prompt, we produced 12 images, yielding a total of 18,744 images for evaluation. Additionally, StyleForge, trained on a target style, generated 6 images per prompt, resulting in a total of 9372 images per style.

5.2. Implementation Details

5.2.1. Ours

As a base diffusion model, we used Stable Diffusion (SD v1.5) that had been pre-trained with realistic images. We employed the Adam optimizer with a learning rate of 1 $\times {10}^{-6}$, setting inference steps at 30, with $\lambda$ value of 1 for the experiments. Fine-tuning the pre-trained text-to-image model for six target styles involved minimizing the loss (4) using 20 StyleRef images and 20 Aux images. All subsequent experiments adhered to training iterations that achieved the best FID/KID scores, see Figure A2 in Appendix A. In Single-StyleForge, we set the StyleRef prompt $\mathbf{p}$ as “a photo of [V] style” and the Aux prompt ${\mathbf{p}}^{\mathrm{aux}}$ as “a photo of style”. In Multi-StyleForge, StyleRef is divided into two components (i.e., person and background), each paired with “a photo of [V] style” and “a photo of [W] style”, respectively.

5.2.2. Baseline Models

For the baseline models, Textual Inversion [15], LoRA [17], DreamBooth [14], CustomDiffusion [50] were trained to achieve the best FID/KID scores using the same text–image pairs for fair comparison. At this time, all CFG scales were set to $7.5$. Some baseline methods, including Textual Inversion [15] and LoRA [17], do not utilize auxiliary prompts; thus, auxiliary images were not used. In the case of DreamBooth [14], auxiliary images were generated by a pre-trained diffusion model using the auxiliary prompt. CustomDiffusion [50] was used as a baeline of Multi-StyleForge, by using the same text–image pairs, with auxiliary images being generated as in DreamBooth. See

Table 1. Details of baseline methods in terms of the fine-tuning method and the use of StyleRef and Aux images. Full and partial tuning indicates a tuning of the entire and a subset of the pre-trained model, respectively.

	DreamBooth	Textual Inversion	LoRA	Custom Diffusion	Single-StyleForge	Multi-StyleForge
Tuning method	Full	Partial	Partial	Partial	Full	Full
StyleRef image	✓	✓	✓	✓	✓	✓
Aux image	✓	✗	✗	✓	✓	✓

for the summary.

5.3. Analysis of StyleRef Images

We assessed the impact of the StyleRef images on the personalization performance. While encapsulating the target style with numerous StyleRef images may ease customization, it could reduce accessibility; however, using only 3–5 images, it is challenging to capture the target style. It is important to maintain diversity while personalizing with a limited image set to effectively portray the desired style. Our empirical findings suggest that around 20 StyleRef images are effective for style personalization. To investigate the composition of StyleRef images, we fine-tuned the base diffusion model using 20 StyleRef images with different combinations, without the use of Aux images. In addition, we provide the dataset configurations for both StyleRef and Aux images, which are available online (https://drive.google.com/drive/folders/1MZDv_NyBJm0x6RLWd2ILn0MybiU-D92S, accessed on 28 September 2025).

• Only backgrounds: 20 landscape images in the target style.
• Only persons: 20 portraits and/or people images in the target style.
• Mixed backgrounds and persons: a mix of 10 landscape and 10 people images.

As depicted in Figure 3, using only landscape images for StyleRef made the model misunderstand the target style due to biased information. Similarly, using only people images resulted in overfitting, as the appearance of a person became a crucial element, as shown in Figure 3b. On the other hand, StyleRef images consisting of a mix of landscapes and people effectively captured the general features of the target style, synthesizing well-aligned images with the prompts, see Figure 3c. Finally,

Table 2. FID and KID ($\times {10}^3$) scores with different compositions of StyleRef images $\mathbf{x}$. Bold indicates the best results.

StyleRef Images	FID Score (↓)			KID Score (↓)
	Realism	Midjourney	Anime	Romanticism	Cubism	Pixel Art
only backgrounds	$22.804$	$24.598$	$34.629$	–	–	–
only persons	$21.708$	$18.812$	$47.588$	–	–	–
mix of backgrounds + persons	$\mathbf{15.196}$	$\mathbf{15.449}$	$\mathbf{22.227}$	$\mathbf{2.022}$	$\mathbf{2.257}$	$\mathbf{0.714}$

shows a performance improvement ranging from $50.07\%$ to $59.22\%$ in quantitative evaluation when using the mixed StyleRef composition.

5.4. Analysis of the Aux Images

5.4.1. Configuration of Aux Images

As discussed in Section 4.2, it is important to properly configure the Aux images properly to improve the performance of Single-StyleForge. When selecting the auxiliary style, we aim to choose a style that is not only similar to the target style but also provides general attributes for creating better images of people. With this purpose in mind, we have observed that styles such as realism, midjourney, anime, and romanticism mainly depict human attributes realistically. In contrast, cubism represents people using polygons and impressionistic shapes, and pixel art portrays realistic human figures using pixels. We have tailored the auxiliary style and images to boost the personalization for each target style. For realism, midjourney, anime, and romanticism target styles, we collected digital painting images that cover the range from realism to abstraction, while impressionism images were selected for the cubism style, and realism images were chosen for the pixel art style.

5.4.2. Auxiliary Binding

We designed the Aux images to play a supporting role in the personalization of the target style. Using the Aux images, we transferred the “style” token from the original fashion style to the artistic style area, facilitating the training process. Our main objective is for the “[V]” token to thoroughly learn the target style and for the “style” token to represent people in the auxiliary style as a booster. In Figure 4, we visualize the attention maps of the tokens “[V]” and “style” to evaluate the auxiliary binding. The attention of “[V]” is evenly distributed across the images, as the StyleRef images contain both the person and the background, whereas the “style” token focuses specifically on the person, as the Aux images only contain the person. The images generated by each token are displayed in the top and bottom rows of Figure 5. It is observed that “style” token effectively captures the attributes of the person itself, while “[V] style” comprehensively produces images of the target style. Finally, the ablation results of using the Aux images are provided in Figure 6, showing that adding the Aux images enhances the FID/KID scores across all target styles, indicating a reduction in overfitting, and there is a slight increase in the CLIP score.

5.4.3. Comparison with DreamBooth

Table 3. Comparison of FID and KID ($\times {10}^3$) with different compositions of Aux images ${\mathbf{x}}^{\mathrm{aux}}$, where we chose the StyleRef composition of backgrounds + persons. Bold indicates the best results.

Method		FID Score (↓)			KID Score (↓)
		Realism	Midjourney	Anime	Roman	Cubism	Pixel-Art
Aux images	Style token [14]	$14.297$	$14.293$	$31.518$	$1.999$	$3.646$	$0.843$
	Illustration style token [14]	$14.093$	$14.466$	$28.570$	–	–	–
	Human-drawn art	$14.263$	$16.366$	$22.836$	–	–	–
	Target style	$15.855$	$13.990$	$29.450$	–	–	–
	Single-StyleForge (ours)	$\mathbf{13.008}$	$\mathbf{12.222}$	$\mathbf{20.718}$	$\mathbf{1.602}$	$\mathbf{1.349}$	$\mathbf{0.704}$

presents the numerical results based on the composition of the Aux images, with examples provided in Appendix A (Figure A1). Encoding useful information into Aux images enhances performance compared to generating unrefined Aux images through a pre-trained diffusion model proposed by DreamBooth [14]. In particular, the performance increases over the Style token were $9.02\%$, $14.49\%$, $34.27\%$, $19.86\%$, $63.00\%$, and $16.49\%$. However, composing Aux images with the same style as the target or including dissimilar information (e.g., human-drawn art) can hinder the model’s generalization abilities and lead to overfitting. In summary, while Aux images are not directly linked to the target style, they should complement the style learning process by providing a more comprehensive understanding of visual features and serving as auxiliary bindings for the target style.

5.5. Multi-StyleForge: Improved Text–Image Alignment Method

Multi-StyleForge separates styles using specific multiple prompts to better distinguish between images generated from different text conditions. This approach clarifies the distinctions between people and backgrounds, thereby improving the text alignment. During inference, the prompt “[V]” is used if the text involves a person, “[W]” for the background, and “[V], [W]” if both are relevant. Figure 5 evaluates the effectiveness of each prompt component in separating people and backgrounds. Specifically, the prompt “[V(person) style]” is observed to generate images of a person in the target style, while the prompt “[W(background) style]” creates images of various backgrounds in the target style. Figure 1, Figure 7, Figure A4 and Figure A5 compare the image outputs using Multi-StyleForge and Single-StyleForge with various prompts, highlighting their effects on text alignment. By examining the texture of six target styles, detailed background representation, and the shapes of people and details of their faces in the generated images in these figures, it becomes evident that Multi-StyleForge excels at rendering these elements based on text descriptions. For instance, when evaluating the presence of “sunglasses” in target style romanticism in the top row of Figure 1, such details are observed only in the images generated by Multi-StyleForge. As shown in the

Table 4. Quantitative comparisons with FID, KID ($\times {10}^3$), and CLIP scores. The table presents FID scores for realism, midjourney, and anime styles, along with KID scores for romanticism, cubism, and pixel art styles, and CLIP scores for all styles. The best and second-best results are indicated in bold and underline, respectively.

Method	FID Score (↓)			KID Score (↓)			CLIP Score (↑)
	Realism	Midjourney	Anime	Roman	Cubism	Pixel-Art	Realism	Midjourney	Anime	Roman	Cubism	Pixel-Art
DreamBooth [14]	$14.093$	$14.293$	$28.570$	$1.999$	$3.646$	$\underline{0.843}$	$28.226$	$29.020$	$28.551$	$27.420$	$27.818$	$26.175$
Textual Inversion [15]	$17.048$	$22.797$	$41.654$	$6.113$	$4.783$	$2.330$	$28.227$	$27.063$	$26.284$	$25.482$	$22.984$	$26.497$
LoRA [17]	$\underline{13.218}$	$16.247$	$24.560$	$8.664$	$13.183$	$2.641$	$\underline{28.926}$	$\underline{29.406}$	$\mathbf{29.015}$	$\underline{29.074}$	$\underline{28.188}$	$\underline{29.534}$
Custom Diffusion [50]	$21.906$	$20.227$	$35.948$	$7.544$	$6.680$	$2.481$	$28.253$	$29.012$	$28.246$	$26.452$	$27.395$	$25.424$
Single-StyleForge (ours)	$\mathbf{13.008}$	$\mathbf{12.222}$	$\mathbf{20.718}$	$\mathbf{1.602}$	$\mathbf{1.349}$	$\mathbf{0.704}$	$28.761$	$28.616$	$27.551$	$27.488$	$27.304$	$26.719$
Multi-StyleForge (ours)	$13.480$	$\underline{12.764}$	$\underline{20.880}$	$\underline{1.912}$	$\underline{1.820}$	$1.216$	$\mathbf{31.215}$	$\mathbf{32.082}$	$\underline{28.662}$	$\mathbf{31.243}$	$\mathbf{30.891}$	$\mathbf{29.852}$

, Multi-StyleForge significantly improves the text alignment compared to other baselines, as evidenced by higher CLIP scores for all target styles except anime, which shows the second-highest score.

5.6. Comparison

Finally, we compared our Single and Multi-StyleForge methods with existing baseline methods. First, quantitative comparisons in terms of FID/KID and CLIP scores are provided in Table 4. We found that the DreamBooth [14] generally performs well, but its effectiveness decreases for styles such as anime, cubism, and pixel art. For Textual Inversion [15], as the target style becomes less realistic, both FID/KID and CLIP scores decrease significantly. Across all target styles, Single-StyleForge demonstrated superior performance in terms of the FID/KID scores, followed by Multi-StyleForge. Regarding the CLIP scores, Multi-StyleForge achieved the best performance by improving the text–image alignment. Compared to Custom Diffusion [50] and LoRA [17], which use the concept of personalizing multi-subjects and/or parameter-efficient fine-tuning, it is evident that our Single/Multi-StyleForge methods using full-tuning perform better in learning the ambiguous subject “style”.

A qualitative comparison with the baselines is illustrated in Figure 7. Some methods often involve a trade-off between reflecting the target artistic style and aligning images with the text. In the case of pixel art, Textual Inversion and LoRA struggle to capture the target style fully. Furthermore, Textual Inversion (in cubism) and Custom Diffusion fail to depict details like “smiling”, with Custom Diffusion often showing a rear view instead. In contrast, our methods faithfully capture both the target style and the text in the generated images. More images generated using other text prompts are provided in Appendix A (see Figure A4 and Figure A5).

5.7. User Study

A user study with 20 participants evaluated our methods against baselines across four metrics—detail attribute reflection, background–person separation, human figure quality, and style consistency—using a 5-point Likert scale (Figure 8). Both Single-StyleForge and Multi-StyleForge consistently outperformed all baselines. Single-StyleForge achieved $35.8$–$46.5\%$ improvements, with the largest gain in style consistency ($46.5\%$). Multi-StyleForge further improved the results, reaching $39.8$–$52.6\%$, validating the effectiveness of dual-binding with multi-token decomposition. Improvements across all metrics (>35%) demonstrate that our approach enhances both the style fidelity and text alignment without trade-offs. The questionnaire is provided in Appendix A.

6. Conclusions

We presented StyleForge, a personalization framework that extends text-to-image synthesis to abstract artistic styles using only a small set of references. Single-StyleForge introduces a dual-binding mechanism that pairs a style token with target-style characteristics and an auxiliary token with curated human-centric images, stabilizing the rendering of people while mitigating overfitting. Multi-StyleForge further disentangles person and background attributes via multi-token decomposition, enabling fine-grained compositional control and improved text–image alignment. Across six styles, our experiments show that StyleForge achieves strong style fidelity and alignment with 15–20 references and without architectural changes to the base diffusion model.

Beyond creative media, these properties make StyleForge practical for smart-city content pipelines in which visual communication is central. Examples include rapidly producing style-consistent assets for urban dashboards and digital signage or adapting imagery to local cultural identities for tourism and citizen engagement, while preserving prompt controllability and reducing design turnaround.

The model’s robustness may degrade under domain shifts, such as night scenes, extreme weather, or densely crowded environments. Additionally, the use of human-centric auxiliary images necessitates careful curation to mitigate potential biases. Future work will focus on incorporating layout and geometric controls for complex urban scenarios, developing lightweight provenance mechanisms to support governance, and conducting human-in-the-loop evaluations to assess clarity and inclusivity in civic applications. In summary, StyleForge advances few-shot controllable style personalization for diffusion models and provides a practical pathway for integrating adaptive visual generation into AI-driven smart city services.