ConWave-LoRA: Concept Fusion in Customized Diffusion Models with Contrastive Learning and Wavelet Filtering

Abstract

Customizing diffusion models via Low-Rank Adaptation (LoRA) has become a standard approach for customized concept injection. However, synthesizing multiple customized concepts within a single image remains challenging due to the parameter pollution problem, where naive fusion leads to gradient conflicts and severe quality degradation. In this paper, we introduce ConWave-LoRA, a novel framework designed to achieve hierarchical disentanglement of object and style concepts in LoRAs. Supported by our empirical validation regarding frequency distribution in the latent space, we identify that object identities are predominantly encoded in high-frequency structural perturbations, while artistic styles manifest through low-frequency global layouts. Leveraging this insight, we propose a Discrete Wavelet Transform (DWT) based filtering strategy that projects these concepts into orthogonal optimization subspaces during contrastive learning, thereby isolating structural details from stylistic attributes. Extensive experiments, including expanded ablation studies on LoRA rank sensitivity and style consistency, demonstrate that ConWave-LoRA consistently outperforms strong baselines, producing high-fidelity images that successfully integrate multiple distinct concepts without interference.

1. Introduction

The advent of latent diffusion [1] has significantly accelerated progress in the field of text-to-image generation. By introducing the latent space that balances generation quality, computational efficiency, and accessibility, latent diffusion has enabled both academic researchers and industry practitioners to explore a wide range of creative and practical applications. At its core, a diffusion model operates through a two-stage process: during training, a clean latent representation of an image is progressively perturbed with Gaussian noise over a sequence of time steps, producing increasingly degraded representations; the model then learns to reverse this process by iteratively denoising the noisy latents, conditioned on text prompts, to recover the original image content [2]. This bidirectional framework allows the model to capture a rich distribution over the data, enabling controllable and high-fidelity generation. More recently, the emergence of flow matching based diffusion models [3], such as SD3 [4], FLUX.1-dev and Qwen Image [5], has further advanced the state of the art, offering the capability to synthesize visually compelling and semantically coherent images with remarkable fidelity. These developments have not only broadened the accessibility of advanced generative techniques but also catalyzed innovations across domains including digital art, content creation, and visual design.

For professionals in content creation and visual design, a key requirement on applying diffusion models is the ability to generate various images of a specified concept while preserving its distinctive features, such as a particular product or model [6,7]. However, a pretrained diffusion model cannot accomplish this task, as it has not been exposed to images of the concept during its training phase. To meet this requirement, a common strategy is to further fine-tune the diffusion model, thereby obtaining a customized version tailored to the concept [8]. Specifically, we employ a dataset containing images of the concept along with their corresponding image captions. The objective of fine-tuning is to enable the diffusion model to generate images, conditioned on the given image captions, that closely resemble those in the dataset. Based on the above approach, we can train diffusion models capable of depicting a wide variety of specific concepts. In fact, several platforms dedicated to sharing such customized models have emerged within the community, such as Civitai [9] and HuggingFace [10], where users can search for and utilize a diverse range of customized diffusion models.

The above method effectively addresses the problem of generating images containing a single specific concept. However, generating images that incorporate multiple specific concepts remains a challenging task [11,12]. On the one hand, due to the absence of a training dataset containing images with multiple specific concepts, it is not feasible to directly fine-tune a diffusion model for this purpose. On the other hand, directly employing multiple customized diffusion models leads to the issue of parameter interference, commonly referred to as the parameter pollution problem.

Figure 1 presents an example on the pollution problem when increasing the number of customized diffusion models being applied in FLUX.1-Dev. The first line shows the generated images using three different models separately, while the second line shows the generated images when applying them sequentially. The results indicate that applying each model separately would yield high quality images that conform to the concept from which the model is trained. In contrast, with the number of models being applied increasing, the image quality degrades drastically due to the parameter conflictions between models. In addition, the distinct behaviors of object and style concepts in image generation further exacerbate the challenge. Specifically, object concepts typically require the generated objects to maintain consistent details and characteristics across different images and prompts. In contrast, style concepts do not demand such consistency in fine-grained features, but instead emphasize uniformity in overall texture and color.

In this paper, we propose a novel method, ConWave-LoRA, to effectively address the challenge of multi-concept fusion. Our approach diverges from existing unified optimization methods by adopting a hierarchical disentanglement strategy. This design is grounded in two key hypotheses that we empirically validate in this work: (1) Concepts in customization tasks can be structurally decoupled into object-oriented identities (requiring strict shape consistency) and style-oriented attributes (emphasizing global texture and palette); and (2) These distinct concept types occupy separable frequency bands within the diffusion latent space. Specifically, we demonstrate that object identities are predominantly encoded in high-frequency structural perturbations (e.g., edges and contours), while artistic styles are manifested through low-frequency global distributions (e.g., color tones and layouts). By mapping these concepts to orthogonal frequency subspaces via Discrete Wavelet Transform (DWT), we can mitigate gradient conflicts during fusion, ensuring that style adaptation does not degrade object structure.

• We formally characterize the parameter pollution problem in multi-LoRA concept fusion as a conflict between optimization directions in shared parameter spaces, and analyze its degradation effects on image generation quality.
• We introduce ConWave-LoRA, a frequency-aware two-stage framework. Unlike prior works treating all parameters uniformly, we propose a DWT-based orthogonal filtering strategy to isolate high-frequency object details from low-frequency style attributes, effectively creating non-conflicting optimization subspaces for different concepts.
• We provide empirical validation of the frequency distribution assumption within the latent space (Section 4.2), visually and statistically confirming the mapping between frequency bands and semantic concepts.
• We conduct extensive experiments and expanded ablation studies (including LoRA Rank sensitivity and Style Consistency metrics). The results demonstrate that ConWave-LoRA achieves state-of-the-art performance in preserving both object fidelity and stylistic coherence compared to existing strong baselines.

The remainder of this paper is structured as follows. Section 2 reviews key topics related to our work. Section 3 presents the technical details and discussion of the proposed method ConWave-LoRA. Section 4 reports extensive experiments demonstrating its effectiveness, and Section 5 concludes the paper.

2. Related Work

2.1. Parameter-Efficient Fine-Tuning

Large-scale diffusion models and vision-language models have demonstrated remarkable performance across various generative tasks, but their deployment in specialized domains is often hindered by the computational and storage costs associated with full fine-tuning. To address this, Low-Rank Adaptation [13] has emerged as a widely adopted method for parameter-efficient fine-tuning. Instead of updating all model parameters, LoRA introduces trainable low-rank matrices into existing layers (e.g., attention and feed-forward blocks), significantly reducing the number of trainable parameters and memory footprint. LoRA has since been extended and adapted to various domains, including image generation, language modeling, and multi-modal tasks [14]. Recent works such as LoRA-CLIP [15] and LoRA-DreamBooth [8] demonstrate that even when only a small subset of weights are adapted, models can still learn novel concepts or styles effectively. Moreover, LoRA enables fast adaptation and merging of multiple skills by supporting modular training and composition, which has led to a surge of community-driven LoRA repositories for sharing and recombining pre-trained LoRA modules, such as Civitai [9] and Huggingface [10]. The proposed method is based on LoRAs trained using existing parameter-efficient finetuning methods, and focuses on how to effectively merge multiple LoRAs to generate images with better image quality, and alignment with corrresponding concepts.

2.2. Concept Fusion with LoRA

Although LoRA facilitates image generation for novel concepts, effectively generating images that incorporate multiple novel concepts remains a challenging task—commonly referred to as the concept fusion problem. Early attempts to address this issue primarily rely on training-free strategies, such as weighted averaging of LoRA parameters [11] or LoRA weight decomposition [16], to combine multiple LoRAs and fuse their associated concepts. Recognizing the redundancy in LoRA parameters, K-LoRA [17] introduces a more refined approach that selectively extracts parameters deemed more important during the merging process. However, these methods often suffer from limitations, including image quality degradation, loss of fine-grained details, and reduced generalization of the fused concepts across diverse contexts. Recent advancements have shifted towards training-based strategies for LoRA fusion, exemplified by methods such as LoRACLR [18] and ZipLoRA [19]. By designing task-specific loss functions tailored to the LoRA merging objective, these approaches achieve improved performance over training-free alternatives. Our proposed method falls within this training-based paradigm but distinguishes itself through a novel hierarchical concept disentanglement framework and a frequency-aware filtering strategy that enhances the merging of multiple LoRAs.

2.3. Contrastive Learning

While diffusion models have demonstrated impressive generative capabilities, integrating contrastive learning into diffusion frameworks has recently gained attention as a means to enhance representation quality [20], improve controllability [21], and stabilize training [22]. Some early approaches incorporate contrastive objectives into the denoising process to learn more semantically meaningful intermediate representations. For instance, contrastive loss can be applied between different noise levels or denoised latents to encourage temporal consistency across the diffusion trajectory [23]. Other works leverage contrastive learning to improve conditional generation, aligning the intermediate representations of noisy inputs with condition embeddings (e.g., prompt, class labels, or images) [18,19]. Additionally, contrastive regularization has been used in latent diffusion models to enforce structure and disentanglement in the latent space, improving downstream tasks such as editing [24], retrieval, or interpolation [25]. The proposed method also falls within the contrastive learning paradigm and uses contrastive learning objectives in separate stages to improve the model performance, and robustness of multiple concepts in different LoRAs after fusion.

3. Proposed Method

In this section, we present the technical details of our proposed method, which is designed to mitigate the pollution problem illustrated in Figure 1. We first introduce the learning paradigm of flow-matching-based diffusion models under parameter-efficient fine-tuning frameworks, with a particular focus on LoRA [13]. A formal problem formulation is then provided. Subsequently, we give a comprehensive overview of the proposed method, elaborating beyond the preliminary intuitions and qualitative insights discussed earlier. The following subsections describe the two major stages of our approach in detail: the object fusion stage, which forms the first stage, and the style fusion stage, which forms the second. Finally, we conclude this section with an analysis of the computational cost and a discussion of the novelty of our method in comparison with existing approaches.

3.1. Learning Paradigm of LoRA and Problem Formulation

State-of-the-art diffusion models for text-to-image generation typically contain billions of parameters. For instance, FLUX.1-dev, released by Black Forest Lab, comprises 12 billion parameters and currently represents the leading performance within the open-source community, while SD3.5-Large from Stability.ai contains 8.1 billion parameters and also demonstrates competitive performance. However, directly fine-tuning such large-scale models is prohibitively expensive, as it requires vast amounts of training data and substantial GPU resources. To mitigate these costs, a variety of parameter-efficient fine-tuning approaches have been proposed. Among these, LoRA [13] has emerged as the most widely adopted technique within the community.

Given a learnable weight matrix $\mathsf{\Delta }\mathbf{W}\in {\mathbb{R}}^{m\times n}$ from the model to be fine-tuned, under the assumption that many parameters in $\mathsf{\Delta }\mathbf{W}$ are redundant, LoRA first decomposes $\mathsf{\Delta }\mathbf{W}$ into two low-rank matrices $\mathbf{D}\in {\mathbb{R}}^{m\times r}$ and $\mathbf{U}\in {\mathbb{R}}^{r\times n}$ with $r\ll min(m,n)$.

$$\mathsf{\Delta }\mathbf{W}\triangleq \mathbf{D}\times \mathbf{U}$$

(1)

The learnable parameters $\mathbf{D}$ and $\mathbf{U}$ are then iteratively updated under a pre-defined training loss, using gradient descent methods such as Stochastic Gradient Descent (SGD) or Adam. Intuitively, the number of learnable parameters is reduced from $m\times n$ to $(m+n)\times r$. Taking FLUX.1-dev as an example, a common case is $m=5072$, $n=1024$ and $r=32$, therefore the number of parameters is reduced from $5072 \times$ 1024 = 5,193,728 to $(5072 + 1024) \times$ 32 = 219,456, accounting for only 4% of the original learnable parameters. As a result, training costs on time and computation resources could be greatly reduced.

Given a training dataset for fine-tuning $\mathcal{D}={\{{\mathbf{x}}^i,{\mathbf{c}}^i\}}_{i=1}^N$, where ${\mathbf{x}}^i$ and ${\mathbf{c}}^i$ denote training images and their corresponding captions, respectively, the fine-tuning process proceeds as follows. First, the encoder of a variational autoencoder (VAE) is used to project the images ${\left\{{\mathbf{x}}^i\right\}}_{i=1}^N$ into a latent space, producing representations ${\left\{{\mathbf{z}}^i\right\}}_{i=1}^N$. Following standard notation, we denote ${\mathbf{z}}_0^i$ as the clean latent representation of an image, and ${\mathbf{z}}_t^i$ as its perturbed version after injecting Gaussian noise ${ϵ}_t$ at diffusion step t.

To optimize the model, we adopt a flow-matching loss built upon the Rectified Flow framework [26]. Rectified Flow reformulates flow matching by explicitly modeling the probability flow ODE in a rectified manner, thereby improving stability and convergence during training. Under this framework, the loss function for fine-tuning is defined as:

$${\mathcal{L}}_{\mathrm{FM}}={\mathbb{E}}_{({\mathbf{x}}^i,{\mathbf{c}}^i)\sim \mathcal{D},t\sim T}\left|\right|({ϵ}_t-{\mathbf{z}}_0^i)-{ϵ}_{\theta }({\mathbf{z}}_t^i,{\mathbf{c}}^i,t){\left|\right|}_2^2$$

(2)

where ${ϵ}_{\theta }({\mathbf{z}}_t^i,{\mathbf{c}}^i,t)$ denotes the velocity predicted by the model at timestep t, conditioned on the noisy latent ${\mathbf{z}}_t^i$ and the text prompt ${\mathbf{c}}^i$, while T is the total number of diffusion timesteps (typically set to 1000). Conceptually, this objective enforces the model to align its predicted velocity with the ground-truth rectified flow field, thereby learning a more accurate transport map between noisy and clean latents across the diffusion trajectory.

Given the learned parameters $\mathsf{\Delta }\mathbf{W}\in {\mathbb{R}}^{m\times n}$ using the flow-matching loss and dataset $\mathcal{D}$, the final fine-tuned diffusion model is obtained by adding $\mathsf{\Delta }\mathbf{W}$ to the original parameter $\mathbf{W}$; that is, for each weight matrix in the model, we have:

$${\mathbf{W}}_{\mathrm{new}}=\mathbf{W}+\alpha \times \mathsf{\Delta }\mathbf{W}=\mathbf{W}+\alpha (\mathbf{D}\times \mathbf{U})$$

(3)

where $\alpha$ is a pre-defined coefficient known as LoRA scale, commonly set as a value between 0.2 and 1 in practice. During the infernece stage of the finetuned model, given an image caption $\mathbf{c}$ and a random Gaussian noise ${\mathbf{z}}_T$, the predicted velocity from the model is iteratively used, and the final result ${\mathbf{z}}_0$ is a clean image in the latent space. Finally, the decoder of VAE is used to transform ${\mathbf{z}}_0$ into an image in the original pixel space.

Formally, given N LoRAs $\{m_1,m_2,\cdots ,m_N\}$ trained from separate datasets, the proposed method aims to merge them into a new LoRA $m_{\mathrm{merged}}$ that better preserves concepts in these datasets. How to simultaneously merge multiple LoRAs for text to image generation remains as a challenging problem, since each LoRA would update the same parameters in the original diffusion model according to Equation (3). Simple averaging (i.e., $m_{\mathrm{merged}}={\displaystyle \frac{1}{N}}{\sum }_{n=1}^N{m}_n$) inevitably causes conflictions, and brings side effects to each other. Such effects lead to the degradation of image quality, known as the model pollution problem in the community.

3.2. Overview

The proposed method addresses the aforementioned challenge through a hierarchical concept disentanglement framework, consisting of the object fusion stage and the style fusion stage. In the first stage, object fusion aims to effectively integrate multiple object LoRAs. We employ a contrastive training strategy to determine the parameters of the fused LoRA. Specifically, the fused LoRA is required to generate images that are semantically similar to those produced by each original object LoRA under its corresponding concept prompt, while simultaneously producing images that are dissimilar to those generated by other LoRAs when conditioned on the same prompt.

The second stage, style fusion, focuses on combining the fused object LoRA from the first stage with style LoRAs. Unlike object LoRAs, which are tailored to represent a single concept or entity, style LoRAs capture broader stylistic patterns that influence the global artistic characteristics of generated images. To better preserve style-specific information, we avoid parameter learning for the fused LoRA as in the first stage. Instead, we introduce a set of learnable coefficients that perform weighted averaging between the fused object LoRA and style LoRAs. These coefficients are optimized using the same contrastive training framework as in the object fusion stage.

Furthermore, motivated by the observation that object LoRAs primarily affect high-frequency structures (e.g., edges and contours), whereas style LoRAs predominantly influence low-frequency components (e.g., textures and colors), we incorporate a discrete wavelet transform (DWT) in both stages. DWT is used to decompose generated images into frequency components, enabling contrastive training to focus on the most informative representations for each LoRA in orthogonal optimization subspaces. An overview of this two-stage framework is illustrated in Figure 2.

3.3. Object Fusion

Given multiple object LoRAs, the object fusion stage focuses on effectively fusing corresponding object concepts into a single merged LoRA, while eliminating the side effects and the model pollution problem when applying all of them, as illustrated in Figure 1. Formally, given N object LoRAs with different parameters $\{m_{{\theta }_1},m_{{\theta }_2},\cdots ,m_{{\theta }_n},\cdots ,m_{{\theta }_N}\}$, with each of them corresponds to a object concept that could be characterized by a prompt $\{{\mathbf{c}}_1,{\mathbf{c}}_2,\cdots ,{\mathbf{c}}_n,\cdots ,{\mathbf{c}}_N\}$, the problem in object fusion stage is to optimize parameters for a merged LoRA $m_{\widehat{\theta }}$, which is capable of generating similar images as each $m_{{\theta }_n}$ using prompt ${\mathbf{c}}_n$. In addition, we are given a set of images generated independently by each LoRA $m_{{\theta }_n}$ using prompt ${\mathbf{c}}_n$, i.e., ${\left\{{\mathbf{x}}_n^i\right\}}_{i=1}^p$, where ${\mathbf{x}}_n^i$ denotes the i-th image generated using LoRA $m_{{\theta }_n}$. Lastly, since the prediction of each LoRA is the velocity used for denoising in latent space at one specific timestep, each image is first preprocessed into its latent representation using VAE encoding, and we have encoded image latents ${\left\{{\mathbf{z}}_n^i\right\}}_{i=1}^p$.

Prior to fine-tuning parameters of the merged LoRA $m_{\widehat{\theta }}$, its initial parameters are set to the average of all object LoRAs using Equation (4). Compared to initializing parameters of the merged LoRA from a random uniform distribution, parameters averaged from LoRAs serve as a better starting point for training loss optimization, commonly leading to fewer training steps before loss convergence.

$$m_{\widehat{\theta }}={\displaystyle \frac{1}{N}}\sum _{n=1}^N{m}_{{\theta }_n}$$

(4)

During each training step, an object LoRA and corresponding prompt are randomly sampled as the positive class $\{m_{{\theta }_n},{\mathbf{c}}_n\}$. In addition, the latent representation of an image generated by $m_{{\theta }_n}$ is also randomly sampled as the positive target image ${\mathbf{z}}_n^i$. Then, all LoRAs aside from $m_{{\theta }_n}$ are treated as the negative class, ${\{m_{{\theta }_j},{\mathbf{c}}_j\}}_{j\ne n}$. Same as the positive class, the latent representation of an image is randomly sampled for each LoRA ${\left\{{\mathbf{z}}_j^i\right\}}_{j\ne n}$. After randomly choosing a diffusion timestep t, the overall training loss is defined as follows:

$${\mathcal{L}}_{\mathrm{object}}=s(m_{\widehat{\theta }}({\mathbf{z}}_{n,t}^i,{\mathbf{c}}_n,t),{\mathbf{z}}_{n,0}^i)+max(0,{\displaystyle \frac{1}{2}}-min\left({\left\{s(m_{\widehat{\theta }}({\mathbf{z}}_{n,t}^i,{\mathbf{c}}_j,t),{\mathbf{z}}_{n,0}^i)\right\}}_{j\ne n}\right))$$

(5)

where $m_{\widehat{\theta }}({\mathbf{z}}_{n,t}^i,{\mathbf{c}}_n,t)$ is the predicted velocity of the target merged LoRA $m_{\widehat{\theta }}$ using prompt ${\mathbf{c}}_n$ and latent ${\mathbf{z}}_{n,t}^i$ at diffusion timestep t, and ${\mathbf{z}}_{n,0}^i$ corresponds to the original latent image before adding noise. $s(·,·)$ is a distance measurement in the latent space:

$$s(\mathbf{v},{\mathbf{z}}_0)=\left|\right|\mathrm{H}({ϵ}_t-{\mathbf{z}}_0)-\mathrm{H}\left(\mathbf{v}\right){\left|\right|}_2^2$$

(6)

where ${ϵ}_t$ is the added noise at t-th diffusion timestep, ${\mathbf{z}}_0$ is the original image encoded into the latent space, $\mathbf{v}$ is the predicted velocity. In the rectified flow framework, ${ϵ}_t-{\mathbf{z}}_0$ is the ground truth velocity. Equation (6) follows the flow matching loss detailed in Equation (2) except the operator H, which denotes discrete wavelet transform (DWT). Using different low-pass and high pass filters, the input data $\mathbf{x}$ could be decomposed into four sub-bands, the low-frequency sub-band ${\mathbf{x}}_{LL}$ and the high-frequency sub-bands ${\mathbf{x}}_{LH}$, ${\mathbf{x}}_{HL}$, ${\mathbf{x}}_{HH}$. More specifically, the Haar wavelet transform is applied channel-wise to the spatial dimensions of the high-frequency sub-bands of the latent tensor ${ϵ}_t-{\mathbf{z}}_0$ and $\mathbf{v}$. In the object fusion stage, since objects and their lines commonly correspond to high frequency parts in the image, the high-frequency sub-band decomposition for the predicted velocity $\mathbf{v}$ and the target ${ϵ}_t-{\mathbf{z}}_0$ are used for distance measurement.

According to Equation (5), the training loss in the object fusion stage requires the merge LoRA to have similar predictions as the LoRA $m_{{\theta }_n}$ given input prompt ${\mathbf{c}}_n$, serving as the positive loss. In addition, it requires the merged LoRA to have predictions different from other LoRAs ${\left\{m_{{\theta }_j}\right\}}_{j\ne n}$, serving as the negative loss. More specifically, a min-max formulation is used, which first locates the most similar prediction in LoRAs ${\left\{m_{{\theta }_j}\right\}}_{j\ne n}$, then requires the merged LoRA to have disimilar prediction to it. Furthermore, the margin in Equation (5) is empirically set to $0.5$. This value serves as an approximate upper bound for the flow matching loss, ensuring that the discriminative term (i.e., the second term in Equation (5)) contributes meaningful gradients throughout the entire object fusion stage. This prevents the inter-class separation loss from vanishing prematurely.

3.4. Style Fusion

Given the merged object LoRA $m_{\widehat{\theta }}$ capable of generating images with multiple concepts, the style fusion stage further conducts style fusion between $m_{\widehat{\theta }}$ and another style LoRA $m_{\beta }$. The final merged LoRA $m_{\mathrm{final}}$ is able to effectively generate images with multiple concepts under the specific image style provided by $m_{\beta }$. In contrast to object LoRAs, style LoRAs are capable of converting images with a broad range of objects into a specific style, such as anime style, oil-painting style, etc. To fully perserve its capability on converting image style, we cannot use contrastive training loss similar to the object fusion stage, since there exist countless objects and it is infeasible to include all of them into the training loss described in Equation (5). Therefore, instead of optimizing model parameters of the final merged LoRA towards generating similar images as the original LoRA using different prompts, the style fusion stage instead focuses on effectively merging $m_{\widehat{\theta }}$ and $m_{\beta }$ without changing their model parameters.

Formally, for parameters in each block ${\left\{i\right\}}_{i=1}^B$ of $m_{\widehat{\theta }}$ and $m_{\beta }$, the style fusion stage aims to find a set of parameters ${\{{\mathbf{w}}_i^o,{\mathbf{w}}_i^s\}}_{i=1}^B$, and the parameters in the block i of the final LoRA $\mathsf{\Delta }{\mathbf{W}}_i$ are determined by:

$$\mathsf{\Delta }{\mathbf{W}}_i={\mathbf{w}}_i^o\times ({\mathbf{D}}_i^o\times {\mathbf{U}}_i^o)+{\mathbf{w}}_i^s\times ({\mathbf{D}}_i^s\times {\mathbf{U}}_i^s),i=1,\cdots ,B$$

(7)

where $\{{\mathbf{D}}_i^o,{\mathbf{U}}_i^o\}$ denotes low-rank matrices of the i-th block parameters in object LoRA $m_{\widehat{\theta }}$, and $\{{\mathbf{D}}_i^s,{\mathbf{U}}_i^s\}$ denotes low-rank matrices of the i-th block parameters in style LoRA $m_{\beta }$. The right part of Figure 2 illustrates the weighted merge used in the style fusion stage. In addition, prior to the training stage, all parameters ${\{{\mathbf{w}}_i^o,{\mathbf{w}}_i^s\}}_{i=1}^B$ are set to initial value 0.5. Therefore, $m_{\mathrm{final}}$ corresponds to the simple average of $m_{\widehat{\theta }}$ and $m_{\beta }$ before training starts.

During each training step i, the VAE encode result of a generated image and corresponding prompt are randomly sampled for the merged object LoRA $m_{\widehat{\theta }}$ and style LoRA $m_{\beta }$, denoted by $\{{\mathbf{z}}_i^o,{\mathbf{c}}_i^o\}$ and $\{{\mathbf{z}}_i^s,{\mathbf{c}}_i^s\}$. Given a randomly chosen timestep t, the training loss is defined as:

$${\mathcal{L}}_{\mathrm{style}}=s(m_{\mathrm{final}}({\mathbf{z}}_{i,t}^o,{\mathbf{c}}_i^o,t),{\mathbf{z}}_{i,0}^o)+s(m_{\mathrm{final}}({\mathbf{z}}_{i,t}^s,{\mathbf{c}}_i^s,t),{\mathbf{z}}_{i,0}^s)+\lambda \left|\right|{\mathbf{w}}_i{\left|\right|}_2^2$$

(8)

where ${\mathbf{z}}_{i,t}^o$ denotes the latent after applying noise at timestep t, ${\mathbf{z}}_{i,0}^o$ denotes the original image in latent space, and $m_{\mathrm{final}}({\mathbf{z}}_{i,t}^o,{\mathbf{c}}_i^o,t)$ is the denoise velocity prediction of the final merged LoRA $m_{\mathrm{final}}$. The last term is the regularization term on learnable parameters. $s(·,·)$ is a distance measurement similar to Equation (6), wheras instead of extracting high-frequency sub-bands as in the object fusion stage, since the image style is mostly represented by textures in the image, the low-frequency sub-band ${\mathbf{x}}_{LL}$ is extracted here and used for measuring similarity. Since $m_{\mathrm{final}}$ is a weighted average of $m_{\widehat{\theta }}$ and $m_{\beta }$ using learnable parameters ${\{{\mathbf{w}}_i^o,{\mathbf{w}}_i^s\}}_{i=1}^B$, optimizing the loss in Equation (8) requires the final merged LoRA to generate similar images to the merged object LoRA and style LoRA using corresponding prompts ${\mathbf{c}}_i^o$ and ${\mathbf{c}}_i^s$. In more detail, ${\mathbf{c}}_i^o$ corresponds to the combination of all object prompts, e.g., <A>&<B>&<C> in Figure 2, and ${\mathbf{c}}_i^s$ corresponds to the style prompt.

3.5. Computational Cost and Novelty of the Proposed Method

The computational cost of the object-fusion stage is comparable to training a LoRA from scratch, whereas the cost of style-fusion stage is far smaller because it only learns a small set of merge coefficients over the (frozen) object and style LoRAs, rather than updating full LoRA weight tensors. As detailed in our ablation experiments, both stages converge within only a few hundred training steps.

While Contrastive Learning and DWT are established techniques individually, our core novelty lies in their structural integration to fundamentally solve the “parameter pollution” problem in multi-concept fusion. We distinguish our approach from existing works in two key aspects:

1.

Orthogonal Optimization Subspaces: Existing methods optimize a unified objective where gradients from object identity and artistic style are mixed, often leading to destructive interference (e.g., style textures overriding object edges). In contrast, ConWave-LoRA introduces a frequency-aware filtering strategy. By applying DWT, we theoretically project the optimization targets into orthogonal subspaces: high-frequency bands for structural identity and low-frequency bands for global style. This ensures that the gradient updates for style adaptation do not pollute the structural integrity of the object, a mechanism validated by our analysis in Section 4.2.

2.

Hierarchical Concept Disentanglement: Unlike prior “joint training” paradigms that treat all concepts uniformly, we propose a ”Structure-First, Style-Second” hierarchical framework. This design is motivated by the inherent semantic hierarchy of image generation—geometric structure (Object) forms the foundation upon which textural attributes (Style) are rendered. Our method respects this hierarchy, effectively decoupling the learning process to avoid the image pollution observed in naive fusion.

The assumption characterizing “objects as high-frequency” and “styles as low-frequency” is a general heuristic that holds for the majority of customization tasks (e.g., photorealistic or painterly styles) but may face limitations in specific edge cases. For instance, “shape-altering” styles (such as abstract geometric deformations or detailed line art) inherently possess significant high-frequency components that overlap with object details. In such scenarios, a strict frequency separation might theoretically constrain the style’s ability to alter the object’s contours. However, in multi-concept fusion, the primary failure mode is often “parameter pollution”, where aggressive style adaptation destroys the subject’s recognizability (as shown in Figure 1). By solidifying the object’s high-frequency structural integrity in the first stage, our framework ensures that the subject remains identifiable, even if it means trading off a degree of geometric deformation in extreme shape-altering styles. This trade-off is essential for maintaining robustness in few-shot customization tasks.

4. Experiment

The experimental section is organized as follows. We first present the implementation details of the proposed method ConWave-LoRA and the baselines used for comparison, along with descriptions of the datasets and evaluation metrics employed during training and evaluating. Next, we present in-depth analysis on the low and high frequency reconstruction results of training images in latent space, demonstrating the validity of our intuition. We then detail the experiments related to image quality evaluation, which directly reflect the model’s performance in image generation. This includes both quantitative results across multiple metrics and qualitative analyses under various scenarios. Finally, we conduct ablation studies on the contributions of discrete wavelet transform (DWT) in the proposed method, and effects of different LoRA ranks in the object fusion stage.

4.1. Experiment Settings

4.1.1. Implementation Details

All experiments are conducted using the open-sourced FLUX.1-dev as the base diffusion model, selected for its strong capability in generating high-fidelity images and adhering to textual prompts. Both object and style LoRAs used for merging are independently trained on FLUX.1-dev, with the LoRA rank r fixed at 16. We adopt the AdamW optimizer with a learning rate of $1\times {10}^{-4}$ to update model parameters during both the object and style fusion stages. The model and training pipeline are implemented in PyTorch 2.5, and all training and evaluation stages are performed on an NVIDIA L20 GPU.

4.1.2. Baselines

We evaluate the proposed method ConWave-LoRA against a broad range of baselines. For training-free approaches, we include Model Average Merge [11], Block Interleaved Merge [11] K-LoRA [17], and TARA [27]. Model Average Merge directly integrates all LoRAs into the diffusion model, as illustrated in Figure 1. Building on the observation that many LoRA parameters are redundant, Block Interleaved Merge improves upon this by separately injecting object LoRAs and style LoRA into the odd- and even-numbered blocks of the diffusion model, respectively. K-LoRA achieves training-free object fusion by dynamically selecting either the object or style LoRA for each attention layer based on the aggregated magnitude of their Top-K weight elements. TARA achieves concept fusion by employing token focus masking to restrict each LoRA module’s influence to its specific class token. For training-based baselines, we include LoRACLR [18] and ZipLoRA [19]. All of K-LoRA, TARA, LoRACLR, and ZipLoRA are competitive approaches for merging multiple LoRAs effectively.

4.1.3. Datasets and Evaluation Metrics

To comprehensively evaluate the effectiveness of the proposed method, we manually curated five datasets, each comprising 10 images depicting a distinct object. The selected objects span a diverse range of categories, including animals, humans, and inanimate objects. Detailed information about the collected datasets is provided in Figure 3. These datasets are employed to train object-specific LoRAs for use in the fusion process. Additionally, style LoRAs are sourced directly from Civitai, a public open-source platform.

For quantitative evaluation, we employ the CLIP score [28] to assess the semantic alignment between the input prompt and the generated image. Specifically, the cosine similarity between the CLIP embeddings of the original and generated images is computed to measure image-level fidelity. In addition, the Human Preference Score (HPS) [29] is adopted to evaluate perceptual quality and overall visual plausibility from a human perspective [30]. For image style comparison after style fusion stage, the VGG style loss [31] is used to measure the style similarity. All experiments are conducted over five independent trials, and results are reported as the mean ± standard deviation to ensure statistical reliability.

4.2. Frequency Analysis on Images in Latent Space

To investigate the distinct roles of high- and low-frequency components, we first randomly sampled an image from each dataset and encoded it into the latent space using the VAE from FLUX.1-dev. Subsequently, we employed a Haar wavelet transform to decompose the latent representation into high-frequency sub-bands (${\mathbf{x}}_{LH}$, ${\mathbf{x}}_{HL}$, ${\mathbf{x}}_{HH}$) and a low-frequency sub-band (${\mathbf{x}}_{LL}$). These sub-bands were then independently decoded back into the pixel space via the VAE decoder. As illustrated in Figure 4, the reconstructions derived from the high-frequency sub-bands predominantly preserve object contours and edges, whereas the low-frequency counterparts retain the global layout and textural information. These observations corroborate our hypothesis that high-frequency components in the latent space correspond to the structural outlines of objects in the original images.

4.3. Image Generation Performance Comparison

Comparisons of image generation performance are organized into three components: (1) single-object generation, which evaluates the ability of the merged LoRA to generate images containing a single target object; (2) multi-object generation, which assesses the model’s capacity to compose multiple objects within a single image; and (3) qualitative analysis of various baseline methods in generating multi-object images under diverse artistic styles.

Table 1. Text alignment results (mean ± std) on single-object image after merging. Best result for each object category is bolded.

Method	Dog	Cat	Vase	Woman	Sofa
Model Average	0.245 ± 0.006	0.261 ± 0.004	0.273 ± 0.005	0.209 ± 0.009	0.257 ± 0.009
Block Interleaved Average	0.239 ± 0.006	0.260 ± 0.005	0.270 ± 0.009	0.201 ± 0.007	0.258 ± 0.005
K-LoRA	0.221 ± 0.003	0.243 ± 0.002	0.261 ± 0.003	0.213 ± 0.005	0.243 ± 0.004
TARA	0.264 ± 0.007	0.263 ± 0.004	0.268 ± 0.007	0.227 ± 0.002	0.273 ± 0.005
LoRACLR	0.270 ± 0.005	0.265 ± 0.003	0.283 ± 0.005	0.235 ± 0.008	0.280 ± 0.007
ZipLoRA	0.262 ± 0.010	0.249 ± 0.008	0.271 ± 0.010	0.230 ± 0.009	0.267 ± 0.009
ConWave-LoRA	0.271 ± 0.004	0.272 ± 0.002	0.292 ± 0.007	0.234 ± 0.010	0.287 ± 0.005

presents the CLIP scores of different baseline methods in generating single-object images from the dataset described in Figure 3. For each object, every method generates five images using distinct random seeds, and the reported results are averaged across all object categories. The CLIP score quantifies the cosine similarity between the text prompt and the generated image in the shared CLIP embedding space, with higher values indicating better text-image semantic alignment. As shown in the table, the proposed method achieves superior performance across all object categories except for <woman>. Furthermore, all training-based approaches—LoRACLR, ZipLoRA, and our method—outperform the training-free baselines, including Model Averaging, Block Interleaving, K-LoRA, and TARA. However, the performance differences among methods remain relatively modest, as the CLIP score primarily reflects semantic relevance and does not account for perceptual image quality or fidelity to the original image content.

Table 2. Image alignment results (mean ± std) on single-object image after merging. Best result for each object category is bolded.

Method	Dog	Cat	Vase	Woman	Sofa
Model Average	0.584 ± 0.013	0.602 ± 0.011	0.632 ± 0.015	0.613 ± 0.020	0.627 ± 0.010
Block Interleaved Average	0.592 ± 0.023	0.594 ± 0.018	0.633 ± 0.023	0.607 ± 0.028	0.601 ± 0.021
K-LoRA	0.597 ± 0.007	0.583 ± 0.010	0.594 ± 0.007	0.580 ± 0.014	0.574 ± 0.005
TARA	0.832 ± 0.024	0.899 ± 0.017	0.873 ± 0.012	0.840 ± 0.032	0.892 ± 0.014
LoRACLR	0.893 ± 0.035	0.927 ± 0.018	0.913 ± 0.024	0.869 ± 0.049	0.938 ± 0.020
ZipLoRA	0.854 ± 0.027	0.901 ± 0.015	0.894 ± 0.018	0.862 ± 0.037	0.910 ± 0.015
ConWave-LoRA	0.912 ± 0.031	0.948 ± 0.020	0.928 ± 0.022	0.874 ± 0.051	0.956 ± 0.013

further presents the image-level alignment performance of different baseline methods on each single object. For each method, five images are independently generated, and the cosine similarity between the CLIP embeddings of the generated image and the original training image (used to train the corresponding object-specific LoRA) is computed. A higher similarity score indicates greater visual fidelity to the source object. As shown in the table, the proposed method consistently achieves the highest scores across all object categories, demonstrating superior object preservation and feature consistency. Consistent with the text-alignment results, all training-based approaches (e.g., LoRACLR, ZipLoRA, and our method) outperform the training-free baselines, highlighting the effectiveness of fine-tuning in preserving object identity during fusion.

Table 3. Human preference score (mean ± std) on single-object image after merging. Best result for each object category is bolded.

Method	Dog	Cat	Vase	Woman	Sofa
Model Average	0.238 ± 0.004	0.232 ± 0.001	0.247 ± 0.002	0.241 ± 0.005	0.243 ± 0.002
Block Interleaved Average	0.240 ± 0.007	0.228 ± 0.003	0.257 ± 0.001	0.253 ± 0.009	0.250 ± 0.004
K-LoRA	0.251 ± 0.006	0.262 ± 0.004	0.270 ± 0.002	0.267 ± 0.003	0.257 ± 0.002
TARA	0.270 ± 0.008	0.281 ± 0.006	0.284 ± 0.008	0.273 ± 0.004	0.275 ± 0.007
LoRACLR	0.264 ± 0.010	0.265 ± 0.007	0.283 ± 0.004	0.271 ± 0.005	0.270 ± 0.009
ZipLoRA	0.280 ± 0.005	0.276 ± 0.005	0.284 ± 0.007	0.279 ± 0.007	0.273 ± 0.010
ConWave-LoRA	0.286 ± 0.008	0.305 ± 0.002	0.302 ± 0.002	0.286 ± 0.011	0.286 ± 0.004

shows the human preference score of different baselines on each object category. For each baseline, five images are independently generated, and the predict scores from HPSv2 model are averaged as the final result. Larger value predicted by HPSv2 model indicates that the image is more visually appealing from the perspective of human eyes. According to the results, the proposed method achieves the best performance among all baselines, demonstrating the effectiveness of the proposed method.

In addition to the quantitative evaluation results above, Figure 5 shows the single object images generated by the proposed method. Comparing the results with images in the training dataset illustrated in Figure 3, it can be clearly shown that the proposed method is capable of generating high-quality images while maintaining high similarity with the original object category.

Table 4. Evaluation results (mean ± std) on multiple-object image after merging. Best result for each evaluation metric is bolded.

Method	Text Alignment	Aesthetic Score
Model Average	0.257 ± 0.022	0.308 ± 0.009
Block Interleaved Average	0.252 ± 0.024	0.307 ± 0.012
K-LoRA	0.243 ± 0.011	0.331 ± 0.004
TARA	0.284 ± 0.015	0.341 ± 0.010
LoRACLR	0.289 ± 0.020	0.340 ± 0.007
ZipLoRA	0.290 ± 0.026	0.346 ± 0.008
ConWave-LoRA	0.301 ± 0.013	0.358 ± 0.006

shows the text alignment and human preference score of all baselines when generating images using the prompt <dog> & <cat> & <vase> & <woman> & <sofa>. Since there is no ground-truth image that contains multiple objects in the training dataset, we cannot evaluate the image alignment, so it is therefore omitted. Same as the experiment setting of the single-object comparison, five images are independently generated for each baseline, and the averaged results are used as the final result. Figure 6 further shows images with multiple objects generated by the proposed method. It can be shown that the proposed method accurately preserves the main characteristics of original objects in the merged images.

Building on the method introduced in the style fusion stage, two style LoRAs: pixel art and Ghibli style are further merged independently into the previously merged object LoRA. For training-free methods (Model Average, Block Interleaved Average, K-LoRA, and TARA), the style LoRA is directly merged into the object LoRA. In contrast, for training-based methods with a single training stage (e.g., LoRACLR and ZipLoRA), all object LoRAs and the style LoRA are jointly trained to obtain the final merged LoRA.

Qualitative results of different baselines after style fusion are shown in Figure 7. It can be observed that Model Average fails to generate images consistent with the intended styles, whereas the other methods successfully preserve the style information from the style LoRA after merging. While several methods can generate objects that resemble those in the training dataset to some extent, the proposed method achieves the most consistent and visually coherent results. In addition,

Table 5. VGG style loss (mean ± std) of different baselines after style fusion, lower value indicates better image style alignment. Best result for each image style is bolded.

Method	Pixel Art Style	Ghibli Style
Model Average	0.068 ± 0.007	0.073 ± 0.011
Block Interleaved Average	0.044 ± 0.008	0.050 ± 0.005
K-LoRA	0.041 ± 0.009	0.051 ± 0.004
TARA	0.034 ± 0.012	0.040 ± 0.008
LoRACLR	0.029 ± 0.014	0.032 ± 0.009
ZipLoRA	0.030 ± 0.007	0.028 ± 0.006
ConWave-LoRA	0.024 ± 0.008	0.025 ± 0.007

presents the VGG style loss of different baselines on the pixel art style and ghibli style. Specifically, five classic images of each style are picked, and the Gram matrix distance of the VGG19 feature map of these images and the generated images from each baseline is reported as the style loss [31].

4.4. Ablation Study

4.4.1. Necessity of Discrete Wavelet Transform

In this section, we investigate the impact of incorporating discrete wavelet transform (DWT) filtering within the proposed method, specifically in terms of its effect on reducing the difficulty of the LoRA merge task through introducing orthogonal optimization subspaces. As shown in Figure 8, the learning curves for the object fusion stage with and without DWT filtering are compared. A similar comparison for the style fusion stage is presented in Figure 9. The results indicate that applying DWT filtering leads to a consistently lower training loss in both object and style fusion stage, suggesting a reduction in task difficulty. Moreover, the convergence speed is significantly improved: with DWT filtering, the object training loss reaches 0.3 after approximately 400 steps, whereas without it, a comparable loss level requires about 600 steps. In addition, the style training loss with DWT filtering reaches 0.21 after the training converges, which is much samller compared to the training loss without DWT filtering.

These findings demonstrate that the use of discrete wavelet transform filtering not only facilitates faster convergence but also contributes to enhanced performance following the LoRA merge. In addition, it can be observed from the training loss curves that the proposed method is computation-efficient and requires very few training steps. In terms of wall-clock time, the object fusion stage requires approximately 3.5 s per training step, whereas the style fusion stage is more efficient, averaging 1.2 s per step. Consequently, the complete fusion process typically converges in under one hour.

4.4.2. Effects of Different LoRA Ranks

In this section, we investigate the impact of LoRA rank variations specifically within the object fusion stage. The style fusion stage is excluded from this analysis, as it employs off-the-shelf style LoRAs with a fixed rank of 16. We train object LoRAs using ranks $\{4,8,16,32,64\}$. As shown in

Table 6. Evaluation results (mean ± std) on multiple objects image after merging trained LoRAs with different LoRA ranks. Best result for each evaluation metric is bolded.

LoRA Rank	Text Alignment	Aesthetic Score
4	0.254 ± 0.007	0.338 ± 0.003
8	0.262 ± 0.012	0.355 ± 0.007
16	0.301 ± 0.013	0.358 ± 0.006
32	0.309 ± 0.015	0.356 ± 0.009
64	0.299 ± 0.019	0.353 ± 0.007

, excessively low ranks yield suboptimal performance due to limited number of learning parameters. Conversely, while higher ranks theoretically offer greater capacity, they require more training iterations to converge due to the increased parameter count. Consequently, under a fixed training budget, the rank-64 model exhibits performance degradation compared to rank 16, as the model remains under-trained.

5. Conclusions

In this paper, we presented ConWave-LoRA, a theoretically grounded framework designed to resolve the parameter pollution problem in multi-concept fusion. Moving beyond heuristic combinations, our approach establishes a hierarchical disentanglement strategy that respects the semantic order of image synthesis—prioritizing structural identity before stylistic rendering. Crucially, we leveraged Discrete Wavelet Transform (DWT) to project conflicting concepts into orthogonal optimization subspaces. This mechanism effectively isolates high-frequency object perturbations from low-frequency style distributions, thereby mitigating gradient interference during the contrastive learning process. Extensive experimental results and ablation studies validate the effectiveness of the proposed method in combining multiple LoRAs while preserving the semantic integrity of each concept. This work offers new insights into the roles of different types of LoRAs in model merging. We plan to explore end-to-end strategies for LoRA fusion to further enhance efficiency and generalizability as next work.