Knit-Edit: A Unified Multi-Task Editing Framework for Knitted Garments

Abstract

Generative Artificial Intelligence has shown immense potential in industrial design. However, applying Diffusion Transformers to precision manufacturing faces a critical bottleneck: the trade-off between flexible multi-task editing and high-fidelity texture preservation. Existing methods often suffer from “texture collapse” when merging multiple adapters, failing to maintain the intricate topological structures required for industrial standards. To address this, we present Knit-Edit, a unified framework for high-precision knitted garment editing. Our core contribution is EditLoRI, a novel task decoupling mechanism utilizing orthogonal Low-Rank Adaptation. By projecting task-specific gradients into orthogonal subspaces, EditLoRI enables the interference-free composition of multiple editing capabilities within a single lightweight model. Furthermore, we introduce a structure-preserving spatial guidance strategy using Bounding Boxes to resolve the localization ambiguity of text prompts. Validated on our constructed KnitEdit dataset, the proposed method significantly outperforms state-of-the-art baselines in controllability and structural fidelity, offering a robust solution for intelligent generative manufacturing.

1. Introduction

The fashion industry is undergoing a paradigm shift from mass production to mass customization, driven by the principles of Industry 4.0 [1,2]. Consumers increasingly demand personalized garments that reflect their unique style, requiring manufacturers to adapt designs rapidly without compromising quality. In this context, generative AI offers a transformative solution, potentially allowing designers to visualize complex modifications—such as changing a fabric pattern or adding a logo—in seconds rather than hours. However, the transition from digital art to industrial-grade design imposes strict constraints: the generated images must not only be aesthetically pleasing but also structurally accurate. For knitted garments, this means preserving the intricate topology of yarn interloops, which is essential for the feasibility of downstream manufacturing.

Current industrial workflows predominantly rely on manual pixel-level editing or traditional CAD software, both of which are labor-intensive, time-consuming, and difficult to scale. While recent advancements in generative models, particularly Diffusion Transformers (DiTs) [3,4], have revolutionized the field of image generation by producing photorealistic visuals, deploying these models in precise industrial editing tasks reveals a significant limitation: the phenomenon of “Texture Collapse”. Standard fine-tuning methods, such as Low-Rank Adaptation (LoRA) [5] or ControlNet [6], often struggle to balance multiple conflicting editing conditions. For instance, when a designer attempts to combine an adapter for “adding a logo” with another for “changing a sleeve style”, the feature representations of these tasks often interfere with one another in the model’s latent space. This interference manifests as a smoothing or blurring of the fine-grained knitting patterns, effectively “collapsing” the rich texture into a flat, unrealistic surface. This degradation is unacceptable in high-end manufacturing, where the visual fidelity of the fabric structure is paramount.

To bridge the critical gap between state-of-the-art generative algorithms and the rigorous quality standards of industrial manufacturing, we propose Knit-Edit (Figure 1), a unified and robust framework built upon the advanced FLUX.1 architecture. We posit that the key to achieving high-quality, multi-task editing lies in the strict orthogonality of task representations. To operationalize this insight, we introduce EditLoRI, a novel adaptation mechanism inspired by Low-Rank Instruction tuning [7]. Unlike traditional LoRA, which allows adapters to update weights in arbitrary directions, EditLoRI constrains the trainable parameters of each task to reside in approximate orthogonal subspaces. By mathematically projecting task-specific gradients into these independent subspaces, EditLoRI enables the model to decompose and recompose multiple editing tasks—such as pattern modification and Virtual Try-On—without destructive interference. This ensures that the semantic modification of a visual pattern does not corrupt the underlying topological structure of the fabric, effectively solving the texture collapse problem.

Figure 1. The Knit-Edit framework demonstrates high-fidelity multi-task editing. By resolving the conflict between semantic modification and structural preservation, our method enables precise pattern manipulation and Virtual Try-On while maintaining the rigorous topological constraints of knitted fabrics.

Furthermore, we address the inherent ambiguity of natural language in defining precise spatial constraints for industrial design. Text descriptions such as “add a logo on the chest” are often too vague to meet the pixel-perfect precision required by manufacturers. To overcome this, we incorporate a Bounding Box Guidance strategy, serving not merely as a visual overlay but as a powerful Spatial Prompt. By tokenizing Bounding Box coordinates and integrating them into the transformer’s attention mechanism, we allow designers to precisely define “where to edit” and “what to edit”. This approach significantly enhances controllability compared to purely text-driven methods, enabling targeted modifications that respect the garment’s geometry.

We validate the efficacy of our approach on the custom-built KnitEdit dataset, a large-scale collection comprising over 6000 paired samples of high-fidelity knitted garments. Extensive experiments demonstrate that Knit-Edit achieves state-of-the-art performance in both texture fidelity and instruction following, significantly outperforming existing baselines. Our framework offers a scalable, high-precision solution for intelligent generative manufacturing, paving the way for the next generation of AI-assisted fashion design.

In summary, our contributions are fourfold:

• We identify and formally characterize the “Texture Collapse” problem in multi-task generative editing, highlighting the specific challenges posed by the high-frequency topology of knitted fabrics.
• We propose EditLoRI, a mathematically grounded orthogonal task decoupling mechanism that enables interference-free multi-task composition, demonstrating superior performance over standard adapter merging techniques.
• We introduce a structure-preserving spatial guidance strategy that synergizes the semantic understanding of DiTs with precise geometric control via Bounding Box tokenization.
• We release the KnitEdit dataset and a comprehensive benchmark, providing a valuable resource for the community to advance research in high-fidelity garment editing and virtual manufacturing.

2. Related Work

2.1. Fashion Image Editing and Virtual Try-On

Fashion image editing aims to intelligently modify the appearance of garments while rigorously preserving the identity of the wearer and the geometric structure of the clothing [8]. This field has evolved significantly over the past decade. Early approaches relied heavily on Generative Adversarial Networks (GANs), which employ a generator-discriminator architecture to synthesize images. For instance, the seminal DeepFashion framework laid the groundwork for feature-based garment synthesis, enabling basic attribute manipulation. Subsequent works, such as VITON-HD [9], introduced high-resolution Virtual Try-On by utilizing misalignment-aware normalization to handle the complex spatial deformations involved in warping clothing to fit human bodies. However, despite their speed, GAN-based methods often struggle with complex textures and large pose variations due to the inherent instability of adversarial training and the phenomenon of mode collapse, where the model fails to capture the full diversity of the data distribution.

The advent of Denoising Diffusion Probabilistic Models (DDPMs) [10] and Latent Diffusion Models (LDMs) [11] has revolutionized this field by offering a more stable and controllable generation process. By iteratively denoising a latent representation, these models can generate photorealistic images with intricate details. Methods like Paint by Example and Multimodal Garment Designer [12] leverage the strong generative priors of pre-trained LDMs to achieve high-quality editing. More recently, TryOnDiffusion [13] proposed a detail-preserved architecture that integrates multimodal signals (e.g., pose keypoints, garment sketches) to enhance fidelity. Similarly, Fashion-RAG introduces retrieval-augmented generation to customize designs based on user preferences, retrieving relevant texture examples from a database to guide generation. Recent advances also include TryOn-Adapter [14] and MFP-VTON [15], which focus on improving identity preservation and enabling mask-free editing, respectively. ClotheDreamer [16] further extends this capability to 3D consistent generation, bridging the gap between 2D images and 3D digital assets. Furthermore, depth-aware synthesis [17] has been proposed to align multi-modal diffusion manifolds for enhanced structural consistency. Despite these impressive advancements, most existing methods are primarily designed for single-task operations. When applied to complex multi-task scenarios—such as simultaneously adding a logo, changing a sleeve style, and adjusting the fabric texture—they often require training separate models for each task or suffer from “texture collapse”, where fine-grained knitting patterns are smoothed out due to feature interference. Our work addresses this critical limitation by proposing a unified framework capable of handling multiple editing tasks without compromising structural integrity.

2.2. Diffusion Transformers for Image Editing

While U-Net based diffusion models have dominated the field for several years, Diffusion Transformers (DiTs) [3,18] are rapidly emerging as a powerful alternative backbone. Unlike U-Nets, which rely on convolutional layers and downsampling operations, DiTs treat images as sequences of flattened patches, similar to tokens in Natural Language Processing (NLP). This architecture allows DiTs to scale effectively with model size and dataset complexity, capturing long-range dependencies via global self-attention mechanisms [19]. Recent works have begun to explore the potential of DiTs for precise image editing. For example, DiT4Edit [20] utilizes DPM-Solver inversion to enable shape-aware editing in high-resolution images, demonstrating superior structure preservation. Lazy Diffusion Transformer [21] proposes an efficient mechanism for localized editing by generating only the masked regions, significantly reducing computational costs while maintaining context consistency. Other notable approaches include SmartEdit [22] for unified generation and editing, and FreeDrag [23] for point-based manipulation. Text-driven region learning [24] has also shown promise in precise control.

In the specific domain of Virtual Try-On, DiT-VTON [25] represents a pioneering effort to adapt DiT architectures for garment transfer. It demonstrates superior detail preservation compared to U-Net baselines, particularly in handling high-frequency textures. However, current DiT-based editing methods still largely rely on global conditioning or simple binary masking. They lack a mechanism to explicitly decouple different editing instructions within the model’s latent space. This limitation becomes critical in industrial manufacturing, where editing a specific region (e.g., a collar) must not inadvertently alter the topological structure of adjacent fabric. Knit-Edit bridges this gap by introducing orthogonal Low-Rank Adaptation to DiTs, ensuring that multiple editing tasks can coexist without destructive interference.

2.3. Parameter-Efficient Fine-Tuning in Vision

Fine-tuning large-scale foundation models for downstream tasks is computationally expensive and storage-intensive. Parameter-Efficient Fine-Tuning (PEFT) methods, such as Adapter, Prefix Tuning, and Low-Rank Adaptation (LoRA) [5], have been widely adopted to mitigate this cost. LoRA, in particular, has become the de facto standard for efficient adaptation. It freezes the pre-trained weights $W_0$ and injects trainable low-rank decomposition matrices A and B, such that $\Delta W=BA$, making the number of trainable parameters a tiny fraction of the original model size. Previous works have explored various conditioning mechanisms, such as T2I-Adapter [26] and ControlNet++ [27], to guide generation with spatial maps (e.g., edges, depth, pose). SDEdit [28], Prompt-to-Prompt [29], and InstructPix2Pix [30] laid the foundation for instruction-based editing, allowing users to modify images via natural language commands. BrushNet [31] further improved inpainting capabilities with dual-branch diffusion, separating the mask processing from the image generation.

In the context of multi-task learning, however, standard LoRA faces significant challenges. Merging multiple task-specific LoRA adapters often leads to performance degradation due to parameter interference, where the gradient updates for one task conflict with those of another. To address this, recent works have proposed improved variants. MTLoRA [32] introduces task-agnostic and task-specific modules to disentangle parameter spaces in hierarchical transformers. UIR-LoRA [33] utilizes multiple low-rank adapters for universal image restoration, employing a composition strategy based on degradation similarity. While these methods improve multi-task performance, they do not explicitly enforce orthogonality between task representations. Our proposed EditLoRI builds upon the concept of orthogonal LoRA [7] from NLP, adapting it to the vision domain. By mathematically projecting task gradients into orthogonal subspaces, EditLoRI eliminates cross-task interference, enabling precise, interference-free multi-task editing for complex knitted structures.

3. Method

3.1. Problem Formulation: Addressing Texture Collapse

We define the core challenge in multi-task knitted garment editing as “Texture Collapse”, a phenomenon where the simultaneous application of multiple editing conditions leads to the degradation of fine-grained structural details. In a standard diffusion model fine-tuning setting, let $\theta$ be the model parameters. Given N distinct editing tasks $\{T_1,\dots ,T_N\}$ (e.g., $T_1$: add logo, $T_2$: Virtual Try-On), we aim to learn task-specific parameter increments $\{\Delta {\theta }_1,\dots ,\Delta {\theta }_N\}$. Ideally, the composed model ${\theta }_{combined}=\theta +\sum {\alpha }_i\Delta {\theta }_i$ should be capable of performing all tasks simultaneously. However, due to the highly overlapping feature space of knitted textures—which are characterized by high-frequency repetitive patterns and strict topological constraints—the parameter updates for different tasks often conflict. Mathematically, this implies that the inner product of the update vectors is non-zero: $⟨\Delta {\theta }_i,\Delta {\theta }_j⟩\ne 0$. This interference results in the averaging or smoothing of topological details, which we term Texture Collapse. Our goal is to enforce orthogonality $⟨\Delta {\theta }_i,\Delta {\theta }_j⟩\approx 0$ to decouple task representations, ensuring that the learning of one task does not negatively impact the performance of another.

3.2. Framework Overview

Our framework, shown in Figure 2 is built upon FLUX.1 Kontext [4], a state-of-the-art flow matching-based [34] Diffusion Transformer. Unlike traditional U-Net architectures that process images via local convolutions, FLUX.1 processes images as flattened sequences of patches, allowing for global context interaction via self-attention layers. This architecture is particularly well-suited for our purpose as it allows for more flexible spatial conditioning. We integrate two key components into this backbone: (1) EditLoRI for orthogonal task decoupling, which ensures that multiple adapters can be merged without interference; and (2) Bounding Box Guidance for precise spatial control, which resolves the ambiguity of text prompts.

Figure 2. Overview of Knit-Edit. We inject extra trainable modules into the multi-layer Transformers of the base model to achieve accurate editing for knitted garment tasks. The framework processes an input image, a reference image, and text instructions to perform editing. We freeze the original parameters during training and solely update the EditLoRI adapters. At the inference stage, the model combines these trained adapters through weighted fusion to enable multi-task editing.

3.3. EditLoRI: Orthogonal Subspace Adaptation

Standard Low-Rank Adaptation (LoRA) approximates the weight update $\Delta W$ as the product of two low-rank matrices B and A:

$$W^{\prime }=W_0+\Delta W=W_0+BA,$$

(1)

where $W_0\in {\mathbb{R}}^{d\times k}$ is the frozen pre-trained weight, $B\in {\mathbb{R}}^{d\times r}$ initializes to zero, and $A\in {\mathbb{R}}^{r\times k}$ initializes with Gaussian noise. While efficient, this formulation does not constrain the direction of the update $\Delta W$, leading to potential conflicts when multiple adapters are summed.

To prevent texture collapse in multi-task scenarios, we adapt the LoRI mechanism [7] for visual generation. While the original LoRI operates on 1D sequence features for NLP tasks, our EditLoRI is adapted for the 2D spatial features of visual generation. This dimensional distinction is crucial, as EditLoRI’s orthogonal constraints are specifically designed to preserve the topological structure of knitted garments against texture collapse, a challenge unique to the high-dimensional visual space. EditLoRI imposes a strict structural constraint on the adapters to enforce orthogonality. For each task i, the update is defined as:

$$\Delta W_i=(B_i\odot M_i)A_i,$$

(2)

Crucially, $A_i$ is a frozen random orthogonal projection matrix. This matrix is shared or distinct across tasks but remains fixed during training. $B_i$ is the trainable low-rank matrix, and $M_i$ is a binary sparsity mask. By fixing A to be orthogonal and enforcing sparsity in B, we mathematically encourage the learned task vectors to reside in orthogonal subspaces of the optimization landscape. Intuitively, this means that each task learns to manipulate a different subset of the feature space, preventing them from stepping on each other’s toes.

During inference, multiple editing tasks are composed linearly:

$$W_{final}=W_0+\sum _{i=1}^N{\lambda }_i(B_i\odot M_i)A_i,$$

(3)

where ${\lambda }_i$ is a scalar weight controlling the intensity of task i. Because the updates $\Delta W_i$ are constrained to be approximately orthogonal, the interference term $\Delta W_i^T\Delta W_j$ is minimized. This ensures that editing the garment’s pattern (Task A) does not degrade the fabric’s topological structure (Task B), effectively solving the texture collapse problem.

EditLoRI enforces orthogonality to isolate these conflicting gradients. This is particularly critical for knitted garments, where visual identity relies on high-frequency micro-structures. By minimizing cross-task conflict, EditLoRI allows for the seamless composition of distinct editing capabilities while preserving crisp topological details. To address the complex requirements of industrial design, we trained three independent adapters for garment editing, Virtual Try-On, and fabric editing, and then merged them with learned weights to form a unified multi-task model, as illustrated in Figure 3.

Figure 3. EditLoRI architecture. Each task has an independent LoRA-structured adapter sharing a frozen projection matrix A. The model combines these adapters to execute various editing tasks.

3.4. Bounding Box Guidance as Spatial Prompts

Knitted garments require pixel-perfect editing precision. Text descriptions like “add a logo on the chest” are often too vague for industrial standards, leading to misplacement or incorrect scaling. To address this, we introduce Bounding Box Guidance not merely as a visual overlay, but as a Spatial Prompt that directly interacts with the model’s attention mechanism.

As shown in Figure 4, during training, we augment the input image with colored Bounding Boxes defining the Region of Interest (ROI). These boxes are tokenized by the VAE encoder along with the image. Since FLUX.1 uses a Transformer backbone, these “box tokens” are treated as additional input tokens that interact globally with “image tokens” via self-attention layers. The model learns to attend specifically to the pixels enclosed by the box coordinates $(x_{min},y_{min},x_{max},y_{max})$ when conditioning on specific text instructions (e.g., “in the red box”).

Figure 4. Schematic of the Bounding Box positioning mechanism. We add Bounding Boxes to the input image to guide the model in locating the target area and content. Text prompts use box colors to distinguish between them. Unlike methods that need extra conditions, we use the model’s own understanding of Bounding Boxes. This allows accurate edit positioning and content mapping without extra cost.

This strategy effectively injects strong spatial inductive biases into the DiT. Unlike mask-based inpainting, which discards information inside the mask, our approach preserves the underlying fabric texture while modulating the semantic content, guided by the EditLoRI adapters. This allows for precise, localized editing that respects the global geometry of the garment.

3.5. Dataset and Training Strategy

To support this framework, we constructed the KnitEdit dataset (detailed in Section 4). We process real images by cropping or collecting references. For generated images, we use knitted garment templates combined with high-definition references (Figure 5). We also designed a pipeline to generate Bounding Box annotations automatically. During training, we use a balanced data loading strategy, selecting half the samples from real data and half from generated data in each batch. This compensates for the scarcity of real data and enhances the model’s generalization ability.

Figure 5. Data processing for the KnitEdit dataset. (a) Processing for real images. (b) Processing for generated images. (c) Bounding Box generation pipeline using image differences.

4. Experiments

4.1. Setup

4.1.1. Datasets

To rigorously evaluate our framework, we utilized two primary datasets for training and evaluation:

• KnitEdit Dataset: We constructed a large-scale, high-fidelity dataset specifically tailored for knitted garment editing, comprising 6000 paired samples. Recognizing the difficulty of obtaining perfectly paired real-world data (i.e., the same garment with and without a specific edit), we leveraged a high-fidelity rendering pipeline to generate 5800 synthetic samples. These samples were created using advanced 3D knitting simulation software, ensuring that they follow “ideal physics” of knitting structures without noise interference. They cover a wide range of knitting patterns (e.g., cable, rib, jersey, intarsia). Additionally, to bridge the domain gap and evaluate real-world performance, we collected 200 high-resolution real-world images. Each sample is meticulously annotated with Bounding Boxes defining the editing regions and corresponding text instructions.
• VITON-HD [9]: For the Virtual Try-On task, we employed the standard VITON-HD dataset, which contains 13,679 image pairs of frontal-view women and upper-body clothing at $1024\times 768$ resolution. We followed the standard train/test split. Although this dataset focuses on human-centric images, unlike the garment-centric KnitEdit dataset, incorporating it allows our unified framework to learn a generalized representation that handles both texture synthesis and spatial deformation tasks, proving the versatility of our approach.

4.1.2. Implementation Details

Our model is implemented in PyTorch 2.8.0 + cu129 and trained on a single NVIDIA GeForce RTX 4090 GPU (24 GB VRAM). We used the FLUX.1 Kontext [4] model as the pre-trained backbone due to its superior flow matching capabilities. The EditLoRI adapters were initialized with a rank $r=8$ and a sparsity ratio of $M_i=0.0$, which we found to be the optimal balance between expressivity and efficiency. We employed the Prodigy optimizer [35], an adaptive method that automatically adjusts the learning rate, starting with an initial learning rate of 1.0 and a weight decay of 0.01 to prevent overfitting. The training was conducted for 3000 steps with a batch size of 1 and gradient accumulation steps of 4, resulting in an effective batch size of 4. Input images were resized to $1024\times 1024$ (or $768\times 1024$ for VITON-HD) to match the backbone’s native resolution.

4.1.3. Baselines

We compared our method with a comprehensive set of state-of-the-art baselines to benchmark its performance. For the Add Symbols task, we compared against ControlNet [6], IP-Adapter [36], and standard FLUX.1 inpainting variants. For the Virtual Try-On task, we compared with SD-VTON [37], StableVITON [38], and D⌃4-VTON [39]. For the Remove Item task, we compared with FLUX.1 variants. All DiT-based methods were evaluated using Q8_0 quantization to ensure a fair comparison of inference efficiency and quality.

4.1.4. Evaluation Metrics

Following the established evaluation protocol of OminiControl [40], we assess our method from three complementary aspects: Controllability, Image Quality, and Alignment. For Controllability, we use Mean Squared Error (MSE) calculated only on the non-edited regions. This metric strictly measures how well the model preserves the original structure of the garment outside the editing ROI. A lower MSE indicates better structural preservation. For Image Quality, we employ widely used metrics including Fréchet Inception Distance (FID) [41], Structural Similarity Index (SSIM), and Peak Signal-to-Noise Ratio (PSNR). Additionally, we use advanced perceptual metrics like CLIP-IQA [42], MAN-IQA [43], and MUSIQ [44] to assess the visual realism and aesthetic quality of the generated images. For Alignment, we use the CLIP Image score [45] to measure the semantic consistency between the generated image and the text instruction/reference image.

4.2. Main Result

4.2.1. Quantitative Results

Table 1 presents our quantitative comparison across the three editing tasks. Our method achieves robust performance across all metrics, maintaining a superior balance between controllability and quality.

Table 1. Quantitative comparison with baseline methods. We evaluate Controllability, Image Quality, and Alignment. The arrows indicate the direction of better performance (↓ lower is better, ↑ higher is better). Best results are in bold, second-best are underlined. Our method achieves the best overall performance.

Task	Methods	Image	Controllability	Image Quality						Alignment
		Condition(s)	MSE ↓	FID ↓	SSIM ↑	CLIP-IQA ↑	MAN-IQA ↑	MUSICQ ↑	PSNR ↑	CLIP Image ↑
Add Symbols	ControlNet [6]	2	1870	48.65	0.67	0.57	0.48	70.04	18.11	0.75
	IP-Adapter [36]	2	1844	50.32	0.56	0.62	0.48	70.66	12.63	0.80
	FLUX.1 [46]	2	2828	49.36	0.68	0.46	0.48	67.94	19.49	0.68
	FLUX.1 Kontext [4]	2	1450	20.48	0.84	0.59	0.59	73.83	17.10	0.89
	Ours	2	1021	5.94	0.89	0.61	0.59	73.41	20.79	0.96
Virtual Try-On	SD-VTON [37]	4	1413	26.62	0.83	0.51	0.43	73.41	13.94	0.94
	StableVITON [38]	4	2566	23.89	0.78	0.42	0.60	75.55	13.99	0.96
	D⌃4-VTON [39]	4	1972	21.41	0.79	0.46	0.59	76.51	14.00	0.96
	FLUX.1 Kontext	2	1933	103	0.84	0.60	0.49	72.76	13.65	0.72
	Ours	2	1951	28.76	0.82	0.54	0.51	75.68	13.70	0.95
Remove Item	FLUX.1	1	5501	345.71	0.42	0.58	0.67	75.21	10.64	0.56
	FLUX.1 Kontext	2	328	7.56	0.89	0.54	0.50	68.85	21.41	0.98
	Ours	2	279	3.01	0.89	0.56	0.53	70.02	21.69	0.99

In the Add Symbols task, our MSE drops to 1021 and FID to 5.94, achieving the best combination of metrics. While IP-Adapter has a slightly higher CLIP-IQA (0.62 vs. 0.61), its overall quality (FID 50.32) and Alignment are significantly lower. This discrepancy suggests that while IP-Adapter generates aesthetically pleasing images, it often fails to respect the original structure of the garment, introducing hallucinations or style shifts that deviate from the input.

In the Virtual Try-On task, D⌃4-VTON performs exceptionally well, which is expected given it is a specialized model designed solely for this task. However, our solution, requiring only 2 condition inputs (image + box) versus D⌃4-VTON’s 4, achieves comparable results (CLIP Image 0.95 vs. 0.96, MUSIQ 75.68 vs. 76.51) while maintaining task universality. This demonstrates that our unified framework does not compromise performance for versatility.

In the Remove Item task, our method achieves the best MSE (279) and FID (3.01). FLUX.1, by comparison, suffers from significant quality degradation (FID 345.71), often introducing artifacts or failing to fill the removed region coherently. Overall, our solution demonstrates stable capabilities without adding extra conditions.

4.2.2. Qualitative Results

Figure 6 displays the qualitative results. Our model successfully adds target symbols while preserving the original knitting structure, showing a clear advantage over baselines, which often blur the yarn details. In the Add Symbols row, we can observe that ControlNet tends to darken the image, while IP-Adapter loses the texture of the sweater entirely. In contrast, Knit-Edit seamlessly integrates the logo. In Virtual Try-On, it restores the garment structure better than SD-VTON and StableVITON, particularly in handling complex necklines and sleeve patterns. For Remove Item, our model cleanly removes objects without leaving “ghosting” artifacts or extra wrinkles, generating images that meet user needs visually.

Figure 6. Qualitative comparison. Our model performs better visually. For Add Symbols, it restores color and brightness effectively. For Virtual Try-On, it maintains garment fidelity. For Remove Item, it cleans up details better than FLUX.1 and other baselines.

4.3. Ablation Studies

4.3.1. Effectiveness of Core Components

Table 2 details the ablation results to isolate the contribution of each module. With neither EditLoRI nor Bounding Box guidance (i.e., standard LoRA with text prompts), performance is suboptimal (MSE 1450, FID 20.49), indicating that the baseline FLUX.1 model struggles with precise industrial editing. Enabling EditLoRI improves metrics significantly, dropping FID to 9.82. This confirms that orthogonal adaptation is crucial for maintaining quality when learning new tasks. Adding the Bounding Box mechanism further optimizes the results, with MSE dropping to 991 and FID to 5.90. This confirms that the synergy of positioning constraints and low-rank editing is key to our performance: EditLoRI handles the texture, while Bounding Boxes handle the geometry.

Table 2. Ablation study on EditLoRI and Bounding Box guidance. The green check mark indicates the component is enabled, and the red cross indicates it is disabled. The arrows indicate the direction of better performance (↓ lower is better, ↑ higher is better). Best results are in bold. The network achieves optimal performance when all components are enabled.

Methods		Controllability	Image Quality						Alignment
EditLoRI	Bounding Box	MSE ↓	FID ↓	SSIM ↑	CLIP-IQA ↑	MAN-IQA ↑	MUSIQ ↑	PSNR ↑	CLIP Image ↑
✗	✗	1450	20.49	0.84	0.59	0.59	73.83	17.10	0.89
✔	✗	1168	9.82	0.88	0.59	0.57	73.11	20.19	0.93
✔	✔	991	5.90	0.89	0.60	0.58	73.11	20.77	0.96

4.3.2. Comparison Between LoRA and EditLoRI on Multiple Adapters

Table 3 compares EditLoRI with the standard Merged LoRA method. EditLoRI outperforms Merged LoRA across most metrics. For example, in the Add Symbols task, EditLoRI achieves an FID of 5.94 compared to 35.95 for Merged LoRA. This validates that the orthogonal constraints in EditLoRI effectively mitigate parameter interference. When standard LoRA adapters are merged, their weight updates sum linearly, often pushing the model weights into a “confused” state where neither task is performed well. EditLoRI’s orthogonality ensures that the tasks remain disentangled in the parameter space, enabling high-quality multi-task editing. Figure 7 provides visual confirmation, where Merged LoRA shows clear signs of texture collapse.

Table 3. Comparison between EditLoRI and Merged LoRA. The arrows indicate the direction of better performance (↓ lower is better, ↑ higher is better). Best results are in bold. EditLoRI outperforms Merged LoRA in all metrics.

Task	Methods	Controllability	Image Quality						Alignment
		MSE ↓	FID ↓	SSIM ↑	CLIP-IQA ↑	MAN-IQA ↑	MUSIQ ↑	PSNR ↑	CLIP Image ↑
Add Symbols	Merged LoRA	1222	35.95	0.84	0.56	0.52	69.45	22.45	0.83
	EditLoRI	1021	5.94	0.89	0.61	0.59	73.41	20.79	0.96
Virtual Try-On	Merged LoRA	2273	51.83	0.81	0.52	0.45	75.08	15.16	0.87
	EditLoRI	1951	28.76	0.82	0.54	0.51	75.68	13.70	0.95
Remove Item	Merged LoRA	672	27.40	0.85	0.57	0.52	69.30	21.23	0.95
	EditLoRI	279	3.01	0.89	0.56	0.53	70.02	21.69	0.99

Figure 7. Qualitative comparison between EditLoRI and LoRA. Both perform well individually, but Merged LoRA exhibits severe texture distortion when fusing multiple adapters, likely due to conflict. EditLoRI maintains high quality.

4.3.3. Impact of Training Steps

Figure 8 illustrates the impact of training steps on model performance. We observe a clear trade-off between Image Quality (FID) and Text Alignment (CLIP Image Score).

Figure 8. Impact of training steps on Image Quality (FID) and alignment (CLIP Image). The model converges rapidly around 3000 steps, achieving the optimal balance between fidelity and control. Extended training (6000 steps) yields marginal gains in FID but risks overfitting, as indicated by the plateauing CLIP score.

Performance improves significantly from 1000 to 3000 steps, with FID dropping from 14.34 to 5.90. Beyond 3000 steps, the gains diminish, and the CLIP Image score stabilizes around 0.965. Balancing computational cost and convergence, we selected 3000 steps as the optimal stopping point. Extended training risks overfitting to the synthetic data distribution, potentially reducing generalization to real-world images.

4.3.4. Impact of LoRA Rank

Table 4 analyzes the effect of LoRA rank on performance. Rank 8 achieves the best balance between Controllability, Image Quality, and Alignment. Lower ranks (e.g., 4) lack the capacity to learn complex knitting patterns, resulting in higher FID. Higher ranks (e.g., 32) improve MSE slightly but compromise subjective quality (MUSIQ), likely due to overfitting the training set. Therefore, we adopt rank 8 as the default setting for all experiments.

Table 4. Ablation study on LoRA ranks. The arrows indicate the direction of better performance (↓ lower is better, ↑ higher is better). Best results are in bold, second-best are underlined. Rank 8 achieves the best overall performance.

Rank	Controllability	Image Quality						Alignment
	MSE ↓	FID ↓	SSIM ↑	CLIP-IQA ↑	MAN-IQA ↑	MUSIQ ↑	PSNR ↑	CLIP Image ↑
4	1030	7.17	0.89	0.59	0.58	73.24	20.20	0.95
8	1021	5.94	0.89	0.61	0.59	73.41	20.79	0.96
16	1020	7.35	0.89	0.59	0.58	73.20	20.36	0.95
32	991	5.90	0.89	0.60	0.58	73.11	20.77	0.96

4.3.5. Layerwise Update Magnitude Analysis

To validate the efficiency of our EditLoRI method, we analyze the update magnitude of the learnable parameters across different layers. The core hypothesis is the “Minimal Intervention Principle”: an effective model editing method should achieve the target effect with minimal parameter modifications, thereby maximally preserving the original knowledge of the pre-trained model.

We calculated the Frobenius norm of the weight update $\Delta W$ for each layer, where $\Delta W=BA$ for Standard LoRA and $\Delta W=(B\odot M)A$ for EditLoRI. Figure 9 illustrates the layerwise comparison.

Figure 9. Layerwise comparison of parameter update magnitudes (Norm of $\Delta W$). The dots represent the raw data points (specific interference magnitude) for each Transformer block. The blue line (EditLoRI) remains consistently lower than the red line (Standard LoRA) across most layers, demonstrating the “minimal intervention” property.

The quantitative results demonstrate a significant advantage for EditLoRI, which maintains a lower average update norm of 6.36 (maximum 14.26) compared to Standard LoRA’s average of 8.04 (maximum 17.63). EditLoRI achieves approximately 20% lower update magnitude compared to Standard LoRA. This indicates that EditLoRI completes the editing task in a more efficient and “lightweight” manner. The reduced parameter variation suggests higher parameter utilization and, crucially, less destructive interference with the original model weights, aligning with the goal of precise and minimal editing.

4.3.6. Interference Magnitude Analysis

To further substantiate the orthogonal superiority of EditLoRI, we analyzed the “Interference Magnitude” between tasks. We calculated the dot product of the Delta Weight updates between different tasks, denoted as $|⟨\Delta W_i,\Delta W_j⟩|$. This metric comprehensively reflects both the directional orthogonality and the magnitude of the updates, directly quantifying the potential interference one task’s modification imposes on another.

The analysis reveals that EditLoRI excels in minimizing inter-task interference, achieving an average interference magnitude of 0.0524, compared to 0.0847 for Standard LoRA. EditLoRI reduces inter-task interference by 38.12%. This implies that the updates generated by EditLoRI are not only more “lightweight” but also less conflicting in the parameter space. Such low-interference characteristics are a key advantage for model merging and continual learning scenarios, proving that EditLoRI effectively decouples the knowledge representation of different tasks.

Figure 10 visually presents the interference magnitude across each Transformer block. It is clearly observable that EditLoRI (blue line) maintains significantly lower interference levels than Standard LoRA (red line) across the vast majority of layers.

Figure 10. Layerwise comparison of interference magnitudes ($|⟨\Delta W_i,\Delta W_j⟩|$). The dots represent the raw data points (specific interference magnitude) for each Transformer block. EditLoRI (blue) consistently exhibits lower interference compared to Standard LoRA (red), validating its effectiveness in task decoupling.

5. Conclusions

In this work, we presented Knit-Edit, a unified and versatile framework that addresses the critical challenge of “Texture Collapse” in generative manufacturing. By introducing EditLoRI, we mathematically enforced the orthogonality of task-specific adapters, allowing the model to decouple and recompose multiple editing capabilities—such as pattern modification and Virtual Try-On—without destructive interference. This mechanism ensures that the learning of semantic attributes does not compromise the underlying topological structure of the fabric. Furthermore, our spatial prompting strategy, enabled by Bounding Box tokenization, bridges the gap between vague textual commands and the precise localization required by industrial standards. Validated on the comprehensive KnitEdit dataset, our approach demonstrates a significant leap in both structural fidelity and multi-task controllability compared to existing diffusion baselines, offering a robust solution for the transition from mass production to mass customization.

It is worth noting that, despite the reliance on synthetic training data, our qualitative evaluations on the real-world samples indicate that the model generalizes effectively to most common manufacturing scenarios, producing satisfactory results for standard fabric textures and lighting conditions. However, several limitations warrant honest discussion. First, our training relies heavily on synthetic data. While this data provides clean “ideal physics” for learning knitting structures, it does not fully capture the complex lighting, material aging, and physical imperfections encountered in real-world photography. This potentially creates a sim-to-real gap during inference, where the model might struggle with highly irregular or worn fabrics. Second, our evaluation is primarily technical, relying on metrics like FID and CLIP scores. While these metrics correlate with quality, they do not fully capture the nuanced preferences of professional fashion designers. We have not yet conducted extensive user studies to assess the tool’s integration into actual production workflows or its ease of use for non-technical practitioners. Finally, the current framework is strictly specialized for the domain of knitted garments. The applicability of EditLoRI’s orthogonal projection to other textile types with different topological constraints or complex multi-material assemblies remains an open question.

Future work will focus on bridging these gaps. We aim to incorporate domain adaptation techniques to enhance robustness on real-world images, potentially by leveraging unsupervised domain adaptation or collecting a larger real-world dataset. We also plan to collaborate with fashion designers to refine the interaction interface, exploring more intuitive control mechanisms such as sketch-based guidance or interactive segmentation. Additionally, we will explore extending the orthogonal adaptation mechanism to a broader range of fabric textures, moving towards a truly universal intelligent design system. Furthermore, we intend to conduct rigorous quantitative analyses, such as measuring the cosine similarity between task-specific subspaces, to theoretically validate the orthogonality of our EditLoRI mechanism and further optimize the subspace projection.