Knit-FLUX: Simulation of Knitted Fabric Images Based on Low-Rank Adaptation of Diffusion Models

Abstract

Generative model-assisted design has become a trend, providing a new paradigm for knitted fabric image generation. The FLUX diffusion model was chosen to generate images in this study and was compared to other generative models. In order to effectively apply the large model to specialized verticals, an efficient fine-tuning method, low-rank adaptation, was used. Experiments showed that the method allows a pre-trained model to stably generate knitted fabric images in batches through easily understandable text prompts. The generated images have clear textures and correct structures, and can display the surface characteristics of knitted fabrics generated by using different yarn specifications and yarn bristles. Moreover, the unit tissue structural similarity index measure (SSIM) is 0.6528, which is very similar to real fabrics. This research expands the application of fabric generation in the field of deep learning. This method is highly efficient, low-cost, and capable of stably simulating knitted fabrics, which can be used to rapidly expand the image design materials of knitted fabrics.

1. Introduction

Computer model-assisted design generation in professional fields is gradually becoming a trend [1]. Classical generative models [2] mainly include Variational Auto-Encoder (VAE), Generative Adversarial Networks (GANs) and Stable Diffusion.

VAE [3] is composed of an encoder and a decoder. The training process involves encoding the input data to obtain a low-dimensional vector and using a decoder to reduce the output data, which leads to low fidelity and high diversity in the training results. With the development of deep learning, VAE is often combined with other generative models [4,5] for apparel generation, blemish detection in textile images [6,7] and yarn recognition and classification [8]. During the training process, VAE is more responsible for the compression and reconstruction of high-resolution images into the low-dimensional potential space, thus enabling other generative models to train and reason more efficiently in the potential space.

In recent years, diffusion models beat GAN [9] to generate high-resolution images from natural language inputs with strong graph generation and generalization capabilities [10]. However, due to its iterative nature, high computational cost, and the problem of there being a long sampling time during inference, many studies have explored the optimization of model performance to improve the sampling efficiency with enhanced training effectiveness [11,12,13]. The stable diffusion model 3 (SD3) [14] innovatively employs the Flow Matching [15] framework to directly accomplish noise-to-data mapping by constructing a continuous-time probabilistic flow, compressing the generation step to 10–20 steps. The model further incorporates the Multimodal Diffusion Transformer Backbone (MM-DiT) [16] and introduces rotational position coding [17] to improve the model performance and enable hardware utilization [18].

Based on this, the FLUX.1 Dev (FLUX) [19] integrates the MM-DiT architecture with Single Transformer Blocks (Single-DiT), which employs parallel attention layers [20], effectively increasing the model’s parameter count to 12 billion and further enhancing the overall performance.

Although the diffusion model already has a strong generalization ability for generating simple and clear fabric textures, it is not able to generate information on microscopic details such as the specific loop structure of yarns in the fabric. This makes it necessary for the design generation of knitted fabrics to integrate process constraints to meet actual design requirements. Parameter-efficient fine-tuning (PEFT) aims to customize diffusion models by training a small number of parameters. After an experimental comparison of 15 different PEFT methods in terms of performance and efficiency [21], low-rank adaptation [22] requires only 0.1% parameter updates to complete the fine-tuning, which significantly reduces the number of parameters and the computational cost, and can provide near-full fine-tuning in resource-constrained environments. The method has achieved better results in clothing style fine-tuning [23], blue print pattern style [24] learning, and woven fabric generation [25].

In summary, the application of diffusion modeling in the textile field focuses on the design generation of styles and patterns [26], repair migration [27] and virtual fitting [28], and less on the research and application of fabric simulation generation. As the curved coils are strung together to form knitted fabrics, the organizational structure is more complex compared to woven fabrics, so the research in the field of fabrics is more focused on the identification and generation of woven fabrics [29]. However, the image generation of knitted fabrics has important application value in many fields such as the digital design of clothing and textile industry applications, but lacks research in the field of intelligent generation. Based on this, this study applies a low-rank adaptation of the FLUX model to generate knitted fabric simulation by means of text-generated image. The contributions of this paper are as follows:

The low-rank adaptation of the FLUX model supports the fast batch generation of knitted fabric images with easy-to-understand textual prompts, and the generated images have clear textures and correct structural features.

We conducted a comparative analysis of the model before and after fine-tuning, as well as between the FLUX and SD1.5 diffusion models. Structural similarity evaluation at the unit organization level demonstrates that the results generated by Knit-FLUX (Low-Rank Adaptation of FLUX model) closely approximate real knitted fabrics compared to Knit-Diff (Low-Rank Adaptation of SD1.5 model).

This study extends the application of knitted fabric image generation in the field of deep learning, and the method is efficient, low-cost, and capable of stably simulating knitted fabrics, which can be used to rapidly expand knitted fabric design materials.

The following are the remaining four sections of this paper. Section 2 provides a detailed theoretical overview of diffusion model concepts and flow-matching diffusion concepts. Section 3 introduces the flow-matching diffusion model architecture and low-rank adaptive fine-tuning methods. Section 4 introduces the dataset, experimental environment, parameter settings, experimental results, and evaluation metrics, and analyzes the experimental results. Section 5 summarizes the entire paper and proposes plans for further research in this field.

2. Theory

Diffusion models are generative probabilistic models that learn to reconstruct data through iterative denoising steps. Recently, Flow Matching has emerged as an efficient training paradigm that constructs a continuous-time probabilistic flow to directly learn the data-to-noise mapping, significantly reducing the number of inference steps. In this section, we provide the theoretical foundation of diffusion modeling, followed by an explanation of the Flow Matching framework used in the FLUX model, which forms the basis of our proposed approach.

2.1. Diffusion Model

The diffusion model [30] is based on the denoising probabilistic framework, which is realized through step-by-step iterations from Gaussian noise to target image generation. As shown in Figure 1, the diffusion model structure realizes the diffusion process through U-Net structure and temporal coding. The diffusion process includes forward diffusion and backward denoising.

During the forward diffusion process, the model gradually adds Gaussian noise to the original data ${\mathrm{z}}_0$, which satisfies the initial distribution $\mathrm{q}\left({\mathrm{z}}_0\right)$, ${\mathrm{z}}_0~\mathrm{q}\left({\mathrm{z}}_0\right)$. For $\mathrm{t}\in \left[1,\mathrm{T}\right]$, ${\mathrm{z}}_{\mathrm{t}}$ and ${\mathrm{z}}_{\mathrm{t}-1}$ satisfy the following equation [31]:

$${\mathrm{z}}_{\mathrm{t}}=\sqrt{{\mathsf{\alpha }}_{\mathrm{t}}}{\mathrm{z}}_{\mathrm{t}-1}+\sqrt{1-{\mathsf{\alpha }}_{\mathrm{t}}}\mathsf{\epsilon },\mathsf{\epsilon }\sim \mathrm{N}\left(0,1\right).$$

(1)

${\mathsf{\beta }}_{\mathrm{t}}$ is the noise variance added at each step of the forward process, which is usually a predefined incremental sequence controlling the rate at which the data is corrupted. ${\mathsf{\alpha }}_{\mathrm{t}}$ is the proportionality coefficient for retaining the original signal, defined as ${\mathsf{\alpha }}_{\mathrm{t}}=\left(1-{\mathsf{\beta }}_{\mathrm{t}}\right)$, which represents the proportion of the original data retained at each step. The noise intensity at each time step $\mathrm{t}$ is controlled by the noise attenuation coefficient ${\mathsf{\alpha }}_{\mathrm{t}}$ and the noise variance ${\mathsf{\beta }}_{\mathrm{t}}$. Changing the scheduling rule of ${\mathsf{\beta }}_{\mathrm{t}}$ can control the rhythm of noise addition and balance the efficiency of signal retention and noise addition. The recursive derivation is obtained from the initial data at step $\mathrm{t}$:

$$\begin{array}{c}{\mathrm{z}}_{\mathrm{t}}=\sqrt{{\overline{\mathsf{\alpha }}}_{\mathrm{t}}}{\mathrm{z}}_0+\sqrt{1-{\overline{\mathsf{\alpha }}}_{\mathrm{t}}}\mathsf{\epsilon },\mathsf{\epsilon }\sim \mathrm{N}\left(0,1\right),\mathrm{t}\in \left(1,\mathrm{T}\right)\end{array},$$

(2)

where ${\overline{\mathsf{\alpha }}}_{\mathrm{t}}$ is the cumulative attenuation coefficient, which is the cumulative product of ${\mathsf{\alpha }}_{\mathrm{t}}$.

In the reverse denoising process, the diffusion model needs to predict the previous moment’s picture ${\mathrm{z}}_{\mathrm{t}-1}$ based on the current moment’s picture ${\mathrm{z}}_{\mathrm{t}}$. This prediction process uses the probabilistic Bayesian formula to calculate the posterior probability:

$$\begin{array}{c}\mathrm{P}\left(\left.{\mathrm{z}}_{\mathrm{t}-1}\right|{\mathrm{z}}_{\mathrm{t}}\right)=\mathrm{N}\left({\displaystyle \frac{1}{\sqrt{{\mathsf{\alpha }}_{\mathrm{t}}}}}\left({\mathrm{z}}_{\mathrm{t}}-{\displaystyle \frac{1-{\mathsf{\alpha }}_{\mathrm{t}}}{\sqrt{1-{\overline{\mathsf{\alpha }}}_{\mathrm{t}}}}}\mathsf{\epsilon }\right),{\displaystyle \frac{\left(1-{\mathsf{\alpha }}_{\mathrm{t}}\right)\left(1-{\overline{\mathsf{\alpha }}}_{\mathrm{t}-1}\right)}{1-{\overline{\mathsf{\alpha }}}_{\mathrm{t}}}}\right)\end{array}.$$

(3)

The denoising process [30] is obtained after derivation:

$$\begin{array}{c}{\mathsf{\mu }}_{\mathsf{\theta }}\left({\mathrm{z}}_{\mathrm{t}},\mathrm{t}\right)={\displaystyle \frac{1}{\sqrt{{\mathsf{\alpha }}_{\mathrm{t}}}}}\left({\mathrm{z}}_{\mathrm{t}}-{\displaystyle \frac{1-{\mathsf{\alpha }}_{\mathrm{t}}}{\sqrt{1-{\overline{\mathsf{\alpha }}}_{\mathrm{t}}}}}{\mathsf{\epsilon }}_{\mathsf{\theta }}\left({\mathrm{z}}_{\mathrm{t}},\mathrm{t}\right)\right)\end{array}.$$

(4)

During the training process, the final optimization goal is to make the network-predicted noise consistent with the real noise, so the loss function [30] needs to be as small as possible; this is given by the following:

$${\min }_{\mathsf{\theta }}\mathcal{L}\left(\mathsf{\theta }\right)={\mathrm{E}}_{\mathrm{t},{\mathrm{z}}_0,\mathsf{\epsilon }}\left[{\left(\mathsf{\epsilon }-{\mathsf{\epsilon }}_{\mathsf{\theta }}\left(\sqrt{{\overline{\mathsf{\alpha }}}_{\mathrm{t}}{\mathrm{z}}_0}+\sqrt{1-{\overline{\mathsf{\alpha }}}_{\mathrm{t}}\mathsf{\epsilon }},\mathrm{t}\right)\right)}^2\right].$$

(5)

2.2. Conditional Flow Matching

The FLUX diffusion model is based on the diffusion model and uses Conditional Flow Matching [32] (CFM) theory to improve the denoising sampling process. It transforms the sampling process of data distribution into a trajectory solution problem of Ordinary Differential Equation (ODE) by constructing a continuous time probabilistic flow field. As shown in Figure 2, conditional flow matching constructs a conditional probabilistic path from the source distribution to each target data point as ${\mathrm{P}}_{\mathrm{t}|{\mathrm{x}}_1}\left(\mathrm{x}\right)$, which satisfies ${\mathrm{P}}_{0|{\mathrm{x}}_1}\left(\mathrm{x}\right)={\mathrm{P}}_0\left(\mathrm{x}\right)$ and ${\mathrm{P}}_{1|{\mathrm{x}}_1}\left(\mathrm{x}\right)=\mathsf{\delta }\left(\mathrm{x}-{\mathrm{x}}_1\right)$. The Gaussian distribution is chosen as the conditional path [15]:

$$\begin{array}{c}{\mathrm{P}}_{\mathrm{t}|{\mathrm{x}}_1}\left(\mathrm{x}\right)=\mathrm{N}\left(\mathrm{x};{\mathsf{\mu }}_{\mathrm{t}}\left({\mathrm{x}}_1\right),{\mathsf{\sigma }}_{\mathrm{t}}{\left({\mathrm{x}}_1\right)}^2\mathrm{I}\right)\end{array}$$

(6)

In the expression of probability distributions, semi-colons “;” are used to separate random variables and distribution parameters. Here, the semicolon indicates that the random variable x follows a normal distribution with a mean of ${\mathsf{\mu }}_{\mathrm{t}}\left({\mathrm{x}}_1\right)$ and a variance of ${\mathsf{\sigma }}_{\mathrm{t}}{\left({\mathrm{x}}_1\right)}^2$. ${\mathsf{\mu }}_{\mathrm{t}}\left({\mathrm{x}}_1\right)$ is the mean path as ${\mathsf{\mu }}_{\mathrm{t}}\left({\mathrm{x}}_1\right)=\mathrm{t}{\mathrm{x}}_1$, which transitions linearly from the source distribution ${\mathrm{p}}_0$ to the target point ${\mathrm{x}}_1$. The standard deviation ${\mathsf{\sigma }}_{\mathrm{t}}\left({\mathrm{x}}_1\right)$ is the variance path as ${\mathsf{\sigma }}_{\mathrm{t}}\left({\mathrm{x}}_1\right)=1-\left(1-{\mathsf{\sigma }}_{\mathrm{m}\mathrm{i}\mathrm{n}}\right)\mathrm{t}$, which decays from ${\mathsf{\sigma }}_0=1$ to ${\mathsf{\sigma }}_1={\mathsf{\sigma }}_{\mathrm{m}\mathrm{i}\mathrm{n}}$. When ${\mathsf{\sigma }}_{\mathrm{m}\mathrm{i}\mathrm{n}}\to 0$, the path degenerates to a deterministic linear interpolation ${\mathrm{x}}_{\mathrm{t}}=\mathrm{t}{\mathrm{x}}_1$.

According to the continuity equation of the probabilistic flow, the corresponding conditional velocity field [15] ${\mathrm{u}}_{\mathrm{t}}\left(\mathrm{x}|{\mathrm{x}}_1\right)$ can be derived as follows:

$${\mathrm{u}}_{\mathrm{t}}\left(\mathrm{x}|{\mathrm{x}}_1\right)={\displaystyle \frac{{\dot{\mathsf{\sigma }}}_{\mathrm{t}}\left({\mathrm{x}}_1\right)}{{\mathsf{\sigma }}_{\mathrm{t}}\left({\mathrm{x}}_1\right)}}\left(\mathrm{x}-{\mathsf{\mu }}_{\mathrm{t}}\left({\mathrm{x}}_1\right)\right)+{\dot{\mathsf{\mu }}}_{\mathrm{t}}\left({\mathrm{x}}_1\right)$$

(7)

Substituting ${\mathsf{\mu }}_{\mathrm{t}}\left({\mathrm{x}}_1\right)=\mathrm{t}{\mathrm{x}}_1$ and ${\mathsf{\sigma }}_{\mathrm{t}}\left({\mathrm{x}}_1\right)=1-\left(1-{\mathsf{\sigma }}_{\mathrm{m}\mathrm{i}\mathrm{n}}\right)\mathrm{t}$, we obtain the following:

$$\begin{array}{c}{\mathrm{u}}_{\mathrm{t}}\left(\mathrm{x}|{\mathrm{x}}_1\right)={\displaystyle \frac{-\left(1-{\mathsf{\sigma }}_{\mathrm{m}\mathrm{i}\mathrm{n}}\right)}{1-\left(1-{\mathsf{\sigma }}_{\mathrm{m}\mathrm{i}\mathrm{n}}\right)\mathrm{t}}}\left(\mathrm{x}-\mathrm{t}{\mathrm{x}}_1\right)+{\mathrm{x}}_1\end{array}$$

(8)

When ${\mathsf{\sigma }}_{\mathrm{m}\mathrm{i}\mathrm{n}}\to 0$,

$$\begin{array}{c}{\mathrm{u}}_{\mathrm{t}}\left(\mathrm{x}|{\mathrm{x}}_1\right)={\displaystyle \frac{{\mathrm{x}}_1-\mathrm{x}}{1-\mathrm{t}}}\end{array}$$

(9)

This means that the velocity field points to the target ${\mathrm{x}}_1$, and $\mathrm{x}$ moves toward the target ${\mathrm{x}}_1$ with a uniform velocity.

Averaging the above conditional path over all target data points yields the global marginal velocity field ${\mathrm{u}}_{\mathrm{t}}\left(\mathrm{x}\right)$, which is the velocity of the overall flow when the conditional variables are not considered, and is the expectation of the conditional velocity field over the target distribution ${\mathrm{q}}_1\left({\mathrm{x}}_1\right)$:

$$\begin{array}{c}{\mathrm{u}}_{\mathrm{t}}\left(\mathrm{x}\right)={\mathbb{E}}_{{\mathrm{x}}_1~{\mathrm{p}}_{1|\mathrm{t}}\left({\mathrm{x}}_1|\mathrm{x}\right)}\left[{\mathrm{u}}_{\mathrm{t}}\left(\mathrm{x}|{\mathrm{x}}_1\right)\right]\end{array}$$

(10)

where ${\mathrm{p}}_{1|\mathrm{t}}\left({\mathrm{x}}_1|\mathrm{x}\right)$ is the posterior distribution of the conditional variable ${\mathrm{x}}_1$, given the current state $\mathrm{x}$. ${\mathrm{q}}_1\left({\mathrm{x}}_1\right)$ is the target distribution, and ${\mathrm{p}}_{\mathrm{t}}\left(\mathrm{x}\right)$ is the marginal distribution at moment $\mathrm{t}$.

$$\begin{array}{c}{\mathrm{p}}_{1|\mathrm{t}}\left({\mathrm{x}}_1|\mathrm{x}\right)={\displaystyle \frac{{\mathrm{p}}_{\mathrm{t}}\left(\left.\mathrm{x}\right|{\mathrm{x}}_1\right){\mathrm{q}}_1\left({\mathrm{x}}_1\right)}{{\mathrm{p}}_{\mathrm{t}}\left(\mathrm{x}\right)}}\end{array}$$

(11)

${\mathrm{u}}_{\mathrm{t}}\left(\mathrm{x}\right)$ can be expressed by the integral formula [15]:

$$\begin{array}{c}{\mathrm{u}}_{\mathrm{t}}\left(\mathrm{x}\right)=\int {\mathrm{u}}_{\mathrm{t}}\left(\mathrm{x}|{\mathrm{x}}_1\right){\displaystyle \frac{{\mathrm{p}}_{\mathrm{t}}\left(\left.\mathrm{x}\right|{\mathrm{x}}_1\right){\mathrm{q}}_1\left({\mathrm{x}}_1\right)}{{\mathrm{p}}_{\mathrm{t}}\left(\mathrm{x}\right)}}\mathrm{d}{\mathrm{x}}_1\end{array}$$

(12)

${\mathrm{u}}_{\mathrm{t}}\left(\mathrm{x}\right)$ is an explicit but implicitly defined reference velocity field defined by the easily sampled conditional path while guaranteeing the transportation of ${\mathrm{p}}_0$ to ${\mathrm{p}}_1$.

The training objective of the conditional flow matching technique can be formulated as predicting the regression of the marginal velocity field ${\mathrm{v}}_{\mathsf{\theta }}\left({\mathrm{x}}_{\mathrm{t}},\mathrm{t}\right)$. The loss function [15] adopts the mean square error between the model predicted flow field ${\mathrm{v}}_{\mathsf{\theta }}\left({\mathrm{x}}_{\mathrm{t}},\mathrm{t}\right)$ and the true probabilistic flow field $\mathrm{u}\left({\mathrm{x}}_{\mathrm{t}},\mathrm{t}\right)$. The loss objective is for the model predicted flow field ${\mathrm{v}}_{\mathsf{\theta }}\left({\mathrm{x}}_{\mathrm{t}},\mathrm{t}\right)$ to be close to the true probabilistic flow field $\mathrm{u}\left({\mathrm{x}}_{\mathrm{t}},\mathrm{t}\right)$ that we obtained analytically.

$$\begin{array}{c}{\mathrm{L}}_{\mathrm{F}\mathrm{L}\mathrm{U}\mathrm{X}}={\mathrm{E}}_{\mathrm{t}~\left(0,1\right),{\mathrm{x}}_1~{\mathrm{q}}_1,{\mathrm{x}}_{\mathrm{t}}~{\mathrm{p}}_{\mathrm{t}}\left({\mathrm{x}}_{\mathrm{t}}|{\mathrm{x}}_1\right)}\left[{\left({\mathrm{v}}_{\mathsf{\theta }}\left({\mathrm{x}}_{\mathrm{t}},\mathrm{t}\right)-{\mathrm{u}}_{\mathrm{t}}\left({\mathrm{x}}_{\mathrm{t}}|{\mathrm{x}}_1\right)\right)}^2\right]\end{array}$$

(13)

${\mathrm{x}}_{\mathrm{t}}={\mathsf{\mu }}_{\mathrm{t}}\left({\mathrm{x}}_1\right)+{\mathsf{\sigma }}_{\mathrm{t}}\left({\mathrm{x}}_1\right)\mathsf{\epsilon },\mathsf{\epsilon }\sim \mathcal{N}\left(0,1\right)$ are the sampling points along the conditional path ${\mathrm{p}}_{\mathrm{t}}\left(\cdot |{\mathrm{x}}_1\right)$, where ${\mathrm{p}}_{\mathrm{t}}\left(\cdot |{\mathrm{x}}_1\right)$ denotes the probability distribution of state x_t at time step t given target point x1.

When the loss convergence tends to zero, it is said that ${\mathrm{v}}_{\mathsf{\theta }}\left({\mathrm{x}}_{\mathrm{t}},\mathrm{t}\right)$ agrees with the reference ${\mathrm{u}}_{\mathrm{t}}\left({\mathrm{x}}_{\mathrm{t}}|{\mathrm{x}}_1\right)$ almost everywhere through implicit learning, at which point the model has learned how to transform the initial distribution of the flow into the target distribution.

3. Methods

3.1. Text-to-Image Architecture

The diffusion process only requires its denoising network to be an image-to-image model with equal dimensions of inputs and outputs, and the convolutional neural network-based U-Net can realize the same dimensions of outputs and inputs, so a basic diffusion model such as SD1.5 uses U-Net for diffusion, which contains a residual-based convolutional module, and at the same time adopts self-attention to enhance the perception of key features. The FLUX model architecture uses a transformer architecture to model the diffusion model to complete the diffusion process compared to U-Net, as shown in Figure 3 below.

The feature extraction module converts the input data into feature vectors that can be learned by the model. The inputs include image and location encoded inputs, text inputs, and guidance scale and timestep.

Text inputs mainly rely on pre-trained Contrastive Language-Image Pre-training (CLIP) [33] and Text-to-Text Transfer Transformer encoders (T5) [34]. While CLIP embeddings provide a global semantic representation of the input text, T5 embeddings extract fine-grained contextual hierarchical information from the text through their sequence-to-sequence pre-training capability, enabling the model to accurately capture the detailed elements of the conditional text.

Firstly, CLIP text encoder is usually based on the Transformer architecture, which first tokenizes the input text ${\mathrm{t}}_{\mathrm{i}}$ to obtain the token sequence $\left\{{\mathrm{w}}_1,{\mathrm{w}}_2,\dots ,{\mathrm{w}}_{\mathrm{n}}\right\}$, which is transformed into the sequence of embedding vectors $\left\{{\mathrm{e}}_1,{\mathrm{e}}_2,\dots ,{\mathrm{e}}_{\mathrm{n}}\right\}$, where each embedding vector has the dimension of $\mathrm{d}$. Then, position coding is added to the sequence of embedding vectors and fed it into the Transformer encoder to obtain the text feature vector ${\mathrm{T}}_{\mathrm{c}\mathrm{l}\mathrm{i}\mathrm{p}}\in {ℝ}^{\mathrm{d}}$, where $ℝ$ represents the real number space, and d represents the dimension of the vector; the T5 text encoder performs the deep contextual modeling of the text through the encoder–decoder architecture to generate the fine-grained feature sequence ${\mathrm{T}}_{\mathrm{t}5}\in {ℝ}^{\mathrm{n}\times \mathrm{d}}$, which preserves the word-level semantic relations.

The image input is first patched using a patch embedding to obtain a series of tokens; after tokenization RoPE (Rotary Position Embedding) is added to obtain the image feature vector ${\mathrm{I}}_{\mathrm{e}}\in {ℝ}^{\mathrm{d}}$, which can be accessed for transformer blocks.

$$\begin{array}{c}{\mathrm{I}}_{\mathrm{e}}={\mathrm{W}}_{\mathrm{i}}\cdot {\mathrm{I}}_{\mathrm{c}\mathrm{l}\mathrm{i}\mathrm{p}}+{\mathrm{b}}_{\mathrm{i}}\end{array}$$

(14)

$$\begin{array}{c}{\mathrm{T}}_{\mathrm{c}\mathrm{l}\mathrm{i}{\mathrm{p}}_{\mathrm{e}}}={\mathrm{W}}_{{\mathrm{t}}_{\mathrm{c}\mathrm{l}\mathrm{i}\mathrm{p}}}\cdot {\mathrm{T}}_{\mathrm{c}\mathrm{l}\mathrm{i}\mathrm{p}}+{\mathrm{b}}_{{\mathrm{t}}_{\mathrm{c}\mathrm{l}\mathrm{i}\mathrm{p}}}\end{array}$$

(15)

$$\begin{array}{c}{\mathrm{T}}_{\mathrm{T}{5}_{\mathrm{e}}}={\mathrm{W}}_{{\mathrm{t}}_{\mathrm{T}5}}\cdot {\mathrm{T}}_{\mathrm{T}5}+{\mathrm{b}}_{{\mathrm{t}}_{\mathrm{T}5}}\end{array}$$

(16)

${\mathrm{W}}_{\mathrm{i}},{\mathrm{W}}_{{\mathrm{t}}_{\mathrm{c}\mathrm{l}\mathrm{i}\mathrm{p}}},{\mathrm{W}}_{{\mathrm{t}}_{\mathrm{t}5}}$ are projection matrices, and ${\mathrm{b}}_{\mathrm{i}},{\mathrm{b}}_{{\mathrm{t}}_{\mathrm{c}\mathrm{l}\mathrm{i}\mathrm{p}}},{\mathrm{b}}_{{\mathrm{t}}_{\mathrm{t}5}}$ are bias terms.

The information ${\mathrm{T}}_{\mathrm{c}\mathrm{l}\mathrm{i}\mathrm{p}}\in {ℝ}^{\mathrm{d}}$ extracted through CLIP is computed with the Timestep and Guidance scale as conditional inputs, which accept the conditional information and compute the parameters through the Adaptive Layer Normalization (adaLN) method, and the generated parameters are controlled through $\mathrm{s}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{e},\mathrm{s}\mathrm{h}\mathrm{i}\mathrm{f}\mathrm{t}$ in modulation.

$$\mathrm{s}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{e},\mathrm{s}\mathrm{h}\mathrm{i}\mathrm{f}\mathrm{t}=\mathrm{L}\mathrm{i}\mathrm{n}\mathrm{e}\mathrm{a}{\mathrm{r}}_{\left(\mathrm{C}\mathrm{o}\mathrm{n}\mathrm{c}\mathrm{a}\mathrm{t}\left({\mathrm{t}}_{\mathrm{C}\mathrm{L}\mathrm{I}\mathrm{P}\mathrm{e}},,\mathrm{T}\mathrm{i}\mathrm{m}\mathrm{e}\mathrm{s}\mathrm{t}\mathrm{e}\mathrm{p},\mathrm{G}\mathrm{u}\mathrm{i}\mathrm{d}\mathrm{a}\mathrm{n}\mathrm{c}\mathrm{e}\right)\right)}$$

(17)

${\mathrm{I}}_{\mathrm{e}}$ and ${\mathrm{T}}_{\mathrm{T}{5}_{\mathrm{e}}}$ are input into the transformer block as image features and text features, and the transformer block includes MM-DiT and Single-DiT. Firstly, ${\mathrm{I}}_{\mathrm{e}}$ and ${\mathrm{T}}_{\mathrm{T}{5}_{\mathrm{e}}}$ are fed into MM-DiT, and are processed separately in the block in the form of double streams, but together they are computed by the attention mechanism. Take the text feature ${\mathrm{T}}_{\mathrm{T}{5}_{\mathrm{e}}}$, which is symmetric with ${\mathrm{I}}_{\mathrm{e}}$, as an example; in each block, it firstly passes through a layer norm, and is then modulated according to the calculated parameter $\mathrm{s}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{e},\mathrm{s}\mathrm{h}\mathrm{i}\mathrm{f}\mathrm{t}$; after passing through a linear layer, it is spliced with the corresponding ${\mathrm{I}}_{\mathrm{e}}$ to compute the Query, Key, Value projection matrices (Q,K,V) in the attention layer. After that, it passes through the linear layer, residual network, layer norm, modulation and Multi-Layer Perceptron (MLP) to complete the calculation of a block.

After completing the computation of MM-DiT, the text and images are spliced together and fed into Single-DiT for processing, which completes the computation of a block in the form of a single stream, which is able to reduce the number of parameters in a single layer and increase the depth of the network. Unlike MM-DiT, Single-DiT performs attention and MLP computations in parallel within a single stream, where MLP consists of a Linear layer, a Gaussian Error Linear Unit activation function (GELU), and another Linear layer.

$$\mathrm{x}=\left[{\mathrm{I}}_{\mathrm{e}};{\mathrm{T}}_{\mathrm{T}{5}_{\mathrm{e}}}\right]$$

(18)

$\mathrm{x}$ denotes that the text and images are spliced together.

The iterative computation continues at MM-DiT and Single-DiT as the timestep continues; when $\mathrm{t}=1$, the obtained final latent variable ${\mathrm{x}}_1$ is mapped back to the pixel space by the VAE decoder, and the final desired knitwear design image $\mathrm{x}$ is output.

$$\begin{array}{c}\mathrm{x}={\mathrm{D}}_{\mathrm{V}\mathrm{A}\mathrm{E}}\left({\mathrm{x}}_1\right)\in {\mathrm{R}}^{\mathrm{H}\times \mathrm{W}\times C}\end{array}$$

(19)

where C represents the number of channels in an image, H represents the number of pixels in the vertical dimension of the image, and W represents the number of pixels in the horizontal dimension of the image.

With the above formulae and steps, the FLUX diffusion model is able to efficiently transform the source distribution into a target distribution and supports batch generation.

3.2. Training Low-Rank Adaption of Model

LoRA (Low-Rank Adaptation) fine-tuning is the core part of this module. The LoRA fine-tuning pre-training model works by introducing low-rank matrices A and B into the Transformer model while the original weight matrix parameters are kept fixed, and then learning task-specific feature representations through low-rank decomposition.

The fine-tuning process is shown below in Figure 4, loading the pre-trained FLUX model with the velocity field network ${\mathrm{v}}_{\mathsf{\theta }}$ and the multimodal encoders T5, CLIP. The original weight matrices ${\mathrm{W}}_0\in {ℝ}^{\mathrm{d}\times \mathrm{k}}$ are frozen and do not participate in the training. Two low-rank matrices A and B are introduced into the attention layer (Query, Value projection matrices) and the modulation parameter generation layer of the velocity field network for operations, and their product $\mathsf{\Delta }\mathrm{W}=\mathrm{B}\mathrm{A}$ approximates the update of the weight matrix. For the original forward propagation $\mathrm{y}={\mathrm{W}}_0\mathrm{x}$, this equation [22] becomes the following:

$$\begin{array}{c}\mathrm{s}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{e},\mathrm{s}\mathrm{h}\mathrm{i}\mathrm{f}\mathrm{t}=\mathrm{y}={\mathrm{W}}_0\mathrm{x}+\mathsf{\Delta }\mathrm{W}\mathrm{x}={\mathrm{W}}_0\mathrm{x}+\left(\mathrm{B}\mathrm{A}\right)\mathrm{x}\end{array}$$

(20)

$\mathsf{\Delta }\mathrm{W}=0$ in the initial state. $\mathrm{A}=\mathcal{N}\left(0,{\mathsf{\sigma }}^2\right),\mathrm{A}\in {ℝ}^{\mathrm{d}\times \mathrm{r}}$ initialized as a stochastic Gaussian distribution. $\mathrm{B}\in {ℝ}^{\mathrm{r}\times \mathrm{k}}$ is initialized as the null matrix, where $\mathrm{r}\ll \mathrm{d}\times \mathrm{r}\ll {\mathrm{d}}^2.$ With this decomposition, the number of fine-tuned parameters is reduced from the original $\mathrm{d}\times \mathrm{d}$ to $2\times \mathrm{r}\times \mathrm{d}$, which significantly reduces the number of parameters and the computational cost.

After training to obtain the low-rank adaptation weights file, load the FLUX diffusion model weight file and LoRA weight file, and dynamically superimpose the updated parameter weights $\left\{\mathrm{A},\mathrm{B}\right\}$ from the LoRA fine-tuning process onto the original model weights $\mathrm{W}$, with the formula $\mathrm{W}\prime =\mathrm{W}0+\mathrm{B}\mathrm{A}$. Then, apply it to the query ${\mathrm{W}}_{\mathrm{Q}}$ and ${\mathrm{W}}_{\mathrm{V}}$ projection matrix in the Spatial Transformer layer of the Attention Mechanism; then the formula of the Attention Mechanism [35] is updated as follows:

$$\begin{array}{c}{\mathrm{W}}_{\mathrm{Q}}^{\prime }={\mathrm{W}}_{\mathrm{Q}0}+{\mathrm{B}}_{\mathrm{Q}}\cdot {\mathrm{A}}_{\mathrm{Q}},{\mathrm{W}}_{\mathrm{V}}^{\prime }={\mathrm{W}}_{\mathrm{V}0}+{\mathrm{B}}_{\mathrm{V}}\cdot {\mathrm{A}}_{\mathrm{V}}\end{array}$$

(21)

$$\begin{array}{c}\mathrm{A}\mathrm{t}\mathrm{t}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}\left(\mathrm{Q},\mathrm{K},\mathrm{V}\right)=\mathrm{S}\mathrm{o}\mathrm{f}\mathrm{t}\mathrm{m}\mathrm{a}\mathrm{x}\left({\displaystyle \frac{\left({\mathrm{W}}_{\mathrm{Q}}^{\prime }\mathrm{h}\right){\left({\mathrm{W}}_{\mathrm{K}}\mathrm{h}\right)}^{\mathrm{T}}}{\sqrt{\mathrm{d}}}}\right)\left({\mathrm{W}}_{\mathrm{V}}^{\prime }\mathrm{h}\right)\end{array}$$

(22)

The updated ${\mathrm{W}}_{\mathrm{Q}}^{\prime }$, ${\mathrm{W}}_{\mathrm{V}}^{\prime }$ changes the projection matrix of the query ${\mathrm{W}}_{\mathrm{Q}}$ and ${\mathrm{W}}_{\mathrm{V}}$, and the scale is adjusted; generation is shifted to make the fine-tuned model pay more attention to the key features in the textual conditions such as double ribbing, and hairless feather yarns, and to make sure that the generated content conforms to the textual constraints.

4. Experiment and Result Analysis

4.1. Dataset Preparation

The low-rank adaptation aims to fine-tune the diffusion model with small batches of data, so each type of feature image is greater than or equal to 20, and the dataset contains a total of 288 image–text pairs of real fabrics. The data requires complete content features, and in order for the model to learn different weave piece length and width ratios, all images are processed as squares with a resolution of $1024\times 1024$.

During the construction of the dataset, the following two types of key parameters affecting fabric texture features are selected: fabric tissue structure and yarn surface features. In this study, six types of commonly used weft knitted fabrics, weft flat knit, two types of ribbed, three types of double reverse, three different yarn surfaces without hairiness, short hairiness and long hairiness, and two different yarn diameters of 4 mm and 2 mm for real fabrics were selected for the experiment. To control the study variables, the knitted fabrics had the same warp and weft densities and the same yarn count and other elements. A schematic diagram of the knitted fabric structure is shown in Figure 5. Weft flat stitch organization plain stitch consists of continuous unit coils in one direction; the front side presents a continuous “V”-shaped appearance along the longitudinal direction formed by the coil columns, and the reverse side presents a continuous corrugated appearance along the transverse direction formed by the coil arcs; ribbed fabrics are configured from the front coil vertical rows and the reverse coil vertical rows in a certain combination, so the front side of the ribbed fabric will have a continuous “V”-shaped appearance and corrugated appearance in the vertical direction. Double reverse is made by alternating one front coil row and one reverse coil row, so the front of double reverse has a continuous corrugated and “V”-shaped appearance in the horizontal direction.

The tags are manually labelled according to image features: set the image data as I1, then the corresponding text features are the three types of feature tags t1, t2, t3 and trigger word tag t4: knitwear, respectively. Its dataset is shown in Figure 6:

4.2. Experimental Setup

In the experimental setup, this study sets the rank dimension to r = 32 for the LoRA fine-tuning approach to balance the model representation capability and parameter efficiency, and adopts the adaptive momentum optimizer Adam with Weight Decay (AdamW) for the parameter updating, with the learning rate fixed at $1\times {10}^{–4}$ to avoid gradient oscillation. In the model training process, the sample size of a single batch is set to 4, and 100 training rounds are used to ensure parameter convergence. In the inference generation stage, 30 iteration steps are set to realize the balance between generation quality and computational efficiency, while the cue word correlation coefficient is adjusted to 7.0 to enhance the control strength of text conditions on image generation.

4.3. Generation Parameters

This experiment takes hairless feather double rib tight fabric as an example, and uses the same labels as the conditions in the denoising stage to test the effect of fine-tuning the weights on the generated results. The image generation effect of the low-rank adaptation model with weights of 0.1~1.0 is shown in Figure 7: with the increase in the weights of the low-rank adaptation model, the bulging stripes in the generated image of the double ribbed knitted fabric gradually change from yarns to string-set coils from 0.4, and the string-set law gradually becomes correct from splicing with errors. The loop structure gradually becomes correct from the splicing error; from 0.7, raising the weight of image generation seems to confuse the generated features, and double ribbed fabrics change single ribbed fabrics, making the generated weight of 0.4 to 0.7 better.

4.4. Generation Results

The fabrics with different organizational structures are generated by hairless roving for example, as shown in Figure 8 below, and the prompts for the knitted fabric organizational structure design include plain fabric, 1-1-ribbed fabric, 2-2-ribbed fabric, 1-over-1-double-faced fabric, 5-over-5-double-faced fabric, and 5-5-checkered-double-faced fabric. The low-rank adaptation of the FLUX model can be used to design knitted fabrics with different organizational structures according to the cue word, and better simulate the real fabric coil string set structure.

Take 1 × 1 double reverse knitted fabric is an example of fabric generation with different characteristic yarns, as shown in Figure 9.

The prompts for yarn thickness are tight knitting and loose knitting, respectively, and the FLUX model can generate the corresponding knitted fabric images accordingly. As shown in Figure 9, from “fine yarn without hair” and “coarse yarn without hair” in the fourth column of the image, it can be clearly seen that the fine yarn fabrics are looser and the coarse yarn fabrics are tighter and thicker.

The prompt words for the characteristics of yarn hairiness are smooth fiber, short hairy feather fiber and long hairy feather fiber. As shown in the picture, the fabric structure of smooth fiber is the most clear, and the organizational structure of the coil can be seen; in the short feather fabric, the basic outline can be clearly seen, and there are short hairs at the edges. The long feathered fabric, on the other hand, has a plush texture like the shaggy-feathered fabric, presenting a fluffier surface.

Based on the above generation results, the low-rank adaptation of the FLUX diffusion model allows the use of easy-to-understand textual prompts to modulate the image generation results, which can be reflected in the generation of the effect of the fabric organizational structure, the hairiness of the yarn, and changes in the realization of the text-generated graphs from the yarn to the fabric shown at two levels.

In order to further prove the correctness of the generated results, a qualitative and quantitative assessment is performed using the fabric organizational structure as an example.

4.5. Evaluation

4.5.1. Qualitative Assessment

In this paper, real fabrics are used as the basis to compare the generation effects. The comparison data include the SD1.5 Model and FLUX Model in group A and Knit-Diff and Knit-FLUX in group B; the results are shown in Figure 10.

Group A FLUX models are able to generate plain knit fabrics but not other types of fabrics, such as double ribbed fabrics with a woolen bulge; Group B models are all enhanced in the fine-tuning process to predict the noise of knitting semantics, and their microstructures are more similar to the real fabrics. The Knit-Diff of Group B is not clear enough compared to the FLUX models of Group A, but the picture quality is not clear enough. The Knit-Diff in group B is more similar to the real fabric than the FLUX model in group A. The Knit-FLUX performs stably in high-resolution generation, surpasses the previous diffusion model in details such as fabric organization, and has the highest similarity to real fabrics among the four groups of data.

4.5.2. Quantitative Assessment

This paper selected two quantitative evaluation indicators, including the Structural Similarity Index (SSIM) and Human Perceptual (HP), to evaluate the structural consistency of the fabric generated. The SSIM and HP values were normalized to the [0, 1] range, where higher values indicate greater similarity between the two images, and a value of 1 denotes identical images. SSIM calculates the brightness, contrast, and structure to quantify the topological consistency between the generated image and the target image. The formula [36] is as follows:

$$\begin{array}{c}\mathrm{S}\mathrm{S}\mathrm{I}\mathrm{M}\left(\mathrm{x},\mathrm{y}\right)={\displaystyle \frac{\left(2{\mathsf{\mu }}_{\mathrm{x}}{\mathsf{\mu }}_{\mathrm{y}}+{\mathrm{c}}_1\right)\cdot \left(2{\mathsf{\sigma }}_{\mathrm{x}\mathrm{y}}+{\mathrm{c}}_2\right)}{\left({\mathsf{\mu }}_{\mathrm{x}}^2+{\mathsf{\mu }}_{\mathrm{y}}^2+{\mathrm{c}}_1\right)\cdot \left({\mathsf{\sigma }}_{\mathrm{x}}^2+{\mathsf{\sigma }}_{\mathrm{y}}^2+{\mathrm{c}}_2\right)}}\end{array}$$

(23)

${\mathsf{\mu }}_{\mathrm{x}}$ is the mean value of x; ${\mathsf{\sigma }}_{\mathrm{x}}^2$ is the variance of x; ${\mathsf{\mu }}_{\mathrm{y}}$ is the mean value of y; ${\mathsf{\sigma }}_{\mathrm{y}}^2$ is the variance of y; ${\mathsf{\sigma }}_{\mathrm{x}\mathrm{y}}$ is the covariance of x and y; and ${\mathrm{c}}_1$ and ${\mathrm{c}}_2$ are constants.

Although SSIM evaluation is an important reference, it also has certain limitations. Therefore, based on the above objective indicators, this study introduced human perception score (HP) as a supplementary evaluation method to supplement the shortcomings of objective indicators and obtain more comprehensive and reliable evaluation results. The HP score was obtained through evaluations conducted by six textile engineering experts.

Under the same parameter settings, the structural similarity comparison indices are presented in

Table 1. Results comparison.

Method	SSIM↑	HP↑
Knit-Diff	0.3537	0.383
Knit-FLUX	0.6528	0.883

. The upward arrows indicate that larger values represent better results, the more similar the two images are. The images generated by Knit-FLUX achieve a Structural Similarity Index Measure (SSIM) of 0.6528 with respect to real fabrics, and a Human Perceptual (HP) score of 0.883. Compared with Knit-Diff, the proposed method yields the highest similarity to real knitted fabrics, demonstrating superior performance in both structural fidelity and human perceptual alignment. These results collectively indicate that the generated knitted fabric images accurately reflect the textile structures.

In this study, fabric pixel coverage was selected as a quantitative evaluation metric to assess the effectiveness of yarn thickness representation in the generated fabric images.

Fabric pixel coverage refers to the proportion of pixels occupied by yarn regions within the entire fabric image. A higher coverage value indicates coarser yarns, denser surface fuzz, and fewer inter-yarn gaps, whereas a lower value suggests finer yarns, less fuzz, and a looser structural arrangement.

To ensure consistency in the evaluation of structural representations, the experiments were conducted on five distinct groups of fabric data. The selected fabric categories cover a range of yarn types and surface textures, including loose knit with smooth fiber, tight knit with smooth fiber, tight knit with smooth fiber, tight knit with short feather fiber, and tight knit with long feather fiber. For each fabric category, both original (real) samples and generated images were included for comparative analysis.

As shown in Figure 11, The results demonstrate that for most fabric types, the generated images closely approximate the structural features of the originals. Comparing the two groups of loose and tight knitting, the coverage of the fine yarn fabrics was significantly lower than that of the coarse yarn fabrics. Specifically, the coverage of the fine yarn fabrics was concentrated between 0.4 and 0.5, while the coverage of the coarse yarn fabrics was concentrated at 0.7. This comparison of performance metrics suggests that the coverage of the fine yarn fabrics was lower. Comparing the three groups of smooth and short feather fibers and long feather fibers, the coverage of long feather fibers is concentrated at 0.8 with the highest value. In addition, the fabric coverage of the two groups of training data and generated data are similar, with an average difference of less than 0.15, indicating that the model is able to effectively learn the fabric size information while maintaining the category learning; however, for the coarse yarn fabrics, the average difference in the coverage between the original data and the generated data is slightly larger, suggesting that there is still room for the optimization of knit-FLUX in terms of the size control during the generation process.

4.5.3. Generation Efficiency

In addition, this study tested and analyzed the model’s image generation time to evaluate the computational efficiency and scalability of Knit-FLUX in practical application scenarios. The generation time directly affects the model’s response speed and user experience, which in turn affects the model’s practicality and market competitiveness. By generating 100 images in batch and calculating the average generation time per image, the average inference speed of the three diffusion models can be quantitatively analyzed. This provides an objective basis for performance evaluation and offers data support for further model optimization.

As shown in

Table 2. Image generation efficiency comparison.

Method	Knit-Diff (SD1.5)	SDXL	Knit-FLUX (FLUX)
Time(s)	1.9	3.76	16.8

, KNIT-SD1.5 demonstrates significant advantages in terms of time consumption, with an average running time of only 1.9 s. KNIT-FLUX has an average runtime of 16.8 s. Despite a relatively higher average generation time than KNIT-SD1.5, KNIT-FLUX delivers superior visual fidelity and supports fast batch generation, proving its effectiveness for real-time design support and scalable fabric material production. Therefore, it remains the preferred solution when both quality and usability are prioritized.

5. Conclusions

This paper is based on the low-rank adaptation of the FLUX model (Knit-FLUX) for knitted fabric image generation, which supports the stable, batch and fast generation of knitted fabric images through easy-to-understand text prompts, and the generated images have clear texture and correct structural features, which are close to the real fabric.

In the typical workflow of fashion design, designers often begin by sketching garment illustrations, yet the representation of fabric textures usually relies on hand-drawn patterns or collage materials, which lack accuracy in reflecting the true structural characteristics of the intended textile. With the integration of the proposed knitted texture generation model, designers can input specific pattern parameters to rapidly generate structurally consistent and visually realistic knitted textures. These generated textures can then be applied to garment illustrations as high-fidelity replacements, thereby enhancing the visual expressiveness and material authenticity of the design renderings. Moreover, in educational settings, this method can assist students in understanding the mapping between knitted structures and surface textures, thus improving the visualization and interactivity of textile pattern design.

Although Knit-FLUX significantly enhances the controllability of image generation with domain-specific feature prompts, when the input prompt contains multiple conceptually or visually similar descriptors, the model may conflate distinct attributes, resulting in inaccurate or ambiguous feature representations in the generated image. This limitation can be attributed to the inherent overlap in the learned latent space and the insufficient disentanglement of features during fine-tuning. Future improvements may involve prompt refinement strategies and feature-space regularization to mitigate such ambiguities.