Enhancing Human Body Generation in Diffusion Models with Dual-Level Prior Knowledge

Abstract

The development of diffusion models (DMs) has greatly enhanced text-to-image generation, outperforming previous methods like generative adversarial networks (GANs) in terms of image quality and text alignment. However, accurately generating human body images remains challenging, often resulting in disproportionate figures and anatomical errors, which limits their practical applications in areas such as portrait generation. While previous methods such as HcP have shown promising results, limitations, including the retention of noisy priors, limited understanding of human representation, and restriction of generalization power, still exist due to the specific design of fully supervised learning with only pose-related information. In this study, we introduce a novel method to enhance pretrained diffusion models for realistic human body generation by incorporating dual-level human prior knowledge. Our approach involves learning shape-level details with the human-related tokens in the original prompts, and learning pose-level priors by adding a learnable pose-aware token to each text prompt. We use a two-stage training strategy to rectify the cross-attentions with a bind-then-generalize process, leveraging multiple novel objectives along with adversarial training. Paired comparisons on SD1.5 and SDXL, under identical prompts and generation seeds, show more coherent human bodies than the pretrained models and a LoRA baseline trained on the same images. Image-level metrics improve on SDXL. Those metrics measure image quality, distributional similarity, and preference. The anatomical comparison is carried by the paired images and the attention maps.

1. Introduction

Despite improvements in AI-generated content fidelity [1,2] and text alignment compared to GANs [3,4], diffusion models [5,6,7,8,9] still struggle with human body images, a common visual in real-world media. These advanced models often produce disproportionate bodies and inaccurately placed, missing, or redundant limbs and hands, limiting their application in areas like portrait generation and image stylization. Since the introduction of the latent diffusion model (LDM) [10], researchers have focused on scaling up model and training data sizes to enhance human body image quality. Stable Diffusion XL 1.0 (SDXL; Stability AI, https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0, accessed on 1 March 2025) still produces these failures, as shown in Figure 1. Published results for Stable Diffusion 3 (SD3) [11] show the same difficulty. We cite SD3 as evidence that increasing model or data size has not by itself produced a reliable human-body representation. SD3 is not a baseline in our experiments.

Figure 1. Comparison between images generated by pretrained SDXL (first row) and SDXL finetuned with our method (second row).

We recognize HcP [12] as one of the few recent works focused on improving human body generation. This work introduces an additional attention map derived from pose information to amend the original cross-attention mechanisms. Unlike other methods [13] that rely on explicit conditions such as pose or depth, HcP can be seamlessly integrated into any text-to-image diffusion model, making it highly valuable. This study highlights a significant issue: the cross-attention modules in diffusion models often struggle to capture the location and shape information of target objects related to human tokens. This challenge is illustrated in Figure 2a, where examples generated by the pretrained SDXL show that cross-attention maps for human-related tokens either marginally highlight or completely neglect human regions, resulting in noisy, large activations in the background.

Figure 2. (a) Images generated by pretrained SDXL and the cross-attention maps of human-related tokens. Those tokens are printed in red in the prompts. In the attention maps, white denotes high activation and black denotes low activation. Pretrained SDXL does not place this activation on the body. (b) The same pose can correspond to different body shapes, so pose alone does not determine the body. (c) Skeletons from a pose detector. Colored lines connect adjacent keypoints and can be assigned to the wrong person or drawn across an occluded part.

While HcP delivers impressive results and maintains a lightweight design, we still identify several issues existing in this approach:

(1)

Retention of Noisy Priors: HcP revises the cross-attention maps of pretrained diffusion models by introducing additional attention maps. This method retains all prior knowledge from the pretrained models, including noisy attention maps. The new branch can highlight the desired regions but cannot suppress other areas, resulting in continued noisy attention maps.

(2)

Limited Understanding of Human Representation: HcP primarily focuses on human poses, neglecting other essential aspects of human representation. A pose alone cannot specify a particular human body type. Factors such as body shape (e.g., thin or heavy) and occlusion by objects (as shown in Figure 2c) are also crucial for accurately depicting human figures. For instance, with the pose illustrated in Figure 2b, it remains unclear whether to generate a slender girl or a heavier one.

(3)

Restriction of Generalization Power: HcP adopts a fully supervised learning strategy, relying on pose annotations paired with each training image. However, real-world images often feature diverse human bodies in unexpected poses. Even well-trained segmenters like SAM [14] and pose detectors such as HrFormer [15] can struggle in various corner cases. As a result, while the pretrained diffusion models may perform well on the training data, their generalization ability is limited.

To address the challenges in generating realistic human bodies, we propose a novel dual-level prior injection method that enhances pretrained diffusion models by learning both shape-level and pose-level human priors. Unlike HcP, which captures only limited human priors, our approach aims for a more generalizable and comprehensive understanding through a two-stage bind-then-generalize strategy. Specifically, we first bind the human-related tokens from the original prompts, along with an additional pose token attached to the text prompt, to their corresponding semantic regions. This binding produces meaningful attention maps for these tokens. We propose a novel composite objective with three distinct terms to facilitate the learning process: The first term suppresses attention activation outside human regions in each image. Building on this, the second term encourages the diffusion UNet to ensure that human-related activation exceeds non-human-related activation, effectively highlighting human body regions. The third term fits the attention map to the mask. Together, these objectives empower the pretrained diffusion models to extract and aggregate critical information from various token types, leading to higher-quality human body generation.

After establishing a one-to-one mapping based on the training data, we guide the model to mimic the distribution of real attention maps. We utilize commonly used generative adversarial networks (GANs) by introducing shape-aware and pose-aware discriminators corresponding to the human-related and pose tokens, respectively. The diffusion UNet is trained against these discriminators in an adversarial minimax game, enabling it to generate cross-attention maps that align with real data distributions. On the paired prompts in Section 5, this second stage keeps a coherent pose and shape for ordinary actions and for less common ones, on both SD1.5 and SDXL.

To show the effectiveness of our proposed method, we conduct experiments across a range of text prompts. Under matched prompts and generation seeds, both SD1.5 and SDXL produce more coherent human bodies than the corresponding pretrained models and LoRA baselines. On SDXL, image-level metrics improve as well. In summary, the contributions of this work are as follows:

(1)

Dual-Level Prior Injection Method: We present a novel method that enhances pretrained diffusion models by learning shape-level and pose-level human priors through a two-stage bind-then-generalize strategy. Our method binds human-related tokens from text prompts and an additional pose token to their corresponding semantic regions, resulting in meaningful attention maps.

(2)

Composite Objective for Learning: We propose a composite objective with three terms: a mean penalty outside the mask, a peak-to-peak margin of $0.1$, and a pixel-level fit to the mask.

(3)

Mimicking Real Attention Map Distribution: We further introduce adversarial training to rectify the cross-attention maps, with shape-aware and pose-aware discriminators enhancing the model’s ability to mimic the real data distribution.

(4)

Improved Model Performance: On SD1.5 and SDXL, paired comparisons show more coherent bodies than the pretrained models and a matched LoRA baseline. Image-level metrics improve on SDXL. The anatomical evidence is the paired images and the attention maps.

2. Related Works

2.1. Refined Human Body Generation

In spite of astonishing image fidelity in general, diffusion models have long suffered from the problem of inaccurate detail, especially with regard to human bodies. To solve this problem and generate better human images, two main strategies are adopted. The first is extra conditions. For example, ControlNet [16] was proposed to add an additional spatial controlling signal to the denoising process through a replicated branch of the diffusion UNet encoder. Following this work, many other works, such as HyperHuman [13], tried to apply more comprehensive conditions including depth and normal maps to diffusion models for better results. Huang et al. [17] proposed generating human bodies in a hierarchical manner, with parts first being generated and then whole bodies. Another important strategy is regularization. HcP [12] tried to learn an additional attention branch with pose-related information, thus making diffusion models aware of the poses and generating more reasonable human bodies. Our paper follows the idea of regularization. ControlNet and HyperHuman belong to the conditional line. They receive a spatial map at inference, such as a pose, a depth map, or surface normals, so much of the body layout is given as input. After training, our sampler follows the original text-to-image procedure of the backbone together with LoRA. It does not take a pose image, a depth map, or a mask, and it does not run SAM, MMPose, or the discriminators. Section 5 compares the pretrained model with a LoRA baseline trained on the same images, which holds the backbone, the data, and the sampler fixed. HcP is the closest text-only method. It adds a pose-derived attention branch and retains the original cross-attention, so the noisy pretrained map remains. It supervises pose and does not train an explicit shape prior, and it learns from fully supervised pairs. A public implementation that can be trained under our backbone, data, and sampler is not available, so the comparison with HcP is the comparison of these training signals. Our first stage suppresses background attention and binds both shape and pose. The second stage replaces one-to-one mask fitting by distribution matching.

2.2. Adversarial Training for Diffusion Models

Adversarial training originated in generative adversarial models (GANs) [18] that learn to mimic the real data distribution as a competing procedure between generator and discriminator, which iteratively improve both models’ performance. Recently, some studies have been focusing on enhancing diffusion models with adversarial training. Most of them are aimed at reducing the sampling steps of DMs. Xiao et al. [19] proposed replacing the minimization of divergence with Gaussian distribution with adversarial divergence with a learned discriminator. Xu et al. [20] further improved this formulation with a simpler objective and better training strategy, thus helping diffusion models generate high-quality images with fewer sampling steps. ADD [21], following the same idea, proposed combining distillation and adversarial training to further enhance the efficiency of diffusion models. On the other hand, adversarial training is also applied to diffusion models for better generation quality. For example, Li et al. [22] proposed discriminating the noisy estimation of generated images with a segmenter. As the result, the diffusion models can better follow the control of input segmentation masks. In addition, Yang et al. [23] proposed leveraging adversarial training to embed structural prior into diffusion models. Our method follows the idea of improving the performance of diffusion models with adversarial training. However, unlike previous works that rely on noisy image estimation, we propose applying the discriminator to the intermediate cross-attention maps, which leads to significantly more efficiency while guaranteeing strong performance.

3. Preliminary: Stable Diffusion

Diffusion models aim to capture the data distribution $p_{\theta }\left({\mathbf{x}}_0\right)$ of clean data ${\mathbf{x}}_0$ by gradually refining a standard Gaussian distribution, with the learning process framed as denoising score matching. Stable Diffusion (SD) builds upon this framework to enable text-to-image generation based on a text prompt p. By leveraging a pre-trained VQ-VAE [24] that includes an encoder $\mathcal{E}$ and a decoder $\mathcal{D}$, SD allows the model to concentrate more on the semantic aspects of the data, thereby enhancing efficiency. A diffusion UNet is utilized to estimate the noise, incorporating an attention mechanism. Specifically, in the l-th layer, self-attention is first employed to facilitate interactions among spatial features: $z^l=Attention(W_Q^l·z^l,W_K^l·z^l,W_V^l·z^l)$, where $Attention$ denotes the attention operator, $z^l$ represents the latent embeddings of the l-th layer, and $W_Q,W_K,W_V$ are the projection layers for self-attention. Following this, cross-attention is applied to incorporate conditioning information such as the text prompt: ${\widehat{z}}^l=Attention({\widehat{W}}_{Q_t}^l·z^l,{\widehat{W}}_{K_t}^l·z_{text},{\widehat{W}}_{V_t}^l·z_{text})$, where $z_{text}$ denotes the text prompt embedding, and ${\widehat{W}}_Q,{\widehat{W}}_K,{\widehat{W}}_V$ are the projection layers for cross-attention. The training objective of SD is formulated as follows:

$${\mathcal{L}}_{noise}={\mathbb{E}}_{\mathcal{E}\left(x\right),ϵ\sim \mathcal{N}(0,1),t}\left[{∥ϵ-{ϵ}_{\theta }\left(z^t,t\right)∥}_2^2\right],$$

(1)

where t is uniformly sampled from $\{0,...,T\}$, and $z^t$ represents the noisy latent at the t-th timestep.

4. Methodology

4.1. Semantic Attention Binding

Previous works have shown that the cross-attention maps of a diffusion UNet between text prompts and image latent embeddings can indicate the coarse location and shape of the target objects. However, as mentioned in Section 1, the pretrained SD fails to produce meaningful cross-attention maps regarding human-related tokens, such as man, woman, etc., which can result in the problem of human body deformity. Consequently, we propose first guiding the cross-attention maps to bind with correct regions, i.e., suppressing non-human-related activation and highlighting human-related activation, as shown in Figure 3. Inspired by previous works, a composite objective is utilized.

Figure 3. Schematic of the proposed method. (Left) Supervised binding: human-related tokens are aligned with the body mask, and the pose token is aligned with the skeleton. (Right) Adversarial rectification: shape and pose discriminators match the distribution of real masks. Arrows indicate the training flow. Masks mark the supervised body region and the skeleton.

• Attention maps. When processing each noisy latent $z^t$ of an image $\mathbf{I}$, we read the cross-attention probabilities of the supervised tokens. Heads of the same layer are averaged. The maps are then grouped by the down, mid, and up blocks at resolutions of 8, 16, 32, and 64. Inside a group, the token loss averages the target tokens and sums the layers, while the pixel loss averages the layers and sums the target tokens. The pixel term is the average of a binary cross-entropy at the native resolution and the same cross-entropy after upsampling. These two losses do not apply a further spatial min–max normalization. Appendix A records the grouping. SAM [14] segments the human bodies in $\mathbf{I}$ and produces the shape mask $M^{shape}$. For a map $\widehat{M}$ and this mask, write ${\Omega }^{+}=\{(i,j)\mid M_{i,j}^{shape}=1\}$ and ${\Omega }^{-}=\{(i,j)\mid M_{i,j}^{shape}=0\}$.
• Suppressing loss. The first term penalizes attention that falls outside the body. It is the mean response on ${\Omega }^{-}$:

  $${\mathcal{L}}_s^{shape}(\widehat{M})={\displaystyle \frac{1}{|{\Omega }^{-}|}}\sum _{(i,j)\in {\Omega }^{-}}{\widehat{M}}_{i,j}.$$

  (2)
• Margin loss. The second term compares the peak response inside the mask with the peak response outside it:

  $$\begin{array}{cc}\hfill f(\widehat{M})& =\underset{(i,j)\in {\Omega }^{+}}{\max }{\widehat{M}}_{i,j},\hfill \end{array}$$

  (3)

  $$\begin{array}{cc}\hfill g(\widehat{M})& =\underset{(i,j)\in {\Omega }^{-}}{\max }{\widehat{M}}_{i,j},\hfill \end{array}$$

  (4)

  $$\begin{array}{cc}\hfill {\mathcal{L}}_{margin}^{shape}(\widehat{M})& =\max (0,g(\widehat{M})-f(\widehat{M})+\delta ),\hfill \end{array}$$

  (5)

  with $\delta =0.1$. The hinge is zero when the strongest human-region response exceeds the strongest non-human response by $0.1$. When it is active, gradient descent raises the inside peak and lowers the outside peak. Both reductions are maxima over their own index sets. They are not a minimum or a maximum of a map that has been multiplied by an indicator, which would insert zeros from the complementary region into the statistic. The token loss on a map is ${\mathcal{L}}_{\mathrm{token}}^{shape}={\mathcal{L}}_s^{shape}+{\mathcal{L}}_{margin}^{shape}$.
• Pixel loss. The token terms control where the attention mass sits. A pixel term then fits the map to the mask with a binary cross-entropy. Masks stored in [−1, 1] are mapped by $(M+1)/2$ before this loss. On each map, the pixel term averages the cross-entropy at the native resolution and the cross-entropy of the upsampled map:

  $${\mathcal{L}}_{scale}^{shape}(\widehat{M})={\displaystyle \frac{1}{2}}\left(\mathrm{BCE}(\widehat{M},{\tilde{M}}^{shape})+\mathrm{BCE}(\mathrm{Up}(\widehat{M}),\mathrm{Up}({\tilde{M}}^{shape}))\right),$$

  (6)

  where ${\tilde{M}}^{shape}=(M^{shape}+1)/2$. The shape-binding objective used in the ablations is the combination of the three terms,

  $$\begin{array}{cc}\hfill {\mathcal{L}}_{bind}^{shape}& ={\mathcal{L}}_s^{shape}+{\mathcal{L}}_{margin}^{shape}+{\mathcal{L}}_{scale}^{shape}.\hfill \end{array}$$

  (7)

In the training run, the noise objective and these attention terms are weighted as 0

$$\mathcal{L}={\mathcal{L}}_{noise}+0.005{\mathcal{L}}_{\mathrm{token}}+0.2{\mathcal{L}}_{\mathrm{pixel}},$$

(8)

where ${\mathcal{L}}_{noise}$ is the mean-squared noise loss in Equation (1), ${\mathcal{L}}_{\mathrm{token}}$ collects the suppressing and margin terms, and ${\mathcal{L}}_{\mathrm{pixel}}$ is the cross-entropy above. SNR weighting is disabled, and the noise offset is 0. With this objective, the cross-attention modules learn which information is carried by the human-related tokens and place it on the body. TokenCompose [25] also proposes regularizing the cross-attention maps of a diffusion UNet with extra objectives. However, since TokenCompose focuses on improving the textual fidelity of diffusion models, it is sufficient for the cross-attentions to provide rough locations for each semantic token. Compared with that, rectifying human body deformity requires more refined attention control, otherwise the unrelated information could easily lead to mistakenly drawn human bodies. Therefore, we propose a composite objective with multiple sub-goals, which can provide stronger supervision as we will show in the experiments.

• Pose tokens. The proposed ${\mathcal{L}}_{bind}^{shape}$ can guide the diffusion UNet to attend to the human-related tokens in text prompts on human-related regions in the images. However, it is hard to correctly portray a human body solely based on the shape information. For example, when a man crosses his arms, $M^{shape}$ can only show that the arms are not stretched, but cannot tell what exact pose the man is holding. As a result, it is also important to embed to pose-related information into the diffusion UNet. In spite of its importance, simply copying ${\mathcal{L}}_{bind}^{shape}$ to also train the human-related tokens with pose information means these tokens have to learn two different levels of knowledge, which would be much harder.

To this end, we propose a simple yet effective alternative method. Specifically, a learnable pose token <sds>, which is aimed at representing pose-related information, is attached to each text prompt. To supervise the pose token with prior knowledge regarding poses, for each image $\mathbf{I}$, we leverage MMPose [26] to detect the keypoints ${\left\{p_i\right\}}_{i=1}^{N_p}$, where $N_p$ denotes the maximum number of keypoints. Then, $\left\{p_i\right\}$ is processed into a skeleton mask $M^{pose}$, in which lines are drawn between adjacent keypoints following human structure (e.g., neck and shoulders, elbows and wrists). After setting up <sds> and $M^{\mathit{pose}}$, we can build the training scheme following the same approach as guiding human-related tokens. Concretely, for each training iteration, we collect the cross-attention maps ${\widehat{\mathcal{M}}}^{\mathit{pose}}$ regarding <sds>, and calculate the following objectives:

$$\begin{array}{cc}\hfill {\mathcal{L}}_{bind}^{\mathit{pose}}& ={\mathcal{L}}_s^{\mathit{pose}}+{\mathcal{L}}_{margin}^{pose}+{\mathcal{L}}_{scale}^{\mathit{pose}},\hfill \end{array}$$

(9)

in which each item follows Equation (7), with the skeleton mask in place of the body mask. The margin on the skeleton likewise uses $\delta =0.1$ and compares the peak on the skeleton with the peak off the skeleton. SAM masks and MMPose skeletons are structural priors. Figure 2c shows typical failures, including a hand assigned to the wrong person and a skeleton drawn through an occluded part. A local mask error changes the outside mean, the two peaks, and the pixel target on the pixels it covers. It is not treated as a label of the whole image.

LoRA parameters [27] are attached to the pretrained UNet and optimized together with the pose-token embedding. The rank is 16, and the scaling coefficient is 16. The adapted projections are the query, key, value, and output projections of the attention layers in the down, mid, and up blocks. The text encoder is frozen. Once the pose token is enabled, the body term and the pose term inside each attention group are averaged, and the group losses are then added in Equation (8).

4.2. Adversarial Attention Rectification

While the above training strategy can guide the model to correctly bind human-related tokens with their semantic regions in the training set, this property can hardly be generalized to unseen cases, since the shape and pose of humans can severely vary among different images according to their ages, genders, events, and even the background scenarios. This makes the distribution of shape and pose data a sparse one, leading to great difficulty for the model to transfer the learned knowledge to a wide range of real cases. To solve this problem, we further propose leveraging adversarial training, which directly learns the data distribution via the supervision of a learnable discriminator rather than reciting the one-to-one mapping.

Specifically, we set up two discriminators, $D^{shape}$ and $D^{pose}$, for shape and pose data, respectively, corresponding to the human-related tokens and extra pose tokens. During each training iteration, we first get the intermediate cross-attention map sets ${\widehat{\mathcal{M}}}^{shape},{\widehat{\mathcal{M}}}^{pose}$, together with their ground truth annotations $M^{shape},M^{pose}$, using the same method as in Section 4.1. The discriminators treat these masks as samples of the real attention distribution, so the second stage matches that distribution rather than copying each training mask. The reported models optimize the Wasserstein objective [28]. The discriminators are updated by minimizing

$$\begin{array}{cc}\hfill {\mathcal{L}}_D^{shape}& ={\displaystyle \frac{1}{|{\widehat{\mathcal{M}}}^{shape}|}}\sum _{\widehat{M}\in {\widehat{\mathcal{M}}}^{shape}}{D}^{shape}(\widehat{M})-D^{shape}(M^{shape}),\hfill \end{array}$$

(10)

$$\begin{array}{cc}\hfill {\mathcal{L}}_D^{pose}& ={\displaystyle \frac{1}{|{\widehat{\mathcal{M}}}^{pose}|}}\sum _{\widehat{M}\in {\widehat{\mathcal{M}}}^{pose}}{D}^{pose}(\widehat{M})-D^{pose}(M^{pose}).\hfill \end{array}$$

(11)

A gradient penalty with coefficient $\alpha =10$ is implemented in this branch. This coefficient is independent of the LoRA scaling coefficient. The reported configuration leaves the penalty inactive. The fake map given to a discriminator is resized to $512\times 512$ and min–max normalized. Accordingly, the diffusion UNet is optimized as follows:

$$\begin{array}{cc}\hfill {\mathcal{L}}_U& ={\mathcal{L}}_{noise}+{\mathcal{L}}_G^{shape}+{\mathcal{L}}_G^{pose},\hfill \end{array}$$

(12)

$$\begin{array}{cc}\hfill {\mathcal{L}}_G^{shape}& =-{\displaystyle \frac{1}{|{\widehat{\mathcal{M}}}^{shape}|}}\sum _{\widehat{M}\in {\widehat{\mathcal{M}}}^{shape}}{D}^{shape}(\widehat{M}),\hfill \end{array}$$

(13)

$$\begin{array}{cc}\hfill {\mathcal{L}}_G^{pose}& =-{\displaystyle \frac{1}{|{\widehat{\mathcal{M}}}^{pose}|}}\sum _{\widehat{M}\in {\widehat{\mathcal{M}}}^{pose}}{D}^{pose}(\widehat{M}).\hfill \end{array}$$

(14)

Through the adversarial minimax game, the diffusion UNet can gradually learn how to adapt the LoRA weights so that the cross-attention maps can be rectified to ideal shapes and poses. Compared with previous works [22] that also adopt adversarial supervision for diffusion models, the most noticeable difference is that our method does not rely on the noisy prediction ${\widehat{z}}^0$ achieved from ${\widehat{z}}^t$, which results in two merits. First, the VAE decoder is not required for calculating discriminator prediction, thus being more efficient. Second, the input of discriminators are binary masks, which are much easier than RGB images, thus making them better functioning for producing guidance supervision for the diffusion UNet.

5. Experiments

5.1. Implementation Details

• Dataset. The training set contains 176,092 images, assembled from Cosmic/CosmicMan, an Asian portrait collection, in-house images, and images gathered by image search. BLIP-2 [29] captions the retained images. One search batch keeps an image when its aesthetic score is above 5, its watermark score is below 3, and its similarity score is above 0.62. That batch contributes 29k images from Cosmic and 28k images from the existing collection. Face parsing then removes incomplete faces and half faces, discarding about 3k images. The retained search results are essentially real single-person portraits. A manual pass removes explicit content, watermarks, truncated bodies, blur, spliced images, and other low-quality frames, and a second pass removes small watermarks and white borders. These images are used to learn the shape and pose priors. Inference does not reload them. For the quantitative comparison, we follow HcP [12] and use the validation captions of HumanArt [30]. The qualitative study adds 50 prompts on which pretrained Stable Diffusion tends to deform the body. Training and sampling hyperparameters are given in Appendix A.

5.2. Quantitative Results

We first present quantitative evaluations in this part. FID [31] and KID [32] compare global image features. CLIP-Score measures prompt–image agreement. HPS-v2 [33] and PickScore [34] predict a general preference. None of these scores counts a missing limb, an extra limb, an abnormal joint, or a broken body proportion, so Table 1 is read as an image-level comparison. The table holds the SDXL backbone, the training images, and the sampler fixed. The LoRA row is trained on the same 176,092 images with the noise objective only. Our model is ahead of both the pretrained model and this LoRA baseline on every column. The LoRA baseline itself improves FID and CLIP-Score, leaves HPS unchanged, and is behind the pretrained model on KID and PickScore. The anatomical comparison is the paired generation protocol in the next subsection. Conditional methods such as ControlNet and HyperHuman take a spatial map at inference and answer a different problem. A public implementation of HcP that can be trained under this matched protocol is not available, so Section 2.1 compares it at the level of the training signal.

Table 1. Quantitative result with SDXL as the backbone. Downward arrows indicate that a smaller score is better, and upward arrows indicate that a larger score is better. The best score in each column is shown in bold.

Methods	FID ↓	KID ↓	CLIP ↑	HPS ↑	PickScore ↑
Pretrain	41.87	12.44	33.94	23.06	22.38
LoRA	40.34	12.54	34.68	23.06	22.31
Ours	38.11	11.77	34.84	23.17	22.61

5.3. Qualitative Results

We present several uncurated results for SDXL in Figure 4. For each prompt, three generation seeds are used, and the pretrained model, the LoRA baseline, and our model share the seed, the prompt, and the sampler. A difference inside one panel is therefore a difference of method under the same initial noise. The same protocol is used for SD1.5 in Figure 5. We read the panels with four visible failure modes. A missing limb is a body part required by the person and the action that is absent. An extra limb is a repeated arm, leg, hand, or head. An abnormal joint bends where there is no joint or in a direction the joint does not allow. A proportion error is a torso, limb, or head whose size disagrees with the rest of the body. Interestingly, we find that while the prompts are generally short and easy for a human to understand, they are especially hard for the pretrained SDXL to generate corresponding images. For example, when generating common actions such as sitting or jumping, which could be abundant in the pretraining dataset, the pretrained SDXL still cannot produce satisfactory results, leaving many artifacts such as unorganized body, redundant arms and hands, and improperly shaped limbs. When it comes to rare actions (e.g., doing yoga, dancing ballet) or rare combinations of actions and objects (e.g., holding a rifle), the deformity becomes more severe. As our baseline method, we find that finetuning SDXL on our dataset directly with LoRA does learn some information from the data, given that the finetuned model can generate much detailed background in many images. Unfortunately, this straightforward method has no positive effect and even makes it worse in many cases with regard to human body generation. For example, when generating according to the prompt ’a woman doing a yoga pose on a yoga mat’, the LoRA-finetuned SDXL can hardly generate a complete human body. This indicates that even if models with sufficient capacity (e.g., SDXL) can be finetuned with specific training data (e.g., high-quality human images), the problem of human body deformity can still not be solved. Compared with these methods, our method can guide the model to learn appropriate human body priors from the training data, resulting in much better human bodies. Meanwhile, our method does not require additional design for the network structure and input conditions, thus combining the best of both worlds.

Figure 4. Qualitative comparison with SDXL as backbone. Each comparison is controlled with the same random seed.

Figure 5. Qualitative comparison with SD1.5 as backbone. Each comparison is controlled with the same random seed.

To further show the generalization ability of our method, we apply it to pretrained SD1.5, whose results are presented in Figure 5. The results are consistent with those of SDXL. In general, since SD1.5 itself is worse than SDXL, the image quality among three methods lags behind images generated by SDXL in terms of the details. Nonetheless, our proposed method can still help the model fix the problem of human body deformity, leading to more reasonable human bodies under different kinds of prompts. These SD1.5 samples use the same prompts and the same shared seeds, so the scheme is assessed on a smaller backbone as well as on SDXL. The prompts mix ordinary actions, such as sitting and jumping, with less common actions, such as yoga and ballet, and with a less common pairing of an action and an object, holding a rifle. Further SDXL panels under the same protocol appear in the Appendix A. The captions used for Table 1 are the HumanArt validation captions, which are separate from the training images. Inference does not reload the training collection.

5.4. Ablation Study

To further verify the effectiveness of our proposed method, we conduct extensive ablation studies using SDXL as the backbone. Since the above adopted quantitative metrics cannot intuitively reflect the quality of generated human bodies, we only provide the qualitative results, with parts of them shown in the main paper. For more a comprehensive comparison, please refer to the Appendix B. Each variant below keeps the SDXL backbone and the prompt fixed and changes one ingredient: the binding terms, the assignment of shape and pose, or the adversarial stage. The generated body and the attention map record that change.

• How can ${\mathcal{L}}_{bind}$ help the model? We first test the efficacy of each item in ${\mathcal{L}}_{bind}$, as illustrated in Equation (7); the generated images together with the averaged cross-attention maps regarding human-related tokens are shown in Figure 6. Similarly to the results in Section 1, we can see that the pretrained SDXL cannot concentrate on the generated human body regarding the human-related tokens. In some cases (e.g., the second and fourth from the left), the model even highlights the background than the human bodies. Consequently, it cannot gather correct information, thus leading to deformed human bodies. Directly finetuning SDXL with LoRA also does not work, resulting in almost the same attention maps as the pretrained model. This indicates that the noise prediction loss ${\mathcal{L}}_{noise}$ cannot solely guide the cross-attention modules to function correctly; hence, leveraging the additional objectives is necessary. As for the three items proposed by us, the suppressing loss reduces background mass and leaves residual activation outside the body. Adding the margin loss, which enforces a gap of $0.1$ between the inside peak and the outside peak, makes the body the main support of the map, at a smaller scale. Adding the pixel loss raises the response on the body toward the mask. The full binding objective is the variant in which the map both covers the body and responds strongly there. These conclusions are read from the images and the attention maps under a fixed backbone and prompt.

Figure 6. Ablation comparison among model variants using different objectives.

Figure 6. Ablation comparison among model variants using different objectives.

• Effectiveness of the pose token. In Figure 7, we present three model variants regarding the learning strategy for human body priors: (1) OnlyShape: we only learn shape-related information with original human-related tokens in the prompts; (2) OnlyPose: similar to OnlyShape, with shape-related information replaced with pose-related information; and (3) Ours: our proposed strategy, as in Section 4.1, i.e., using an extra pose token to learn pose information. To make the results simple for understanding, we average attention activation among all human-related tokens and the pose token. Basically, solely learning shape-level information can help the model better concentrate on the human bodies. However, since pose information is not injected, the model cannot avoid the problem of abnormal poses, i.e., missing or additional limbs. On the other hand, it is also difficult for human-related tokens to learn pose information, considering the results that the attention maps can only concentrate on specific regions such as faces. Compared with these two variants, our proposed strategy can help the model leverage attention maps that truly highlight the reasonable human body regions, thus generating human body images with higher quality.

Figure 7. Ablation of how shape and pose are assigned. Columns compare OnlyShape, OnlyPose, and our dual-level tokens. Each column shows the generated image and the averaged cross-attention map.

Figure 7. Ablation of how shape and pose are assigned. Columns compare OnlyShape, OnlyPose, and our dual-level tokens. Each column shows the generated image and the averaged cross-attention map.

• Effectiveness of adversarial training. In Figure 8, we compare three model variants including pretrained SDXL; SDXL finetuned with only the semantic attention binding stage, as proposed in Section 4.1; and the model finetuned with both semantic attention binding and adversarial attention rectification. We find that the first stage can help the model generate better human bodies in some cases. For example, in the first column, while the pretrained SDXL generates an additional head, the one-stage model can revise this mistake. However, the human bodies generated by this one-stage model are often stilted, while also being prone to deformity, which shows its limited generalization ability due to fully supervised learning. In comparison, the model trained with both stages, as shown in the last row, can be generalized to different prompts, resulting in more appropriate human poses and shapes. This comparison keeps the backbone and the prompts fixed and changes only the presence of the adversarial stage. The prompts are those described above, covering ordinary actions and less common ones. Stage 1 removes errors such as the extra head and still leaves stiff bodies. The two-stage model is the variant in which the pose and the shape stay coherent. Occlusion enters the study as a failure of the automatic targets, illustrated in Figure 2c. The second stage treats the collection of masks as the real distribution, so the UNet is guided by that population of body configurations.

Figure 8. Ablation of adversarial training. From top to bottom: pretrained SDXL, binding only, and binding followed by adversarial rectification.

Figure 8. Ablation of adversarial training. From top to bottom: pretrained SDXL, binding only, and binding followed by adversarial rectification.

6. Discussion

The adaptation is complementary to scaling. Figure 2 shows that pretrained SDXL can place human-token attention on the background, and Figure 6 shows that a LoRA update with only the noise loss leaves maps of the same kind. The same binding and adversarial objectives improve both SD1.5 and SDXL. On SDXL, the gain is also visible in Table 1. On SD1.5, it is visible in the paired samples of Figure 5. The training cost is the two-stage schedule in Appendix A. At inference, the added computation is the UNet LoRA. The user does not supply a structural condition.

The seeds in Figure 4 and Figure 5 are shared generation seeds. Together with the matched training images, the shared sampler, and the repeated prompts on two backbones, they are the protocol used to read the comparison. Initialization, batch order, and the adversarial updates remain part of a fine-tuning run. The same dependence on initialization, data partitioning, and training configuration has been reported for automated fault detection of air-handling units, in a different application domain [35,36]. A multi-run mean would summarize that training variation. The present evidence is the controlled comparison above.

SAM and MMPose supply the masks used by both stages. Their characteristic errors are illustrated in Figure 2c. The outside mean, the peak-to-peak margin, and the pixel term all respond to the mask on the pixels it marks. The adversarial stage then uses the collection of masks as the real distribution, which is how the method reduces dependence on any single noisy target.

Table 1 measures the proposed objectives on a fixed text-to-image backbone. The LoRA baseline is trained on the same 176,092 images with the noise objective and without the binding losses or the adversarial stage. ControlNet and HyperHuman receive a spatial condition at sampling time and are discussed in Section 2.1 as conditional generators. HcP is discussed in the same section through its training signal: a pose branch added on top of the original attention, pose supervision, and fully supervised pairs. Our method suppresses background attention, binds shape and pose, and then matches the distribution of attention maps. The ablations supply the corresponding contrasts. The suppressing and margin terms move human-token mass onto the body. Shape-only supervision covers the silhouette and still shows missing or extra limbs. Pose supervision on the original human tokens concentrates on local regions such as the face. The separate pose token is the variant whose averaged map follows a coherent body. Figure 8 compares the binding stage with both stages on the same prompts, including sitting, jumping, yoga, ballet, and holding a rifle. Stage 1 can remove an extra head and still produces stiff bodies. The two-stage model keeps a coherent pose and shape on those prompts, and the same shared-seed protocol is repeated on SD1.5. The HumanArt validation captions used in the table are separate from the training images.

The support reported here for the adversarial stage is this paired comparison, together with the selected images and attention maps. A dedicated evaluation would extend the same question to three further settings: actions and articulations held out from the poses seen in training; cases in which the body is heavily occluded by an object or by another person; and configurations that vary body shape, age, viewpoint, and the number of people. Those settings measure the generalization claim more directly than the panels above, and they are the direction in which this study extends. The qualitative ablations are the evidence reported here for the contribution of each component. A quantitative comparison of the same variants on the reported metrics is left to follow-up work.

7. Conclusions

To address anatomical errors in text-to-image diffusion models arising from inadequate human shape and pose representations, we presented a dual-level prior injection method for improving human body generation. Our method associates shape priors with human-related tokens and pose priors with a learnable pose token through a two-stage bind-then-generalize strategy. Semantic attention binding suppresses background responses and aligns token attention with the corresponding human regions, while adversarial attention rectification learns shape and pose distributions beyond individual supervision targets. Paired qualitative comparisons on SD1.5 and SDXL show more coherent human bodies than the pretrained models and LoRA baselines trained on the same images. On SDXL, our method improves all five reported image-level metrics, reducing FID from 41.87 to 38.11 relative to the pretrained model. Qualitative ablations further support the contributions of the binding objectives, the separate pose token, and adversarial rectification.