Multi-Scale Optimal Transport Transformer for Efficient Exemplar-Based Image Translation

Abstract

Exemplar-based image translation generates an output image by transferring appearance from a reference exemplar to a content image. Existing works only consider the local correspondences between two modalities, and ignore the global distributions in each modality, struggling to obtain fine-grained details with efficient computation. In this paper, we propose OTFormer, a multi-scale Optimal Transport transformer for exemplarbased image translation. We formulate cross-modal alignment as a multi-scale optimal transport problem, which progressively provides a globally coherent matching. In addition, we design a lightweight multi-scale fusion block to extract and fuse features efficiently. Experiments on CelebA-HQ and DeepFashion demonstrate that OTFormer improves both image fidelity and style adherence, while reducing model parameters by 62% and achieving faster inference compared with strong baselines. These results highlight OTguided global alignment as an effective and deployable solution for high-fidelity exemplarbased image translation.

1. Introduction

Image-to-image translation is an important problem in computer vision, which has many applications, such as conditional image editing [1,2] and controllable synthesis [3,4,5]. Among its variants, exemplar-based image translation aims to generate an output by transferring appearance attributes (e.g., texture, color, and local details) from a reference exemplar to a content image while preserving the underlying structure. Despite rapid progress, high-fidelity transfer, i.e., generating images that are both photorealistic and accurately preserve the structure of the content image while inheriting the appearance of the exemplar, remains difficult. A method must estimate reliable cross-image correspondences under semantic mismatch, and it should handle pose or layout changes and occlusions. Moreover, it must still satisfy deployment constraints on model size and inference speed. This setting exposes a persistent tension between correspondence accuracy and model efficiency.

Current methods largely fall into two streams. Each stream addresses a different side of the tension.

• Attention-based methods [6,7,8,9] use dense pairwise attention to compute correspondences. They rely on attention mechanisms with local similarity matching. However, the local matching only considers the correlation between cross-modal local features, ignoring global structure in each domain. This weakness can produce misaligned features and artifacts, such as distorted textures or abnormal style placement. Besides, the quadratic complexity in attention mechanisms and large parameter footprint increase computation and memory demands, which makes real-world deployment harder.
• Diffusion-based models [10,11] can achieve strong generation fidelity. However, exemplar-based translation rarely offers precisely aligned exemplar–content pairs. This data scarcity limits direct supervision for the denoising process [10,11]. Furthermore, diffusion models also require iterative sampling at inference time, which slows inference and increases computational cost [12]. These properties reduce their appeal in efficiency-critical scenarios.

We argue that the core issue is the lack of a principled framework for cross-domain feature alignment. A useful framework should be globally coherent and computationally efficient. Optimal Transport (OT) theory [13,14] provides a good foundation and mathematical tool for alignment. OT casts alignment as a minimal-cost mapping between two distributions. UNITE [15] pioneered unbalanced OT for exemplar-based translation. However, it still relies on a single-scale OT formulation and a conventional CNN decoder. These choices limit fine-grained transfer and multi-scale style control. More importantly, existing methods do not fully leverage OT in lightweight multi-scale designs. Moreover, previous methods also fail to use OT to support local semantic similarity and global structural preservation at the same time.

To address these problems, we propose OTFormer by integrating OT theory into a hierarchical Transformer framework for exemplar-based translation. OTFormer replaces local similarity attention with a multi-scale OT formulation for correspondence learning, which encourages semantic matching and enforces spatial coherence across domains. Besides, we stack several OTFormer blocks at different resolutions for multi-stage refinement. Early blocks operate at a low resolution and captures global style layout, and later blocks operate at a higher resolution and refine local textures and details. To achieve parameter-efficient modeling, we also introduce a lightweight Multi-Scale Fusion (MSF) block. Specifically, we adopt depthwise convolution for efficient feature extraction and introduce parallel depthwise convolutions with different receptive fields to achieve multi-scale feature aggregation. The experimental results demonstrate the effectiveness of our method compared with existing methods.

The main contributions can be summarized as follows:

• We propose OTFormer for exemplar-based translation, which provides a globally coherent and theoretically grounded alternative to local attention matching.
• We design a progressive alignment scheme and a lightweight MSF block, which supports coarse-to-fine style transfer while maintaining strong parameter efficiency.
• We show that OTFormer outperforms GAN-based and diffusion-based methods in visual quality and semantic consistency, offering a better efficiency profile in parameters and inference.

The remainder of this paper is organized as follows. Section 2 reviews related work on exemplar-based image translation, covering GAN-based, diffusion-based, and optimal transport-based methods. Section 3 details the proposed OTFormer framework, including the multi-scale optimal transport formulation and the lightweight multi-scale fusion block. Section 4 presents experimental results and ablation studies on CelebA-HQ and DeepFashion. Section 5 concludes the paper.

2. Literature Review

Exemplar-based image translation has mainly progressed through two generative paradigms. The first paradigm uses Generative Adversarial Networks (GANs), while the second paradigm uses diffusion models. We also include the previous OT-based methods related to exemplar-based translation.

2.1. GAN-Based Methods

GAN-based methods [16,17,18] for exemplar-based translation typically learn a mapping from a content image and an exemplar to an output by leveraging adversarial training. Early approaches focused on learning style transfer through cycle consistency or feature matching, but often struggled with fine-grained alignment. Later works introduced explicit correspondence learning mechanisms to improve transfer quality.

Generative Adversarial Networks (GANs) [19,20] have been a core tool for exemplar-based translation. GAN-based methods can transfer visual style while preserving content structure in many settings [21,22]. CoCosNet [6] introduced a dual-encoder design with pixel-wise attention by estimating correspondences between the content and the exemplar. CoCosNet delivers strong results, but the dense matching in pixel space is expensive. CoCosNet-v2 [7] addresses this cost with a ConvGRU-assisted PatchMatch strategy. This strategy improves matching efficiency across multiple resolutions.

Transformer architectures have further pushed this line of work. DyNAST [8] uses dynamic sparse attention to refine feature transformation. MAT [9] proposes masked adaptive transformers to achieve better alignment for high-quality image synthesis. Chiu et al. [23] propose a method that incorporates object co-saliency awareness into the colorization process, leading to more natural and visually appealing results. HyperplaneGAN [24] proposes a novel unified translation framework to achieve facial-attribute editing and synthesis. These methods improve flexibility, but they still struggle to obtain fine-grained details with efficient computation. This is because standard attention mainly models pairwise cross-image similarities, and ignores intra-image structural dependencies, which can lead to style inconsistency or content blur in generated images.

2.2. Diffusion-Based Methods

Diffusion models have recently emerged as a powerful class of generative models. For exemplar-based translation, they are typically adapted by conditioning the denoising process on both content and exemplar images. The iterative refinement process of diffusion enables high-quality synthesis, but at the cost of slow inference and large model sizes.

Diffusion models [25,26] have recently achieved strong performance in image generation. These models synthesize images through iterative denoising. A learned reverse process gradually transforms noise into an image.

Several works adapt diffusion models to exemplar-based translation. MIDMS [10] propose a matching-and-generation framework with exemplar guidance. EBDM [11] propose a Brownian bridge diffusion model for inter-domain transformation. These approaches can produce high-quality images, but they face practical constraints. Exemplar-based translation rarely provides tightly aligned training pairs. This data scarcity makes it hard to accurately supervise denoising trajectories. Kosuji et al. [27] address this issue by leveraging the pre-trained diffusion model to achieve exemplar-based color translation. Jin et al. [28] propose a novel exemplar-based image synthesis framework by adapting the powerful diffusion models using the proposed appearance matching adapter. Diffusion inference also requires many sampling steps, which increases computational cost and slows inference [12,29,30]. These limitations reduce feasibility for real-time or resource-limited deployment.

2.3. Optimal Transport-Based Methods

Instead of attention mechanisms, Optimal Transport (OT) offers a principled alternative for feature alignment [13,31]. OT defines correspondence as a transport plan between feature distributions, which can be obtained through cost minimization by solving the OT problem. This global optimization perspective enforces spatial coherence and is robust to local ambiguities.

UNITE [15] introduces unbalanced OT [32,33] for cross-domain alignment. However, it adopts a single-resolution OT formulation and a conventional CNN decoder, which limits the capacity of high-quality image translation. In contrast, multi-scale OT can capture cross-image matching and within-image structure more directly, but prior GAN-based methods do not fully explore this potential.

Our OTFormer integrates multi-scale OT into progressive correspondence estimation. This design treats alignment as a global optimization problem rather than a purely local matching step, which preserves cross-modal similarity and intra-modal structure during refinement. A lightweight multi-scale fusion module further improves efficiency, reducing parameters while maintaining synthesis quality.

3. Methodology

In this section, we present the proposed OTFormer framework. We first describe the overall architecture, then detail the optimal transport formulation within each OTFormer block, and finally introduce the lightweight multi-scale fusion block.

3.1. Optimal Transport Transformer

We propose OTFormer, a new framework for exemplar-based image translation. OTFormer integrates optimal transport theory with hierarchical feature learning. This design targets key limitations in prior work. GAN-based methods [6,7,8,9] often depend on expensive attention mechanisms, while diffusion-based methods [10,11] often require slow iterative sampling and large models. OTFormer takes a different route by formulating feature alignment as a multi-scale mass transport optimization problem. This formulation not only encourages globally coherent correspondences across images, but also preserves inter-modal style consistency. At the same time, the formulation maintains intra-modal structural fidelity.

As illustrated in Figure 1, the framework consists of dual encoders extracting multi-scale features from input content image $I_c$ and exemplar image $I_e$, respectively, followed by a decoder comprising multiple cascaded OTFormer blocks. Each OTFormer block solves an optimal transport problem between feature distributions, and produces a minimal-cost transport plan. Then, we can warp the exemplar features to align with the content features using the transport plan. Then, we adopt a lightweight multi-scale fusion block to extract and fuse features, which reduces parameters while achieving superior synthesis quality.

3.1.1. Encoders

OTFormer employs symmetric dual encoders $E_c$ and $E_e$ to extract hierarchical features from input content image and input exemplar image. Given $I_c,I_e\in {\mathbb{R}}^{H\times W\times 3}$, the encoders produce multi-scale feature pyramids:

$${\left\{I_c^l\right\}}_{l=0}^2=E_c\left(I_c\right),{\left\{I_e^l\right\}}_{l=0}^2=E_e\left(I_e\right),$$

(1)

where $I^l$ denotes the feature map at pyramid level l. Each encoder contains three strided convolutional blocks, and each block increases the receptive field progressively. This design supports robust multi-scale representation learning, which enables coarse-to-fine correspondence estimation. Note that the model uses encoders with the same architecture but different weights for the content and exemplar branches.

3.1.2. OTFormer Block

The decoder is formed by a cascade of OTFormer blocks, each tasked with estimating precise cross-modal correspondences and refining generated features with low computational cost. As depicted in Figure 2, each OTFormer block receives three inputs: the current generated feature map $I_g^0$, the content feature $I_c^1$, and the exemplar feature $I_e^1$. Central to the block is a differentiable, entropically regularized optimal transport solver (implemented via the Sinkhorn algorithm), which computes a transport plan T between the generated and exemplar feature distributions. This plan is used to warp exemplar features into the spatial domain of the generated features, followed by efficient multi-scale fusion with the content features to produce the refined output $I_g^1$.

• Optimal transport We summarize the discrete optimal transport formulation used in the OTFormer block. Let

$$X={\left\{x_i\right\}}_{i=1}^n,Y={\left\{y_j\right\}}_{j=1}^m$$

be two sets of feature vectors obtained by spatially flattening $I_g^0$ and $I_e^1$, respectively, where $x_i,y_j\in {\mathbb{R}}^C$. Their marginal weight vectors are denoted $a\in {\Delta }_n$ and $b\in {\Delta }_m$ (default uniform distributions: $a_i={\displaystyle \frac{1}{n}}$, $b_j={\displaystyle \frac{1}{m}}$). The cost matrix $C\in {\mathbb{R}}^{n\times m}$ is defined by a dissimilarity measure; we adopt cosine-based dissimilarity:

$$C_{ij}=1-{\displaystyle \frac{\langle x_i,y_j\rangle }{\parallel x_i\parallel \parallel y_j\parallel }}.$$

(2)

We choose cosine dissimilarity because deep features tend to be high-dimensional and their magnitudes can vary due to factors like contrast or illumination; cosine similarity focuses on the direction of feature vectors, which better captures semantic content. Moreover, it is differentiable and has been widely used in attention mechanisms, making it a natural fit for our end-to-end training.

The hyperparameters in Algorithm 1 are chosen as follows: $ϵ={10}^{-8}$ prevents division by zero; the regularization parameter $\tau =0.03$ balances convergence speed and solution accuracy, selected empirically based on validation performance; $\gamma =1$ controls the unbalancedness (here we use balanced OT). The maximum number of iterations is set to 20, which is sufficient for convergence in practice.

Algorithm 1 Sinkhorn-Knopp Algorithm

1: procedure  Initialization 2:     Normalize features: $I_g^0\leftarrow I_g^0/\sqrt{\parallel I_g^0{\parallel }_2^2+ϵ}$                             ▹$ϵ={10}^{-8}$ 3:     Compute cost matrix: $C\leftarrow 1-\langle I_g^0,I_e^1\rangle \in {\mathbb{R}}^{B\times L\times L}$ 4:     Initialize kernel: $K\leftarrow exp(-C/\tau )$                                       ▹$\tau =0.03$ 5:     Set $\rho \leftarrow \gamma /(\gamma +ϵ)$                                                                          ▹$\gamma =1$ 6:     Initialize $a\leftarrow {\mathbb{1}}_L/L$, $b\leftarrow {\mathbb{1}}_L/L$ 7: end procedure 8: procedure  SinkhornIteration 9:     for $k=1$ to $max\text{\_}iter$ do 10:         $b\leftarrow {\left({\displaystyle \frac{{\mathbb{1}}_L/L}{K^{\top }a+ϵ}}\right)}^{\rho }$                                                         ▹ Target marginal 11:         $a\leftarrow {\left({\displaystyle \frac{{\mathbb{1}}_L/L}{Kb+ϵ}}\right)}^{\rho }$                                                           ▹ Source marginal 12:     end for 13:     $T\leftarrow \mathrm{diag}\left(a\right)K\mathrm{diag}\left(b\right)\in {\mathbb{R}}^{B\times L\times L}$ 14:     $T^{*}\leftarrow T/{\sum }_{j=1}^L{T}_{:,:,j}$                                                  ▹ Column normalization 15: end procedure

We solve the entropically regularized Kantorovich problem:

$$\begin{array}{cc}\hfill T^{*}& =arg\underset{T\in \mathcal{U}(a,b)}{min}\langle C,T\rangle -\epsilon H\left(T\right),\hfill \end{array}$$

(3)

$$\begin{array}{cc}\hfill & =arg\underset{T\in \mathcal{U}(a,b)}{min}\sum _{i,j}{C}_{ij}{T}_{ij}+\epsilon \sum _{i,j}{T}_{ij}log{T}_{ij},\hfill \end{array}$$

(4)

where $\mathcal{U}(a,b)=\{T\in {\mathbb{R}}_{+}^{n\times m}\mid T{1}_m=a,T^{\top }{1}_n=b\}$, $H\left(T\right)$ is the entropy term, and $\epsilon >0$ is the entropic regularization coefficient. Problem (3) is efficiently solved using Sinkhorn–Knopp iterations [34,35], implemented in the log-domain for numerical stability. Taking the optimization between the generated feature $I_g^0$ and the exemplar feature $I_e^1$ as an example, the Sinkhorn–Knopp algorithm can be written as Algorithm 1. The resulting transport plan $T^{*}$ is differentiable, enabling end-to-end training.

• Warping and fusion Using the transportation map $T^{*}$ and exemplar features $I_e^1$, we warp the exemplar features to obtain style-consistent generated features:

$$I_w^1=T^{*}\times I_e^1,$$

(5)

where × denotes matrix multiplication. The warped feature $I_w^1$ is roughly aligned with the generated feature $I_g^0$. These are fused by simple addition, followed by bilinear upsampling if higher resolution is required. Subsequently, a lightweight multi-scale fusion (MSF) block refines the fused feature:

$$I_g^{\prime }=\mathrm{MSF}\left(\mathrm{Up}(I_w^1+I_g^0)\right).$$

(6)

To improve consistency with the content image, each OTFormer block concatenates content features along the channel dimension. This concatenation encourages the generated features to remain aligned with the content structure. Then, two MSF blocks are used to fuse the concatenated features. These MSF blocks produce the final generated feature $I_g^1$.

• Multi-scale Fusion Block Exemplar-based translation needs effective feature reuse and strong multi-scale context aggregation. Many existing methods use large networks to boost capacity. However, these networks can produce unstable feature statistics during training, and can also miss important multi-scale context or lose information during processing. These issues limit the modeling of complex structural dependencies.

Therefore, we introduce a multi-scale fusion (MSF) block to aggregate multi-scale spatial context efficiently. Figure 3 presents the overall architecture of the MSF block. The MSF block extracts spatial context at multiple scales and fuses it efficiently. Specifically, the block takes an input feature tensor $F\in {\mathbb{R}}^{L\times C}$. Here, $L=H\times W$ denotes the flattened spatial dimension, and the symbol C denotes the number of channels. We first adopt Layer Normalization [36] to stabilize feature statistics. Then, we adopt a pointwise layer to mix channel information, which is implemented using a linear projection and a GELU activation [37]:

$$F_{\mathrm{pw}}=\mathrm{GELU}(W_1\widehat{F}+b_1),W_1\in {\mathbb{R}}^{C\times C},$$

(7)

where $W_1$ and $b_1$ are learnable parameters.

After that, we reshape the feature to $F_{\mathrm{sp}}\in {\mathbb{R}}^{H\times W\times C}$ for spatial processing. Three depthwise convolutions are used in parallel to extract multi-scale spatial context. These convolutions use kernel sizes $3\times 3$, $5\times 5$, and $7\times 7$:

$$\begin{array}{cc}\hfill F_3& ={\mathrm{DWConv}}_{3\times 3}\left(F_{\mathrm{sp}}\right),\hfill \end{array}$$

(8)

$$\begin{array}{cc}\hfill F_5& ={\mathrm{DWConv}}_{5\times 5}\left(F_{\mathrm{sp}}\right),\hfill \end{array}$$

(9)

$$\begin{array}{cc}\hfill F_7& ={\mathrm{DWConv}}_{7\times 7}\left(F_{\mathrm{sp}}\right),\hfill \end{array}$$

(10)

where ${\mathrm{DWConv}}_{k\times k}(·)$ denotes depthwise convolution [38].

Then, the multi-scale features are concatenated with the channel dimension. We also fuse $F_{\mathrm{pw}}$ in the concatenation to preserve low-level information:

$$F_{\mathrm{cat}}=\mathrm{Concat}(F_3,F_5,F_7,F_{\mathrm{pw}})\in {\mathbb{R}}^{H\times W\times 4C}.$$

(11)

Finally, another pointwise projection is used to compress the channel dimension:

$$F_{\mathrm{out}}=W_2{F}_{\mathrm{cat}}+b_2,W_2\in {\mathbb{R}}^{4C\times C},$$

(12)

where $W_2$ and $b_2$ are learnable parameters. A standard convolution would require $\mathcal{O}\left(k^2{C}^2\right)$ parameters per scale, while the depthwise branch uses $\mathcal{O}\left(k^{2}C\right)$ parameters per scale, keeping the computation lightweight.

The proposed MSF block improves local feature extraction and preserves efficiency. This block also strengthens multi-scale context aggregation, which benefits translation quality. In Figure 1, MSF blocks fuse the transformed features $I_w^1$ and the generated features $I_g^0$. A convolution layer further transforms the content feature $I_c^2$ before concatenation. Two additional MSF blocks then process the fused representation, and output the final generated feature $I_g^1$.

A stack of OTFormer blocks produces the final output image $I_g$ by a convolution layer, which maps the final features to the RGB space. The model also predicts RGB outputs at multiple resolutions following the multi-scale synthesis from StyleGAN2 [39]. The model aggregates these outputs upsampled to the output resolutions through pixel-wise summation to obtain the final image $I_g$.

Discussion: For attention-based correspondence, the computational cost is $O\left(n^2\right)$ for computing similarity and weighting. In our optimal transport (OT) based approach, each Sinkhorn iteration also costs $O\left(n^2\right)$; with a fixed number of iterations L (e.g., 20), the total cost becomes $O\left(L{n}^2\right)$. However, due to our multi-scale design, the OT problems are solved at low resolutions—for instance, $n=256$ for a $16\times 16$ feature map. This makes $O\left(L{n}^2\right)$ comparable to or even smaller than the cost of attention at high resolutions (e.g., $n=4096$ for a $64\times 64$ map). Furthermore, the lightweight MSF block further reduces overall FLOPs. These factors explain why our method achieves faster inference despite using iterative optimization.

3.2. Loss Function

To train the OTFormer end to end, we adopt a combination of loss functions to encourage photorealistic and faithful synthesis.

Correspondence Constraint Loss: To enforce the structural alignment between the warped exemplar image and the content image, we minimize the ${\ell }_1$ distance between them. Given the warped image

$$I_w^{\prime }=T\times I_{e\downarrow },$$

(13)

where ↓ denotes downsampling (via bilinear interpolation) to match the spatial resolution of the transport map T. The correspondence loss is

$${\mathcal{L}}_w={\parallel I_w^{\prime }-I_{c\downarrow }\parallel }_1.$$

(14)

This loss is applied to all OTFormer blocks.

Perceptual Loss: We measure content consistency between generated and content images using a perceptual loss based on pre-trained VGG [40] features:

$${\mathcal{L}}_p={\parallel {\varphi }_l\left(I_g\right)-{\varphi }_l\left(I_c\right)\parallel }_2,$$

(15)

where ${\varphi }_l$ denotes activations from the l-th layer.

Style loss. To encourage stylistic similarity, we minimize the difference between the Gram matrices of generated and exemplar features:

$${\mathcal{L}}_s={\parallel \mathrm{Gram}\left({\varphi }_l\left(I_g\right)\right)-\mathrm{Gram}\left({\varphi }_l\left(I_e\right)\right)\parallel }_2,$$

(16)

where $\mathrm{Gram}$ computes the Gram matrix [41].

Contextual Loss: We further enhance style fidelity using the contextual loss [42]:

$${\mathcal{L}}_{con}=-log\left(\sum _l{W}_l\mathrm{CX}\left({\varphi }_l\left(I_e\right),{\varphi }_l\left({\widehat{I}}_t\right)\right)\right),$$

(17)

where $\mathrm{CX}$ measures contextual similarity and $W_l$ represents weighting coefficients for VGG-19 layers [40].

Adversarial Loss: A generative adversarial network (GAN) framework encourages photorealism by pitting the generator G, i.e., OTFormer, against a discriminator D:

$$\begin{array}{c}{\mathcal{L}}_{\mathrm{adv}}^D=-\mathbb{E}\left[h\left(D\left(I_t\right)\right)\right]-\mathbb{E}\left[h(-D\left(I_g\right))\right],\end{array}$$

(18)

$$\begin{array}{c}{\mathcal{L}}_{\mathrm{adv}}^G=-\mathbb{E}\left[D\left(I_g\right)\right],\end{array}$$

(19)

where $h\left(t\right)=min(0,-1+t)$ is the hinge loss.

3.3. Implementation Details

We implemented OTFormer using PyTorch 1.8.0 and trained on a NVIDIA GeForce RTX 4090 workstation. We use learning rates of $2\times {10}^{-4}$ and $4\times {10}^{-4}$ for the OTFormer and discriminator, respectively. During training, we set the batch size to 7. The batch size of 7 was chosen as the largest value that fits into the memory of a single NVIDIA GeForce RTX 4090 GPU (24 GB) given the model size and input resolution. This setting allows for stable training while maximizing GPU utilization. Following [9], on the CelebA-HQ dataset, we train the whole model for 60 epochs, and for the DeepFashion dataset, we train for 100 epochs.

4. Experimental Results

In this section, we empirically evaluate OTFormer on two public datasets. We first describe the datasets and evaluation metrics, then compare our method with state-of-the-art approaches both quantitatively and qualitatively. Finally, we present ablation studies to analyze the contribution of each component.

4.1. Datasets and Metrics

We conduct comparison experiments on two public datasets, i.e., CelebA-HQ [43] and DeepFashion [44]. CelebA-HQ dataset has 30,000 face images, while DeepFashion dataset has 52,712 person images. Following [6,9], we perform the edge-to-image task for CelebA-HQ dataset, where the edge image is obtained by connecting the facial landmarks for facial regions and detecting the Canny edges for the background regions. The DeepFashion dataset paired images for the same identity with different poses; therefore, we perform the pose-to-image task on DeepFashion.

Following previous works [6,7,9], we adopt several metrics to evaluate the quality of generated results. We evaluate the distribution-level quality of the generated results using Fréchet inception score (FID) and sliced Wasserstein distance (SWD). With reference to the CelebA-HQ dataset, we evaluate the style relevance and the semantic consistency following [6,9]. The style relevance for color and texture is computed based on the features of relu$1\text{\_}2$ and relu$2\text{\_}2$, which came from the pre-trained VGG-19 network. The semantic consistency is evaluated by making use of the average cosine similarity among high-level features from the VGG-19 network. The data pairs in DeepFashion dataset have ground-truth images; therefore, we adopt structural similarity (SSIM), peak signal to noise ratio (PSNR) to measure the quality of the generated results at the pixel level. To better align human perception, we also adopt learned perceptual image patch similarity (LPIPS) to evaluate the perceptual similarity. In addition, to assess the efficiency of our method, we also report the number of parameters (Params), floating-point operations (FLOPs) and the inference time (time) of different methods. The inference time is computed as the average runtime over 1000 images on the same machine.

4.2. Comparison Results

We compare OTFormer with several state-of-the-art methods, including CoCosNet [6], CoCosNet-v2 [7], UNITE [15], DynaST [8], MIDMs [10], MAT [9], and EBDM [11]. For fair comparison, we report numbers from the original papers or re-evaluated using their official code when available. The codes of MIDMs and EBDM are not publicly available, so we cite their reported numbers.

Quantitative results. The quantitative results on CelebA-HQ and DeepFashion highlight the clear advantages of our method. Our method improves generation quality across multiple metrics, and maintains strong efficiency in parameters and computation.

The quantitative results on DeepFashion dataset are shown in

Table 1. Quantitative comparison on DeepFashion [44] datasets.

Model	FID ↓	SWD ↓	SSIM ↑	PSNR ↑	LPIPS ↓	Params ↓	FLOPs ↓	Time ↓
CoCosNet [6]	14.4	17.2	0.501	15.0	0.286	146.3	396.5	0.088
CoCosNet-v2 [7]	13.0	16.7	0.628	17.3	0.195	45.6	394.4	0.176
UNITE [15]	13.1	16.7	-	-	-	186.7	474.3	0.105
DynaST [8]	8.4	11.8	0.703	18.5	0.160	91.1	340.7	0.061
MIDMs [10]	10.9	10.1	-	-	-	-	-	-
MAT [9]	8.2	11.0	0.661	17.6	0.181	103.6	128.0	0.043
EBDM [11]	10.6	12.4	-	-	-	765.0	-	-
Ours	6.9	9.2	0.704	18.6	0.157	17.4	50.0	0.037

. Our method outperforms all other methods across all reported metrics. For distribution-based metrics, our OTFormer achieves the best FID and SWD scores, which indicates superior ability to generate realistic images. For pixel-level metrics, our method achieves the highest SSIM and PSNR, while also obtaining the lowest LPIPS score, which demonstrates better structural similarity and perceptual quality compared to other methods. Furthermore, our method is more efficient than all other methods in terms of parameters, FLOPs and inference time. Specifically, our method achieves 17.4 million parameters, reducing by 62% compared to the previous method with the fewest parameters (CoCosNet-v2, 45.6 million), which suggests that our method is more practical for real-world applications.

Table 2. Quantitative comparison on CelebA-HQ [43] datasets.

Model	FID ↓	SWD ↓	Texture ↑	Color ↑	Semantic ↑
CoCosNet [6]	14.3	15.2	0.958	0.977	0.949
CoCosNet-v2 [7]	13.2	14.0	0.954	0.975	0.948
UNITE [15]	13.2	14.9	0.952	0.966	0.950
DynaST [8]	12.0	12.4	0.959	0.978	0.952
MIDMs [10]	15.7	12.3	0.962	0.982	0.915
MAT [9]	11.5	13.2	0.965	0.986	0.949
EBDM [11]	11.8	12.1	0.968	0.984	0.920
Ours	11.4	13.1	0.970	0.988	0.948

presents the quantitative results on the CelebA-HQ dataset. Our method achieves competitive performance on FID, texture and color metrics, while the semantic consistency is comparable to baselines.

Figure 4 visualizes the FID and texture scores of different methods on the CelebA-HQ dataset. Our method achieves the best FID and texture scores, which indicates superior generation quality and style consistency. At the same time, our method has significantly fewer parameters than all other methods, which highlights the efficiency of our method.

Qualitative results. Figure 5 shows qualitative comparisons of CelebA-HQ. OTFormer produces more photorealistic results while better preserving exemplar style cues, such as hair color and eye details. Competing methods often introduce artifacts or lose fine-grained textures.

On DeepFashion (as shown in Figure 6), OTFormer not only preserves pattern continuity even under occlusion but also infers reasonable textures for invisible parts. For instance, it generates plausible sleeve patterns in occluded areas and maintains consistent shirt patterns near the bottom. More qualitative results can be found in the supplementary document.

4.3. Ablation Study

To verify the effectiveness of our contributions, we carry out ablation studies on the architecture design and loss functions.

4.3.1. Architecture Design

We studied several variants of the model architecture. All models were trained under the same loss functions and training strategy for fair comparison.

The Model with Attention Mechanism (w Att): We adopt a traditional cross-attention mechanism to replace the optimal transport solver to evaluate the effectiveness of the optimal transport formulation in modeling correspondences.

The Model without Multi-Scale Fusion Block (w/o MSF): In this variant, the multi-scale fusion block is replaced by a standard single-scale convolutional layer, which lacks the ability to explicitly capture multi-scale spatial context. This tests the necessity of multi-scale fusion in effectively integrating features.

The Model without Multi-Scale OTFormer Block (w/o MSOTF): This model omits the multi-scale design in the OTFormer blocks, i.e., it only models correspondences at a single low-resolution scale, removing the coarse-to-fine refinement process.

The Model without Multi-Scale Depth Convolution (w/o MSD): Within the multi-scale fusion block, parallel depthwise convolution layers with varying kernel sizes are replaced by a single depthwise convolution layer. This variant assesses the impact of capturing multi-scale receptive fields in feature fusion.

Table 3. Ablation study on architecture design on CelebA-HQ.

Model	FID ↓	SWD ↓	Texture ↑	Color ↑	Semantic ↑
w Att	12.4	14.3	0.965	0.984	0.944
w/o MSF	11.8	13.4	0.962	0.982	0.948
w/o MSOTF	12.1	12.9	0.968	0.987	0.946
w/o MSD	11.6	13.7	0.987	0.968	0.947
Ours	11.4	13.1	0.970	0.988	0.948

and Figure 7 show the quantitative and qualitative results on the CelebA-HQ dataset. We can see that our full model consistently outperforms all ablated variants across most metrics. Notably, replacing the optimal transport solver with a standard attention mechanism (w Att) leads to decreased performance with worse FID and semantic consistency, which suggests the effectiveness of the optimal transport formulation for exemplar-based image translation. This indicates that the optimal transport solver can better model the complex correspondences between the content and exemplar features, leading to improved generation quality. Furthermore, the model without multi-scale OTFormer block (w/o MSOTF) also performs worse than the full model, which indicates that the multi-scale design is important for modeling correspondences at different scales and improving generation quality. Moreover, by comparing the model without multi-scale fusion block (w/o MSF) and the model without multi-scale depth convolution (w/o MSD) with the full model, we can see the effectiveness of the proposed blocks and designs. From Figure 7, we can obtain similar conclusions, which further validate the effectiveness of the proposed architecture design.

4.3.2. Loss Functions

To validate the effectiveness of the loss design, we conduct experiments on the CelebA-HQ dataset with different loss combinations.

Table 4. Ablation study on loss functions on CelebA-HQ.

Model	FID ↓	SWD ↓	Texture ↑	Color ↑	Semantic ↑
w/o Cor	11.9	12.1	0.987	0.969	0.947
w/o CX	11.5	12.6	0.983	0.960	0.950
w/o Adv	16.1	16.9	0.988	0.967	0.946
w/o Per	12.6	155.7	0.987	0.974	0.915
w/o Style	11.8	12.2	0.979	0.960	0.950
Ours	11.4	13.1	0.970	0.988	0.948

and Figure 8 show the quantitative and qualitative results. From Table 4, we can see that each loss function is important for improving the generation quality. The model without correspondence constraint loss (w/o Cor) achieves the best SWD, but the other metrics all degrade, which indicates that it is useful to improve the spatial alignment between the final output and the content image. Compared with the full model, the model without contextual loss (w/o CX) fails to contain accurate style in the exemplar, leading to a significant decrease in color and texture metrics. The model without perceptual loss (w/o Per) obtains the worst SWD score, which indicates that the perceptual loss is important for preserving the semantic consistency between the generated image and the content image. For the style loss, it can be seen that it is important for style consistency. Overall, all the loss terms are effective for improving the final results. Figure 8 shows qualitative results, which suggests similar conclusions. Without the perceptual loss, the model fails to contain the structures in the content image. Without the style loss, it is difficult to obtain the accurate style in the exemplar. In contrast, the full model can preserve the structure in the content image and obtain a more faithful style regarding the exemplar.

4.4. Limitations and Future Work

Despite its strong performance, OTFormer has limitations. As shown in Figure 9, it can struggle under extreme occlusion or when the exemplar and content have large pose differences, leading to structural distortions. Ambiguous style cues, such as textureless regions, may result in blur. Future work will explore incorporating semantic priors and more robust cost functions to handle such cases. Additionally, extending OTFormer to video translation is a promising direction.

5. Conclusions

In this paper, we propose OTFormer, a new framework for exemplar-based image translation. OTFormer integrates optimal transport into a Transformer architecture by introducing a lightweight multi-scale fusion block, which reduces correspondence misalignment between content and exemplar features. Extensive experiments on two public benchmark datasets validate the effectiveness of the proposed method. OTFormer can not only preserve content structure more consistently, but can also transfer exemplar style attributes more faithfully. Furthermore, our model uses 62% fewer parameters than existing methods and runs faster at inference time, which improves its practical applicability.