UDDSFormer: An Efficient Hardware-Oriented Transformer Framework for Unified Demosaicing, Denoising, and Super-Resolution

Abstract

Reconstructing high-quality, high-resolution RGB images from raw sensor data is a critical task in the image signal processing (ISP) pipeline. However, cascading these processing stages often leads to error accumulation and imposes excessive computational and memory overheads, thus limiting efficient deployment on resource-constrained embedded edge devices. To address these challenges, we propose UDDSFormer—a hardware-oriented Transformer framework designed specifically for the unified RAW-to-RGB reconstruction task. First, the framework introduces an adaptive deformable sampling mechanism that aggregates features aligned with image structures by predicting offsets relative to a regular grid. Subsequently, a geometry decomposition module models horizontal and vertical dependencies through two directional branches and sums their separately normalized axial value responses for feature aggregation. Furthermore, on the Kodak24 test set, UDDSFormer achieves an 8.63× reduction in parameter count while improving PSNR by about 4.22 dB compared to the reference lightweight model, benefiting from unified modeling with deformable sampling. Finally, quantization-aware training experiments show that configurations with 8-bit or 6-bit weights and 8-bit activations achieve validation PSNR values approximately 0.2 dB below the FP32 baseline, whereas reducing activation precision to 4 bits leads to a substantial performance decrease. Overall, these results demonstrate that the unified framework achieves a favorable balance among reconstruction quality, computational efficiency, and hardware deployment potential.

1. Introduction

In computational photography, reconstructing high-quality and high-resolution red–green–blue (RGB) images from degraded RAW sensor measurements is a fundamental problem in the image signal processing (ISP) pipeline [1,2,3]. Unlike processed RGB images, RAW data retain minimally processed sensor measurements, providing greater flexibility for reconstruction. However, Bayer-pattern sensors record only one color component per pixel, requiring demosaicing to estimate the missing components and reconstruct a full-color image [4,5,6]. In practical imaging systems, RAW measurements are also affected by photon shot noise, readout noise, optical degradation, and limited sensor resolution [7,8,9,10,11]. Therefore, denoising, demosaicing, and super-resolution are closely related operations for reconstructing high-quality RGB images from noisy, low-resolution RAW images [12,13]. In traditional ISP systems, denoising, demosaicing, and super-resolution are typically implemented as independent stages executed sequentially. While this cascaded design is simple to implement and facilitates the individual optimization of each component, it fails to account for the strong dependencies among reconstruction tasks. For example, applying denoising directly to mosaiced RAW data may remove weak high-frequency signals together with sensor noise, thereby reducing the color and structural information available to the subsequent demosaicing stage. Applying demosaicing before denoising changes the original noise distribution and may introduce spatially correlated color artifacts [10,11,14,15,16,17]. When super-resolution is subsequently performed, residual noise, zipper artifacts, false colors, and interpolation errors can be further amplified. It can be concluded that reconstruction errors introduced at an early stage can propagate through the entire processing pipeline, becoming increasingly difficult to correct. Besides error accumulation, cascaded ISP pipelines may introduce redundant computation across independently designed modules. These modules may repeatedly extract similar low-level features and convert data between RAW and RGB domains, increasing computation, memory usage, and data transfer overhead. Moreover, optimizing each task independently may not achieve the best overall RAW-to-RGB reconstruction. A unified end-to-end framework can jointly optimize denoising, color reconstruction, and detail recovery while reducing unnecessary intermediate processing.

Convolutional neural networks (CNNs) have achieved substantial progress in low-level vision and have been widely applied to denoising, demosaicing, and super-resolution. Compared with conventional handcrafted ISP algorithms, learning-based methods can directly estimate complex nonlinear mappings from degraded measurements to high-quality images. Existing CNN-based studies have investigated individual or joint RAW restoration tasks, including joint demosaicing and denoising [18,19,20,21,22,23], as well as unified demosaicing, denoising, and super-resolution [12,13,24,25]. These methods demonstrate that jointly learned feature representations can reduce the incompatibility between independently optimized ISP components.

These advances also motivate compact architectures that balance spatial feature modeling with the resource constraints of embedded ISP systems. Whereas the compared CNN-based pipelines predominantly rely on convolutional backbones, UDDSFormer adopts a compact Transformer architecture with content-dependent attention. It decomposes dense window interactions into parallel horizontal and vertical branches and combines their value responses, reducing the attention-product and probability-tensor requirements at the same window size and feature width. In this paper, we compare representative recent methods in terms of backbone, restoration task, parameter count, and reconstruction quality. While traditional CNNs are effective, they primarily rely on spatially invariant convolutional kernels to extract features. Their receptive field is determined by the kernel size, network depth, and sampling strategy. Although stacking more convolutional layers can expand the effective receptive field, doing so also increases network depth and computational cost. More importantly, conventional convolutions sample features from fixed spatial locations; consequently, they may struggle to model irregular edges, spatially varying textures, and content-dependent degradations. These limitations are particularly critical in the unified RAW reconstruction task, which requires the network to simultaneously perform noise suppression, infer missing color components, and recover high-frequency spatial details. Furthermore, artificial intelligence has been widely applied across various fields [26,27,28,29,30,31], while the need for lightweight deployment has driven network designs aimed at balancing feature representation with computational efficiency. Recent restoration approaches also explore attention, modular coordination and generative priors. NOP-GAN combines noise optimization with pyramid coordinate attention for image denoising [32]. MoA-VR investigates an all-in-one video restoration system based on a mixture of agents [33]. MCLL-Diff uses multiple conditions in diffusion-based low-light enhancement [34].

Transformer architectures capture dependencies between spatially separated features through self-attention [26,35,36,37,38,39]. By integrating contextual information based on feature similarity, they have shown promise in low-level vision tasks [40,41]. Compared with local convolution, self-attention directly models long-range interactions, supporting the reconstruction of repetitive patterns, directional textures, and continuous edges. However, global self-attention incurs computational and memory costs that grow quadratically with the number of spatial tokens, making it expensive for high-resolution image reconstruction. Window-based self-attention reduces these costs by restricting feature interactions to non-overlapping local windows [42,43]. With a fixed window size, its complexity grows linearly with the number of spatial tokens. This efficiency, however, comes at the expense of interactions across window boundaries. Shifted windows enable information exchange between neighboring regions, but their fixed sampling grids offer limited adaptability to irregular image structures. Enlarging the windows captures broader context but increases computational and memory costs. These limitations motivate an attention mechanism that combines content-adaptive sampling with efficient long-range dependency modeling. Practical ISP models must also meet the resource constraints of embedded edge devices. Attention operations and large intermediate feature maps can impose substantial computational and memory costs [44]. Low-precision inference can reduce model storage and support efficient integer computation, but post-training quantization may degrade reconstruction quality through clipping and rounding errors. In addition, quantization-aware training (QAT) simulates these errors during optimization to improve robustness to low-precision inference [45].

Our previous work implemented a hardware-friendly joint denoising and demosaicing system on an FPGA [44]. Building on this experience, the present study investigates a compact Transformer architecture and its low-precision robustness. The Transformer hardware accelerator is under development; “hardware-oriented” describes the design motivation and deployment potential.

To address these challenges, we propose UDDSFormer, an efficient end-to-end Transformer framework designed to unify the tasks of denoising, demosaicing, and super-resolution. The network reconstructs high-resolution RGB images directly from noisy, low-resolution Bayer RAW images. Specifically, a RAW preprocessing module first packs the input into four channels and extracts shallow features; a backbone network based on deformable attention samples Keys and Values at content-adaptive locations while maintaining Queries on a regular grid, thereby achieving feature aggregation aligned with image structures; a geometric decomposition module further organizes spatial interactions into horizontal and vertical branches and sums their separately normalized value responses. Two local attention upsampling stages are positioned between the backbone groups to recover spatial details through content-adaptive feature reorganization; subsequently, an RGB reconstruction head generates the high-resolution output. Furthermore, we introduce quantization-aware training to evaluate reconstruction quality under reduced numerical precision, assessing the framework’s potential for deployment on resource-constrained devices.

The main contributions of this work are summarized as follows:

• We propose UDDSFormer, a lightweight end-to-end RAW-to-RGB reconstruction framework with a compact Swin-T backbone. The framework jointly optimizes denoising, demosaicing, and super-resolution within a single network, with the full model containing only 0.80 M parameters.
• We propose DDNA–GDM, which captures richer structural information through deformable key–value sampling while retaining regular-grid queries for spatial correspondence. GDM models horizontal and vertical dependencies through two parallel directional branches. Each branch aggregates the corresponding sampled values, and their responses are summed.
• The two-stage LAU strategy progressively enhances the feature resolution during upsampling. Unlike fixed feature rearrangement, LAU predicts content-dependent local weights at each position and applies them to combine neighboring low-resolution features, recovering high-resolution details.
• We systematically evaluate the reconstruction performance, computational complexity, component effectiveness, and low-precision robustness of UDDSFormer. Additionally, we conduct quantization-aware training to assess its potential for deployment on resource-constrained embedded imaging platforms.

2. The Proposed Method

In this section, we introduce UDDSFormer for unified denoising, demosaicing, and super-resolution of degraded low-resolution Bayer RAW images. The proposed method is described in detail in the following subsections.

2.1. Overall Structure

As illustrated in Figure 1, UDDSFormer reconstructs a high-resolution (HR) RGB image from a noisy low-resolution (LR) Bayer RAW input through joint denoising, demosaicing, and super-resolution. The network consists of a RAW preprocessing module (RPM), a DDNA-GDM-based feature extraction backbone, local attentional upsampling (LAU) modules, and an RGB reconstruction head, organized into four stages.

Figure 1. Overall architecture of UDDSFormer.

In Stage 1, RPM packs the Bayer input into a four-channel representation corresponding to the R, $G_1$, $G_2$, and B samples. The packed input is concatenated with a noise map and projected into a shallow feature space, providing both image content and noise-level information for subsequent processing. Stages 2 and 3 form the feature extraction backbone. Each stage contains six attention blocks, alternating between deformable window blocks (D-WB) and deformable shifted-window blocks (D-SWB). D-WB supports content-adaptive feature aggregation within local windows, while D-SWB enables information exchange across window boundaries. The backbone contains $N=12$ blocks in total, with $L=4$ attention heads and a feature dimension of $C=64$. The two feature extraction stages are followed by LAU1 and LAU2, respectively. These modules progressively increase spatial resolution through content-adaptive feature reassembly. This arrangement allows feature extraction and resolution recovery to be jointly optimized. In Stage 4, the RGB reconstruction head maps the refined features to the HR RGB output ${\widehat{I}}^{HR}$. The RPM, DDNA-GDM, LAU, and RGB reconstruction head are described in detail below.

The backbone is a compact restoration network that adopts Swin-style window partitioning, cyclic shifts, relative-position bias, pre-normalization, residual connections and feed-forward modules. It uses two groups of six blocks, a fixed width of $C=64$, four heads, $8\times 8$ windows, alternating shifts of zero and four positions, and an MLP expansion ratio of four. It does not use the canonical Swin-T classification hierarchy, patch merging or an ImageNet classification head. The approximately 0.80 M parameter count refers to this restoration configuration; its exact value depends on the task and noise-conditioning channels.

• RPM. For a noisy LR Bayer image $M_n^{LR}\in {\mathbb{R}}^{H\times W\times 1}$, RPM first rearranges the Bayer samples into a compact four-channel representation using pixel unshuffle, as follows:

  $$M_p^{LR}={\mathcal{P}}_{\downarrow 2}\left(M_n^{LR}\right)\in {\mathbb{R}}^{{\displaystyle \frac{H}{2}}\times {\displaystyle \frac{W}{2}}\times 4},$$

  (1)

  where ${\mathcal{P}}_{\downarrow 2}$ denotes pixel unshuffle with a spatial reduction factor of two. The four channels correspond to the R, $G_1$, $G_2$, and B samples within each Bayer pattern.

The packed representation is concatenated with the spatially aligned noise map $\mathit{N}$ and mapped to C-channel features by a $3\times 3$ convolution.

$${\mathit{F}}_0={\mathcal{C}}_{3\times 3}\left(Concat(M_p^{LR},\mathit{N})\right),{\mathit{F}}_0\in {\mathbb{R}}^{{\displaystyle \frac{H}{2}}\times {\displaystyle \frac{W}{2}}\times C}.$$

(2)

where Concat denotes channel-wise concatenation, and ${\mathcal{C}}_{3\times 3}$ denotes the convolutional projection. The resulting shallow features ${\mathit{F}}_0$ are fed into the feature extraction backbone network.

• DDNA-GDM. As depicted in Figure 2, DDNA-GDM integrates dual-scale offset prediction, deformable key-value sampling, a directional attention mechanism, and residual feature fusion. Sampling offsets are predicted based on the input features $\mathbf{X}$. Queries are generated directly from the original features while retaining their regular grid positions, whereas keys and values are derived from features sampled at the learned offset locations. Subsequently, the GDM organizes attention interactions along the horizontal and vertical axes. Thus, deformable sampling determines the locations from which key and value information is gathered, while directional decomposition dictates how queries associate with these keys and aggregate the corresponding values.
  

  Figure 2. Architecture of the DDNA-GDM module used in D-WB and D-SWB. Offsets predicted from input features guide deformable key-value sampling, while queries retain their regular grid positions. The Geometric Decomposition Mechanism (GDM) organizes interactions between queries and keys along horizontal and vertical axes. The $A_{\mathrm{geo}}$ conceptually represents the combined aggregation; the two directional branches separately aggregate sampled values and sum their responses, as detailed in Figure 3.

For the input feature $\mathbf{X}$, two offset prediction branches with $3\times 3$ and $7\times 7$ depthwise convolutions capture spatial context at different scales. Their outputs are fused to predict bounded sampling offsets:

$$\Delta \mathbf{P}=\gamma \tanh \left(f_{3\times 3}^{\mathrm{DW}}\left(\mathbf{X}\right)+f_{7\times 7}^{\mathrm{DW}}\left(\mathbf{X}\right)\right),$$

(3)

where $f_{3\times 3}^{\mathrm{DW}}(·)$ and $f_{7\times 7}^{\mathrm{DW}}(·)$ denote the two offset predictors, and $\gamma$ controls the sampling range.

Queries are projected from the original features on the regular reference grid $\mathbf{P}$, while keys and values are projected from deformably sampled features as follows:

$$\left\{\begin{array}{cc}\hfill \mathbf{Q}& ={Proj}_Q\left(\mathbf{X}\right),\hfill \\ \hfill {\mathbf{K}}_d& ={Proj}_K\left(\mathcal{B}(\mathbf{X},\mathbf{P}+\Delta \mathbf{P})\right),\hfill \\ \hfill {\mathbf{V}}_d& ={Proj}_V\left(\mathcal{B}(\mathbf{X},\mathbf{P}+\Delta \mathbf{P})\right).\hfill \end{array}\right.$$

(4)

where $\mathcal{B}(·)$ denotes bilinear sampling, $\Delta \mathbf{P}$ represents the predicted sampling offsets, and ${Proj}_Q$, ${Proj}_K$, and ${Proj}_V$ are learnable feature projections. The subscript d indicates features derived from deformable sampling. This formulation allows keys and values to capture information from content-adaptive locations while maintaining regular grid query positions.

Figure 3 illustrates the directional decomposition in GDM. Inspired by directional organization [46], the implementation computes horizontal and vertical attention in parallel. For each window and head, let $(r,c)$ denote a regular-grid query position, m the window side length, and $d=C/h$ the head dimension. Let ${\mathbf{q}}_{r,c}$ be a query and ${\mathbf{k}}_{r,c}^d$ and ${\mathbf{v}}_{r,c}^d$ the sampled key and value features in window coordinates. The directional logits are

$$\begin{array}{cc}\hfill s_{r,c,j}^x& ={\displaystyle \frac{{\mathbf{q}}_{r,c}^{\mathsf{T}}{\mathbf{k}}_{r,j}^d}{\sqrt{d}}}+b_{c-j}^x+{\mu }_{r,c,j}^x,\hfill \\ \hfill s_{c,r,i}^y& ={\displaystyle \frac{{\mathbf{q}}_{r,c}^{\mathsf{T}}{\mathbf{k}}_{i,c}^d}{\sqrt{d}}}+b_{r-i}^y+{\mu }_{c,r,i}^y,\hfill \end{array}$$

(5)

where $b^x$ and $b^y$ are the relative-position biases, and ${\mu }^x$ and ${\mu }^y$ are the corresponding shifted-window masks. Softmax is applied separately along each set of directional keys:

$$a_{r,c,j}^x={\displaystyle \frac{\exp \left(s_{r,c,j}^x\right)}{{\sum }_{j^{\prime }=1}^m\exp \left(s_{r,c,j^{\prime }}^x\right)}},a_{c,r,i}^y={\displaystyle \frac{\exp \left(s_{c,r,i}^y\right)}{{\sum }_{i^{\prime }=1}^m\exp \left(s_{c,r,i^{\prime }}^y\right)}}.$$

(6)

Geometric fusion adds the two value responses:

$${\mathbf{z}}_{r,c}={\displaystyle \sum _{j=1}^m}{a}_{r,c,j}^x{\mathbf{v}}_{r,j}^d+{\displaystyle \sum _{i=1}^m}{a}_{c,r,i}^y{\mathbf{v}}_{i,c}^d.$$

(7)

Thus, ${\mathcal{F}}_{\mathrm{geo}}$ denotes response-level addition. No full learned window-attention matrix is constructed in the directional forward path. The row/column intersection contributes through both branches, and their outputs are summed without further normalization. Values do not participate in computing the logits.

Figure 3. Parallel directional attention in GDM. Regular-grid queries and deformably sampled keys produce independent row-wise and column-wise attention weights. Each branch aggregates the corresponding sampled values. Geometric fusion sums the two value responses; it does not multiply the directional attention arrays or materialize a full learned window-attention matrix.

The aggregated features are projected, restored to their spatial positions, and added to the input:

$$\mathbf{Y}=\mathbf{X}+Restore\left({Proj}_O\left(\mathbf{Z}\right)\right).$$

(8)

Here ${Proj}_O$ is a learnable output projection and Restore restores the window-wise tokens to the regular query grid. Residual addition uses matching spatial and channel dimensions. Regular-grid queries preserve pixel-level correspondence, while deformable sampling selects the key–value locations and the two directional branches aggregate their responses.

• LAU. In unified RAW reconstruction tasks, upsampling is crucial for recovering spatial resolution. While traditional interpolation methods rely on predefined sampling rules, PixelShuffle rearranges features into a higher-resolution grid. Although these operations are highly efficient, they do not explicitly predict content-dependent aggregation weights for each output location. Inspired by the local attention mechanism in [47], we introduce Local Attention Upsampling (LAU), designed to enable the feature aggregation process to adapt to the local structure of the image.

LAU utilizes high-resolution guiding features to predict spatially varying weights within local neighborhoods. These weights are applied to adjacent low-resolution value features to reconstruct high-resolution features. Subsequently, the reconstructed features are combined with the upsampled residual branch and undergo local refinement. Furthermore, the LAU1 and LAU2 follow the two backbone groups, respectively, enabling the joint optimization of spatial resolution recovery and feature extraction.

As illustrated in Figure 4, the k-th LAU module takes the backbone feature ${\mathbf{F}}_{B,k}\in {\mathbb{R}}^{B\times C\times H_k\times W_k}$ as input. It first constructs an upsampled feature, an HR guide feature, and an LR value feature as

$$\left\{\begin{array}{c}{\mathbf{F}}_k^{\mathrm{up}}={\mathcal{B}}_{s_k}\left({\mathbf{F}}_{B,k}\right),\hfill \\ {\mathbf{F}}_k^{\mathrm{g}}={\mathcal{E}}_k^{\mathrm{g}}\left({\mathbf{F}}_k^{\mathrm{up}}\right),\hfill \\ {\mathbf{F}}_k^{\mathrm{v}}={\mathcal{C}}_{3\times 3}^{\mathrm{v},k}\left({\mathbf{F}}_{B,k}\right),\hfill \end{array}\right.$$

(9)

where ${\mathcal{B}}_{s_k}$ denotes bilinear upsampling with scale factor $s_k$, ${\mathcal{E}}_k^{\mathrm{g}}$ contains two $3\times 3$ convolution–GELU layers, and ${\mathcal{C}}_{3\times 3}^{\mathrm{v},k}$ denotes the value projection.

Figure 4. Architecture of the LAU module. (For $\times 2$ super-resolution, the upsampling factor is set to 2 for both LAU modules. For $\times 4$ super-resolution, the two upsampling factors must satisfy $s_1{s}_2=8$ to account for Bayer packing. Both modules use $D=9$ sampling positions in a $3\times 3$ neighborhood. The guide branch contains two $3\times 3$ convolution–GELU layers followed by a $1\times 1$ convolution to predict attention logits. The value branch contains a $3\times 3$ convolution, while the skip branch is an identity mapping or a $1\times 1$ projection. The residual scale ${\alpha }_k$ is initialized to $0.1$. The refinement block consists of a depthwise $3\times 3$ convolution, a pointwise $1\times 1$ convolution, and ReLU).

The HR guide feature predicts spatially varying attention weights over the local neighborhood:

$${\mathbf{A}}_k={Softmax}_D\left({\varphi }_k\left({\mathbf{F}}_k^{\mathrm{g}}\right)\right),$$

(10)

where ${\varphi }_k$ denotes the attention-logit projection and ${Softmax}_D$ normalizes the weights over the D local sampling positions.

The attention weights aggregate neighboring LR value features at each HR position $\mathbf{p}$:

$${\mathbf{F}}_k^{\mathrm{att}}\left(\mathbf{p}\right)={\displaystyle \sum _{d=1}^D}{A}_{k,d}\left(\mathbf{p}\right){\mathbf{F}}_k^{\mathrm{v}}\left({\mathbf{q}}_k\left(\mathbf{p}\right)+{\mathbf{r}}_d\right),$$

(11)

where ${\mathbf{q}}_k\left(\mathbf{p}\right)$ maps the HR position $\mathbf{p}$ to its corresponding LR position, and ${\mathbf{r}}_d$ denotes the d-th fixed sampling offset.

The aggregated feature is then combined with the upsampled skip feature and locally refined:

$$\begin{array}{cc}\hfill {\mathbf{F}}_k^{\mathrm{skip}}& ={\mathcal{P}}_k\left({\mathbf{F}}_k^{\mathrm{up}}\right),\hfill \\ \hfill {\mathbf{F}}_{\mathrm{LAU},k}& ={\mathcal{R}}_k\left({\mathbf{F}}_k^{\mathrm{skip}}+{\alpha }_k{\mathbf{F}}_k^{\mathrm{att}}\right),\hfill \end{array}$$

(12)

where ${\mathcal{P}}_k$ denotes the skip projection, ${\alpha }_k$ is a learnable residual scaling factor, and ${\mathcal{R}}_k$ denotes the refinement block.

Content-dependent local reassembly is also used by CARAFE [48] and Local Attender [47]. Unlike CARAFE, which predicts reassembly kernels from LR features, LAU generates aggregation weights from an HR guide obtained through explicit upsampling. The weights aggregate a separately projected LR value branch. The result is fused with an interpolated skip branch through learnable residual scaling and local refinement. This guided residual design is integrated into two progressive RAW restoration stages.

• RGB reconstruction head. The RGB reconstruction head maps the refined features from LAU2 to three RGB channels to reconstruct the final HR image:

  $${\widehat{I}}^{HR}={\mathcal{H}}_{\mathrm{RGB}}\left({\mathbf{F}}_{\mathrm{LAU},2}\right)\in {\mathbb{R}}^{B\times 3\times sH\times sW},$$

  (13)

  where ${\mathcal{H}}_{\mathrm{RGB}}$ denotes the learnable RGB projection, and $s\in \{2,4\}$ is the super-resolution scale factor relative to the original Bayer input. Since Bayer packing halves each spatial dimension, the two LAU upsampling factors satisfy $s_1{s}_2=2s$.

2.2. Computational Complexity Analysis

$T=HW$ denote the number of padded feature positions at one stage, m the window side length, C the feature channel dimension, and h the number of attention heads. There are $T/m^2$ windows. Dense window attention requires $\mathcal{O}\left(T{m}^{2}C\right)$ operations for query–key products and value aggregation. The directional path separately computes horizontal and vertical interactions of length m, requiring $\mathcal{O}\left(2TmC\right)$ operations. Its two probability tensors contain $2hTm$ elements in total, compared with $hT{m}^2$ for dense attention. For a window containing $M=m^2$ tokens, the corresponding product costs are $\mathcal{O}\left(2M^{3/2}C\right)$ and $\mathcal{O}\left(M^{2}C\right)$, respectively.

The directional path sums the separately normalized axial value responses without constructing a full learned window-attention matrix. The current shifted-window mask builder nevertheless allocates a dense mask containing $T{m}^2$ elements. Projection activations, offset prediction, sampling, normalization and feed-forward layers also contribute to runtime and memory. Therefore, the reduction in stored attention probabilities does not imply the same reduction in total block memory or runtime.

For example, at a feature resolution of $128\times 128$ and a window side of 32, dense and directional FP32 attention tensors occupy 256 and 16 MiB, respectively, while the corresponding first-forward block peaks are 648.65 and 220.35 MiB. These measurements use an NVIDIA RTX 3060, batch size one, 64 channels and four heads. Peak allocation includes input/model buffers and intermediate tensors. At fixed feature resolution, doubling the window side increases dense attention storage fourfold and directional storage twofold. At fixed window size, both scale linearly with the number of feature positions.

Table 1 retains the stage-wise notation: $N_i$ denotes the number of windows, $M_i$ the number of positions per window, and $C_i$ the channel dimension, so that $T_i=N_i{M}_i$ and $m_i=\sqrt{M_i}$. Deformable key–value sampling adds $\mathcal{O}\left(N_i{M}_i{C}_i\right)$ work for a fixed interpolation neighborhood. The analysis excludes feature projections, offset prediction, guide extraction and refinement. For LAU, local weight prediction and aggregation require $\mathcal{O}\left(nD(C_g+C_v)\right)$ operations, where n counts output positions across both modules. With $D=9$ and fixed channel dimensions, this term is $\mathcal{O}\left(n\right)$.

Table 1. Comparison of computational complexity for attention products, deformable sampling and LAU. GDM separately aggregates horizontal and vertical value responses and adds them. Feature projections, offset prediction, guide extraction and refinement are excluded. The dense shift-mask allocation described in the text is additional memory overhead. LAU uses a fixed neighborhood size and channel dimensions.

Model	DDNA	GDM	LAU	Complexity
Baseline	×	×	×	$\mathcal{O}\left(N_i{M}_i^2{C}_i\right)$
Baseline + DDNA	✓	×	×	$\mathcal{O}\left(N_i{M}_i^2{C}_i\right)+\mathcal{O}\left(N_i{M}_i{C}_i\right)$
Baseline + DDNA–GDM	✓	✓	×	$\mathcal{O}\left(2N_i{M}_i^{3/2}{C}_i\right)+\mathcal{O}\left(N_i{M}_i{C}_i\right)$
UDDSFormer	✓	✓	✓	$\mathcal{O}\left(2N_i{M}_i^{3/2}{C}_i\right)+\mathcal{O}\left(N_i{M}_i{C}_i\right)+\mathcal{O}\left(n\right)$

2.3. Three-Stage Reconstruction Loss

We construct a joint loss function for UDDSFormer that combines ${\mathsf{\ell }}_1$ losses tailored to the three specific tasks: denoising (DN), demosaicing (DM), and super-resolution (SR). The loss function simultaneously supervises intermediate predictions and the final reconstruction, enabling end-to-end optimization of these three tasks.

The noisy LR Bayer image $M_n^{LR}$ is fed into the network to obtain a denoised LR RAW prediction ${\widehat{M}}^{LR}$, a demosaiced LR color prediction ${\widehat{I}}^{LR}$, and an HR color reconstruction ${\widehat{I}}^{HR}$. Each prediction is supervised by its corresponding ground-truth target. The task-specific losses are

$$\left\{\begin{array}{cc}\hfill {\mathcal{L}}_{\mathrm{DN}}& ={\mathsf{\ell }}_1\left({\widehat{M}}^{LR},M_{\mathrm{gt}}^{LR}\right),\hfill \\ \hfill {\mathcal{L}}_{\mathrm{DM}}& ={\mathsf{\ell }}_1\left({\widehat{I}}^{LR},I_{\mathrm{gt}}^{LR}\right),\hfill \\ \hfill {\mathcal{L}}_{\mathrm{SR}}& ={\mathsf{\ell }}_1\left({\widehat{I}}^{HR},I_{\mathrm{gt}}^{HR}\right).\hfill \end{array}\right.$$

(14)

where ${\mathsf{\ell }}_1$ denotes the mean absolute error, and the subscript $\mathrm{gt}$ denotes the ground-truth target. Each loss is averaged independently over its tensor elements to account for differences in output size.

The joint training loss function is defined as,

$${\mathcal{L}}_{\mathrm{joint}}={\lambda }_{\mathrm{DN}}{\mathcal{L}}_{\mathrm{DN}}+{\lambda }_{\mathrm{DM}}{\mathcal{L}}_{\mathrm{DM}}+{\lambda }_{\mathrm{SR}}{\mathcal{L}}_{\mathrm{SR}}$$

(15)

where each supervision switch ${\lambda }_{\mathrm{DN}}$, ${\lambda }_{\mathrm{DM}}$, and ${\lambda }_{\mathrm{SR}}$ takes a value from the two-element set $\{0,1\}$: 0 disables the corresponding loss, and 1 enables it. These switches enable the auxiliary losses for the intermediate predictions defined by each pipeline configuration. For example, setting ${\lambda }_{\mathrm{DN}}={\lambda }_{\mathrm{DM}}={\lambda }_{\mathrm{SR}}=1$ jointly activates the DN, DM, and SR losses.

For instance, the denoised RAW prediction is fed into the demosaicing stage, and the resulting LR color prediction serves as input to the super-resolution stage. During training, the intermediate predictions remain connected, allowing the final reconstruction loss to be backpropagated to the preceding stages; these intermediate losses provide direct supervision for their respective outputs. Disabling a specific loss does not result in the removal of the corresponding image restoration operation. For configurations where intermediate predictions are not defined, the associated auxiliary losses are omitted. Furthermore, color supervision adopts RGB targets for Gaussian noise and camera-linear RGB targets from inverse ISP for Gaussian–Poisson noise.

3. Experiments

This section presents the implementation details and a comprehensive evaluation of UDDSFormer, including comparisons with state-of-the-art methods, ablation studies, attention visualization, and hardware-oriented quantization analysis.

3.1. Implementation Detail

All models are implemented in Python 3.7 with PyTorch 1.10.1, and experiments are conducted on an AutoDL cloud server running Ubuntu 20.04 and equipped with a single NVIDIA GeForce RTX 4090 GPU. The random seed is fixed at 0 solely to ensure that the recorded training process is reproducible; it does not imply anything about the degree of experimental fluctuation that might occur with other random seeds. The Gaussian benchmark models are trained on the DIV2K dataset, which contains 800 high-resolution training images and 100 validation images. The training images are divided into overlapping $480\times 480$ sub-images with a stride of 240, resulting in 32,207 training samples. A subset of 208 sub-images from the validation set is selected for model validation. During training, $128\times 128$ patches are randomly cropped, and data augmentation includes horizontal and vertical flips. During validation, center cropping is applied without random geometric augmentation. The UDDSFormer variants are trained from scratch for 500 epochs with the ${\mathsf{\ell }}_1$ loss function and the AdamW optimizer. The hyperparameters are set to ${\beta }_1=0.9$, ${\beta }_2=0.999$, $ϵ={10}^{-8}$, and a weight decay coefficient of $0.05$. The learning rate increases linearly from ${10}^{-5}$ to a peak value of ${10}^{-3}$ during the first 10 epochs. Cosine annealing then reduces it to ${10}^{-6}$ over the remaining 490 epochs. The learning rate scheduler is updated at the end of each epoch. The batch size is set to 64 for UDDSFormer-V1, V2, and V4, and to 32 for UDDSFormer-V3. Four data-loading workers are employed, and gradients are clipped to a maximum ${\mathsf{\ell }}_2$ norm of $1.0$. The random seed is fixed at 0. Validation is performed after each epoch, and the checkpoint with the highest validation PSNR is selected for final evaluation.

• Noise conditioning and task configuration. For GP experiments, we follow the TENet/TENet++ camera-linear protocol [13]. For RGB-source datasets, inverse ISP reverses tone mapping, gamma compression, color correction and white-balance/exposure gains. The same sampled camera parameters are used for HR and LR images. PixelShift200 supplies camera-linear measurements directly; camera metadata are retained for rendering predictions and references into RGB.

The M denote clean packed LR RAW data with intensities normalized to $[0,1]$. Gaussian noise is added as

$$Y=M+{\displaystyle \frac{\sigma }{255}}\epsilon ,\epsilon \sim \mathcal{N}(0,1),N_{\mathrm{G}}={\displaystyle \frac{\sigma }{255}}.$$

(16)

Independent standard-normal samples are generated using torch.randn. A constant noise-level map is concatenated with the four packed RAW channels, yielding a five-channel input. For GP noise, shot/read parameters are sampled as

$$\begin{array}{cc}\hfill \log \alpha & \sim \mathcal{U}(\log {10}^{-4},\log 0.012),\hfill \\ \hfill \log \beta & =2.18\log \alpha +1.20+\eta ,\eta \sim \mathcal{N}(0,0.{26}^2).\hfill \end{array}$$

(17)

The noisy RAW observation and its conditioning maps are

$$Y=M+\sqrt{\alpha M+\beta }\epsilon ,N_{\mathrm{GP}}=\alpha Y+\beta .$$

(18)

Noise generation uses the clean signal M, whereas the four conditioning maps use the noisy observation Y. Concatenating these maps with packed RAW produces an eight-channel input in both training and evaluation. The GP model represents photon shot noise and read noise.

The configuration options in_type, mid_type, out_type, noise_model, scale and model specify the input degradation/domain, supervised intermediate targets, final target, noise model, scale and network implementation. For example, noisy_raw→raw specifies RAW denoising; lr_raw→raw specifies RAW SR; raw→linrgb specifies demosaicing; and noisy_lr_raw→linrgb specifies joint DN–DM–SR. Each configuration is trained as a separate model. For P1–P4, the common input/output pair is noisy_lr_raw→linrgb. P1 uses mid_type=lr_raw,lr_linrgb for DN→DM→SR with intermediate supervision; P2 uses lr_linrgb for joint DN+DM→SR; P3 uses lr_raw for DN→joint DM+SR; and P4 uses no intermediate target for end-to-end joint restoration. These pipeline arrangements are distinct from the task-specific V1–V4 variants.

3.2. Comparison with State-of-the-Art Method

To evaluate UDDSFormer across different RAW restoration tasks, four variants are compared with representative methods: DN (V1), DM–SR (V2), DN–DM (V3), and DN–DM–SR (V4). Table 2 represents a comparison of parameter counts and reconstruction performance on Kodak24 and Urban100. On the Kodak24 and Urban100 datasets, UDDSFormer-V1 achieves PSNR improvements of 1.00 dB and 1.00 dB, respectively, compared to RNANet [23]. Compared to FLUDNet [49], the improvements are 1.92 dB and 2.18 dB, respectively. With only 0.66 M parameters, V1 reduces the parameter count by a factor of 5.86 (an 82.95% reduction) compared to FLUDNet while improving denoising accuracy. In the joint denoising and demosaicing (DN–DM) task, UDDSFormer-V3 outperforms FlexISP [50] by 4.66 dB in PSNR on Urban100. Compared to SGNet [21], V3 achieves a PSNR gain of 0.03 dB and an SSIM increase of 0.0184 on Urban100. Its parameter count is 0.73 M, only 2.82% higher than the 0.71 M parameters of SGNet, achieving better reconstruction quality with a comparable model size. In the joint demosaicing and super-resolution (DM–SR) task, UDDSFormer-V2 reduces the parameter count from 3.39 M for JDSR-DDEB-FS-DFR [24] to 0.80 M, representing a reduction by a factor of 4.24 (76.40%). Its PSNR values on Kodak24 and Urban100 are lower by 3.12 dB and 3.90 dB, respectively, reflecting the trade-off between model size and reconstruction accuracy.

Table 2. Comparison of reconstruction performance and parameter count for different tasks on the Kodak24 and Urban100 datasets.

Model	Publication	Backbone	Task	Params. (M)	Kodak24		Urban100
					PSNR (dB)	SSIM	PSNR (dB)	SSIM
RNANet [23]	ICLR 2019	ResNet	DN	–	33.18	–	32.40	–
FLUDNet [49]	IEEE SPL 2021	U-Net	DN	3.87	32.26	–	31.22	–
FlexISP [50]	TOG 2014	CNN	DN&DM	–	–	–	27.51	0.8196
SGNet [21]	CVPR 2020	CNN	DN&DM	0.71	–	–	32.14	0.9229
JDSR-DDEB-FS-DFR [24]	IEEE TCSVT 2021	DDEB-CNN	DM&SR	3.39	33.98	0.9475	31.55	0.9490
$J{D}_N{D}_{M}SR$Net [12]	CVPR 2021	CNN	DN&DM&SR	78.20	28.80	0.8942	26.89	0.8922
TENet [13]	ICCP 2022	DenseNet	DN&DM&SR	20.22	28.70	0.8963	26.53	0.8872
FastLDN [25]	ICCECE 2023	CNN	DN&DM&SR	3.84	24.12	0.7923	26.62	0.8904
JDDSNet [51]	ICATEEE 2025	DenseNet	DN&DM&SR	6.90	24.15	0.6480	23.21	0.6638
UDDSFormer-V1	Ours	Swin-T	DN	0.66	34.18	0.9311	33.40	0.9665
UDDSFormer-V2	Ours	Swin-T	DM&SR	0.80	30.86	0.8819	27.65	0.8865
UDDSFormer-V3	Ours	Swin-T	DN&DM	0.73	33.81	0.9119	32.17	0.9413
UDDSFormer-V4	Ours	Swin-T	DN&DM&SR	0.80	28.37	0.8046	26.98	0.8935

Note: Bold values indicate the results of UDDSFormer variants. DN: denoising; DM: demosaicing; SR: super-resolution. Swin-T: the compact restoration backbone described in Section 2.1; it is not the canonical Swin-T classifier. The DN–DM–SR comparisons involving $J{D}_N{D}_{M}SR$Net, TENet, FastLDN and JDDSNet use identical degradation, noise, preprocessing and PSNR/SSIM evaluation settings based on the TENet framework.

In the joint DN–DM–SR task, the CNN-based methods JDDSNet [51] and FastLDN [25] employ fewer parameters than TENet [13]. However, their PSNR values on Kodak24 are lower by 4.55 dB and 4.58 dB, respectively, while their SSIM values decrease by 0.2483 and 0.1040, respectively. These comparisons illustrate a dataset-dependent trade-off between model size and reconstruction quality. UDDSFormer-V4 contains only 0.80 M parameters. Compared to $J{D}_N{D}_{M}SR$Net [12], V4 reduces the parameter count by a factor of 97.75 (a 98.98% reduction). Its PSNR is lower by 0.43 dB on Kodak24 but higher by 0.09 dB on Urban100. Compared to TENet [13], V4 reduces the parameter count by a factor of 25.28 (a 96.04% reduction). On Kodak24, its PSNR and SSIM are lower by 0.33 dB and 0.0917, respectively. On Urban100, its PSNR and SSIM are higher by 0.45 dB and 0.0063, respectively. Table 2 illustrates the trade-off between reconstruction quality and model size. At fixed numerical precision, the smaller parameter count directly lowers weight-storage requirements. This is useful for embedded ISP applications with limited storage when the reconstruction meets the application’s fidelity requirements. The results demonstrate deployment potential, rather than universally superior reconstruction quality or measured target-device acceleration and energy savings.

3.3. Ablation Study

We evaluate the contributions of deformable sampling (DS) within DDNA, directional decomposition (DD) implemented by GDM, and local attentional upsampling (LAU). In this section, we conduct comprehensive ablation experiments to evaluate the task-level and architectural design choices of UDDSFormer. First, stage-wise, partially joint, and fully joint restoration strategies are compared under different noise conditions and super-resolution scale factors. Next, the individual and combined contributions of deformable sampling (DS), directional decomposition (DD), and the two LAU stages are investigated. Finally, intermediate attention responses are visualized to analyze the progressive feature refinement across successive backbone blocks.

3.3.1. Joint Restoration Strategy Analysis

Table 3 compares the performance of four restoration pipelines across five datasets, two upsampling factors ($\times 2$ and $\times 4$), and four noise conditions. The fully joint pipeline achieved the highest PSNR and SSIM values in all 40 evaluation settings. Averaged across the five datasets and four noise conditions, the pipeline reached 28.0505 dB/0.8290 at $\times 2$ and 23.9980 dB/0.6637 at $\times 4$; compared to the staged pipeline, its performance improved by 1.6725 dB/0.0361 and 1.6020 dB/0.0452, respectively. Under Gaussian noise with $\sigma =5$, the fully joint pipeline outperforms the best-performing alternative pipeline on each dataset by an average of 0.190 dB in PSNR and 0.0090 in SSIM at $\times 2$. Using the same comparison criterion, the average PSNR advantage increases to 1.54 dB at $\sigma =10$ but decreases to 0.71 dB at $\sigma =20$, indicating that the advantage does not increase monotonically with the intensity of Gaussian noise. For the fully joint pipeline, PSNR and SSIM values across the five datasets showed a continuous downward trend as the Gaussian noise level increased from $\sigma =0$ to $\sigma =20$. At $\times 2$ scaling, the average PSNR dropped from 29.472 dB to 25.884 dB; at $\times 4$ scaling, it fell from 24.774 dB to 22.864 dB. Increasing the upsampling factor from $\times 2$ to $\times 4$ also reduced reconstruction quality across all corresponding datasets and noise conditions. Furthermore, the two partial joint pipelines each offer distinct advantages. At $\times 2$ scaling, the $D_N\to \mathrm{Joint}{D}_{M}SR$ pipeline achieved a slightly higher overall average PSNR, whereas the $\mathrm{Joint}{D}_N{D}_M\to SR$ pipeline yielded a higher average SSIM. These results indicate that the relative performance of the partial joint pipelines depends on the noise model and the upscaling factor.

Table 3. Comparison of different restoration pipelines under Gaussian noise with different super-resolution (SR) upscaling factors ($\times 2$ and $\times 4$).

Solution Pipeline	Noise Model	SR $\times 2$ (PSNR/SSIM)					SR $\times 4$ (PSNR/SSIM)
		McMaster	DIV2K	Kodak24	Waterloo	Urban100	McMaster	DIV2K	Kodak24	Waterloo	Urban100
Stage-wise $D_N\to D_M\to SR$	$\sigma =0$ $\sigma =5$ $\sigma =10$ $\sigma =20$	29.91/0.8627 29.94/0.8511 25.53/0.7483 24.25/0.6915	31.39/0.8966 30.94/0.8716 25.29/0.7936 24.63/0.7337	29.41/0.8530 29.11/0.8194 25.28/0.7379 24.01/0.6758	25.35/0.8035 25.33/0.7880 22.54/0.6934 21.56/0.6231	27.69/0.9215 27.65/0.9062 24.23/0.8284 23.52/0.7594	23.48/0.6537 23.32/0.6598 23.22/0.6431 21.75/0.5879	24.95/0.7223 23.69/0.7072 24.18/0.6846 23.07/0.6412	23.90/0.6436 23.07/0.6330 23.32/0.6132 22.39/0.5773	20.61/0.5529 20.14/0.5478 20.47/0.5266 19.48/0.4809	22.15/0.6494 21.69/0.6462 22.01/0.6265 21.03/0.5744
Partially joint $\mathrm{Joint}{D}_N{D}_M\to SR$	$\sigma =0$ $\sigma =5$ $\sigma =10$ $\sigma =20$	28.66/0.8426 27.52/0.7983 27.20/0.7818 25.73/0.7206	29.66/0.8842 28.43/0.8377 27.92/0.8134 26.59/0.7557	28.24/0.8407 27.11/0.7815 26.66/0.7528 25.57/0.6904	24.49/0.7816 23.95/0.7352 23.68/0.7170 22.67/0.6492	26.67/0.9136 25.94/0.8668 25.55/0.8502 24.53/0.7848	24.44/0.6883 24.48/0.6859 23.98/0.6537 22.70/0.6088	25.92/0.7373 25.72/0.7279 25.02/0.6929 23.71/0.6525	24.70/0.6589 24.47/0.6506 24.11/0.6197 23.03/0.5888	21.18/0.5723 21.18/0.5716 20.76/0.5356 19.93/0.4983	22.71/0.6722 22.69/0.6767 22.21/0.6346 21.42/0.5998
Partially joint $D_N\to \mathrm{Joint}{D}_{M}SR$	$\sigma =0$ $\sigma =5$ $\sigma =10$ $\sigma =20$	29.37/0.8504 27.91/0.8026 27.42/0.7759 25.98/0.7279	31.03/0.8914 29.30/0.8394 28.38/0.8062 26.72/0.7535	28.99/0.8436 27.74/0.7820 27.12/0.7463 25.74/0.6883	24.99/0.7918 24.23/0.7416 23.82/0.7088 22.75/0.6495	27.14/0.9247 26.20/0.8737 25.85/0.8446 24.66/0.7274	25.18/0.7085 24.85/0.6926 24.19/0.6512 23.21/0.6266	26.36/0.7504 25.94/0.7258 25.32/0.6959 24.20/0.6608	25.11/0.6742 24.78/0.6474 24.28/0.6220 23.44/0.5919	21.49/0.5921 21.26/0.5700 20.81/0.5384 20.20/0.5064	21.47/0.6263 21.37/0.6148 21.02/0.5925 20.10/0.5364
Fully joint $\mathrm{Joint}{D}_N{D}_{M}SR$	$\sigma =0$ $\sigma =5$ $\sigma =10$ $\sigma =20$	30.72/0.8860 30.20/0.8575 29.44/0.8387 26.93/0.7600	32.32/0.9095 31.09/0.8749 30.57/0.8461 27.58/0.7781	30.03/0.8673 29.26/0.8240 28.37/0.8046 26.47/0.7131	25.82/0.8226 25.46/0.7930 24.95/0.7704 23.35/0.6838	28.47/0.9393 27.91/0.9320 26.98/0.8935 25.09/0.7859	25.99/0.7433 25.72/0.7244 24.97/0.6937 23.70/0.6473	27.26/0.7729 26.80/0.7498 25.92/0.7193 24.60/0.6750	25.66/0.6889 25.36/0.6669 24.79/0.6421 23.80/0.6051	21.88/0.6157 21.73/0.5993 21.34/0.5739 20.49/0.5261	23.08/0.6998 22.83/0.6790 22.31/0.6428 21.73/0.6093

Red and blue indicate the best PSNR and SSIM results, respectively. All PSNR/SSIM values are computed in the RGB domain and averaged over the images in each dataset (McMaster, DIV2K, Kodak24, Waterloo, and Urban100). The four entries in each cell correspond, from top to bottom, to Gaussian noise levels $\sigma =0,5,10,20$, respectively. The noisy input is generated as $M_n^{LR}=M^{LR}+\mathbf{n}$, where $\mathbf{n}\sim \mathcal{N}(0,{(\sigma /255)}^2)$. In addition, the “stage-wise” row corresponds to a three-stage architecture in which the stages are jointly optimized with intermediate supervision.

Overall, the fully joint pipeline achieves higher PSNR and SSIM than both the stage-wise and partially joint pipelines across all 40 evaluation settings, demonstrating the effectiveness of jointly modeling denoising, demosaicing, and super-resolution across the evaluated datasets, noise levels, and upsampling factors.

PixelShift200 is a real-world dataset captured using a Sony ILCE-7RM3 camera with pixel-shift technology [13]. Four spatially shifted RAW captures provide directly sampled full-color reference images, preserving camera-acquired scene textures and color information. These references support the joint reconstruction of clean HR RGB images from noisy LR Bayer inputs. The experiments add GP noise with variance $\alpha M^{LR}+\beta$ to model shot and read noise.

PixelShift200 was selected because its full-color references suit joint denoising, demosaicing and SR. SID primarily addresses short-exposure low-light recovery, while FiveK provides expert photographic adjustments; their supervisory targets and native evaluation protocols differ from this controlled joint-restoration setting. Native RAW denoising benchmarks such as SIDD and DND would additionally require a defined SR degradation and compatible full-color target/rendering protocol for the complete joint task.

Table 4 summarizes the reconstruction performance and estimated operation counts of TENet and UDDSFormer. The most notable difference is the substantial reduction in the number of parameters for UDDSFormer. At the $\times 2$ scale, UDDSFormer utilizes only 0.8017 M parameters compared to TENet’s 20.22 M, representing a reduction of approximately 96.04%. At the $\times 4$ scale, the parameter count drops from 20.37 M to 0.80 M, a decrease of approximately 96.07%. In summary, regardless of the scale, UDDSFormer requires only about one-twenty-fifth of the parameters used by TENet.

Table 4. The reconstruction performance and computational complexity of the joint restoration model on the real-world PixelShift200 dataset under Gaussian–Poisson (GP) noise with variance $\alpha M^{LR}+\beta$, where $\alpha$ and $\beta$ denote the shot noise and read noise parameters, respectively [13].

Model	SR $\times 2$			SR $\times 4$
	PSNR (dB)	Params. (M)	Operations (G)	PSNR (dB)	Params. (M)	Operations (G)
TENet (RRDB, C64) †	35.90	20.22	828.24	32.03	20.37	2825.76
UDDSFormer-V4 (Ours)	34.94	0.80	44.15	31.02	0.80	156.96

Notes: ^† The PSNR values for TENet are taken from the fully joint model in the reference study; complexity is evaluated locally based on the RRDB-C64 implementation (configured as $D_N+SR\to D_M$). Both models use the same $1\times 8\times 128\times 128$ packed-RAW/noise input and counting convention. The estimates extend fvcore counts (one fused multiply-add = one operation) with supplementary counts for previously unsupported arithmetic. Basic operations and nonlinear primitives receive unit counts; GELU and softmax are expanded into component operations. Comparisons, integer/index operations and data movement are recorded separately. Bold text indicates lower complexity.

The extended operation estimates decrease from 828.24 G for TENet to 44.15 G for UDDSFormer at $\times 2$, and from 2825.76 G to 156.96 G at $\times 4$, corresponding to reductions of 94.67% and 94.45%, respectively. These estimates use identical input dimensions and counting rules. Supplementary arithmetic counting changes the reduction relative to the reproduced fvcore baseline by only 0.27 and 0.28 percentage points at the two scales; the numerical effect on the comparative reduction is therefore small.

The reconstruction PSNR values remain 34.94 dB and 31.02 dB for UDDSFormer at $\times 2$ and $\times 4$, compared with the reference TENet values of 35.90 dB and 32.03 dB. Thus, the compact model trades some reconstruction quality for substantially lower parameter and operation counts. These arithmetic estimates are distinct from measured runtime.

3.3.2. Component Ablation and Reconstruction Performance

Table 5 compares six UDDSFormer configurations on Urban100 under Gaussian noise with $\sigma =10$ and $\times 2$ super-resolution. We evaluate the contributions of deformable sampling (DS), directional decomposition (DD), and local-attention upsampling (LAU1 and LAU2) in terms of parameter count, computational cost, and the associated changes in reconstruction performance. The DS-only configuration achieves PSNR and SSIM values of 25.04 dB and 0.8257, respectively. The full configuration achieves the highest PSNR and SSIM, at 26.98 dB and 0.8935, with 0.80 M parameters. The contribution of deformable sampling (DS) was evaluated by comparing the DD-only configuration with the DS + DD configuration. Enabling DS improved PSNR from 25.21 dB to 26.69 dB and SSIM from 0.8138 to 0.8864, representing gains of 1.48 dB and 0.0726, respectively. These improvements came at the cost of an additional 0.06 M parameters and 2.08 G FLOPs. These results demonstrate the effectiveness of DS when DD is enabled and both LAU modules are disabled. Directional decomposition (DD) was evaluated by comparing the DS-only configuration with DS + DD. Adding DD reduced the parameter count from 0.97 M to 0.96 M and FLOPs from 43.16 G to 40.14 G, corresponding to reductions of 1.03% and 7.00%, respectively. Meanwhile, PSNR increased by 1.65 dB and SSIM by 0.0607. Thus, in this configuration, DD reduced computational cost while improving reconstruction quality. The two LAU modules further reduced the model size, although their effects on reconstruction quality differed. Incorporating LAU1 into DS + DD reduced the parameter count from 0.96 M to 0.78 M, an 18.75% reduction, while increasing FLOPs from 40.14 G to 40.48 G. Meanwhile, PSNR and SSIM decreased by 0.13 dB and 0.0234, respectively. Subsequently enabling LAU2 increased the parameter count by 0.02 M to 0.80 M and increased FLOPs by 0.39 G to 40.87 G, while improving PSNR by 0.42 dB and SSIM by 0.0305. Together, the two LAU modules reduced the parameter count by 16.67% relative to DS+DD at the cost of a 1.82% increase in FLOPs, with gains of 0.29 dB in PSNR and 0.0071 in SSIM. These results demonstrate that the combined LAU modules improve model compactness and reconstruction quality with additional computational cost. Compared with the original Swin-T baseline, the full UDDSFormer configuration reduced the parameter count by 33.88% and FLOPs by 2.88%, while improving PSNR by 2.94 dB and SSIM by 0.2675. Compared with the DS-only configuration, it reduced the parameter count by 17.53% and FLOPs by 5.31%, with increases of 1.94 dB in PSNR and 0.0678 in SSIM.

Table 5. Reconstruction performance and computational complexity of different UDDSFormer component configurations on Urban100.

Uddsformer Components				Evaluation Metrics
DS	DD	LAU1	LAU2	PSNR (dB)	SSIM	Params. (M)	FLOPs (G)
✗	✗	✗	✗	24.04	0.6260	1.21	42.08
✓	✗	✗	✗	25.04	0.8257	0.97	43.16
✗	✓	✗	✗	25.21	0.8138	0.90	38.06
✓	✓	✗	✗	26.69	0.8864	0.96	40.14
✓	✓	✓	✗	26.56	0.8630	0.78	40.48
✓	✓	✓	✓	26.98	0.8935	0.80	40.87

Notes: DS denotes deformable sampling used to generate Key/Value features. DD represents the directional decomposition of window attention into parallel horizontal and vertical branches. LAU1 and LAU2 denote local-attention upsampling at the first and second stages, respectively; results are averaged over Urban100 images for the $\sigma =10$ Gaussian noise and $\times 2$ super-resolution (SR) tasks. The operation counts in this component ablation retain the original fvcore convention; the supplementary arithmetic counting in Table 4 applies to the GP comparison.

Overall, DS improves reconstruction quality and DD reduces computational cost, while the composite LAU module significantly lowers the parameter count and further enhances performance compared to the DS + DD configuration. UDDSFormer achieves the best PSNR and SSIM among the evaluated configurations while requiring fewer parameters and FLOPs than the baseline.

3.3.3. Qualitative Comparison and Attention Map Visualization

Figure 5 compares six selected examples, with two samples drawn from each dataset (kodim01/12, McMaster1/9, and Urban100_002/100). The evaluated checkpoints correspond to the full V4 model addressing “RGB-to-Bayer” direct degradation, Gaussian noise with $\sigma =10$ at an 8-bit intensity scale, and 2× super-resolution tasks. Additionally, attention heatmaps from Stages 2 and 3 of the feature extraction backbone are displayed to illustrate the evolution of attention features. Shallower modules primarily capture local edges and repetitive high-frequency textures, whereas deeper modules progressively aggregate broader contextual information while preserving structural details. This progressive enhancement of response and improved coherence demonstrate that the stacked UDDSFormer modules effectively integrate local details with long-range dependencies, thereby facilitating accurate image reconstruction.

Figure 5. The visualization results for each stage summarize the spatial responses of the six core DDNA modules. (Additionally, for the overlaid displays, the heatmaps are blended with the ground truth (GT) using a heatmap weight of 0.55.) All four views utilize identical cropping coordinates, and the cropped regions are not independently re-normalized. Columns (6) and (7) are used to compare responses across stages, while columns (5) and (8) facilitate a comparison of details between the reference and reconstructed images. The analysis involves examining the underlying heatmaps alongside the overlays to distinguish between object colors and response intensities. The JET colormap represents normalized attention responses, with warmer colors indicating stronger relative responses within each visualization. The heatmap and underlying image have blending weights of 0.55 and 0.45, respectively.

• Kodak24: window structures, masonry textures, and human contours. Figure 5a contains both regular window structures and irregular wall textures. In the reference image a(5), the window frames and horizontal dividers have clear boundaries; the interior of the shutters is relatively uniform, while the surrounding wall exhibits subtle textural variations. The second-stage attention map a(6) displays a broad, smooth response region around the window; in contrast, the third-stage attention map a(7) shows more concentrated responses along the vertical window frames, horizontal dividers, and the bottom edge of the window, with weaker responses inside the window. This shift indicates that attention responses in deeper stages correspond more precisely to major structural boundaries. However, the subtle wall textures do not elicit similarly distinct responses. The final reconstruction a(8) preserves the window’s primary geometry and contours, though some wall details are smoothed out compared to a(5), suggesting that the preservation of major structures is superior to the recovery of fine material textures in this sample. Figure 5b illustrates the difference in attention responses between the human figure and the water background. In the global attention maps b(2) and b(3), responses near the figure are strong, while responses across most of the water surface are weak. Close-up views reveal that the second-stage map b(6) primarily shows a broad response covering parts of the figure, whereas the third-stage map b(7) exhibits clearer localized distinctions: responses are more pronounced along the outer body contours, between limbs, and at clothing boundaries, while responses within the torso are relatively weak. This indicates that the response shifts from broad regional distributions to specific structural boundaries across the two stages, rather than uniformly enhancing the entire figure. The final reconstruction result, b(8), preserves the main contours of the body and the color of the clothing, although some subtle tonal variations are smoothed out. Thus, these visualizations demonstrate a clear spatial correspondence between the model’s attention response and structural features such as straight lines, irregular contours, and region boundaries, reflecting a selective focus on boundaries of varying morphological structures.

• McMaster: color partitions, fruit interiors, and basket boundaries. Figure 5c features bright glass areas, dark dividing lines, and slanted decorative borders. In the second stage, the attention map c(6) exhibits broad response bands near some horizontal dividing lines and slanted edges. In the third stage, c(7) displays finer local variations near these boundaries, while responses within some glass areas remain relatively low. Since the yellow glass also appears warm-toned in the overlaid image, response intensity must be assessed in conjunction with the underlying heatmap. It is evident that significant responses primarily correspond to certain dividing lines and slanted boundaries. The reconstruction result c(8) preserves the main color zones and structural boundaries, though some fine glass textures appear smoother compared to the reference image c(5). Figure 5d features a flat wall surface, plant leaves, fruit, and a basket. The global attention maps d(2) and d(3) show weak responses across most of the wall area, with more distinct responses in areas containing objects. The zoomed-in reference region d(5) includes the interiors of fruits, the boundaries between adjacent fruits, and the basket’s rim and fine structural elements. The second-stage map d(6) shows relatively scattered responses, whereas the third-stage map d(7) reveals more pronounced local variations at the basket rim and the boundaries between adjacent fruits. The underlying heatmap further indicates that responses within some fruits are lower than at their surrounding boundaries; thus, in the overlaid image, the red color of the tomatoes themselves must be distinguished from the high-response colors of the heatmap. The reconstruction result d(8) retains the general arrangement of the fruit and the basket’s outline, though some surface details are smoothed out. Figure 5c,d demonstrate a clear distinction between attention responses within objects and those at structural boundaries. The responses in Figure 5c primarily correspond to regular dividing lines and slanted edges, whereas the responses in Figure 5d correspond more to the curved boundaries of the fruit, object junctions, and the structure of the basket. These results demonstrate the selective response of attention to boundaries of different forms, as well as the more refined response distribution near these boundaries at deeper stages.

• Urban100: intersecting frameworks and repetitive curved balconies. Figure 5e shows the glass and metal framework of a curved enclosure. e(5) depicts the thicker support frame, transverse connectors, and a fine diagonal structure. The attention map e(6) displays broad response regions in the second stage, with some areas interconnected; in the third stage, e(7) reveals more detailed variations in response intensity near the framework. These responses are unevenly distributed across the various glass panels and support structures, demonstrating spatial selectivity regarding the complex intersecting geometry. The reconstruction e(8) preserves the orientation of the main support frame and some intersection points, though—compared to e(5)—finer support structures and subtler textures appear smoothed out. Figure 5f illustrates a series of repetitive balcony structures arranged along a curved path. The attention map f(6) shows broad response bands near some of the arc-shaped structures, with connections or gaps between response regions visible in the second stage. In the third stage, f(7) displays several response bands following the curvature of the balconies, separated by zones of lower response, thereby clarifying the correspondence between the response distribution and the repetitive curved forms. The reconstruction f(8) retains the arrangement, primary curvature, and overall silhouette of the balconies, although some railings and surface details are smoothed. Figure 5e,f demonstrate that distinct structural layouts correspond to different patterns of attention response. The multi-directional intersecting framework in Figure 5e corresponds to complex local response variations, whereas the repetitive arc structures in Figure 5f correspond to response bands aligned with the curves. These examples illustrate the spatial correspondence between attention responses and the orientation, intersection points, and repetitive arrangements of the underlying structures.

The six examples reveal that the attention response in the second stage is relatively smooth and broad, whereas the third stage exhibits clearer, more detailed local variations along major contours, structural boundaries, and region interfaces. Compared to the second stage, the response distribution in the third stage demonstrates a more distinct spatial correspondence with straight lines, curves, and intersecting structures, thereby better highlighting the image’s primary structural features. Comparisons between the ground truth and the reconstruction results indicate that the model effectively preserves major contours, color regions, and geometric layouts; however, some fine textures—such as wall textures, glass details, tonal variations on fruit surfaces, small support structures, and railings—are still subject to smoothing. This suggests that while the model successfully retains major structures, there remains room for improvement in the recovery of fine-grained textures.

Overall, these visualizations demonstrate the improvements in structural response distribution achieved in the third stage: compared to the second stage, the spatial correspondence between the response and regular straight lines, irregular contours, repetitive curves, and intersecting structures is clearer, allowing for better emphasis on major boundaries and regional transitions. Furthermore, the reconstruction results effectively preserve the image’s primary contours, color regions, and geometric layouts, demonstrating robust recovery capabilities across various structural types, although some fine textures and low-contrast details still undergo a degree of smoothing.

3.4. Hardware-Oriented Quantization Analysis

To evaluate the reconstruction quality of UDDSFormer at different quantization precisions, we conducted quantization-aware training (QAT) [45] using six combinations of weight and activation bitwidths. Figure 6 shows the validation PSNR curves during fine-tuning, while Table 6 summarizes the corresponding DIV2K validation set results. During fine-tuning, fake quantization operations simulated clipping and rounding errors, enabling the network parameters to adapt to perturbations introduced by low-precision representations.

Figure 6. Validation set PSNR curves for UDDSFormer-V4 during Quantization-Aware Training (QAT) under six bit-width configurations: (a) $W_8{A}_8$, (b) $W_8{A}_6$, (c) $W_6{A}_8$, (d) $W_6{A}_6$, (e) $W_4{A}_8$, (f) $W_8{A}_4$. $W_x{A}_y$ denotes x-bit weights and y-bit activations. Red asterisks mark the highest PSNR for each configuration. The curves represent unsmoothed validation results; each iteration corresponds to an optimizer update, and the end of the curve indicates the final recorded validation iteration.

Table 6. Reconstruction performance of UDDSFormer-V4 on the DIV2K validation set under different QAT bit-width configurations.

Configuration	Last Iteration	Best Iteration	PSNR (dB)	$\mathbf{\Delta }$ (dB)
(a) W8A8	2000	2000	31.53	0.19
(b) W8A6	13,300	13,000	30.30	1.42
(c) W6A8	2900	2000	31.51	0.20
(d) W6A6	9200	8400	30.11	1.61
(e) W4A8	4500	3900	30.30	1.42
(f) W8A4	21,100	20,100	25.74	5.98

Notes: $\Delta ={\mathrm{PSNR}}_{\mathrm{FP}32}-{\mathrm{PSNR}}_{\mathrm{QAT}}$, with an FP32 reference of 31.72 dB; lower is better. Bold indicates the best QAT results.

QAT selectively simulates low-precision weights and input/output activations for convolutional and linear projections, including the Q/K/V projections. Normalization, attention matrix products, softmax, bilinear sampling and residual additions remain floating point. Accordingly, W8A8 denotes the selected weight/activation precision in the simulation, rather than an integer-only implementation of every operator. The $W_8{A}_8$ and $W_6{A}_8$ configurations achieve peak validation PSNR values of 31.53 dB and 31.51 dB, respectively, at iteration 2000. Compared to the FP32 baseline (31.72 dB), these represent drops of only 0.19 dB and 0.20 dB, demonstrating that the model effectively maintains its original reconstruction quality under higher-precision configurations. Furthermore, the PSNR difference between $W_8{A}_8$ and $W_6{A}_8$ is merely 0.0176 dB. This indicates that, while maintaining 8-bit activations, reducing the weight precision from 8 bits to 6 bits has a negligible impact on reconstruction quality. However, when the weight precision is further reduced to 4 bits, the peak PSNR for $W_4{A}_8$ drops to 30.30 dB—a decrease of 1.23 dB relative to $W_8{A}_8$—demonstrating that performance degradation intensifies as the degree of weight compression increases.

In contrast, the performance impact of reducing activation bit-width is more pronounced. With weights maintained at 8 bits, reducing activations from 8 bits to 6 bits caused the best PSNR to drop from 31.53 dB to 30.30 dB— a decrease of 1.24 dB—which is significantly greater than the loss incurred by reducing weights from 8 bits to 6 bits. Starting from the 6-bit activation baseline, further reducing weights to 6 bits resulted in a $W_6{A}_6$ performance of 30.11 dB, a further drop of 0.19 dB compared to $W_8{A}_6$. It is evident that, under the current experimental setup, preserving activation precision is more beneficial for maintaining reconstruction quality than preserving weight precision at the same bit-width.

This difference is even more striking in the 4-bit configurations. The best validation PSNR for $W_4{A}_8$ is 30.30 dB, whereas for $W_8{A}_4$ it was only 25.74 dB—a difference of 4.57 dB. Although both configurations reduced one type of numerical representation to 4 bits, the performance loss caused by reducing activation bit-width was far greater than that caused by reducing weight bit-width. Furthermore, $W_4{A}_8$ achieved peak performance at the 3900th iteration, while the best checkpoint for $W_8{A}_4$ occurred at the 20,100th iteration. Even with the extended fine-tuning observed, the 4-bit activation configuration failed to recover to the quality levels of the other configurations, indicating that increasing fine-tuning iterations is insufficient to fully compensate for the quantization loss associated with this configuration.

Overall, the six configurations demonstrate a clear trade-off regarding precision: moderately reducing weight bit-width effectively preserves reconstruction quality, whereas reducing activation bit-width requires greater caution. Among them, $W_6{A}_8$ maintains validation performance close to that of $W_8{A}_8$ while employing lower bit-widths for weights, making it a configuration worthy of priority consideration for subsequent low-precision hardware deployment [44,52].

Figure 7 compares the reconstruction performance of six QAT configurations across seven benchmark datasets in terms of PSNR and SSIM, using $W_8{A}_8$ as the quantization baseline. This comparison examines the effects of weight and activation bit-widths and their variations across datasets.

Figure 7. Quantitative analysis of UDDSFormer-V4 on seven benchmark datasets under different quantization-aware training (QAT) configurations: (a) W8A8, (b) W8A6, (c) W6A8, (d) W6A6, (e) W4A8, and (f) W8A4. WxAy denotes x-bit weights and y-bit activations. Blue and magenta bars represent PSNR (dB, left axis) and SSIM (right axis), respectively. Higher values indicate better reconstruction quality.

First, the $W_6{A}_8$ configuration maintains reconstruction quality close to that of $W_8{A}_8$ across all evaluated datasets. On McMaster, the respective PSNR/SSIM values are 29.12/0.8207 and 29.13/0.8213; on DIV2K, they are 29.71/0.8375 and 29.72/0.8374. On Kodak24 and Set14, the PSNR values are identical when rounded to two decimal places. Although $W_6{A}_8$ achieves slightly higher SSIM on DIV2K and Set5, the differences are small. These results indicate that reducing weight precision from 8 bits to 6 bits while maintaining 8-bit activations has minimal impact on reconstruction quality.

Second, the comparison of 4-bit configurations highlights distinct differences in the impact of weight versus activation quantization. Compared to W8A8, the PSNR for W4A8 drops by only about 0.54–1.05 dB, whereas for W8A4, it drops by approximately 3.05–4.86 dB, with a simultaneous SSIM decline of about 0.20–0.27. Notably, the PSNR loss is greatest on the McMaster dataset, where results fall from 29.13/0.8213 to 24.27/0.5532; the loss is smallest on the Waterloo dataset, yet it still drops from 24.87/0.7572 to 21.82/0.5399. Furthermore, W4A8 outperforms W8A4 on both metrics across all datasets, indicating that under the evaluated configurations, reducing the activation bit-width is more detrimental to reconstruction quality than reducing the weight bit-width. A comparison between W6A6 and W8A6 reveals that a higher nominal weight bit-width does not necessarily yield better results. Except for Kodak24, W6A6 achieves superior PSNR and SSIM values compared to W8A6 on the other six datasets. For instance, on DIV2K, W6A6 reaches 28.78/0.7980, whereas W8A6 scores 28.12/0.7712; on Kodak24, the ranking is reversed, with scores of 27.44/0.7351 and 27.57/0.7441, respectively.

Additionally, a further comparison of the results for W4A8, W6A6, and W8A6 shows that the optimal choice among intermediate configurations depends on the dataset. On DIV2K and Set5, W4A8 outperforms the other two configurations on both metrics, achieving 28.89/0.8125 and 28.68/0.8132, respectively; on Urban100, however, W6A6 achieves better results, reaching 24.54/0.7790, surpassing W4A8’s 24.39/0.7782 and W8A6’s 24.26/0.7627. Therefore, there is no universal performance ranking for intermediate configurations that applies to all datasets. The rankings of configurations based on PSNR and SSIM are not entirely consistent. On McMaster, Waterloo, and Set14, W6A6 achieves a slightly higher PSNR than W4A8, yet W4A8 yields a higher SSIM. For instance, on the McMaster dataset, the respective values are 28.20/0.7826 and 28.08/0.7917. Kodak24 exhibits a similar pattern: W8A6 achieves a higher PSNR than W4A8 but a lower SSIM, with values of 27.57/0.7441 and 27.47/0.7510, respectively. This demonstrates that when evaluating intermediate bit-width configurations, one must consider both pixel-level fidelity and structural similarity, rather than relying on a single metric.

For Kodak24 and Urban100, we further compare the quantized results in Figure 7 with the fully joint results under Gaussian noise with $\sigma =10$ and $\times 2$ super-resolution in Table 3. On Kodak24, both W8A8 and W6A8 achieve a PSNR of 28.27 dB, which is only 0.10 dB lower than the reported fully joint result of 28.37 dB. On Urban100, W8A8 and W6A8 achieve 25.19 dB and 25.17 dB, respectively, corresponding to differences of 1.79 dB and 1.81 dB relative to the fully joint result of 26.98 dB. These comparisons show that the differences between the reported quantized and fully joint results vary across datasets. Within the QAT configurations, reducing weight precision from 8 bits to 6 bits while retaining 8-bit activations leaves the Kodak24 PSNR unchanged at two-decimal precision and reduces the Urban100 PSNR by only 0.02 dB. In contrast, retaining 8-bit weights while reducing activation precision from 8 bits to 6 bits decreases PSNR by 0.70 dB on Kodak24 and 0.93 dB on Urban100. These results further support preserving activation precision while moderately reducing weight precision.

Overall, W6A8 consistently maintains reconstruction quality close to that of W8A8 across datasets, whereas W8A4 consistently shows significant performance degradation; this supports the strategy of prioritizing activation precision while moderately reducing weight bit-widths. The variations observed in intermediate configurations indicate that final performance is jointly influenced by the dataset, evaluation metrics, and fine-tuning settings.Furthermore, the results in Figure 7 were evaluated on the complete test sets of McMaster, DIV2K, Kodak24, Waterloo, Urban100, Set5 and Set14. For large images, tiled inference uses $128\times 128$ core regions with a 64-pixel context border on each side in packed-RAW coordinates. Each forward pass processes at most $256\times 256$ input positions. Border outputs are discarded, and core outputs are stitched into the full image on the CPU, limiting the image area processed on the accelerator per pass. This controls the processed input area without changing the FP32 dtype of the measured attention tensors.

4. Conclusions

This paper proposes UDDSFormer, a lightweight Transformer model designed to unify demosaicing, denoising, and super-resolution tasks. The model captures spatial dependencies within a unified framework and facilitates joint reconstruction by sharing intermediate features across tasks. Additionally, its configurable restoration pipeline accommodates both single-task and multi-task processing, making it suitable for image restoration across varying noise levels and super-resolution scaling factors. Through component ablation studies, this paper analyzes the impact of individual modules on reconstruction quality, parameter count, and computational complexity; it also visualizes intermediate attention maps to demonstrate the spatial correspondence between responses at different stages and image structures. Furthermore, quantization-aware training (QAT) experiments reveal the distinct effects of weight and activation precision on reconstruction quality, providing a basis for bit-width configuration in subsequent low-precision deployments. Evaluations on multiple benchmark datasets validate UDDSFormer’s reconstruction capabilities across diverse restoration settings. Combined with analyses of model complexity and quantization results, UDDSFormer demonstrates potential for deployment in resource-constrained embedded ISP systems. Future work will investigate hardware-supported mixed-precision execution, operator fusion and memory-aware tiling. Building on our earlier FPGA implementation of joint denoising and demosaicing [44], we are developing a Transformer hardware accelerator. Target-device integer inference and energy measurements remain future work. Evaluation with paired authentic sensor-noise data and across camera systems will further assess generalization.