Attention-Edge-Assisted Neural HDRI Based on Registered Extreme-Exposure-Ratio Images

Abstract

In order to improve image visual quality in high dynamic range (HDR) scenes while avoiding motion ghosting artifacts caused by exposure time differences, innovative image sensors captured two registered extreme-exposure-ratio (EER) image pairs with complementary and symmetric exposure configurations for high dynamic range imaging (HDRI). However, existing multi-exposure fusion (MEF) algorithms suffer from luminance inversion artifacts in overexposed and underexposed regions when directly combining such EER image pairs. This paper proposes a neural network-based framework for HDRI based on attention mechanisms and edge assistance to recover missing luminance information. The framework derives local luminance representations from a convolution kernel perspective, and subsequently refines the global luminance order in the fused image using a Transformer-based residual group. To support the two-stage process, multi-scale channel features are extracted from a double-attention mechanism, while edge cues are incorporated to enhance detail preservation in both highlight and shadow regions. The experimental results validate that the proposed framework can alleviate luminance inversion in HDRI when inputs are two EER images, and maintain fine structural details in complex HDR scenes.

1. Introduction

In natural conditions, the dynamic range of a scene can reach up to ten orders of magnitude. However, mainstream image capture and display devices can only acquire and present low dynamic range (LDR) content. HDRI technology successfully addresses this limitation, benefiting applications in medical imaging, security, remote sensing, and industrial inspection. Conventional HDR techniques expand the luminance, contrast, and color gamut of the synthesized image by merging images with diverse exposure settings, significantly enhancing the realism and immersion of visual experiences. To generate HDR images, various capture methods have emerged, like staggered HDR [1], temporal exposure control [2], and global shutter with regional exposure [3]. These state-of-the-art techniques can obtain different exposure time information in a single shot, effectively preventing the motion artifacts that often arise from multi-exposure images synthesis in HDRI [4,5,6,7].

However, capturing images with excessively different exposures often comes at the cost of spatial resolution. Thus, it is typical to obtain a pair of EER images consisting of one long-exposure image and one short-exposure image with symmetric and complementary exposure settings, as shown in Figure 1a. A larger exposure interval is deliberately applied to the EER images to better capture a wider dynamic range, leading to hardly any overlap between the radiance ranges of the two images. The MEF algorithms based on pyramid weight map (e.g., [8,9,10]) may occur luminance inversion artifacts that degrade visual realism, as illustrated in Figure 1b. Consequently, the pixel values in the highlight areas of the low-exposure image are lower than those in the dark regions of the large-exposure image. As shown in Figure 1b, the outdoor areas appear brighter than the indoor areas at night.

Recent research has focused on designing end-to-end deep approaches for MEF by leveraging data captured at varying exposure times to address luminance inversion artifacts [11,12,13,14,15]. These methods have proven effective in reducing such luminance reversal issues. However, as shown in Figure 1c, these convolutional-based methods cannot preserve the sharp contrast in high- and low-brightness areas. By focusing on local regions, CNNs are able to detect and combine fine-grained details from different exposures effectively. However, when fusing two EER images, it is critical to consider not only localized features but also the overall contextual cues contained in the image. In contrast to CNNs, Transformer models have demonstrated effectiveness in a wide range of vision tasks, owing to the self-attention mechanism that captures wide-range spatial dependencies between features [16,17,18]. Intuitively, this attention structure can model the global brightness representation, making it advantageous for fusing two EER images. Nevertheless, the self-attention is generally applied to small independent patches, which can produce artifacts when handling image-related tasks. To address this issue, SwinIR [18] proposed an interesting Swin Transformer model for image restoration [19], which adopts a shifting window method to capture long-range dependencies. Despite the use of window-based Transformers for image tasks, blocking artifacts persist in the smooth areas of the images. This occurs because the shifted window strategy fails to establish adequate connections between adjacent local windows.

This paper proposes an HDRI framework based on an attention-edge-assisted neural network (AEAN), aiming to fuse two registered EER images. The network combines convolutional kernels with a multi-head attention mechanism to enhance both localized and holistic feature representation. We first use a residual network to roughly learn fusion features. Due to the limited representation ability of preliminary fusion, the coarse fusion result can be regarded as a noisy image for real images. Therefore, this paper embeds Swin Transformer into the residual CNN group (STRCG) to form a short jump connection, which can retain shallow information and pass it to the deep layer, so as to learn the luminance differences between varying exposure images. Simultaneously, the residual CNNs are injected into the Swin Transformer block (CSTB) to ensure that the fused image can effectively represent the global brightness smoothness and local detail features. Specifically, a multi-scale channel attention (MSCA) and an edge-assisted enhancement (EH) model are designed in the CSTB structure to extract features with stronger representation ability. Notably, the entire proposed framework is cascaded through the residual structure, so that the fused features have sufficient multipath transmission ability. Therefore, the algorithm can solve both halo artifacts and luminance inversion artifacts, as demonstrated in Figure 1d, outperforming methods that rely solely on CNNs. In addition to the network structure, we also design suitable loss functions to maintain color realism and brightness spatial consistency. We use the color cosine loss function to alleviate the risk of color inconsistency in the fused output, and use SSIM and spatial loss functions to enhance spatial coherence. Experimental results indicate that our AEAN effectively suppresses luminance inversion artifacts while maintaining details in shadow and highlight areas. Our main contributions are as follows

• An attention-edge-assisted neural framework is proposed to fuse two EER images, which preserves the information of the highlight and shadow areas and avoids halo-like artifacts present in the fusion output.
• To retain the overall contrast structure and simultaneously enhance fine local details of the fused image, we embed the CNNs into the Transformer module as the core component. We parallelly concatenate the dual attention mechanism and residual gradient information into the window-based multi-head attention, taking into account both local and long-range information of the input images.

The remainder of this paper is organized as follows. Section 2 reviews existing work on multi-exposure fusion. Section 3 details the proposed AEAN algorithm. Experimental results comparing our AEAN with current exposure fusion algorithms are given in Section 4, and Section 5 concludes the paper.

2. Related Works

Under the assumption that the scenes in all images to be fused are aligned, many MEF algorithms for HDR imaging have been proposed. The core objective of these algorithms is to integrate information from all input exposure images as much as possible to generate a high-definition image. Existing MEF methods include traditional algorithms, CNN-based frameworks, and Transformer-based architectures.

The classic traditional MEF method in [8] decomposed the image into a pyramid and fused the features at each scale level to maintain the multi-scale expression of the image. Then, at each scale level, an appropriate fusion strategy was adopted and weights were carefully designed to fuse pixels in different exposure images at multiple scales. Finally, the fused image was reconstructed to produce the final high-quality output. This approach captures a wider range of brightness and provides richer colors in the fused image, allowing details in both dark and bright areas to be visible. However, the algorithm in [8] faces a fundamental difficulty in preserving information from both the highlights and shadows of an HDR scene [20]. As shown in [21], such pyramid-based methods [9,22,23,24] can solve this problem, but they also suffer from brightness reversal in overexposed areas. Adding a suitable virtual exposure image into the input sequence has proven to be one of the effective methods to solve the luminance reversal problem [25,26,27].

The CNN-based methods can automatically extract the features and learn the patterns from the image through the characteristics of deep learning, significantly improving the fusion effect. The first CNN network for MEF named DeepFuse [11] obtained the richly textured image by decomposing the images into YUV space and then fusing luminance cues from two LDR images using an end-to-end framework. Since then, a large number of MEF algorithms based on combining traditional models and CNNs have been proposed. Similar to [11], Wu et al. proposed a DMEF network [28], which obtains the initial illumination map from the Retinex model [29] as the input of CNN, so as to learn the fused illumination map. Ma et al. [12] introduced an MEF-Net, where the weight map is learned from downsampled images via unsupervised learning on top of MEF-SSIM [30] and then upsampled to full size via [31]. Moreover, Jiang et al. [13] proposed a fast MEF method, which used a one-dimensional lookup table to take pixel values as input to produce fusion weights. The lookup table is learned for each exposure using unsupervised learning and GGIF. Zheng [14] proposed a filter prediction-driven fusion paradigm to generate spatially flexible filters based on level-by-level represented features to carry out pixel-level dynamic fusion. The algorithms in [12,13,14] calculate the coefficients of filter once, which means that the methods in [12,13,14] may not be able to transfer the structure of the brightness component to the weighted graph well. There are also methods that obtain information-rich image fusion structures by improving the network structure, such as MEF-GAN [32] and MIN-MEF [33]. These interactive methods learn brightness information and may not be able to preserve scene depth and local contrast well. They perform well if enough images are captured and if the captured images cover as much radiance as possible in the HDR scene. Unfortunately, this is not always true. For example, the brightness in overexposed and underexposed areas of the fused image might occur inverted if only two exposure images of an HDR scene are fused.

The self-attention mechanism in Transformer enables capturing wide-range spatial correlations across multi-exposure images. Therefore, compared to CNNs that focus on processing local information, Transformers are able to model the global features of images, avoiding possible missed details or artifacts. Vs et al. [34] proposed an MEF framework named IFT by training an auto-encoder at multiple scales and using a spatial Transformer fusion strategy. Qu et al. [35] proposed a framework entitled TransMEF by introducing self-supervised learning and multi-task learning. Liu et al. [36] designed an encoder that integrates CNNs and Transformer to capture multi-level abstractions while incorporating holistic context. These Transformer-based methods attempt to use Transformer’s self-attention mechanism to extract abundant features as much as possible. IET [34] and TransMEF [35] pay attention to the availability of the Transformer module in MEF, while Liu et al. [36] focus on utilizing CNN and Transformer to extract short and long-distance features, respectively. In a word, the above methods aim to make up for the shortcomings of long-range extraction in CNN, but directly using Transformer will produce block artifacts in the fused image. To address this issue, the proposed AEAN algorithm first applies a CNN to perform coarse fusion of the two EER images, followed by a coarse-to-fine fusion strategy that integrates Transformer and CNN frameworks.The approach leverages multi-scale spatial attention and edge-assisted features, enabling the model to produce a single, information-rich fused image that faithfully replicates the radiometric characteristics of the HDR scene.

3. The Proposed AEAN Method

In this part, Section 3.1 first outlines the fusion process of two EER images. Section 3.2 introduces the proposed attention-edge-assisted neural (AEAN) HDRI method, which learns both global and detailed fusion representations from low-exposure and high-exposure images. Finally, Section 3.3 describes the training strategy and objectives.

3.1. Motivation and Overview

When there is a pair of EER images $I_1$ and $I_2$, assuming that the exposure times of $I_1$ and $I_2$ are $\Delta t_1$ and $\Delta t_2$, respectively, $\Delta t_1<\Delta t_2$. Assume $f_{i,j}(·)$ is the pixel mapping function of images $I_i$ to $I_j$. This function is the CRF inverse mapping and can be easily calculated by the algorithms in [37,38]. When the radiation values between the two images hardly overlap, as shown in Figure 2c, the relationship between the pixels in the two images may be

$$f_{1,2}\left(I_1\left(p_h\right)\right)<f_{2,1}\left(I_2\left(p_l\right)\right),$$

(1)

where $p_h$ and $p_l$ denote the highlight pixel in image $I_1$ and the low-light pixel in image $I_2$, respectively [or Figure 2a,b], $I_1\left(p_h\right)$ and $I_2\left(p_l\right)$ are their corresponding intensity values. According to common sense, the exposure time does not change the holistic luminance distribution across the image. The high-brightness area is still the brightest in the image when the exposure time is very short. This feature also applies to the low-brightness area. However, if we directly fuse these two images, the pixel brightness of the fused image $I_f$, as demonstrated in [26], will be reversed as

$$I_f\left(p_h\right)<I_f\left(p_l\right),$$

(2)

as one example shown in Figure 2d. All in all, considering both global luminance and local detail preservation when fusing two EER images is challenging. When there is hardly any overlap in the radiance range of the two images, directly fusing them will introduce luminance order inversion [or Figure 2d] and only convolution-based modules for fusion will limit global brightness smoothing [or Figure 1c]. Although good gradient details can be maintained by convolution, the overall contrast of the fused result in bright and shadow parts will be lost. To accurately extract complementary information from both EER images, the proposed exposure fusion method aims to fuse the pair into a single information-rich image, while ensuring global and local brightness. Therefore, we first proposed STRCG, which incorporates the Swin Transformer into the residual network structure to capture the global contrast between bright and dark areas. In each STRCG, we then designed several CSTB modules, which embed CNNs within the Swin Transformer to reduce the blockiness artifacts associated with the window-based Transformer and to learn multi-scale gradient detail features.

3.2. Structure of the Proposed AEAN Method

This subsection presents the entire architecture of the proposed AEAN method. As shown in Figure 3, the proposed method comprises a three-stage framework, including shallow feature fusion, global and detail level feature enhancement, and fused image reconstruction.

Taking two EER images as an example, AEAN takes $I_1,I_2{\in }^{ H\times W\times 3}$ as inputs, where H and W are the height and the width of the input, and 3 corresponds to the color channel. As shown in Figure 3, a shallow feature fusion is first performed, consisting of a $3\times 3$ convolution followed by a normalization layer and an activation function, 4 residual blocks with 2 at $1/2$ resolution and 2 at $1/4$ resolution, and one $1\times 1$ convolution. An additional $1\times 1$ convolution is then applied for preliminary fusion, producing the rough fusion feature $F_{f0}{\in }^{ 2C\times \frac{H}{4}\times \frac{W}{4}}$. Then, the global and detail level feature enhancement strategies are adopted through the Swin Transformer combined residual CNN group (STRCG) to further learn the global brightness and detail fusion features of the fused image. The core part of STRCG is the residual CNN-embedded Swin Transformer block (CSTB). In the core module CSTB, we use CNN self-supervised learning to preserve the local texture features. Meanwhile, we use the STRCG to supplement the missing global brightness information of the two EER images. Details of CSTB can be found in Section 3.2.2. Then, the descriptor integration is achieved through a global residual connection that combines both shallow features and global detail fusion outputs. Finally, the image reconstruction module is responsible for generating the final fused image. It consists of a reconstruction trunk (Recon-trunk) composed of 10 residual blocks, followed by two upsampling layers. Each upsampling layer includes a $3\times 3$ convolution, a PixelShuffle operation with an upscale factor of 2 for spatial resolution enhancement, and a ReLU activation function. The reconstruction process concludes with a final $3\times 3$ convolution layer that produces the final fused image. Next, we explain the structures of STRCG and CSTB from the perspectives of global-level design and detail-level design, respectively.

3.2.1. Global-Level Design

At the global level, we incorporate Transformer into Resnet to learn the global brightness characteristics. The global level consists of three STRCGs and one convolutional layer, as shown in Figure 3. Given an input feature ${F^{\prime }}_{f_{i-1}}$, the output of STRCG is computed as

$${F^{\prime }}_{f_i}={\mathrm{H}}_{STRC{G}_i}\left({F^{\prime }}_{f_{i-1}}\right),i=1,2,3,$$

(3)

where ${\mathrm{H}}_{STRC{G}_i}$ refers to the i-$th$ STRCG, and then the output ${F^{\prime }}_{f_i}$ enters the $3\times 3$ convolution layer ${\mathrm{H}}_{CONV}$ and residual addition $F_{f0}$ to obtain the output of global level $F_{f_3}$

$$F_{f_3}={\mathrm{H}}_{CONV}\left({F^{\prime }}_{f_3}\right)+F_{f0}.$$

(4)

In particular, each STRCG has four CSTBs; that is, for a given feature ${F^{\prime }}_{f_{i,j-1}}$ of the $\{j-1\}$-$th$ CSTB in i-$th$ STRCG, the intermediate output feature is

$$F_{f_{i,j}}^{\prime }={\mathrm{H}}_{CST{B}_{i,j}}\left({F^{\prime }}_{f_{i,j-1}}\right),i=1,2,3;j=1,2,3,4,$$

(5)

where ${\mathrm{H}}_{CST{B}_{i,j}}$ is the j-$th$ CSTB in the i-$th$ STRCG. Following processing by multiple CSTBs, the convolution layer ${\mathrm{H}}_{CONV}$ is preserved before the residual connection in the subsequent STRCG, as formulated below:

$$F_{f_{i,j}}={\mathrm{H}}_{CONV}(F_{f_{i,j}}^{\prime })+{F^{\prime }}_{f_{i-1,j}},i=1,2,3;j=1,2,3,4,$$

(6)

where ${F^{\prime }}_{f_{i-1,j}}$ is the output of the j-$th$ CSTB in $\{i-1\}$-$th$ STRCG. The whole process is similar to the CNN-injected Transformer in [16]; thus, the global contrast of the fused image is well preserved.

3.2.2. Detail-Level Design

At the detail level, we embed the CNNs into Swin Transformer block (CSTB) to efficiently leverage multi-scale texture and global contrast from different images, and the structure of CSTB is shown in Figure 4. We propose multi-scale spatial-channel attention and edge-assisted enhancement (MSCA-EH) to preserve edge details while effectively utilizing spatial and exposure time information from inputs. As part of MSCA-EH, a window-based Transformer architecture is selected as the core module component to acquire holistic context. As presented in Figure 4, our MSCA mainly consists of multi-scale spatial attention (MSA), channel attention (CA), and window-based multi-head self-attention (W-MSA). In addition, different from the commonly used spatial-channel attention structure, the edge-assisted enhancement (EH) is multiplied after the attention to effectively improve the details of the fused result.

This embedding process can be formulated as

$$Y=Y_{MSCA}\times Y_{EH}+X,$$

(7)

where $Y_{MSCA}$, $Y_{EH}$ denote the output of MSCA, EH, and X is the initialization feature of the input in CSTB. Figure 4 displays the detailed configuration of the CSTB.

• Multi-scale Spatial Attention. Given a shallow fusion image of two EER images, multi-scale spatial attention (MSA) is designed followed by layer normalization (LN) to further fuse them in a coarse-to-precise manner. As presented in Figure 4, input $X_L$ is first split into three branches, each undergoing initial feature extraction with convolutions of size $1\times 1$, $3\times 3$, and $5\times 5$, respectively. After each layer, average and max pooling are sequentially executed to capture global spatial information at different receptive fields. The pooled features from each branch are concatenated along the channel dimension and then passed through a $3\times 3$ convolution followed by a Sigmoid activation to generate spatial attention maps corresponding to each scale. The multi-branch attention maps are further concatenated and processed by a $3\times 3$ convolution, a GELU activation, and another $3\times 3$ convolution for feature calibration. Finally, a residual connection adds the calibrated features to $X_L$, producing the enhanced feature $X_{MSA}$. Similar to the algorithms in [11,14], the proposed algorithm is not based on weighted graphs like the algorithms in [12,13]. Therefore, the proposed method and the algorithms in [11,14] can effectively avoid brightness reversal artifacts in the fused images.
• Channel Attention. The channel attention (CA) is inserted in parallel with W-MSA and MSA. As shown in Figure 4, $X_L$ first passes through a $3\times 3$ convolution. Global context is captured via average pooling and is then processed by another $3\times 3$ convolution with GELU activation to model non-linear channel dependencies. A third $3\times 3$ convolution followed by Sigmoid generates channel-wise attention, which is multiplied with $X_L$ to produce the refined output $X_{CA}$. The whole process is similar to the hierarchical Transformer in [10,16] by reshaping the flattened token sequence back to the spatial grid to refine the local information, so as to reduce the blocking effect caused by the window segmentation mechanism. At the same time, because CA contains global information, the introduction of CA can dynamically adjust the weight of each channel, so that more pixels are activated. Similar to [18] in W-MSA, we use shift W-MSA in continuous CSTB in a periodic manner.

Then, the dual attention block MSCA can be described as

$$X_k=WMSA\left(LN\left(X\right)\right)+MSA\left(LN\left(X\right)\right)\overline{)\mathrm{C}}CA\left(LN\left(X\right)\right),$$

(8)

where $WMSA(·)$, $MSA(·)$, and $CA(·)$ denote the output of W-MSA, MSA, and CA.

• Edge-assisted Enhancement. In order to achieve local image detail refinement and facilitate global exposure fusion, edge-assisted enhancement (EH) is incorporated in parallel with the transformer module. As shown in Figure 4, input X goes through three $3\times 3$ convolutions with ReLU activations, with skip connections from the first two to the third layer. The output is $Y_{EH}$. Similar to reference [16], adding detailed features after fusion can sharpen the image and improve the image expression in gradient texture. We use the instance normalization (IN) in [39] on half of the channels, with the other half preserving their original context through identity (ID) mapping.

The proposed AEAN net is designed upon the CSTB, which possesses two appealing advantages: $\left(1\right)$ the CSTB adopts a residual network architecture, which can reuse features to mitigate vanishing gradients, and $\left(2\right)$ an efficacious MSCA-EH structure is employed within the CSTB to suppress non-essential features and selectively propagate only high-information features, which results in effectively improving the fused image quality. As illustrated in Figure 4, the CSTB mainly includes W-MSA, MSA, CA, and EH. It combines a CA block and an MSA block in channel-wise and pixel-wise features, respectively, so that CSTB performs differential processing of distinct features and pixels, which can confer enhanced adaptability when dealing with heterogeneous types of information. Moreover, the proposed multi-scale structure is more conducive to preserving fine details and scene depth for the fused image.

3.3. Training Procedure and Objectives

The proposed AEAN undergoes end-to-end training. When there is hardly any radiance range overlap between two input images, directly fusing them will result in an imbalanced brightness of the overall image. Therefore, the essence of the proposed AEAN is to recover the lost brightness details of the two images. Similar to [25,26], that simulate the loss of brightness information by interpolating virtual exposure images, the proposed framework uses a set of real exposure images as ground truth to enable the AEAN to obtain the missing illumination cues of the inputs. Inspired by the algorithms in [36], the loss functions are defined based on the fused image, ensuring that the result closely resembles the set of input images. Our reference fused images are created from multiple real exposure images, whereas the network takes only two EER images as input. The overall training objective is thus defined as

$$L=L_{re}+{\lambda }_1{L}_s+{\lambda }_2{L}_c,$$

(9)

where $L_{re}$ is the rebuilding loss, $L_s$ is the consistency constraint, and $L_c$ is the color loss. Color loss $L_c$ is inherited from [40], since the EER image fusion problem is similar to a dark image brightened in terms of color goals. Additionally, the similarity constraint $L_s$, taken from [30], effectively measures the quality of MEF results and is also employed as an objective function. We empirically set the weights ${\lambda }_1=0.5$ and ${\lambda }_2=0.5$. The description is given as follows.

• Rebuilding loss. The $L_{re}$ loss function is defined between the reconstructed pixels in the fused images $I_f$ and the ground truth images $I_{gt}$ as follows:

  $$L_{re}={\Vert I_f\left(p\right)-I_{gt}\left(p\right)\Vert }_1.$$

  (10)

  Similarity constraint. We use SSIM between the fused images and the reference images to constrain the coherence of the intensity, constant, and structure. The SSIM loss $L_s$ is defined as

  $$L_s={\displaystyle \frac{1}{N}}{\displaystyle \sum _{i=1}^N}(1-{\displaystyle \frac{2{\mu }_{I_f}{\mu }_{I_{gt}}+c_1}{{{\mu }_{I_f}}^2+{{\mu }_{I_{gt}}}^2+c_1}}·{\displaystyle \frac{2{\sigma }_{I_f{I}_{gt}}+c_2}{{{\sigma }_{I_f}}^2+{{\sigma }_{I_{\mathrm{gt}}}}^2+c_2}}),$$

  (11)

  where N denotes the total pixel count of input. ${\mu }_{I_f}$ and ${\mu }_{I_{gt}}$ are the mean intensity of the $I_f$ and $I_{gt}$. Since a higher contrast means the pixel with a higher quality, ${{\sigma }_{I_f}}^2$ and ${{\sigma }_{I_{gt}}}^2$ are the variances of $I_f$ and $I_{gt}$, respectively, and ${\sigma }_{I_f{I}_{gt}}$ is the covariance between $I_f$ and $I_{gt}$. $c_1$ and $c_2$ represent a pair of small positive constants to prevent the possible instability.
• Color loss. In the RGB color space, the cosine angle between the three channels can be used to describe the consistency of color. To avoid potential color distortion and to maintain the inter-channel relationships, we utilize color loss $L_c$ as

  $$L_c=cos({I_f}^h,{I_{gt}}^h),h\in \{R,G,B\},$$

  (12)

  where ${I_f}^h$ and ${I_{gt}}^h$ indicate the pixel value of the h-$th$ color channel in the fused output and the target image, respectively. The color channel h is from the three color channels $\{R,G,B\}$.

4. Experiments and Results

The proposed AEAN is trained on the dataset collected by [15,21]. The dataset provides 800 8-bit sRGB images under diverse exposure configurations. To ensure the images depict static scenes, camera shake and object movement were strictly controlled during acquisition. A random split of the dataset assigns 700 sequences to the training set and the remaining 100 to the test set. Without loss of generality, for each scene, we select the images with the shortest and longest exposure times as inputs, and the fused image generated from all exposure images is used as the ground truth. In order to validate the generalization capability of ten MEF methods, the performance of qualitative and quantitative experiments on two datasets (i.e., [8] and MEFB [41]) are evaluated. The dataset in [8] is a classic dataset for traditional MEF algorithms. The MEFB database [41] encompasses a diverse set of image types, including numerous images commonly used in recent fusion algorithms. During the training phase, the EER inputs are carefully aligned across exposures and then randomly cropped or resized to $512\times 352$ blocks to ensure consistency. The batch size is set to 1. The learning rate is initially set to ${10}^{-5}$ and then decayed following a cosine annealing schedule for training 1000 epochs, using the Adam optimizer with parameters ${\beta }_1=0.9,{\beta }_2=0.99$. We train our model on two NVIDIA 2080 Ti GPUs.

4.1. Evaluation of Different MEF Algorithms

Our AEAN algorithm is first evaluated in comparison with four conventional MEF algorithms in [8,9,22,23] and five data-driven exposure fusion algorithms in [11,12,13,14,32].

• Quantitative Results. To quantify the luminance reversal in the fused result, two indices defined in [26] are calculated for the ten MEF methods. One is $\eta$, which describes the number of pixel pairs whose brightness order changes. Let $\mathsf{\Theta }(I_1,I_2)=\left\{(p_h,p_l)\right|I_1\left(p_h\right)\ge I_1\left(p_l\right),I_f\left(p_h\right)<I_f\left(p_l\right)\}$ and N represent the total number of the fused result, the brightness luminance reversal ratio is given by

  $$\eta ={\displaystyle \frac{\mathsf{\Theta }(I_1,I_2)}{N}}.$$

  (13)

The other is $\epsilon$ to describe the average difference between the pixel values of these pixel pairs after reversal. The mean value of the difference is given by

$$\epsilon ={\displaystyle \frac{{\sum }_{(p_h,p_l)\in \mathsf{\Theta }(I_1,I_2)}|I_f\left(p_h\right)-I_f\left(p_l\right)|}{N}}.$$

(14)

The moreserious the brightness inversion of the fused result, the larger the values of $\eta$ and $\epsilon$. The luminance reversal levels corresponding to ten fusion algorithms are summarized in

Table 1. The $\eta \%$ and $\epsilon$ of ten different algorithms.

Dataset	[8]		[13]		[41]		AVG.
Method	$\mathit{\eta }\%$	$\mathit{\epsilon }$	$\mathit{\eta }\%$	$\mathit{\epsilon }$	$\mathit{\eta }\%$	$\mathit{\epsilon }$	$\mathit{\eta }\%$	$\mathit{\epsilon }$
Mertens [8]	3.586	1.296	0.955	0.334	3.306	1.119	2.616	0.916
Kou [9]	5.128	2.335	0.963	0.321	4.958	2.161	3.683	1.606
Li [23]	3.519	1.157	1.04	0.34	3.251	1.079	2.603	0.859
PESPD [22]	1.409	0.266	0.499	0.037	1.684	0.231	1.197	0.178
Deepfuse [11]	0.374	0.014	0.171	0.004	0.501	0.043	0.349	0.02
MEFLUT [13]	5.076	2.319	0.278	0.054	5.165	2.618	3.506	1.664
MEFGAN [32]	0.38	0.025	0.271	0.007	0.574	0.038	0.408	0.023
MEFNET [12]	5.224	3.411	0.722	0.437	5.637	4.225	3.861	2.691
FFMEF [14]	0.422	0.033	0.236	0.038	0.557	0.014	0.405	0.017
Ours	0.219	0.021	0.127	0.019	0.091	0.008	0.146	0.016

. This demonstrates that the proposed algorithm maintains the true brightness order in the fused image with respect to both indices. Moreover, the methods reported in [11,14,22,32] show promising results as well.

Furthermore, all ten methods are also compared from the perspective of MEF-SSIM and PSNR. The MEF-SSIM and PSNR calculate structural similarity and mean squared error (MSE) between the fused result and multiple exposure-bracketed reference images. According to

Table 2. The MEF-SSIM and PSNR of ten different algorithms.

Dataset	[8]		[13]		[41]		AVG.
Method	MEF-SSIM	PSNR	MEF-SSIM	PSNR	MEF-SSIM	PSNR	MEF-SSIM	PSNR
Mertens [8]	0.8591	38.69	0.9331	41.90	0.8614	38.35	0.8845	39.64
Kou [9]	0.8482	38.78	0.9249	43.41	0.8364	38.21	0.8698	40.13
Li [23]	0.8701	39.72	0.9304	42.00	0.8683	37.66	0.8896	39.79
PESPD [22]	0.8657	41.11	0.9084	46.52	0.8773	39.52	0.8838	42.38
Deepfuse [11]	0.8328	42.02	0.8616	35.93	0.8818	40.94	0.8587	39.63
MEFLUT [13]	0.7029	37.04	0.8313	43.23	0.6323	38.41	0.7222	39.56
MEFGAN [32]	0.8363	42.23	0.8869	41.43	0.8884	40.35	0.8705	41.34
MEFNET [12]	0.7948	41.03	0.9026	42.49	0.7689	38.05	0.8221	40.52
FFMEF [14]	0.8151	39.53	0.9173	44.27	0.8412	39.98	0.8579	41.26
Ours	0.9127	40.61	0.9439	42.39	0.9273	38.85	0.9280	40.62

, the proposed AEAN attained the highest MEF-SSIM score for fusing two EER images. The performance of PSNR is also well, ranking fourth among the ten algorithms, just two points lower than that of [22]. Figure 5 is the box plot depicting the distribution of the overall quality assessment results of the ten algorithms evaluated on the three test datasets. As shown in Figure 5a, the proposed algorithm achieves the highest median score under the MEF-SSIM metric, which indicates that it provides superior overall visual quality compared to the other algorithms. Furthermore, according to the PSNR metric in Figure 5b, the performance of all ten algorithms is relatively consistent, with small differences. Among them, the PESPD [22] has the highest PSNR median score, while the proposed method ranks fourth. These results demonstrate that the fused images obtained by the proposed algorithm effectively preserve structural details while maintaining global luminance consistency.

• Qualitative Results. In this subsection, we compare the ten algorithms from subjective perspectives. Specifically, the comparison is made in terms of brightness retention, information of the brightest and darkest regions, and scene depth. Figure 6 and Figure 7 show the visualization performance of EER image fusion on datasets [8] and MEFB [41], respectively.

As shown in Figure 6 and Figure 7, all ten methods can generate the fused image from two EER images. However, as shown in Figure 6, the algorithms based on traditional weight graphs in [8,9,12,22,23,25] have brightness reversal artifacts, while the algorithms in [9,12,22] have halo artifacts, although the algorithms in [12,22] are much simpler than those in [11,14]. As shown in Figure 6, the improved inverse tangent function in [22,23] preserves the relative brightness order better than the Gaussian curve in [8,9]. Therefore, the brightness reversal artifacts of the fused image in algorithms such as [8,9] are more serious, but it can be seen from the enlarged portion of Figure 7 that the Gaussian curve can better reserve the highlight and shadow details. The single-resolution MEF methods in [11,12,13] fail to preserve scene depth and local contrast as well as the MEF algorithms in [8,9,22,23,25]. Therefore, it can be seen from Figure 7 that although these single-scale fusion algorithms can express details well, all elements from foreground to background are clear and sharp. However, such fused images feel sterile and struggle to guide attention or evoke mood. The learning-based algorithm in [14] has a hierarchical structure but cannot preserve local contrast well, such as the grass and flower in Figure 7. The GAN-based algorithm in [32] is based on supervised learning, which produces severe color distortion. The algorithms in [11,14,32] are not based on weight maps, so they have advantages in avoiding halo artifacts in the fused result, but cannot preserve the information in the highlight and shadow areas well. Comparatively, the proposed fusion method preserves the authentic brightness distribution and depth of the scene, thereby effectively avoiding unnatural brightness reversal and halo artifacts. As a result, the fused image exhibits excellent visual realism, with both local and global details harmoniously preserved.

Figure 6. The EER images and the fused results using the dataset [8]. (a1,a2,a3) are the input low-exposure images, (b1,b2,b3) are the input high-exposure images, (c1,c2,c3) are fused images by Mertens [8], (d1,d2,d3) are by Kou [9], (e1,e2,e3) are by Li [23], (f1,f2,f3) are by PESPD [22], (g1,g2,g3) are by Deepfuse [11], (h1,h2,h3) are by FFMEF [14], (i1,i2,i3) are by MEFGAN [32], (j1,j2,j3) are by MEFNET [12], (k1,k2,k3) are by MEFLUT [13], and (l1,l2,l3) are by our algorithm.

Figure 6. The EER images and the fused results using the dataset [8]. (a1,a2,a3) are the input low-exposure images, (b1,b2,b3) are the input high-exposure images, (c1,c2,c3) are fused images by Mertens [8], (d1,d2,d3) are by Kou [9], (e1,e2,e3) are by Li [23], (f1,f2,f3) are by PESPD [22], (g1,g2,g3) are by Deepfuse [11], (h1,h2,h3) are by FFMEF [14], (i1,i2,i3) are by MEFGAN [32], (j1,j2,j3) are by MEFNET [12], (k1,k2,k3) are by MEFLUT [13], and (l1,l2,l3) are by our algorithm.

Figure 7. The EER images and the fused results using the dataset [41]. (a1,a2) are the low-exposure images and (b1,b2) are the high-exposure images. (c1,c2) are the fused results by Mertens [8], (d1,d2) are by Kou [9], (e1,e2) are by Li [23], (f1,f2) are by PESPD [22], (g1,g2) are by Deepfuse [11], (h1,h2) are MEFLUT [13], (i1,i2) are by MEFGAN [32], (j1,j2) are by MEFNET [12], (k1,k2) are by FFMEF [14], and (l1,l2) are by our algorithm.

Figure 7. The EER images and the fused results using the dataset [41]. (a1,a2) are the low-exposure images and (b1,b2) are the high-exposure images. (c1,c2) are the fused results by Mertens [8], (d1,d2) are by Kou [9], (e1,e2) are by Li [23], (f1,f2) are by PESPD [22], (g1,g2) are by Deepfuse [11], (h1,h2) are MEFLUT [13], (i1,i2) are by MEFGAN [32], (j1,j2) are by MEFNET [12], (k1,k2) are by FFMEF [14], and (l1,l2) are by our algorithm.

To facilitate a more effective comparison of the color fidelity among the ten algorithms, we captured a pair of color chart EER images and generated fused images using each algorithm. The fused images are shown in Figure 8. It can be seen that although the weight-map-based methods in [8,9,23] exhibit brightness inversion between light and dark regions, they generally preserve the true color of the original images, as shown in Figure 8d–f. This is because the fused images produced by these methods derive their color information entirely from the input EER images, thereby avoiding the introduction of new color artifacts. In contrast, all the data-driven methods [11,12,13,14] introduce noticeable color distortions in the fused results. In comparison to other algorithms, our AEAN algorithm effectively maintains the natural color fidelity in the fused images.

4.2. Ablation Study

To analyze the contribution of each part of our AEAN structure and the effectiveness of the loss function, we designed the following ablation studies. The quantitative outcomes of the ablation study for our AEAN method are shown in

Table 3. Ablation study for investigating components of our algorithm.

EH	MSCA	${\mathit{L}}_{\mathit{re}}$	${\mathit{L}}_{\mathit{c}}$	${\mathit{L}}_{\mathit{s}}$	MEF-SSIM	PSNR
		✓	✓	✓	0.4796	8.66
✓		✓	✓	✓	0.7766	36.98
	✓	✓	✓	✓	0.8527	37.26
✓	✓	✓			0.8834	38.13
✓	✓	✓	✓	✓	0.9127	40.61

. As shown in the first line, directly utilizing the Transformer to exposure fusion fails to attain gratifying performance. The next two lines discuss the effectiveness of the MSCA and EH modules. In comparison with the fused image generated from the window-based Transformer, MSCA and EH embedded into the Transformer module can improve the MEF-SSIM. This illustrates the efficacy of our AEAN algorithms for two EER image fusion, because the MSCA and EH modules can effectively preserve local details of the fused image while reducing blocking artifacts. By adding two CNN modules at the same time, the performance can be further improved, as presented in the last row of Table 3. To comprehensively validate the effectiveness of each component, we carried out experiments with the model trained from scratch. The SSIM and PSNR values at different training epochs are presented in Figure 9. When the Swin Transformer is directly employed for fusing EER images, the quality of the fused images shows only limited improvement, primarily due to the persistence of block-like artifacts. By incorporating the MSCA or EH module individually, both SSIM and PSNR showed consistent enhancement, indicating their positive contribution. Notably, when MSCA and EH are all integrated into the W-WSA in a parallel and cascaded manner, the fusion performance is significantly improved. Furthermore, the experiments by training the model from scratch to evaluate the contribution of each loss term are also conducted. It is evident from Table 3 that color and SSIM loss are effective in improving the performance of our AEAN network.

Apart from the objective comparison, Figure 10 provides an intuitive visualization of the contribution of each AEAN module to exposure fusion. It is noticeable that the blocking artifacts appear in the fused result when only the window-based Transformer model is used, as shown in Figure 10c. Moreover, in Figure 10d,e, the fused images only implanting MSCA or EH to the Transformer can reduce blocking artifacts. When the three are added together, the model performs better in local hue and depth of field, as shown in Figure 10f.

5. Conclusions

This paper proposes an attention-edge-assisted neural network (AEAN) for generating HDR images from a pair of registered EER images with symmetric and complementary exposure settings. Quantitative evaluations demonstrate that AEAN provides effective performance, with an average MEF-SSIM gain of 0.0384 across three benchmark datasets and a PSNR of 40.62 dB. Furthermore, AEAN substantially reduces luminance inversion artifacts, as measured by the brightness reversal ratio and mean value, thereby ensuring improved structural fidelity and visual realism compared with existing methods. These results underscore the effectiveness of AEAN in preserving structural consistency and enhancing visual realism in HDRI. Despite these advantages, AEAN presents several practical limitations. The framework requires accurate spatial alignment between input images, as even small misalignments can lead to artifacts such as ghosting and detail loss. Moreover, the model focuses on real-world scenarios, while its robustness on synthetic images requires further validation.

Building on these findings and limitations, our future work will focus on developing alignment-robust architectures capable of handling multi-frame inputs despite spatial misregistration. At the same time, the AEAN framework will be extended to support exposure interpolation and extrapolation, allowing more flexible HDR synthesis under diverse capture conditions. Together, these advancements aim not only to enhance the reliability of HDR generation but also to broaden the applicability of AEAN to unconstrained imaging pipelines.

In summary, AEAN demonstrates potential as a practical framework for HDRI, offering tangible improvements over existing methods. Its impact is particularly relevant for computational photography, augmented and virtual reality, medical imaging, and intelligent surveillance, where faithful luminance recovery and artifact suppression are critical for downstream analysis and human perception.