MDIE-Net: Multi-Scale Detailed Information Exploration Network for Low-Light Image Enhancement

Abstract

Low-light image enhancement technology has significant application value in various fields. To improve the image quality under challenging conditions, the recently proposed approaches effectively extract deep feature representations from spatial or frequency domain via well-designed neural networks. Although the overall brightness and the critical structural information of images have been carefully restored, the detailed information is still easy to lose. To address this issue, a multi-scale detailed information exploration network is proposed. First, a Haar wavelet encoding (HWE) block is designed, which consists of the proposed multi-scale feature enhancement (MFE) module and the Haar wavelet transform. In this way, more intrinsic structural information of input low-light images is extracted. Furthermore, a cross-scale valuable information exploration (CVIE) module is constructed, establishing the information interactions between the frequency and spatial domains across different scales. Thus, more illumination and texture details of images are progressively restored. Finally, an edge enhancement decoding (EED) block is devised to further preserve the boundary information of low-light images during the reconstruction process. Qualitative and quantitative experiments are conducted on three widely used public datasets, including both real-world indoor and outdoor scenarios. The experimental results demonstrate that the proposed network outperforms other state-of-the-art image enhancement approaches in both visual effects and evaluation metrics.

1. Introduction

Low-light image enhancement technology has been widely applied in various fields. Although numerous methods have been proposed, severe challenges such as extremely low illumination, complex noise interference, and large dynamic range variations across scenes pose great challenges for this task [1,2,3]. Traditional low-light image enhancement methods primarily improve the visual effects of original images via optimizing the intensity distribution of pixels [4]. These approaches are computationally efficient. However, they are prone to occur overexposure or color distortion phenomena, since they usually only rely on global or local statistical information to improve the quality of images. In recent years, deep learning techniques have been widely used in the low-light enhancement task [5,6]. These data-driven approaches effectively model the complex mapping relationships between low-light and normal illumination environments via well-designed deep networks, thereby improving the overall brightness and visibility of degraded images. Nevertheless, these methods tend to encounter limitations when handling the degradation issues caused by illumination inhomogeneity and serious information attenuation. More recently, to extract more information from images, inspired by signal processing theory, some researchers have incorporated the frequency domain analysis into low-light enhancement frameworks [7,8,9]. By transforming the feature representation of original images into the frequency domain, the detailed information of foreground objects and background noise can be enhanced or suppressed separately on different frequency components, thus producing the superior enhancement results. Although the existing methods improve the image quality via concatenation connections across the space and frequency domains, the detailed information is still hard to extract, especially under extremely low-light environments. To solve the above issue, a multi-scale detailed information exploration network is proposed. The main contributions of this paper are as follows:

(1)

The multi-scale feature enhancement module is proposed and incorporated into both encoding and decoding blocks. In this way, the discriminative feature extraction capability of the network is enhanced.

(2)

The Haar wavelet encoding block is devised to preserve the structural consistency of original image. Additionally, the edge enhancement decoding block is designed to recover the critical boundary information of images progressively.

(3)

A cross-scale valuable information exploration module is designed to establish the information interaction between the frequency and spatial domains across different scales. The overall brightness and the critical structure information is explored across multiple scales, resulting in more natural and clear enhancement results.

2. Related Works

Benefiting from the powerful capabilities of deep learning technology, in recent years, numerous networks have been proposed for the low-light image enhancement task. For instance, Fu et al. [10] proposed an unsupervised low-light image enhancement method. In this approach, the Retinex theory was introduced to impose the network and learn the adaptive constraints from low-light image pairs. Besides, the inappropriate features of images were removed based on the self-supervised mechanism, hence the image details could be effectively revealed in various scenes. Guo et al. [11] proposed a divide-and-rule principle-based low-light image enhancement network. The images were first converted from the RGB color space into the luminance chrominance space. Then, an adjustable noise suppression network was proposed to execute noise removal based on the estimated illumination map, and generate realistic colors along with light adjustment. Yao et al. [12] proposed a gradient-aware and contrastive-adaptive learning framework to address the impact of imprecise gradient information on low-light image enhancement. The low-light image was first fed into the feature separation module to extract multi-scale feature representations. After that, a part of feature representation was fed into the gradient extraction module to calculate a clean gradient map as the structural prior, and then fused with the other part of feature representation via the information fusion module to help the network to learn the sparse gradients. Finally, the preliminarily enhanced image was further refined through a multi-sample contrastive constrained regularization module to effectively suppress the occurrence of artifacts. Sheng et al. [6] introduced the clustering contrastive learning into the low-light image enhancement task, and proposed a gradient-class activation map feature alignment loss to reduce the feature difference between low-light and normal-light images. The above approaches effectively improve the image qualities under low-light environments by accurately modeling the mapping from low light to normal light conditions via elaborate deep neural networks. However, these methods solely recover the image contents at the full resolution, which usually leads to contextual information loss.

To compensate for more critical information from different scales, Fan et al. [13] proposed a multi-scale low-light image enhancement network with an illumination constraint. The squeeze-and-excitation-Res2Net block was introduced as a base unit to extract multi-scale deep feature representation. Then, the illumination constraint was calculated to generate greater weights for dark regions and lower weights for bright areas. In this way, underexposure, overexposure, and uneven exposure issues could be alleviated. Zamir et al. [14] presented an image enhancement architecture to learn an enriched set of features with more contextual information across multiple scales. Besides, the non-local attention mechanism and attention-based multi-scale feature aggregation were introduced, hence the precise high-resolution spatial details were preserved. Qu et al. [15] embedded a coarse enhancement module and a Laplacian of Gaussian-based gradient enhancement module to enhance the color and edge features of original images simultaneously through decomposition and reconstruction processes based on an encoder–decoder structure. Lin et al. [16] proposed a multi-scale spatial-channel transformer encoder to enhance the feature extraction ability of network. Besides, an adaptive feature fusion module was devised. Since the features from both image and event streams were aggregated, the generalization performance has been further enhanced.

Although the above methods restore more detailed information which results in better visual quality, they remain confined to local perception within the spatial domain. In recent years, the frequency transformation has attracted extensive attention due to the structural information extraction and decomposition ability, and has also been applied to the field of low-light image enhancement task. For example, Wang et al. [7] proposed a two-stage Fourier-based method for low-light image enhancement task. In the first stage, the network explored the positive correlation between the magnitude of amplitude and lightness to improve the lightness of low-light images in the Fourier space. In the second stage, the signal-to-noise-ratio map was further introduced to extract more global Fourier frequency information of the image, hence the image details could be effectively recovered. Shang et al. [17] argued that the luminance information in images was primarily encoded in low-frequency components; thus, a frequency-aware network was designed to effectively preserve more high-frequency details of images during the enhancement process via a selective low-frequency adjustment mechanism. Hu et al. [18] proposed a wavelet-based image enhancement network that combined the convolution layers and Transformer blocks. A wavelet calibrate layer was designed to convert the image features into the wavelet domain through multiple calibration filters and distribute them to corresponding areas, thereby restoring the image detailed information. Besides, a contrast adjustment layer was introduced to recognize the noise amplification during the wavelet learning process and improve the image contrast. In the frequency space, the lightness degradation primarily exists in the amplitude component; hence, the enhancement operations can be more precisely applied by separately manipulating the amplitude and phase components. Thus, remarkable image enhancement results have been produced. Although some methods attempt to incorporate both frequency and the spatial information to achieve more generalization and stability performance, most of them simply concatenate the extracted features from both domains in a heuristic manner, failing to dynamically model the intricate and non-linear relationships between spatial and frequency components, especially across different scales.

3. Proposed Method

This section first presents the overall architecture of the proposed framework. Then, the detailed descriptions of each key module and block are elaborated, followed by the loss constraints employed in this work.

3.1. Network Architecture

As shown in Figure 1, to restore more detailed information in enhancement results, the proposed network adopts an encoder–decoder architecture through a multi-scale feature extraction and progressive reconstruction process. With an input low-light image $I_l$, the initial feature representation $f_l$ of image $I_l$ is first extracted via a $3\times 3$ convolution operation. Then, in the encoding stage, the $f_l$ undergoes four cascaded Haar wavelet encoding (HWE) blocks for multi-scale feature extraction. In each HWE, the multi-scale feature enhancement (MFE) module is further designed to improve the feature extraction capability of the network. After that, the enhanced feature representations are decomposed into low-frequency and high-frequency components, respectively, from which more useful information is extracted. Furthermore, to facilitate the feature interaction among multiple scales, a cross-scale valuable information exploration (CVIE) module is devised. This module establishes the connection between the frequency and spatial domains across different scales, which highlights the key feature representations of images while suppressing redundant noise information. Thereby, the high quality enhancement results could be obtained via more accurate feature representation. For the decoding process, an edge enhancement decoding (EED) block is further designed to progressively generate the restored images under different spatial resolutions via a supervised training process. On one hand, the extracted information could effectively enhance the critical contours of the multi-scale restored images. On the other hand, the progressive reconstruction process ensures the illumination consistency across different scales. In the training process, to effectively optimize the parameters of the designed network, three loss constraints are introduced in total. The MSE and contrastive learning-based loss constraints measure the differences among enhancement results, normal-light reference images, and low-light images to guarantee the effectiveness of the network. Additionally, the perceptual loss is further added to ensure the structural similarity between the restored results and reference images.

Figure 1. Overall structure of the proposed network.

3.2. The Haar Wavelet Encoding Block

The detailed structure of each HWE block is shown in the bottom left of Figure 1. For the first HWE, based on the extracted feature representation $f_l$, the multi-scale feature enhancement (MFE) modules are first designed and incorporated to improve the feature extraction ability of the network. The detailed structure of MFE module is shown in Figure 2. For the first MFE module, the input feature representation $f_l$ undergoes a $3\times 3$ convolution operation, and then is split into two sub-features $f_l^{\left(1\right)},f_l^{\left(2\right)}$ along the channel dimension. These two sub-features undergo the $3\times 3$ and $5\times 5$ convolution operations respectively to extract the feature representation from different receptive fields. Then, the concatenated feature representation $f_c$ is generated, followed by a $1\times 1$ convolution and a SiLU activation layer. In addition, the initial feature $f_l$ undergoes a $1\times 1$ convolution operation and a SiLU activation layer for the channel dimension adjustment, and is added with $f_c$. Subsequently, the deeper feature representation $f_x$ is generated, followed by an additional $1\times 1$ convolution operation. The above operations could be formulated as follows:

$$f_l^{\left(1\right)},f_l^{\left(2\right)}=Split\left({\mathrm{Conv}}_{3\times 3}\left(f_l\right)\right)$$

(1)

$$f_c={\sigma }_s({\mathrm{Conv}}_{1\times 1}\left(cat\left({\mathrm{Conv}}_{3\times 3}\left(f_l^{\left(1\right)}\right),{\mathrm{Conv}}_{5\times 5}\left(f_l^{\left(2\right)}\right),{\mathrm{Conv}}_{3\times 3}\left(f_l\right)\right)\right)$$

(2)

$$f_x={\mathrm{Conv}}_{1\times 1}\left(f_c+{\sigma }_s\left({\mathrm{Conv}}_{1\times 1}\left(f_l\right)\right)\right)$$

(3)

where ${\sigma }_s$ denotes the SiLU activation function, and $cat$ denotes the concatenation operation along channel dimension. Benefiting from multiple convolution operations with different kernel sizes, the network can capture more associations in feature maps from different local region.

Figure 2. The structure of multi-scale feature enhancement module.

To explore more useful information from a low-light image, the $f_x$ further goes through a double-path bottleneck structure for feature enhancement. The global max pooling and global average pooling operations are applied respectively, and followed by a $1\times 1$ convolution operation, GeLU activation layer, and an additional $1\times 1$ convolution operation sequentially to calculate the attention map $A_m$ and $A_e$ respectively as follows:

$$\begin{array}{cc}\hfill A_m& ={\mathrm{Conv}}_{1\times 1}\left({\sigma }_g\left({\mathrm{Conv}}_{1\times 1}\left(\mathrm{GMP}\left(f_x\right)\right)\right)\right)\hfill \end{array}$$

(4)

$$\begin{array}{cc}\hfill A_e& ={\mathrm{Conv}}_{1\times 1}\left({\sigma }_g\left({\mathrm{Conv}}_{1\times 1}\left(\mathrm{GAP}\left(f_x\right)\right)\right)\right)\hfill \end{array}$$

(5)

where GMP denotes the global max pooling, GAP denotes the global average pooling, and ${\sigma }_g$ denotes the GeLU activation function.

Then, the $A_m$ and $A_e$ are added and activated by a Sigmoid function $\sigma$; thus, the channel attention map $A_c$ is calculated. The $A_c$ is multiplied by the feature representation of $f_x$ to generate the channel attention enhanced feature $f_{ce}$. The $f_{ce}$ further undergoes two $3\times 3$ convolution operations followed by GeLU and Sigmoid activation layers respectively; thus, the spatial attention map $A_s$ is calculated. After that, the $A_s$ is multiplied by $f_{ce}$, and the spatial attention enhanced feature $f_{cse}$ is generated. Then, the $f_{cse}$ is added with the original input feature representation $f_l$ through a residual connection to produce the final enhanced feature representation $f_{ef}$. The above operations could be formulated as follows:

$$\begin{array}{cc}\hfill f_{ce}& =\sigma (A_m+A_e)\odot f_x\hfill \end{array}$$

(6)

$$\begin{array}{cc}\hfill f_{ef}& =f_l+\sigma \left({\mathrm{Conv}}_{3\times 3}\left({\sigma }_g\left({\mathrm{Conv}}_{3\times 3}\left(f_{ce}\right)\right)\right)\right)\odot f_{ce}\hfill \end{array}$$

(7)

where $\sigma$ denotes the sigmoid activation function, and ⊙ denotes the element-wise multiplication.

For other MFE module, the input is the output of the previous MFE. In this paper, the number of MFE is set to 2, 4, 4, 6 for each HWE block. With the help of proposed MFE module, more important information within the feature representations is further calculated from spatial and channel dimensions. Thus, more robust and informative feature representations are obtained.

Based on the enhanced feature representation, the Haar wavelet transform (HWT) is incorporated. Through low-pass and high-pass filters [19], the feature map is separated into four frequency sub-bands via Haar transforms. The low-frequency component $f_{LL}$ preserves the main structure of the image, the horizontal and vertical high-frequency components $f_{LH}$, $f_{HL}$ emphasize the vertical and horizontal boundary information of image respectively, and the diagonal high-frequency component $f_{HH}$ emphasizes the corner textures of the image. After that, the valid frequency components are adaptively selected via a $1\times 1$ convolution operation to produce the frequency domain enhanced feature $f_{fe}$. Finally, the $f_{ef}$ goes through a $3\times 3$ convolution operation and is concatenated with $f_{fe}$ to generate the feature representation $\widehat{f_E^k}$ for the $k_{th}$ HWE block as follows:

$$\widehat{f_E^k}=cat\left({\mathrm{Conv}}_{3\times 3}\left(f_{ef}\right),{\mathrm{Conv}}_{1\times 1}\left(\left[f_{LL},f_{LH},f_{HL},f_{HH}\right]\right)\right)$$

(8)

where $k=1,2,3,4$.

3.3. The Cross-Scale Valuable Information Exploration Module

With the designed HWE blocks, multi-scale feature representations $\widehat{f_E^1}$, $\widehat{f_E^2}$, $\widehat{f_E^3}$, and $\widehat{f_E^4}$ are extracted. Then, before the decoding process, a cross-scale valuable information exploration (CVIE) module is further designed to establish the association between encoder and decoder. Thus, the propagation of the CVIE is from deeper to shallower layers. The detailed structure of this module is shown in Figure 3. For each CVIE, the inputs are two adjacent features $\widehat{f_E^{k+1}}$ and $\widehat{f_E^k}$ calculated from two adjacent HWE modules. First, to align the spatial dimensions across different scales, the pixel shuffle is applied to the small scale feature representation. Then, the feature representations of adjacent scales are fused via an element-wise addition to produce $f_{ea}^k$ and followed by a layer normalization. Subsequently, the multi-scales fused features $f_{mf}^k$ are transformed from the spatial domain to the frequency domain via the fast Fourier transform (FFT) [20], aiming to capture more structural information from images. In the frequency domain, the features further undergo a $1\times 1$ convolution operation and GeLU activation for information interaction along the cross-channel, and then are transformed to the spatial domain via the inverse fast Fourier transform (IFFT). After that, the obtained feature representation goes through a $1\times 1$ convolution operation for channel dimension adjustment and added with $f_{ea}^k$ through a residual connection to yield frequency-refined feature representation $f_{mffr}^k$. The above operations could be formulated as follows:

$$f_{mf}^k=\left(\mathrm{LN}(\widehat{f_E^k}+PS(\widehat{f_E^{k+1}}))\right)$$

(9)

$$f_{mffr}^k=f_{ea}^k+{\mathrm{Conv}}_{1\times 1}\left({\mathcal{F}}^{-1}\left({\sigma }_g\left({\mathrm{Conv}}_{1\times 1}(\mathcal{F}(f_{mf}^k))\right)\right)\right)$$

(10)

where $PS$ denotes the pixel shuffle operation, $LN$ denotes the layer normalization, $\mathcal{F}$ denotes the fast Fourier transform, and ${\mathcal{F}}^{-1}$ denotes the inverse fast Fourier transform.

Figure 3. The structure of cross-scale valuable information exploration module.

After frequency refinement, the $f_{mffr}^k$ is further adaptively refined in the spatial domain. The $f_{mffr}^k$ undergoes a layer normalization and $1\times 1$ convolution operation, and then followed by a gated operation to adaptively select features. Then, via a $1\times 1$ convolution operation and a residual connection, the spatial domain refined feature representation $f_{mfsr}^k$ is obtained. Finally, the $f_{mffr}^k$ and $f_{mfsr}^k$ are fused. To dynamically balance the contributions of spatial and frequency domains, two learnable weighting factors ${\beta }_f$ and ${\beta }_s$ are introduced, hence the cross-scale detail information refined feature representation $f_{mfcr}^k,k=1,2,3$ is obtained as follows:

$$f_{mfsr}^k=f_{mffr}^k+{\mathrm{Conv}}_{1\times 1}(\mathcal{G}({\mathrm{Conv}}_{1\times 1}(\mathrm{LN}(f_{mffr}^k))))$$

(11)

$$f_{mfcr}^k={\beta }_f{f}_{mffr}^k+{\beta }_s{f}_{mfsr}^k$$

(12)

where $\mathcal{G}$ denotes gated operation, ${\beta }_f$ and ${\beta }_s$ are learned adaptively via a lightweight weight predictor [21]. The whole operation is calculated from deeper to shallower layers as the information is propagated across the full multi-scale hierarchy; thus, the $f_{mfcr}^3$, $f_{mfcr}^2$, and $f_{mfcr}^1$ are obtained sequentially. With the help of CVIE, more valuable detailed information is extracted from different components across different scales.

3.4. The Edge Enhancement Decoding Block

For the decoding process, the edge enhancement decoding (EED) block is further designed to progressively recover the global brightness and local details of input images. For the last EED block, since there are no deeper feature representations for cross-scale valuable information exploration, the $\widehat{f_E^4}$ calculated from HWE is directly passed through the designed MFE modules to calculate the enhanced feature $f_{de}$. For other EED blocks, the feature representations $I_E^{k+1}$ and $f_{mfcr}^k$ generated from the previous decoding stage and CVIE module are added after feature dimension alignment to generate fused feature $f_{fd}$. In this way, more contextual information across different scales is incorporated. Then, similar to the encoding process, the fused feature $f_{fd}$ is passed through the designed MFE module to calculate the enhanced feature $f_{de}$, thereby capturing richer detail information. In this paper, the number of MFE for each EED block is set to be the same as that in HWE blocks. Subsequently, the global average pooling is employed to extract the global information of the image, which is then subtracted from the enhanced feature map. In this way, the low-frequency component can be removed from the feature representation. Then, a $1\times 1$ convolution operation is added for feature redistribution; the contribution of the high-frequency component is thereby further amplified. Finally, with the help of a residual connection, the potential boundary information is enhanced. With the above operations, more contour information is explored; thus, clearer image enhancement results $I_E^k,k=1,2,3,4$ can be obtained via an additional $3\times 3$ convolution operation from deeper to shallower layers progressively. The above operations could be formulated as follows:

$$I_E^k={\mathrm{Conv}}_{3\times 3}\left(f_{de}+{\mathrm{Conv}}_{1\times 1}\left(f_{de}-\mathrm{GAP}\left(f_{de}\right)\right)\right)$$

(13)

3.5. The Loss Constraints

The proposed network aims to achieve image enhancement results based on the more accurate feature extraction across multiple scales, hence three loss constraints are totally introduced.

The MSE loss [22] is applied to measure the pixel-wise differences between enhancement results and reference normal-light images in the spatial domain. This loss constraint could be formulated as follows:

$${\mathcal{L}}_{\mathrm{MSE}}={\displaystyle \sum _{k=1}^K}{w}_S^k{∥{\psi }_{rgb}\left(I_R^k\right)-{\psi }_{rgb}\left(I_E^k\right)∥}_2^2$$

(14)

where $w_S^k$ represents the weight coefficient at the $k_{th}$ scale, and $K=4$ represents the total number of feature scales. ${\parallel ·\parallel }_2^2$ represents the squared L2 norm. ${\psi }_{rgb}\left(I_R^k\right)$ and ${\psi }_{rgb}\left(I_E^k\right)$ represent the RGB feature map of the reference normal-light image and the image enhancement result generated from the proposed network at scale k respectively.

In order to encourage the network to learn the characteristics from normal illumination images while pulling away from low-light samples, the contrastive learning strategy [23] is further introduced in the frequency domain. This loss constraint could be formulated as follows:

$${\mathcal{L}}_{\mathrm{Freq}}={\displaystyle \sum _{k=1}^K}{w}_F^k{\displaystyle \frac{{∥\mathcal{F}\left(I_E^k\right)-\mathcal{F}\left(I_R^k\right)∥}_1}{{∥\mathcal{F}\left(I_E^k\right)-\mathcal{F}\left(I_l^k\right)∥}_1}}$$

(15)

where $w_F^k$ represents the weight coefficient at $k_{th}$ scale, $\mathcal{F}(·)$ denotes the fast Fourier transform operator, and ${\parallel ·\parallel }_1$ denotes the L1 norm.

Additionally, to ensure the structural similarity of the enhancement result in high-level feature space, the perceptual loss [24] is introduced on the scale of $k=1$ as follows:

$${\mathcal{L}}_{\mathrm{Percep}}={∥{\psi }_{VGG}\left(I_E^1\right)-{\psi }_{VGG}\left(I_R^1\right)∥}_1$$

(16)

where the ${\psi }_{VGG}(·)$ represents the feature representation extracted from different levels of VGG networks.

Thus, the total loss function could be formulated as follows:

$${\mathcal{L}}_{\mathrm{Total}}={\mathcal{L}}_{\mathrm{MSE}}+{\lambda }_1{\mathcal{L}}_{\mathrm{Freq}}+{\lambda }_2{\mathcal{L}}_{\mathrm{Percep}}$$

(17)

where ${\lambda }_1$ and ${\lambda }_2$ are the weight coefficients for different loss constraints.

4. Experimental Results and Analysis

4.1. Dataset and Parameter Setting

To evaluate the effectiveness of the proposed network, we conduct quantitative and qualitative comparisons on three widely used datasets LOL [25], LOLv2-real [26], and DICM [27]. These datasets cover a wide range of real-world indoor and outdoor scenes under extremely low illumination conditions. All experiments are conducted on a single NVIDIA RTX 2080 Ti GPU with 11 GB memory. During the training process, the batch size is set to 2, and the input images are randomly cropped into 256 × 256 patches. No more data augmentation strategy has been added. The Adam optimizer is utilized for parameter learning. The initial learning rate is set to 0.0001, and the momentum term is set to ${\beta }_1$ = 0.9, ${\beta }_2$ = 0.999. The number of training epochs is set to 200, and the weight coefficients of the loss function are set to ${\lambda }_1=0.5$, ${\lambda }_2=0.3$. The multi-scale weight coefficients $w_S^k$ and $w_F^k$ are set to $1,0.5,0.25,0.125$ for $k=1,2,3,4$ respectively. During the testing process, the original images are fed into the network directly without any cropping or resizing.

4.2. Quantitative Analysis

For quantitative analysis, the peak signal-to-noise ratio (PSNR), structural similarity index (SSIM), and learned perceptual image patch similarity (LPIPS) are employed to evaluate the performance of various methods on LOL and LOLv2-real datasets. The higher values of PSNR, SSIM, and the lower value of LPIPS indicate better enhancement results. Twelve more recently proposed methods: EnlightenGAN [28], Uformer [29], SCI [30], IAT [31], PairLIE [10], DDNet [15], GACA [12], LightenDiffusion [32], SRENet [33], WENet [18], FT-LLIE [34], and SSDSNet [5] are chosen for comparison. The EnlightenGAN has combined generative adversarial network with U-Net structure. The self-regularized perceptual loss and the attention mechanism have been introduced to further improve the visual quality of enhancement results. Uformer has introduced the locally enhanced window Transformer block into U-Shaped structure to capture both local and global dependencies for image enhancement. SCI has developed the self-calibrated module and unsupervised training loss, and obtained the restoration results according to the Retinex theory. Similarly, PairLIE has achieved favorable performance based on Retinex decomposition and reference-free losses. IAT has extracted both local and global image components via pixel-wise enhancement module and global prediction module through Transformer style network. DDNet has designed an encoder–decoder structure and divided the image enhancement processing into color and gradient enhancement sub-tasks via devised coarse enhancement module and LoG-based gradient enhancement module. GACA has also employed gradient information as a structural prior, and introduced a regularization constraint to guide the image restoration process. LightenDiffusion has designed a content-transfer decomposition network based on Retinex decomposition and diffusion model. SRENet has integrated the saliency extraction module, local enhancement module, and fusion module into encoder–decoder network to obtain more flexibility enhancement results. WENet has proposed a wavelet-based enhancement network which combined the convolution layer with Transformer block. FT-LLIE has designed a Fourier-based image enhancement method. Both spatial and frequency domain information is extracted and the complementary information is obtained via mutual learning to enhance image details. SSDSNet has designed a two-stage framework via perceptual loss and composite loss for detail preservation and brightness enhancement. The above approaches cover the classical and popular network structures which are widely used in more recently years. For a fair comparison, each approach utilizes the same training and testing samples for model training and evaluation. The experimental results are obtained based on the published papers or run via publicly available codes provided by the authors.

As shown in Table 1 and Table 2, compared with the above methods, our proposed network achieves the best performance in terms of all metrics. Compared with U-Net- or U-Shaped-based methods, our proposed network outperforms EnlightenGAN by 6.25, 4.95, and Uformer by 7.37, 4.81 in terms of PSNR. With the help of the devised Haar wavelet encoding block and edge enhancement decoding block, our network preserves more structural and detailed information of the original image during the enhancement process. Hence, it achieves 0.152 and 0.061 gain over SCI and PairLIE in terms of LPIPS. Compared with other encoder–decoder structure-based methods, such as DDNet and SRENet, although additional information has been extracted from different domains, our network still outperforms them by 1.87, 1.78 and 0.015, 0.056 in terms of PSNR and SSIM on LOL dataset. Benefiting from the designed cross-scale valuable information exploration module, our network outperforms GACA by 1.83, 0.011, 0.083 in terms of PSNR, SSIM, and LPIPS on LOL dataset. Compared with WENet, which has combined the Wavelet transform with Transformer block, benefiting from our devised multi-scale feature enhancement modules in Haar wavelet encoding block, our network outperforms it by 1.39, 0.033, 0.111, and 2.09, 0.026, 0.024 in terms of PSNR, SSIM, and LPIPS on two datasets respectively. Compared with FT-LLIE, which has also fused the spatial and frequency domain information, our proposed network explores more valid information across different scales, thus producing the superior performance on two datasets. Compared with latest proposed methods, such as SSDSNet, our network outperforms it by 3.31, 0.047 and 2.7, 0.041 in terms of PSNR and SSIM, respectively. Overall, our network demonstrates superior advantages compared with other state-of-the-art approaches.

Table 1. Quantitative comparison results of different methods on LOL dataset.

Methods	PSNR	SSIM	LPIPS
EnlightenGAN [28]	17.48	0.677	0.322
Uformer [29]	16.36	0.771	0.321
SCI [30]	14.78	0.522	0.339
PairLIE [10]	19.51	0.736	0.248
DDNet [15]	21.86	0.832	0.318
GACA [12]	21.90	0.836	0.270
LightenDiffusion [32]	20.45	0.803	0.192
SRENet [33]	21.95	0.791	0.254
WENet [18]	22.34	0.814	0.298
FT-LLIE [34]	22.46	0.790	-
SSDSNet [5]	20.42	0.800	0.201
Ours	23.73	0.847	0.187

Table 2. Quantitative comparison results of different methods on LOLv2-real dataset.

Methods	PSNR	SSIM	LPIPS
EnlightenGAN [28]	18.68	0.632	0.385
Uformer [29]	18.82	0.771	0.347
SCI [30]	17.30	0.555	0.355
IAT [31]	21.30	0.815	0.325
PairLIE [10]	19.88	0.739	0.351
GACA [12]	19.72	0.812	0.346
LightenDiffusion [32]	19.31	0.712	-
WENet [18]	21.54	0.825	0.305
FT-LLIE [34]	20.59	0.781	-
SSDSNet [5]	20.93	0.810	-
Ours	23.63	0.851	0.281

To better demonstrate the generalization of our proposed network, three no-reference image quality assessment metrics are chosen to evaluate the enhancement results of different methods on DICM dataset. The BRISQUE [35] measures the image distortion in the spatial domain based on the statistical characteristics of normal-light images. NIQE [36] evaluates the distances of feature distributions between normal-light images and enhancement results. Lower values of BRISQUE and NIQE indicate higher image quality. CEIQ [37] evaluates the brightness and clarity of restored images, where higher values indicate better enhancement results. From Table 3 it can be found that our proposed network can achieve favorable image enhancement results on real-world dataset under various evaluation metrics.

Table 3. Quantitative comparison results of different methods on DICM dataset.

Methods	BRISQUE	NIQE	CEIQ
EnlightenGAN [28]	23.11	3.66	3.14
Uformer [29]	24.61	4.12	3.08
SCI [30]	26.43	3.69	2.72
PairLIE [10]	25.32	3.54	3.17
GACA [12]	23.48	3.59	3.26
Ours	22.81	3.48	3.33

Table 4 further presents the computational complexity of several representative approaches. It can be found that the model parameters and floating-point operations are suitable for our proposed network. The number of images processed per second basically meets the real-time requirements. The network makes a relatively good balance between enhancement quality and computational complexity.

Table 4. Comparison of computational complexity.

Methods	EnlightenGAN [28]	Uformer [29]	GACA [12]	Ours
Params	54.41 M	50.88 M	5.74 M	17.31 M
FLOPs	72.59 G	89.46 G	375.05 G	190.28 G
FPS	8.42	9.35	14.73	13.59

4.3. Qualitative Analysis

The qualitative results of different approaches on different datasets under both indoor and outdoor scenarios are shown in Figure 4. It can be found that the enhancement results produced from EnlightenGAN usually suffer from insufficient brightness and color deviation issues. The restored outcomes from Uformer demonstrate superior brightness uniformity. Besides, the colors of the restored results are more natural. However, the structure restoration capability of this method is relatively weak. It usually fails to clearly recover the detailed textures in images. The enhancement results generated by PairLIE sometimes appear with a grayish tone color deviation. Besides, there exists a shortcoming in illumination restoration, resulting in certain differences in terms of brightness compared to the ground truth. The GACA effectively solves the above problem. The enhancement results of this approach are usually under satisfied lighting. However, it sometimes fails to successfully restore the color of the original images. Our proposed network effectively overcomes the aforementioned limitations of other methods. More texture detail information in the images is effectively restored, which ensures that the final output closely aligns with the ground truth.

Figure 4. Enhancement results of different methods on different datasets. The area within the red box is further enlarged for a more detailed comparison.

Figure 5 and Figure 6 present the zoomed-in visual comparisons in indoor scenarios. It can be found that, sometimes the halo phenomenon is appeared in the enhancement result obtained by EnlightenGAN. Furthermore, it suffers from color saturation issue in the gymnasium scenario. The illumination restored by Uformer and PairLIE is overly smooth, and the texture information is loss in the decorative painting and window areas. GACA effectively improves the brightness of raw images; however, there is a certain degree of color cast in the enhancement results. Benefiting from our devised Haar wavelet encoding and edge enhancement decoding blocks, the brightness of the enhancement results obtained by our network is more natural. Furthermore, with the help of cross-scale valuable information exploration module, the contour and detailed texture are effectively restored in both foreground and background areas.

Figure 5. Zoomed-in visual comparison of the area within the red box in an indoor bowling alley scenario.

Figure 6. Zoomed-in visual comparison of the area within the red box in an indoor gymnasium scenario.

Figure 7 and Figure 8 present the zoomed-in visual comparisons in outdoor scenarios. It can be found that, the enhancement results of EnlightenGAN sometimes suffer from color deviation issue. Additionally, some detailed information regarding the car in the parking lot and building beside the statue is lost. The high-frequency information restoration ability of Uformer and PairLIE is relatively poor. The license plate of car and the windows of building are a little bit blurred. The brightness of enhancement results obtained by GACA has a certain color cast problem. In contrast, our proposed network avoids overexposure issue during the enhancement process, and effectively restores clear details and textures for license plate, statue, and building. The above experimental results under real-world scenarios demonstrate that our proposed network can obtain superior enhancement results.

Figure 7. Zoomed-in visual comparison of the area within the red box in an outdoor parking lot scenario.

Figure 8. Zoomed-in visual comparison of the area within the red box in an outdoor statue scenario.

Figure 9 further shows the detection results of both original low-light images and enhanced images to demonstrate the usefulness of restored structural and high-frequency information for downstream vision task. The CrossKD [38] is utilized for object detection. It can be found that, benefiting from image enhancement, the phenomenon of missed detections can be effectively suppressed. Besides, the confidence scores for correct detection results have also been improved. The above experimental results demonstrate the practical application value of enhanced images.

Figure 9. Object detection results for original low-light and enhanced images.

4.4. Ablation Study

To analyze the contribution of each module in our network, an ablation study is conducted on the LOL dataset. As shown in Table 5, the full network achieves optimal performance. Removing Haar wavelet encoding (HWE) blocks causes a decrease of PSNR, SSIM and LPIPS by 1.56, 0.031 and 0.034. It is demonstrated that the HWE could effectively capture more information of images, hence improving the enhancement results. When the edge enhancement decoding (EED) block is removed, the PSNR is reduced to 22.04, which achieves the lowest accuracy among all ablation configurations. The SSIM and LPIPS decline by 0.019 and 0.032, respectively. It indicates that the edge enhancement decoding mechanism is essential for pixel-level reconstruction. Without this module, although the structural similarity of images could be partially preserved, the detail restoration ability of the network is compromised. When the cross-scale valuable information exploration (CVIE) module is removed, the SSIM declines significantly. It confirmed the critical role of CVIE in maintaining structural and illumination consistency. The above experimental results prove the effectiveness of each designed module. For the computational complexity, it can be found that EED has a relatively high computational complexity, whereas the CVIE needs the fewest parameters. In terms of FLOPs, the consumption costs of HWE and CVIE are nearly identical. Combined with Table 4, the results indicate that, compared with other approaches, the overall computational complexity of each proposed module is reasonable. They make good balances between complexity increase and performance improvement.

Table 5. Ablation experimental results on LOL dataset.

HWE	EED	CVIE	PSNR	SSIM	LPIPS	Params (M)	FLOPs (G)
✕	✓	✓	22.17	0.816	0.221	11.64	144.82
✓	✕	✓	22.04	0.828	0.219	10.37	129.99
✓	✓	✕	22.68	0.808	0.193	15.17	140.81
✓	✓	✓	23.73	0.847	0.187	17.31	190.28

Table 6 further shows the detail effectiveness of different components in different modules. The Haar decomposition operation in HWE, the edge enhancement operations in EED, and the frequency and spatial refinement operations in CVIE are removed sequentially to better demonstrate the effectiveness of each designed module. The results indicate that, the component in each module has a positive impact on the overall image enhancement results.

Table 6. The detail effectiveness of different components.

Component	PSNR	SSIM	LPIPS
W/o Haar decomposition in HWE	23.08	0.834	0.205
W/o edge enhancement in EED	22.97	0.821	0.201
W/o frequency refinement in CVIE	23.26	0.837	0.196
W/o spatial refinement in CVIE	23.41	0.841	0.192
Our full network	23.73	0.847	0.187

We further analyze the effectiveness of different loss constraints. The results in Table 7 demonstrate that only introducing the pixel-level MSE loss constraint yields 22.57, 0.809, and 0.243, in terms of PSNR, SSIM and LPIPS. It indicates that the absence of high-level semantic constraints results in limited restored outputs. Introducing the perceptual loss increases PSNR, SSIM and LPIPS by 0.48, 0.007, and 0.002. It confirms the effectiveness in enhancing visual quality through VGG feature constraints. Introducing the contrastive learning-based frequency domain loss constraint increases PSNR, SSIM and LPIPS by 0.21, 0.017, and 0.02. This loss constraint encourages the frequency components of the enhancement results to be closer to those of reference images while remaining far from the original low-light inputs. The critical spectral components in the frequency domain can effectively preserve the high-frequency detail information and structural consistency in the restored images, thus producing favorable results. The further incorporation all loss constraints produces significant improvements. It proves that each loss constraint has a positive effect on improving the quality of restored images.

Table 7. The effectiveness of different loss constraints.

${\mathit{L}}_{\mathit{MSE}}$	${\mathit{L}}_{\mathit{Percep}}$	${\mathit{L}}_{\mathit{Freq}}$	PSNR	SSIM	LPIPS
✓	✕	✕	22.57	0.809	0.243
✓	✓	✕	23.05	0.816	0.241
✓	✕	✓	22.78	0.826	0.223
✓	✓	✓	23.73	0.847	0.187

5. Conclusions

In this paper, a multi-scale detailed information exploration network has been proposed for low-light image enhancement. The Haar wavelet encoding blocks are devised to extract more detailed and structural information of input images via multi-scale feature enhancement module and frequency decomposition, while the edge enhancement decoding blocks are designed to effectively reconstruct the critical boundary information of images progressively. Moreover, a cross-scale valuable information exploration module is constructed to further leverage more information from both spatial and frequency domains across different scales. Quantitative and qualitative evaluations on widely used datasets demonstrate that the proposed network outperforms other state-of-the-art low-light enhancement methods in both illumination uniformity and detailed texture preservation. Future work will explore more lightweight network architectures to further improve the image enhancement results.