# Fractional-Order Fusion Model for Low-Light Image Enhancement

^{*}

## Abstract

**:**

## 1. Introduction

**Contribution:**The main contributions of the proposed model are:

- As compared to integer-order, we apply fractional calculus to process the original images without logarithmic transformation. Remarkable results have been achieved in preserving the natural character of images.
- A novel fusion framework is introduced to extract more contents in the dark areas while preserving the visual appearance of images.
- The experimental results compared with other image enhancement algorithms show that the proposed model can reveal more hidden contents in dark regions of the images.

## 2. Background

#### 2.1. Fractional Calculus

#### 2.2. Retinex Theory

## 3. Fractional-Order Fusion Model Based On Retinex

#### 3.1. Reflectance and Illumination Based On Fractional-Order

#### 3.1.1. Reflectance

#### 3.1.2. Illumination

- For color images, three color channels (R, G, B) share the same illumination map [27].

#### 3.1.3. The Energy Function

#### 3.1.4. Adjust Illumination

#### 3.2. Fusion Framework

## 4. Implementation of FFM

#### 4.1. Optimization of the Energy Function

#### 4.1.1. R Sub-Problem

#### 4.1.2. L Sub-Problem

#### 4.2. Implementation of the Fusion Process

## 5. Experiments and Analysis

#### 5.1. Fractional Order Impact

#### 5.2. Comparison with Other Algorithms

#### 5.2.1. Visual Contrast

#### 5.2.2. Lightness Order Error

#### 5.2.3. Images Quality Assessment

#### 5.2.4. Images Similarity Assessment

## 6. Conclusions

## Author Contributions

## Funding

## Acknowledgments

## Conflicts of Interest

## Appendix A

## References

- Berclaz, J.; Fleuret, F.; Turetken, E.; Fua, P. Multiple object tracking using k-shortest paths optimization. IEEE Trans. Pattern Anal. Mach. Intell.
**2011**, 33, 1806–1819. [Google Scholar] [CrossRef] [PubMed] - Redmon, J.; Divvala, S.; Girshick, R.; Farhadi, A. You only look once: Unified, real-time object detection. Comput. Vis. Pattern Recognit.
**2016**, 779–788. [Google Scholar] - Wang, R.; Wang, G. Medical X-ray image enhancement method based on tv-homomorphic filter. In Proceedings of the International Conference on Image, Vision and Computing, Chengdu, China, 2–4 June 2017; pp. 315–318. [Google Scholar]
- Cheng, H.D.; Shi, X.J. A simple and effective histogram equalization approach to image enhancement. Digit. Signal Process.
**2004**, 14, 158–170. [Google Scholar] [CrossRef] - Reza, A.M. Realization of the contrast limited adaptive histogram equalization for real-time image enhancement. J. VLSI Signal Process. Syst. Signal Image Video Technol.
**2004**, 38, 35–44. [Google Scholar] [CrossRef] - Dong, X.; Wang, G.; Pang, Y.; Li, W.; Wen, J.; Meng, W.; Lu, Y. Fast efficient algorithm for enhancement of low lighting video. In Proceedings of the IEEE International Conference on Multimedia and Expo, Barcelona, Spain, 11–15 July 2011; pp. 1–6. [Google Scholar]
- Li, L.; Wang, R.; Wang, W.; Gao, W. A low-light image enhancement method for both denoising and contrast enlarging. In Proceedings of the IEEE International Conference on Image Processing, Quebec City, QC, Canada, 27–30 September 2015; pp. 3730–3734. [Google Scholar]
- Land, E.H.; Mccann, J.J. Lightness and retinex theory. J. Opt. Soc. Am.
**1971**, 61, 1–11. [Google Scholar] [CrossRef] [PubMed] - Cai, B.; Xu, X.; Guo, K.; Jia, K.; Hu, B.; Tao, D. A joint intrinsic-extrinsic prior model for retinex. In Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy, 22–29 October 2017; pp. 4020–4029. [Google Scholar]
- Kimmel, R.; Elad, M.; Shaked, D.; Keshet, R.; Sobel, I. A variational framework for retinex. Int. J. Comput. Vis.
**2003**, 52, 7–23. [Google Scholar] [CrossRef] - Ying, Z.; Li, G.; Ren, Y.; Wang, R.; Wang, W. A new low-light image enhancement algorithm using camera response model. In Proceedings of the IEEE International Conference on Computer Vision Workshop, Venice, Italy, 22–29 October 2017; pp. 3015–3022. [Google Scholar]
- Wang, S.; Zhang, J.; Hu, H.-M.; Li, B. Naturalness preserved enhancement algorithm for non-uniform illumination images. IEEE Trans. Image Process.
**2013**, 22, 3538–3548. [Google Scholar] [CrossRef] [PubMed] - Fu, X.; Zeng, D.; Yue, H.; Liao, Y.; Ding, X.; Paisley, J. A fusion-based enhancing method for weakly illuminated images. Signal Process.
**2016**, 129, 82–96. [Google Scholar] [CrossRef] - Lore, K.G.; Akintayo, A.; Sarkar, S. Llnet: A deep autoencoder approach to natural low-light image enhancement. Pattern Recognit.
**2017**, 61, 650–662. [Google Scholar] [CrossRef] - Gharbi, M.; Chen, J.; Barron, J.T.; Hasinoff, S.W.; Durand, F. Deep bilateral learning for real-time image enhancement. ACM Trans. Graph.
**2017**, 36, 118. [Google Scholar] [CrossRef] - Cai, J.; Gu, S.; Lei, Z. Learning a deep single image contrast enhancer from multi-exposure images. IEEE Trans. Image Process.
**2018**, 27, 2049–2062. [Google Scholar] [CrossRef] [PubMed] - Chongyi, L.; Guo, J.; Porikli, F.; Pang, Y. Lightennet: A convolutional neural network for weakly illuminated image enhancement. Pattern Recognit. Lett.
**2018**, 104, 15–22. [Google Scholar] - Pu, Y.-F.; Zhang, N.; Zhang, Y.; Zhou, J.L. A texture image denoising approach based on fractional developmental mathematics. Pattern Anal. Appl.
**2016**, 19, 427–445. [Google Scholar] [CrossRef] - Pu, Y.-F. Fractional calculus approach to texture of digital image. In Proceedings of the International Conference on Signal Processing, Beijing, China, 16–20 November 2006; pp. 1002–1006. [Google Scholar]
- Aygören, A. Fractional Derivative and Integral; Gordon and Breach Science Publishers: Philadelphia, PA, USA, 1993. [Google Scholar]
- Koeller, R.C. Applications of fractional calculus to the theory of Viscoelasticity. Trans. ASME J. Appl. Mech.
**1984**, 51, 299–307. [Google Scholar] [CrossRef] - Rossikhin, Y.A.; Shitikova, M.V. Applications of Fractional Calculus to Dynamic Problems of Linear and Nonlinear Hereditary Mechanics of Solids; McGraw-Hill: New York, NY, USA, 1997. [Google Scholar]
- Oldham, K.; Spanier, J. The Fractional Calculus Theory and Applications of Differentiation and Integration to Arbitrary Order; Elsevier: Amsterdam, The Netherlands, 1974. [Google Scholar]
- Lee, S. An efficient content-based image enhancement in the compressed domain using retinex theory. IEEE Trans. Circuits Syst. Video Technol.
**2007**, 17, 199–213. [Google Scholar] [CrossRef] - Pu, Y.-F.; Siarry, P.; Chatterjee, A.; Wang, Z.N.; Zhang, Y.I.; Liu, Y.G.; Zhou, J.L.; Wang, Y. A fractional-order variational framework for retinex: Fractional-order partial differential equation based formulation for multi-scale nonlocal contrast enhancement with texture preserving. IEEE Trans. Image Process.
**2018**, 27, 1214–1229. [Google Scholar] [CrossRef] [PubMed] - Fu, X.; Zeng, D.; Huang, Y.; Zhang, X.P.; Ding, X. A weighted variational model for simultaneous reflectance and illumination estimation. In Proceedings of the Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 27–30 June 2016; pp. 2782–2790. [Google Scholar]
- Guo, X.; Li, Y.; Ling, H. Lime: Low-light image enhancement via illumination map estimation. IEEE Trans. Image Process.
**2017**, 26, 982–993. [Google Scholar] [CrossRef] [PubMed] - Dabov, K.; Foi, A.; Katkovnik, V.; Egiazarian, K. Image denoising by sparse 3-D transform-domain collaborative filtering. IEEE Trans. Image Process.
**2007**, 16, 2080–2095. [Google Scholar] [CrossRef] [PubMed] - Tseng, P. Convergence of a block coordinate descent method for nondifferentiable minimization. J. Optim. Theory Appl.
**2001**, 109, 475–494. [Google Scholar] [CrossRef] - Candès, E.J.; Wakin, M.B.; Boyd, S.P. Enhancing sparsity by reweighted lsb1 minimization. J. Fourier Anal. Appl.
**2007**, 14, 877–905. [Google Scholar] [CrossRef] - Xu, L.; Yan, Q.; Xia, Y.; Jia, J. Structure extraction from texture via relative total variation. ACM Trans. Graph.
**2012**, 31, 139. [Google Scholar] [CrossRef] - Farbman, Z.; Fattal, R.; Lischinski, D.; Szeliski, R. Edge-preserving decompositions for multi-scale tone and detail manipulation. ACM Trans. Graph.
**2008**, 27, 1–10. [Google Scholar] [CrossRef] - Burt, P.J.; Adelson, E.H. The laplacian pyramid as a compact image Code. Read. Comput. Vis.
**1987**, 31, 671–679. [Google Scholar] - Chakrabarti, A.; Scharstein, D.; Zickler, T.E. An empirical camera model for internet color vision. BMVC
**2009**, 1, 4. [Google Scholar] - Gu, K.; Zhai, G.; Lin, W.; Yang, X.; Zhang, W. No-reference image sharpness assessment in autoregressive parameter space. IEEE Trans. Image Process. Publ. IEEE Signal Process. Soc.
**2015**, 24, 3218–3231. [Google Scholar] - Gu, K.; Li, L.; Lu, H.; Min, X.; Lin, W. A fast reliable image quality predictor by fusing micro- and macro-structures. IEEE Trans. Ind. Electron.
**2017**, 64, 3903–3912. [Google Scholar] [CrossRef] - Zhang, L.; Zhang, L.; Mou, X.; Zhang, D. Fsim: A feature similarity index for image quality assessment. IEEE Trans. Image Process. Publ. IEEE Signal Process. Soc.
**2011**, 20, 2378. [Google Scholar] [CrossRef] - Pu, Y.-F.; Zhou, J.L.; Yuan, X. Fractional differential mask: A fractional differential-based approach for multiscale texture enhancement. IEEE Trans. Image Process.
**2010**, 19, 491–511. [Google Scholar]

**Figure 2.**Comparison of different illumination adjustment strategy. (

**a**) the under-exposure image; (

**b**) the Gamma transformation according to Equation (8); (

**c**) the original camera response function (CRF) according to Equation (9); (

**d**) the modified result of CRF; (

**e**) the well-exposure image. (

**f**), (

**g**), (

**h**), (

**i**) and (

**j**) are the three color channels (R, G, B) histograms of (

**a**), (

**b**), (

**c**), (

**d**) and (

**e**), respectively.

**Figure 3.**Multi-scale fusion process weights setting. (

**a**) the original image; (

**b**) the first enhanced image; (

**c**) the second enhanced image; (

**d**) the result of fusion; (

**e**) the weight function.

**Figure 4.**One-dimensional signal processing. (

**a**) the original signal; (

**b**) the signal with Gaussian noise added; (

**c**) the result from Equation (25), v is 2.4 and ${\lambda}_{3}$ is $1.125$; (

**d**–

**f**) the results are from Equation (23), ${\lambda}_{1}$ is 0.6, 0.8, 1 respectively; (

**g**–

**i**) the outputs are from Equation (24), ${\lambda}_{2}$ is 1.6, 2.4, 3.2, respectively.

**Figure 5.**The impact of fractional-order. (

**a**) the original image; (

**b**) the reflectance from first-order differential is more prominent than fractional-order differential; (

**c**) the reflectance measured with fractional-order differential is more prominent than first-order differential.

**Figure 7.**Comparison of enhancement schemes for images which are slightly degraded from low-light conditions. (

**a**,

**i**,

**q**) are original images; (

**b**,

**j**,

**r**) MF [13]; (

**c**,

**k**,

**s**) LightenNet [17]; (

**d**,

**l**,

**t**) CRM [11]; (

**e**,

**m**,

**u**) NPE [12]; (

**f**,

**n**,

**v**) JIEP [9]; (

**g**,

**o**,

**w**) Our enhanced images of FFM(1); (

**h**,

**p**,

**x**) are final results.

**Figure 8.**An example of a failure case. (

**a**) the original image; (

**b**) the effect of FFM(1); (

**c**) the result of FFM(2).

Dataset | NPE [12] | CRM [11] | JIEP [9] | MF [13] | LightenNet [17] | FFM(1) | FFM(2) |
---|---|---|---|---|---|---|---|

Middlebury | 359 | 240 | 260 | 207 | 919 | 124 | 224 |

MF-data | 316 | 402 | 241 | 289 | 807 | 135 | 215 |

NPEpart1 | 220 | 575 | 337 | 342 | 720 | 182 | 216 |

NPEpart2 | 210 | 504 | 264 | 272 | 747 | 167 | 205 |

NPEpart3 | 259 | 568 | 354 | 311 | 785 | 211 | 246 |

**Table 2.**Results of image sharpness quality assessment using ARISM [35].

Dataset | Assessment | NPE [12] | CRM [11] | JIEP [9] | MF [13] | LightenNet [17] | FFM(1) | FFM(2) |
---|---|---|---|---|---|---|---|---|

Middlebury | ARISM1 | 3.1909 | 2.9654 | 2.8468 | 2.952 | 3.0898 | 2.821 | 2.7284 |

ARISM2 | 3.4643 | 3.2278 | 3.1103 | 3.2258 | 3.3712 | 3.0866 | 3.0351 | |

MF-data | ARISM1 | 2.7961 | 2.7584 | 2.6982 | 2.7286 | 2.7624 | 2.6858 | 2.6421 |

ARISM2 | 3.0527 | 3.0445 | 2.9511 | 2.987 | 3.0127 | 2.9418 | 2.9083 | |

NPEpart1 | ARISM1 | 2.7833 | 2.7521 | 2.758 | 2.7356 | 2.7396 | 2.7404 | 2.7064 |

ARISM2 | 3.0252 | 3.0219 | 3.0001 | 2.979 | 2.9819 | 2.9801 | 2.9519 | |

NPEpart2 | ARISM1 | 2.7278 | 2.6827 | 2.6877 | 2.6723 | 2.6991 | 2.6688 | 2.6403 |

ARISM2 | 2.991 | 2.9676 | 2.9458 | 2.9373 | 2.9573 | 2.9306 | 2.9118 | |

NPEpart3 | ARISM1 | 2.9495 | 2.8561 | 2.8417 | 2.8468 | 2.9352 | 2.8031 | 2.7727 |

ARISM2 | 3.1976 | 3.1346 | 3.0886 | 3.0976 | 3.1785 | 3.0516 | 3.0299 |

Dataset | Assessment | NPE [12] | CRM [11] | JIEP [9] | MF [13] | LightenNet [17] | FFM(1) | FFM(2) |
---|---|---|---|---|---|---|---|---|

Middlebury | FSIM1 | 0.7696 | 0.8117 | 0.8659 | 0.8019 | 0.7427 | 0.9085 | 0.8906 |

FSIM2 | 0.7589 | 0.8054 | 0.8605 | 0.7924 | 0.7325 | 0.9034 | 0.8832 | |

PSIM | 0.9954 | 0.9967 | 0.9976 | 0.9960 | 0.9952 | 0.9982 | 0.9976 | |

MF-data | FSIM1 | 0.8072 | 0.8127 | 0.8648 | 0.8084 | 0.8179 | 0.9123 | 0.8957 |

FSIM2 | 0.8012 | 0.8078 | 0.8609 | 0.8024 | 0.8122 | 0.9086 | 0.8905 | |

PSIM | 0.9964 | 0.9966 | 0.9977 | 0.9966 | 0.9967 | 0.9984 | 0.9979 | |

NPEpart1 | FSIM1 | 0.8911 | 0.8834 | 0.9125 | 0.8869 | 0.8915 | 0.9492 | 0.9457 |

FSIM2 | 0.8883 | 0.8806 | 0.9105 | 0.8839 | 0.8893 | 0.9474 | 0.9433 | |

PSIM | 0.9980 | 0.9978 | 0.9985 | 0.9979 | 0.9982 | 0.9989 | 0.9987 | |

NPEpart2 | FSIM1 | 0.8813 | 0.8553 | 0.9096 | 0.8808 | 0.8708 | 0.9421 | 0.9376 |

FSIM2 | 0.8779 | 0.8520 | 0.9073 | 0.8774 | 0.8675 | 0.9398 | 0.9344 | |

PSIM | 0.9977 | 0.9973 | 0.9983 | 0.9977 | 0.9976 | 0.9988 | 0.9986 | |

NPEpart3 | FSIM1 | 0.8955 | 0.8752 | 0.9225 | 0.8865 | 0.9038 | 0.9522 | 0.9480 |

FSIM2 | 0.8915 | 0.8710 | 0.9202 | 0.8821 | 0.9006 | 0.9499 | 0.9449 | |

PSIM | 0.9976 | 0.9974 | 0.9985 | 0.9976 | 0.9980 | 0.9989 | 0.9987 |

**Table 4.**Running time comparison of the various methods. These values represent the average time, in seconds.

© 2019 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Dai, Q.; Pu, Y.-F.; Rahman, Z.; Aamir, M.
Fractional-Order Fusion Model for Low-Light Image Enhancement. *Symmetry* **2019**, *11*, 574.
https://doi.org/10.3390/sym11040574

**AMA Style**

Dai Q, Pu Y-F, Rahman Z, Aamir M.
Fractional-Order Fusion Model for Low-Light Image Enhancement. *Symmetry*. 2019; 11(4):574.
https://doi.org/10.3390/sym11040574

**Chicago/Turabian Style**

Dai, Qiang, Yi-Fei Pu, Ziaur Rahman, and Muhammad Aamir.
2019. "Fractional-Order Fusion Model for Low-Light Image Enhancement" *Symmetry* 11, no. 4: 574.
https://doi.org/10.3390/sym11040574