Multi-Scale FPGA-Based Infrared Image Enhancement by Using RGF and CLAHE
Abstract
:1. Introduction
2. Proposed Method
2.1. Rolling Guidance Filter
2.1.1. Removal of Small Structures
2.1.2. Edge Recovery
2.2. Image Enhancement Strategy
2.2.1. Image Decomposition
2.2.2. Detail Layer Enhancement
2.2.3. Base Layer Enhancement
Algorithm 1: Redistribution process |
Input: the histogram value h(n), the clip limit β Output: the histogram value after redistributing h(n) |
1. excess = 0; 2. for (n = 0; n < N; ++ n) { 3. if (h[n] > β) { 4. excess += h [n] − β;}} 5. m = excess/N; 6. for (n = 0; n < N; ++ n) { 7. if (h[n] < β − m) { 8. h [n] += m; 9. excess −= m;} 10. else if (h[n] < β) { 11. excess += h [n] − β; 12. h[n] = β;}} 13. while (excess > 0) { 14. for (n = 0; n < N; ++ n) { 15. if (excess > 0) { 16. if (h[n] < β) { 17. h [n] += 1; 18. excess −= 1;}}}} excess: the value above the threshold |
2.2.4. Image Reconstruction
3. Algorithm Experiment and Analysis
3.1. Subjective Analysis
3.2. Objective Analysis
4. Hardware Implementation
4.1. Hardware Architecture
4.2. RGF Unit Design
4.3. CLAHE Unit Design
4.3.1. Histogram Calculation
4.3.2. Histogram Clipping and Redistribution
4.3.3. Mapping Function
4.3.4. Interpolation
4.4. FPGA Implementation Results
5. Discussion
6. Conclusions
Author Contributions
Funding
Data Availability Statement
Conflicts of Interest
References
- Zhang, H.; Qian, W.; Wan, M.; Zhang, K. Infrared image enhancement algorithm using local entropy mapping histogram adaptive segmentation. Infrared Phys. Technol. 2022, 120, 104000. [Google Scholar] [CrossRef]
- Min, X.; Zhai, G.; Zhou, J. A Multimodal Saliency Model for Videos with High Audio-Visual Correspondence. IEEE Trans. Image Process. 2020, 29, 3805–3819. [Google Scholar] [CrossRef] [PubMed]
- Min, X.; Zhai, G.; Zhou, J. Study of Subjective and Objective Quality Assessment of Audio-Visual Signals. IEEE Trans. Image Process. 2020, 29, 6054–6068. [Google Scholar] [CrossRef] [PubMed]
- Min, X.; Zhai, G.; Hu, C.; Gu, K. Fixation prediction through multimodal analysis. In Proceedings of the 2015 Visual Communications and Image Processing (VCIP), Singapore, 13–16 December 2015. [Google Scholar]
- Zhang, L.; Yang, X.; Wan, Z.; Cao, D.; Lin, Y. A real-time FPGA implementation of infrared and visible image fusion using guided filter and saliency detection. Sensors 2022, 22, 8487. [Google Scholar] [CrossRef]
- Wu, X.; Hong, D.; Chanussot, J. UIU-Net: U-Net in U-Net for Infrared Small Object Detection. IEEE Trans. Image Process. 2023, 32, 364–376. [Google Scholar] [CrossRef]
- Wang, K.; Du, S.; Liu, C.; Cao, Z. Interior Attention-Aware Network for Infrared Small Target Detection. IEEE Trans. Geosci. Remote Sens. 2022, 60, 3163410. [Google Scholar] [CrossRef]
- Zhao, M.; Li, W.; Li, L. Single-Frame Infrared Small-Target Detection: A survey. IEEE Trans. Geosci. Remote Sens. 2022, 10, 87–119. [Google Scholar] [CrossRef]
- Kandlikar, S.G.; Perez-Raya, I.; Raghupathi, P.A.; Gonzalez-Hernandez, J.L.; Dabydeen, D. Infrared imaging technology for breast cancer detection–Current status, protocols and new directions. Int. J. Heat Mass Transf. 2017, 108, 2303–2320. [Google Scholar] [CrossRef]
- Silva, N.C.M.; Castro, H.A.; Carvalho, L.C.; Chaves, É.C.L.; Ruela, L.O.; Iunes, D.H. Reliability of infrared thermography images in the analysis of the plantar surface temperature in diabetes mellitus. J. Chiropr. Med. 2018, 17, 30–35. [Google Scholar] [CrossRef]
- Wan, M.; Gu, G.; Qian, W.; Ren, K.; Chen, Q. Robust infrared small target detection via non-negativity constraint-based sparse representation. Appl. Opt. 2016, 55, 7604–7612. [Google Scholar] [CrossRef]
- Wan, M.; Ren, K.; Gu, G.; Zhang, X.; Qian, W.; Chen, Q.; Yu, S. Infrared small moving target detection via saliency histogram and geometrical invariability. Appl. Sci. 2017, 7, 569. [Google Scholar] [CrossRef]
- Yuan, L.T.; Swee, S.K.; Ping, T.C. Infrared image enhancement using adaptive trilateral contrast enhancement. Pattern Recognit. Lett. 2015, 54, 103–108. [Google Scholar] [CrossRef]
- Min, X.; Gu, K.; Zhai, G.; Yang, X. Screen Content Quality Assessment: Overview, Benchmark, and Beyond. ACM Comput. Surv. 2022, 54, 1–36. [Google Scholar] [CrossRef]
- Dhal, K.G.; Das, A.; Ray, S. Histogram Equalization Variants as Optimization Problems: A Review. Arch. Comput. Methods Eng. 2021, 28, 1471–1496. [Google Scholar] [CrossRef]
- Abin, D.; Thepade, S.D. Illumination Inconsistency Reduction in Video Frames using DSIHE with Kekre’s LUV Color Space. In Proceedings of the 2021 Third International Conference on Intelligent Communication Technologies and Virtual Mobile Networks (ICICV), Tirunelveli, India, 4–6 February 2021. [Google Scholar]
- Chen, S.D.; Ramli, A.R. Minimum mean brightness error bi-histogram equalization in contrast enhancement. IEEE Trans. Consum. Electron. 2003, 49, 1310–1319. [Google Scholar] [CrossRef]
- Fan, X.; Wang, J.; Wang, H.; Xia, C. Contrast-Controllable Image Enhancement Based on Limited Histogram. Electronics 2022, 11, 3822. [Google Scholar] [CrossRef]
- Chaudhary, S.; Bhardwaj, A.; Rana, P. Image enhancement by linear regression algorithm and sub-histogram equalization. Multimed. Tools Appl. 2022, 81, 29919–29938. [Google Scholar] [CrossRef]
- Acharya, U.K.; Kumar, S. Image sub-division and quadruple clipped adaptive histogram equalization (ISQCAHE) for low exposure image enhancement. Multidimens. Syst. Signal Process. 2023, 34, 25–45. [Google Scholar] [CrossRef]
- Mehdizadeh, M.; Tavakoli, T.K.; Soltani, P. Evaluation of histogram equalization and contrast limited adaptive histogram equalization effect on image quality and fractal dimensions of digital periapical radiographs. Oral Radiol. 2023, 39, 418–424. [Google Scholar] [CrossRef]
- Min, X.; Zhou, J.; Zhai, G.; Callet, P.L. A Metric for Light Field Reconstruction, Compression, and Display Quality Evaluation. IEEE Trans. Image Process. 2020, 29, 3790–3804. [Google Scholar] [CrossRef]
- Min, X.; Zhai, G.; Gu, K.; Yang, X. Objective Quality Evaluation of Dehazed Images. IEEE Trans. Intell. Transp. Syst. 2019, 20, 2879–2892. [Google Scholar] [CrossRef]
- Min, X.; Zhai, G.; Gu, K.; Zhu, Y. Quality Evaluation of Image Dehazing Methods Using Synthetic Hazy Images. IEEE Trans. Multimed. 2019, 21, 2319–2333. [Google Scholar] [CrossRef]
- Branchitta, F.; Diani, M.; Corsini, G.; Romagnoli, M. New technique for the visualization of high dynamic range infrared images. Opt. Eng. 2009, 48, 096401. [Google Scholar] [CrossRef]
- Zuo, C.; Chen, Q.; Liu, N.; Ren, J.; Sui, X. Display and detail enhancement for high-dynamic-range infrared images. Opt. Eng. 2011, 50, 127401. [Google Scholar] [CrossRef]
- Liu, N.; Zhao, D. Detail enhancement for high-dynamic-range infrared images based on guided image filter. Infrared Phys. Technol. 2014, 67, 138–147. [Google Scholar] [CrossRef]
- Xie, J.; Liu, N. Detail enhancement of infrared image based on BEEPS. In Proceedings of the Second Target Recognition and Artificial Intelligence Summit Forum, Shenyang, China, 28–30 August 2019. [Google Scholar]
- Dong, C.; Loy, C.C.; He, K.; Tang, X. Learning a Deep Convolutional Network for Image Super-Resolution. In Proceedings of the Computer Vision—ECCV 2014, Zurich, Switzerland, 6–12 September 2014. [Google Scholar]
- Kim, J.; Lee, J.K.; Lee, K.M. Accurate Image Super-Resolution Using Very Deep Convolutional Networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 27–30 June 2016. [Google Scholar]
- Zhang, Y.; Li, K.; Li, K.; Wang, L.; Zhong, B.; Fu, Y. Image Super-Resolution Using Very Deep Residual Channel Attention Networks. In Proceedings of the Computer Vision—ECCV 2018, Munich, Germany, 8–14 September 2018. [Google Scholar]
- Kuang, X.; Sui, X.; Liu, Y.; Qian, C.; Gu, G. Single infrared image enhancement using a deep convolutional neural network. Neurocomputing 2019, 332, 119–128. [Google Scholar] [CrossRef]
- Wang, D.; Lai, R.; Guan, J. Target attention deep neural network for infrared image enhancement. Infrared Phys. Technol. 2021, 115, 103690. [Google Scholar] [CrossRef]
- Kokufuta, K.; Maruyama, T. Real-time processing of contrast limited adaptive histogram equalization on FPGA. In Proceedings of the 2010 International Conference on Field Programmable Logic and Applications (FPL), Milan, Italy, 31 August–2 September 2010. [Google Scholar]
- Unal, B.; Akoglu, A. Resource efficient real-time processing of contrast limited adaptive histogram equalization. In Proceedings of the 2016 26th International Conference on Field Programmable Logic and Applications (FPL), Lausanne, Switzerland, 29 August–2 September 2016. [Google Scholar]
- Chen, Y.; Kang, J.U.; Zhang, G.; Cao, J.; Xie, Q.; Kwan, C. Real-time infrared image detail enhancement based on fast guided image filter and plateau equalization. Appl. Opt. 2020, 59, 6407–6416. [Google Scholar] [CrossRef]
- Yang, Y.; Xiong, Y.; Cao, Y. Fast bilateral filter with spatial subsampling. Multimed. Syst. 2023, 29, 435–446. [Google Scholar] [CrossRef]
- He, K.; Sun, J.; Tang, X. Guided image filtering. IEEE Trans. Pattern Anal. Mach. Intell. 2012, 35, 1397–1409. [Google Scholar] [CrossRef]
- Singh, P.; Bhandari, A.K.; Kumar, R. Naturalness balance contrast enhancement using adaptive gamma with cumulative histogram and median filtering. Optik 2022, 251, 168251. [Google Scholar] [CrossRef]
- Zhang, Q.; Shen, X.; Xu, L.; Jia, J. Rolling guidance filter. In Proceedings of the Computer Vision—ECCV 2014, Zurich, Switzerland, 6–12 September 2014. [Google Scholar]
- Wu, Q.; Tang, H.; Liu, H.; Chen, Y. Masked Joint Bilateral Filtering via Deep Image Prior for Digital X-Ray Image Denoising. IEEE J. Biomed. Health Inform. 2022, 26, 4008–4019. [Google Scholar] [CrossRef] [PubMed]
- Alexander, T. TNO Image Fusion Dataset; TNO: The Hague, The Netherlands, 2014. [Google Scholar]
- Liu, J.; Fan, X.; Huang, Z.; Wu, G.; Liu, R. Target-aware Dual Adversarial Learning and a Multi-scenario Multi-Modality Benchmark to Fuse Infrared and Visible for Object Detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA, 21–24 June 2022. [Google Scholar]
- Zhai, G.; Min, X. Perceptual image quality assessment: A survey. Sci. China Inf. Sci. 2020, 63, 211301. [Google Scholar] [CrossRef]
- Min, X.; Ma, K.; Gu, K.; Zhai, G. Unified Blind Quality Assessment of Compressed Natural, Graphic, and Screen Content Images. IEEE Trans. Image Process. 2017, 26, 5462–5474. [Google Scholar] [CrossRef]
- Min, X.; Zhai, G.; Gu, K.; Liu, Y. Blind Image Quality Estimation via Distortion Aggravation. IEEE Trans. Broadcast. 2018, 64, 508–517. [Google Scholar] [CrossRef]
- Min, X.; Gu, K.; Zhai, G.; Liu, J. Blind Quality Assessment Based on Pseudo-Reference Image. IEEE Trans. Multimed. 2018, 20, 2049–2062. [Google Scholar] [CrossRef]
- Huang, Y.; Bi, D.; Wu, D. Infrared and visible image fusion based on different constraints in the non-subsampled shearlet transform domain. Sensors 2018, 18, 1169. [Google Scholar] [CrossRef]
- Lv, M.; Li, L.; Jin, Q.; Jia, Z.; Chen, L.; Ma, H. Multi-Focus Image Fusion via Distance-Weighted Regional Energy and Structure Tensor in NSCT Domain. Sensors 2023, 23, 6135. [Google Scholar] [CrossRef]
- Lv, H.; Shan, P.; Shi, H.; Zhao, L. An adaptive bilateral filtering method based on improved convolution kernel used for infrared image enhancement. Signal Image Video Process. 2022, 16, 2231–2237. [Google Scholar] [CrossRef]
HE | CLAHE | GF&DDE | BEEPS&DDE | IE-CGAN | Proposed | |
---|---|---|---|---|---|---|
Scene1 | 3.3064 | 5.75 | 4.7207 | 6.162 | 3.0171 | 9.4747 |
Scene2 | 4.7369 | 7.7667 | 8.6033 | 9.9665 | 2.3355 | 10.6338 |
Scene3 | 5.1166 | 5.515 | 4.7766 | 6.1097 | 4.8409 | 6.807 |
Scene4 | 6.591 | 9.8609 | 9.1665 | 11.1179 | 3.1194 | 11.2412 |
Scene5 | 4.6784 | 8.901 | 7.4432 | 9.2324 | 5.2663 | 9.6896 |
Scene6 | 6.7287 | 8.808 | 8.0154 | 9.7518 | 4.1704 | 10.0586 |
Scene7 | 6.7745 | 7.5473 | 6.5898 | 8.4726 | 4.6242 | 8.1753 |
Scene8 | 6.2241 | 8.6474 | 7.5217 | 9.2147 | 3.8121 | 9.4747 |
Scene9 | 8.0002 | 13.6018 | 12.2886 | 13.8019 | 4.071 | 14.7438 |
Scene10 | 3.9875 | 5.8685 | 4.7549 | 6.7469 | 6.1466 | 6.351 |
HE | CLAHE | GF&DDE | BEEPS&DDE | IE-CGAN | Proposed | |
---|---|---|---|---|---|---|
Scene1 | 34.7727 | 60.3802 | 49.837 | 63.9985 | 32.1226 | 97.3956 |
Scene2 | 48.8454 | 81.3168 | 91.5198 | 104.4826 | 24.3485 | 112.2565 |
Scene3 | 51.934 | 55.893 | 49.5355 | 62.0235 | 52.3518 | 67.612 |
Scene4 | 62.5707 | 92.6684 | 86.2605 | 103.9665 | 32.9632 | 105.8907 |
Scene5 | 45.44 | 86.6194 | 72.4972 | 88.771 | 52.4775 | 94.9481 |
Scene6 | 65.1722 | 85.4135 | 79.2866 | 95.0336 | 43.7175 | 98.7767 |
Scene7 | 66.2712 | 72.4612 | 64.925 | 82.1765 | 47.7164 | 79.061 |
Scene8 | 63.9582 | 88.6129 | 77.6044 | 93.9924 | 39.005 | 97.3956 |
Scene9 | 84.0962 | 143.5655 | 129.0542 | 144.5164 | 42.2968 | 155.592 |
Scene10 | 40.2207 | 59.1701 | 48.294 | 67.3686 | 65.662 | 64.0221 |
HE | CLAHE | GF&DDE | BEEPS&DDE | IE-CGAN | Proposed | |
---|---|---|---|---|---|---|
Scene1 | 3.7445 | 6.4858 | 5.3914 | 7.0231 | 3.5633 | 11.0912 |
Scene2 | 5.8501 | 9.3404 | 10.0945 | 11.9584 | 2.8753 | 12.5943 |
Scene3 | 6.3087 | 6.7871 | 5.6667 | 7.4641 | 5.5845 | 8.7062 |
Scene4 | 9.4174 | 14.0102 | 12.7917 | 15.6909 | 3.6518 | 15.8017 |
Scene5 | 6.7108 | 12.6621 | 10.5362 | 13.1964 | 6.7881 | 13.6354 |
Scene6 | 9.5789 | 12.4054 | 10.907 | 13.5917 | 5.373 | 13.8656 |
Scene7 | 9.6476 | 10.888 | 9.0738 | 11.8983 | 5.939 | 11.6588 |
Scene8 | 7.3154 | 10.1447 | 8.8202 | 10.8575 | 5.0632 | 11.0912 |
Scene9 | 9.1861 | 15.4576 | 14.0613 | 15.8 | 4.927 | 16.7094 |
Scene10 | 4.8271 | 7.0696 | 5.7111 | 7.9857 | 7.0285 | 7.65 |
HE | CLAHE | GF&DDE | BEEPS&DDE | IE-CGAN | Proposed | |
---|---|---|---|---|---|---|
Scene1 | 74.5077 | 58.695 | 78.034 | 82.9624 | 83.5156 | 58.9593 |
Scene2 | 74.9025 | 62.5509 | 81.5486 | 85.7531 | 95.0626 | 75.8319 |
Scene3 | 74.6886 | 46.0226 | 77.7295 | 80.7774 | 82.7147 | 50.6077 |
Scene4 | 74.9016 | 49.5487 | 79.2614 | 83.9679 | 49.5168 | 58.9766 |
Scene5 | 74.8368 | 54.0383 | 78.8685 | 83.7585 | 75.446 | 62.5467 |
Scene6 | 74.6253 | 46.9971 | 79.7722 | 83.2655 | 90.9194 | 57.3257 |
Scene7 | 74.7973 | 46.8248 | 78.2011 | 83.1262 | 63.522 | 53.9331 |
Scene8 | 74.6965 | 51.6546 | 79.214 | 83.8128 | 70.1383 | 58.9593 |
Scene9 | 74.787 | 56.3033 | 81.8486 | 85.9174 | 60.9261 | 65.8268 |
Scene10 | 72.1822 | 68.7966 | 75.1887 | 84.2537 | 71.5311 | 74.834 |
Algorithm | AG | EI | FD | RMSC |
---|---|---|---|---|
HE | 5.6144 | 56.3281 | 7.2587 | 74.4926 |
CLAHE | 8.2267 | 82.6101 | 10.5251 | 54.1432 |
GF&DDE | 7.3881 | 74.8814 | 9.3054 | 78.9667 |
BEEPS&DDE | 9.0576 | 90.6330 | 11.5466 | 83.7595 |
IE-CGAN | 4.1403 | 43.2661 | 5.0793 | 74.3293 |
Proposed | 9.6650 | 97.2950 | 12.2804 | 61.7801 |
Resource | Used | Available | % of All |
---|---|---|---|
BRAM_18K | 408 | 1488 | 27 |
DSP48E | 126 | 3528 | 3 |
FF | 374600 | 682560 | 5 |
LUT | 97685 | 341480 | 28 |
FPGA Maximum Clock Frequency | 114 MHz |
FPGA Maximum Frame Rate | 147 fps |
PC/MATLAB R2021a (i7-12700H @ 2.30 GHz) | 5 fps |
Speedup | 29.4× |
AG | EI | FD | RMSC | |
---|---|---|---|---|
PC | 8.9718 | 92.4214 | 10.7972 | 61.8020 |
FPGA | 8.0301 | 83.4944 | 9.5328 | 54.7012 |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2023 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Liu, J.; Zhou, X.; Wan, Z.; Yang, X.; He, W.; He, R.; Lin, Y. Multi-Scale FPGA-Based Infrared Image Enhancement by Using RGF and CLAHE. Sensors 2023, 23, 8101. https://doi.org/10.3390/s23198101
Liu J, Zhou X, Wan Z, Yang X, He W, He R, Lin Y. Multi-Scale FPGA-Based Infrared Image Enhancement by Using RGF and CLAHE. Sensors. 2023; 23(19):8101. https://doi.org/10.3390/s23198101
Chicago/Turabian StyleLiu, Jialong, Xichuan Zhou, Zhenlong Wan, Xuefei Yang, Wei He, Rulong He, and Yingcheng Lin. 2023. "Multi-Scale FPGA-Based Infrared Image Enhancement by Using RGF and CLAHE" Sensors 23, no. 19: 8101. https://doi.org/10.3390/s23198101
APA StyleLiu, J., Zhou, X., Wan, Z., Yang, X., He, W., He, R., & Lin, Y. (2023). Multi-Scale FPGA-Based Infrared Image Enhancement by Using RGF and CLAHE. Sensors, 23(19), 8101. https://doi.org/10.3390/s23198101