A Dynamic Fusion of Local and Non-Local Features-Based Feedback Network on Super-Resolution
Abstract
:1. Introduction
- We introduce the Convolution-based block and Attention-based block as the base block of our Feedback-based SR; the Convolution-based block focuses on extracting local feature, and the Attention-based block focus on extracting non-local feature. We also propose the Delivery–Adjust–Fusion framework to hold these blocks, making them work on what they do best.
- We proposed a Dynamic Weighting block (DW block) to generate the right weight values for different inputs under different iterations, and fuse both branches’ feature maps together.
- We introduce the MAConv layer as the input block, which is critical for our two branch-based feedback algorithms. By cascading 4 MAConv layers as the input layer, we can obtain a deeper input layer while easy to train.
2. Related Works
3. The Dynamic Fusion of Local and Non-Local Features-Based Feedback Network (DLNFN) for SR
3.1. The Network Overall Architecture of Our DLNFN
3.2. The Dynamic Fusion of Local and Non-local Features-Based Feedback Block (DLN Block)
3.3. Other Implementation Details
4. Experimental Results
4.1. Datasets, Reuse Strategy, and Evaluation Metrics
4.2. Ablation Study
4.3. Comparisons With State-of-the-Arts
4.4. Visualized Analysis
5. Conclusions
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Conflicts of Interest
Appendix A
Acronyms and Notations | Description |
---|---|
SISR | Single Image Super-Resolution |
SR | Super-Resolution |
HR | High-Resolution |
LR | Low-Resolution |
Super-Resolution image | |
Low-Resolution image | |
High-Resolution image | |
the loss function | |
the In block | |
the output features of the In block | |
the DLN block | |
the output features of the DLN block | |
the Reconstruction block | |
our proposed DLNFN algorithm | |
the i-th iteration output of the DLN block with dimension as LR | |
the i-th iteration output of the DLN block with dimension as SR | |
the output of FB branch with dimension as LR in the i-th iteration | |
the output of CSNL branch with dimension as LR in the i-th iteration | |
the output of FB branch with dimension as SR in the i-th iteration | |
the output of CSNL branch with dimension as SR in the i-th iteration | |
the weight value of FB branch in the i-th iteration | |
the weight value of CSNL branch in the i-th iteration | |
concatenate all the inputs on feature dimension |
References
- Wang, Z.; Chen, J.; Hoi, S.C. Deep Learning for Image Super-resolution: A Survey. IEEE Trans. Pattern Anal. Mach. Intell. 2020, 43, 3365–3387. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Mao, X.; Shen, C.; Yang, Y. Image Restoration Using Very Deep Convolutional Encoder-Decoder Networks with Symmetric Skip Connections. In Proceedings of the Advances in Neural Information Processing Systems (NIPS), Barcelona, Spain, 5–10 December 2016; Curran Associates, Inc.: New York, NY, USA; pp. 2802–2810. [Google Scholar]
- Li, Z.; Yang, J.; Liu, Z.; Yang, X.; Jeon, G.; Wu, W. Feedback Network for Image Super-Resolution. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA, 15–20 June 2019; IEEE: Piscataway, NJ, USA; pp. 3867–3876. [Google Scholar] [CrossRef] [Green Version]
- Mei, Y.; Fan, Y.; Zhou, Y.; Huang, L.; Huang, T.S.; Shi, H. Image Super-Resolution with Cross-Scale Non-Local Attention and Exhaustive Self-Exemplars Mining. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA, 13–19 June 2020; pp. 5690–5699. [Google Scholar] [CrossRef]
- Chen, H.; Gu, J.; Zhang, Z. Attention in attention network for image super-resolution. arXiv 2021, arXiv:2104.09497. [Google Scholar]
- Liang, J.; Sun, G.; Zhang, K.; Van Gool, L.; Timofte, R. Mutual affine network for spatially variant kernel estimation in blind image super-resolution. In Proceedings of the IEEE/CVF International Conference on Computer Vision (CVPR), Nashville, TN, USA, 21–24 June 2021; pp. 4096–4105. [Google Scholar]
- Dong, C.; Loy, C.C.; He, K.; Tang, X. Learning a Deep Convolutional Network for Image Super-Resolution. In Proceedings of the Computer Vision—ECCV 2014, Zurich, Switzerland, 6–12 September 2014; Springer International Publishing: Cham, Switzerland; pp. 184–199. [Google Scholar] [CrossRef]
- Kim, J.; Lee, J.K.; Lee, K.M. Accurate Image Super-Resolution Using Very Deep Convolutional Networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 27–30 June 2016; pp. 1646–1654. [Google Scholar] [CrossRef] [Green Version]
- Kim, J.; Lee, J.K.; Lee, K.M. Deeply-Recursive Convolutional Network for Image Super-Resolution. In Proceedings of the IEEE Comput Soc Conf Comput Vision Pattern Recognit (CVPR), Las Vegas, NV, USA, 27–30 June 2016; IEEE: Piscataway, NJ, USA; pp. 1637–1645. [Google Scholar] [CrossRef] [Green Version]
- Song, D.; Wang, Y.; Chen, H.; Xu, C.; Xu, C.; Tao, D. Addersr: Towards energy efficient image super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA, 20–25 June 2021; pp. 15648–15657. [Google Scholar]
- Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A.N.; Kaiser, Ł.; Polosukhin, I. Attention Is All You Need. In Proceedings of the Advances in Neural Information Processing Systems, Long Beach, CA, USA, 4–9 December 2017; pp. 5998–6008. [Google Scholar]
- Li, B.; Xiong, S.; Xu, H. Channel Pruning Base on Joint Reconstruction Error for Neural Network. Symmetry 2022, 14, 1372. [Google Scholar] [CrossRef]
- Woo, S.; Park, J.; Lee, J.Y.; Kweon, I.S. CBAM: Convolutional Block Attention Module. In Proceedings of the European conference on computer vision (ECCV), Munich, Germany, 8–14 September 2018; pp. 3–19. [Google Scholar] [CrossRef] [Green Version]
- Li, B.; Lu, Y.; Pang, W.; Xu, H. Image Colorization using CycleGAN with semantic and spatial rationality. Multimed. Tools Appl. 2023, 82, 1–15. [Google Scholar] [CrossRef]
- Zhao, H.; Kong, X.; He, J.; Qiao, Y.; Dong, C. Efficient image super-resolution using pixel attention. In Proceedings of the Computer Vision–ECCV 2020 Workshops, Glasgow, UK, 23–28 August 2020; Springer: Berlin/Heidelberg, Germany, 2020; pp. 56–72. [Google Scholar]
- Niu, B.; Wen, W.; Ren, W.; Zhang, X.; Yang, L.; Wang, S.; Zhang, K.; Cao, X.; Shen, H. Single image super-resolution via a holistic attention network. In Proceedings of the Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, 23–28 August 2020; Springer: Berlin/Heidelberg, Germany, 2020; pp. 191–207. [Google Scholar]
- Xia, B.; Hang, Y.; Tian, Y.; Yang, W.; Liao, Q.; Zhou, J. Efficient non-local contrastive attention for image super-resolution. In Proceedings of the AAAI Conference on Artificial Intelligence, Virtual Online, 22 February–1 March 2022; Volume 36, pp. 2759–2767. [Google Scholar]
- Mei, Y.; Fan, Y.; Zhou, Y. Image super-resolution with non-local sparse attention. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA, 20–25 June 2021; pp. 3517–3526. [Google Scholar]
- Behjati, P.; Rodriguez, P.; Fernández, C.; Hupont, I.; Mehri, A.; Gonzàlez, J. Single image super-resolution based on directional variance attention network. Pattern Recognit. 2023, 133, 108997. [Google Scholar] [CrossRef]
- Liu, Y.; Chu, Z.; Li, B. A Local and Non-Local Features Based Feedback Network on Super-Resolution. Sensors 2022, 22, 9604. [Google Scholar] [CrossRef] [PubMed]
- Liu, J.; Zhang, W.; Tang, Y.; Tang, J.; Wu, G. Residual Feature Aggregation Network for Image Super-Resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA, 13–19 June 2020; pp. 2359–2368. [Google Scholar] [CrossRef]
- Wang, Z.; Bovik, A.C.; Sheikh, H.R.; Simoncelli, E.P. Image Quality Assessment: From Error Visibility to Structural Similarity. IEEE Trans. Image Process. 2004, 13, 600–612. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Lim, B.; Son, S.; Kim, H.; Nah, S.; Mu Lee, K. Enhanced Deep Residual Networks for Single Image Super-Resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA, 21–26 July 2017; pp. 136–144. [Google Scholar] [CrossRef] [Green Version]
- Zhang, Y.; Li, K.; Li, K.; Wang, L.; Zhong, B.; Fu, Y. Image Super-Resolution Using Very Deep Residual Channel Attention Networks. In Proceedings of the European Conference on Computer Vision (ECCV), Rupnagar, India, 26–28 November 2018; pp. 286–301. [Google Scholar] [CrossRef] [Green Version]
- Bevilacqua, M.; Roumy, A.; Guillemot, C.; Alberi-Morel, M.L. Low-Complexity Single-Image Super-Resolution based on Nonnegative Neighbor Embedding. In Proceedings of the 23rd British Machine Vision Conference (BMVC), Surrey, UK, 3–7 September 2012. [Google Scholar] [CrossRef] [Green Version]
- Zeyde, R.; Elad, M.; Protter, M. On Single Image Scale-Up using Sparse-Representation. In Proceedings of the International Conference on Curves and Surfaces, Avignon, France, 24–30 June 2010; Springer: Berlin/Heidelberg, Germany, 2010; pp. 711–730. [Google Scholar] [CrossRef]
- Martin, D.; Fowlkes, C.; Tal, D.; Malik, J. A Database of Human Segmented Natural Images and its Application to Evaluating Segmentation Algorithms and Measuring Ecological Statistics. In Proceedings of the Eighth IEEE International Conference on Computer Vision. ICCV 2001, Vancouver, BC, Canada, 7–14 July 2001; Volume 2, pp. 416–423. [Google Scholar] [CrossRef] [Green Version]
- Huang, J.B.; Singh, A.; Ahuja, N. Single Image Super-resolution from Transformed Self-Exemplars. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA, 7–12 June 2015; pp. 5197–5206. [Google Scholar] [CrossRef]
- Matsui, Y.; Ito, K.; Aramaki, Y.; Fujimoto, A.; Ogawa, T.; Yamasaki, T.; Aizawa, K. Sketch-based manga retrieval using manga109 dataset. Multimed. Tools Appl. 2017, 76, 21811–21838. [Google Scholar] [CrossRef] [Green Version]
- Gu, J.; Dong, C. Interpreting Super-Resolution Networks with Local Attribution Maps. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA, 20–25 June 2021; pp. 9199–9208. [Google Scholar] [CrossRef]
- Xiong, S.; Li, B.; Zhu, S. DCGNN: A single-stage 3D object detection network based on density clustering and graph neural network. Complex Intell. Syst. 2022, 9, 1–10. [Google Scholar] [CrossRef]
Algorithm | Local-Feature-Based | Non-Local-Feature-Based | Local and Non-Local Features-Based | ||
---|---|---|---|---|---|
SRFBN-L | DLNFN-Local | CSNLN-L | DLNFN-Non-local | DLNFN-L (our) | |
PSNR | 37.77 | 37.84 | 37.91 | 37.85 | 38.02 |
Algorithm | Conv-DLNFN-L | Fix-DLNFN-L | DLNFN-L (our) |
---|---|---|---|
PSNR | 37.90 | 37.75 | 38.02 |
Algorithm | convolutional Layer Based | MAConv Layer Based | ||
---|---|---|---|---|
DLNFN-2Conv | DLNFN-4Conv | DLNFN-2MA | DLNFN-L (4MA, our) | |
PSNR | 37.96 | 37.97 | 38.00 | 38.02 |
Algorithm | Scale | Params | Set5 | Set14 | B100 | Urban100 | Manga109 | |||||
---|---|---|---|---|---|---|---|---|---|---|---|---|
PSNR | SSIM | PSNR | SSIM | PSNR | SSIM | PSNR | SSIM | PSNR | SSIM | |||
SRCNN [7] | 0.02M | 36.45 | 0.9527 | 32.32 | 0.9043 | 31.04 | 0.8838 | 29.11 | 0.8887 | 34.82 | 0.9627 | |
VDSR [8] | 0.67M | 37.58 | 0.9587 | 33.16 | 0.9133 | 31.94 | 0.8964 | 31.05 | 0.9164 | 37.44 | 0.9737 | |
AN [5] | 1.04M | 38.06 | 0.9608 | 33.75 | 0.9194 | 32.22 | 0.9002 | 32.43 | 0.9311 | 38.87 | 0.9769 | |
SRFBN [3] | 2.14M | 38.11 | 0.9609 | 33.82 | 0.9196 | 33.29 | 0.9010 | 32.62 | 0.9328 | 39.08 | 0.9779 | |
DiVANet+ [19] | 0.9M | 38.23 | 0.9618 | 33.88 | 0.9201 | 32.36 | 0.9018 | 32.67 | 0.9330 | 39.15 | 0.9780 | |
CSNLN [4] | 3.06M | 38.28 | 0.9616 | 34.12 | 0.9223 | 32.40 | 0.9024 | 33.25 | 0.9386 | 39.37 | 0.9785 | |
RCAN [24] | 15.44M | 38.27 | 0.9614 | 34.12 | 0.9216 | 32.41 | 0.9027 | 33.34 | 0.9384 | 39.44 | 0.9786 | |
EDSR [23] | 40.73M | 38.25 | 0.9613 | 33.97 | 0.9205 | 32.36 | 0.9020 | 32.98 | 0.9361 | 39.17 | 0.9781 | |
DLNFN (our) | 5.82M | 38.28 | 0.9617 | 34.22 | 0.9231 | 32.41 | 0.9026 | 33.39 | 0.9390 | 39.41 | 0.9786 | |
SRCNN [7] | 0.02M | 32.52 | 0.9052 | 29.09 | 0.8160 | 28.10 | 0.7781 | 25.84 | 0.7869 | 29.62 | 0.8999 | |
VDSR [8] | 0.67M | 33.76 | 0.9225 | 29.96 | 0.8347 | 28.85 | 0.7986 | 27.32 | 0.8324 | 32.41 | 0.9356 | |
AN [5] | 1.04M | 34.47 | 0.9279 | 30.44 | 0.8437 | 29.14 | 0.8059 | 28.41 | 0.8570 | 33.78 | 0.9458 | |
SRFBN [3] | 2.83M | 34.70 | 0.9292 | 30.51 | 0.8461 | 29.24 | 0.8084 | 28.73 | 0.8641 | 34.18 | 0.9481 | |
DiVANet+ [19] | 0.9M | 34.66 | 0.9289 | 30.53 | 0.8452 | 29.26 | 0.8077 | 28.66 | 0.8610 | 34.02 | 0.9473 | |
CSNLN [4] | 6.01M | 34.74 | 0.9300 | 30.66 | 0.8482 | 29.33 | 0.8105 | 29.13 | 0.8712 | 34.45 | 0.9502 | |
RCAN [24] | 15.63M | 34.74 | 0.9299 | 30.65 | 0.8482 | 29.32 | 0.8111 | 29.09 | 0.8702 | 34.44 | 0.9499 | |
EDSR [23] | 43.68M | 34.74 | 0.9297 | 30.50 | 0.8461 | 29.24 | 0.8095 | 28.76 | 0.8651 | 34.01 | 0.9481 | |
DLNFN (our) | 9.59M | 34.84 | 0.9307 | 30.78 | 0.8498 | 29.36 | 0.8119 | 29.45 | 0.8764 | 34.72 | 0.9515 |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2023 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Liu, Y.; Chu, Z. A Dynamic Fusion of Local and Non-Local Features-Based Feedback Network on Super-Resolution. Symmetry 2023, 15, 885. https://doi.org/10.3390/sym15040885
Liu Y, Chu Z. A Dynamic Fusion of Local and Non-Local Features-Based Feedback Network on Super-Resolution. Symmetry. 2023; 15(4):885. https://doi.org/10.3390/sym15040885
Chicago/Turabian StyleLiu, Yuhao, and Zhenzhong Chu. 2023. "A Dynamic Fusion of Local and Non-Local Features-Based Feedback Network on Super-Resolution" Symmetry 15, no. 4: 885. https://doi.org/10.3390/sym15040885
APA StyleLiu, Y., & Chu, Z. (2023). A Dynamic Fusion of Local and Non-Local Features-Based Feedback Network on Super-Resolution. Symmetry, 15(4), 885. https://doi.org/10.3390/sym15040885