# A Non-Linear Convolution Network for Image Processing

^{1}

^{2}

^{3}

^{4}

^{*}

## Abstract

**:**

## 1. Introduction

- the linear convolution process used in standard CNNs is replaced by a Non-Linear Convolution (NLC) that more thoroughly exploits the relationships that exist among adjacent image pixels;
- the input-dependent weights generated inside the NLC are normalized to preserve the dynamic range of the data while the input image is transformed into a set of features, processed in the hidden layers, and finally projected into the desired output image;
- the NLC modules are trained end-to-end by a standard back-propagation algorithm.

- a new typology of neural network layer, realizing a non-linear convolution is introduced;
- we prove that such a layer performs like a space-variant linear convolution and we suggest how to configure it accordingly to the desired behavior;
- we propose some strategies to combine profitably these layers in larger architectures;
- complete architectures which adopt this layer are proposed and used to solve different image-processing problems.

## 2. Non-Linear Convolution Methods in CNN

## 3. Single Layer Architecture

#### 3.1. The Main Layer

#### 3.2. One-Channel Case

#### 3.3. Multi-Input-Output Channel Case

## 4. Non-Linear Layer Discussion

- usually, a network is developed to deal with a specific application, and is trained on this application. Therefore, comparisons make sense only if limited to that single application. On the opposite, we want to test the versatility of the proposed layer;
- the depth of a network, the number of features and the way they are combined, the types of connection, the adoption of processing strategies such as “inception”, “residual”, etc., all affect its performance. Thus, it is difficult to distinguish the effects of the network architecture from those of the design of the single layer.

## 5. Network Architectures

#### 5.1. Cascade

#### 5.2. Cascade + Bypass

#### 5.3. Cascade + Bypass + Mixing Layer

#### 5.4. Parallel Architecture to Increase Receptive Field

## 6. Performance Comparison

## 7. Network Applications

#### 7.1. Edge-Preserving Smoothing Architecture

#### 7.2. NLCN Performance Showcase for Noise Removal

#### 7.3. NLCN Performance Showcase for JPEG Artifacts Removal

## 8. Perspectives for an FPGA Realization of the NLCN

## 9. Conclusions

## Author Contributions

## Funding

## Acknowledgments

## Conflicts of Interest

## References

- Mitra, S.K.; Sicuranza, G.L. Preface. In Nonlinear Image Processing; Mitra, S.K., Sicuranza, G.L., Eds.; Communications, Networking and Multimedia; Academic Press: San Diego, CA, USA, 2001; pp. 9–15. [Google Scholar] [CrossRef]
- Cha, S.; Moon, T. Fully Convolutional Pixel Adaptive Image Denoiser. arXiv
**2019**, arXiv:cs.CV/1807.07569. [Google Scholar] - Zoumpourlis, G.; Doumanoglou, A.; Vretos, N.; Daras, P. Non-linear Convolution Filters for CNN-Based Learning. In Proceedings of the 2017 IEEE International Conference on Computer Vision (ICCV), Venice, Italy, 22–27 October 2017; pp. 4771–4779. [Google Scholar]
- Wang, C.; Yang, J.; Xie, L.; Yuan, J. Kervolutional Neural Networks. In Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA, 16–20 June 2019; pp. 31–40. [Google Scholar]
- Cui, Y.; Zhou, F.; Wang, J.; Liu, X.; Lin, Y.; Belongie, S. Kernel Pooling for Convolutional Neural Networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA, 21–26 July 2017. [Google Scholar]
- Lin, M.; Chen, Q.; Yan, S. Network in Network. arXiv
**2014**, arXiv:cs.NE/1312.4400. [Google Scholar] - Min, M.R.; Ning, X.; Cheng, C.; Gerstein, M. Interpretable Sparse High-Order Boltzmann Machines. In Proceedings of Machine Learning Research, Proceedings of the Seventeenth International Conference on Artificial Intelligence and Statistics, Reykjavik, Iceland, 22–25 April 2014; Kaski, S., Corander, J., Eds.; PMLR: Reykjavik, Iceland, 2014; Volume 33, pp. 614–622. [Google Scholar]
- Sabour, S.; Frosst, N.; Hinton, G.E. Dynamic Routing between Capsules. In Proceedings of the 31st International Conference on Neural Information Processing Systems NIPS’17, Long Beach, CA, USA, 4–9 December 2017; Curran Associates Inc.: Red Hook, NY, USA, 2017; pp. 3859–3869. [Google Scholar]
- Dai, J.; Qi, H.; Xiong, Y.; Li, Y.; Zhang, G.; Hu, H.; Wei, Y. Deformable Convolutional Networks. arXiv
**2017**, arXiv:1703.06211. [Google Scholar] - Wang, X.; Zhang, B.; Li, C.; Ji, R.; Han, J.; Cao, X.; Liu, J. Modulated Convolutional Networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA, 18–22 June 2018. [Google Scholar]
- Zhang, K.; Zuo, W.; Chen, Y.; Meng, D.; Zhang, L. Hybrid nonlinear convolution filters for image recognition. Appl. Intell.
**2020**. [Google Scholar] [CrossRef] - Zhang, R.; Tang, S.; Zhang, Y.; Li, J.; Yan, S. Perspective-Adaptive Convolutions for Scene Parsing. IEEE Trans. Pattern Anal. Mach. Intell.
**2020**, 42, 909–924. [Google Scholar] [CrossRef] [PubMed] - Kroner, S.; Ramponi, G.G. Design constraints for polynomial and rational filters. In Proceedings of the EEE-EURASIP Workshop on Nonlinear Signal and Image Processing (NSIP’99), Antalya, Turkey, 20–23 June 1999; pp. 501–505. [Google Scholar]
- He, K.; Zhang, X.; Ren, S.; Sun, J. Deep Residual Learning for Image Recognition. In Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 26 June–1 July 2016; pp. 770–778. [Google Scholar] [CrossRef] [Green Version]
- Zhang, Y.; Tian, Y.; Kong, Y.; Zhong, B.; Fu, Y. Residual Dense Network for Image Super-Resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA, 18–22 June 2018. [Google Scholar]
- Liu, Y.; Zhang, X.; Wang, S.; Ma, S.; Gao, W. Progressive Multi-Scale Residual Network for Single Image Super-Resolution. arXiv
**2020**, arXiv:eess.IV/2007.09552. [Google Scholar] - Kim, J.; Lee, J.K.; Lee, K.M. Deeply-Recursive Convolutional Network for Image Super-Resolution. In Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 27–30 June 2016; pp. 1637–1645. [Google Scholar] [CrossRef] [Green Version]
- Shi, W.; Caballero, J.; Huszár, F.; Totz, J.; Aitken, A.P.; Bishop, R.; Rueckert, D.; Wang, Z. Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network. In Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 26 June–1 July 2016; pp. 1874–1883. [Google Scholar]
- Zhang, X.; Wei, K.; Kang, X.; Liu, J. Super Resolutions-Browse State-of-the-Art. 2013. Available online: https://paperswithcode.com/task/image-super-resolution (accessed on 1 January 2021).
- Li, Y.; Huang, J.; Ahuja, N.; Yang, M. Joint Image Filtering with Deep Convolutional Networks. IEEE Trans. Pattern Anal. Mach. Intell.
**2019**, 41, 1909–1923. [Google Scholar] [CrossRef] [PubMed] [Green Version] - Liu, S.; Pan, J.; Yang, M.H. Learning Recursive Filters for Low-Level Vision via a Hybrid Neural Network. In Computer Vision—ECCV; Leibe, B., Matas, J., Sebe, N., Welling, M., Eds.; Springer International Publishing: Cham, Switzerland, 2016; pp. 560–576. [Google Scholar]
- Xu, L.; Ren, J.S.J.; Yan, Q.; Liao, R.; Jia, J. Deep Edge-Aware Filters. In Proceedings of the 32nd International Conference on International Conference on Machine Learning ICML’15, Lille, France, 6–11 July 2015; Volume 37, pp. 1669–1678. [Google Scholar]
- Kohli, M.D.R.M. Indiana University Chest X-ray Collection. 2013. Available online: https://openi.nlm.nih.gov/imgs/collections/NLMCXR_png.tgz (accessed on 1 January 2021).
- Zhang, K.; Zuo, W.; Zhang, L. FFDNet: Toward a Fast and Flexible Solution for CNN-Based Image Denoising. IEEE Trans. Image Process.
**2018**, 27, 4608–4622. [Google Scholar] [CrossRef] [PubMed] [Green Version] - Lecouat, B.; Ponce, J.; Mairal, J. Fully Trainable and Interpretable Non-Local Sparse Models for Image Restoration. In Proceedings of the European Conference on Computer Vision (ECCV), Glasgow, UK, 23–28 August 2020. [Google Scholar]
- Zhang, K.; Zuo, W.; Chen, Y.; Meng, D.; Zhang, L. Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising. IEEE Trans. Image Process.
**2017**, 26, 3142–3155. [Google Scholar] [CrossRef] [PubMed] [Green Version] - Sledevič, T.; Serackis, A. mNet2FPGA: A Design Flow for Mapping a Fixed-Point CNN to Zynq SoC FPGA. Electronics
**2020**, 9, 1823. [Google Scholar] [CrossRef] - Duarte, J.; Han, S.; Harris, P.; Jindariani, S.; Kreinar, E.; Kreis, B.; Ngadiuba, J.; Pierini, M.; Rivera, R.; Tran, N.; et al. Fast inference of deep neural networks in FPGAs for particle physics. J. Instrum.
**2018**, 13, P07027. [Google Scholar] [CrossRef] - Meloni, P.; Capotondi, A.; Deriu, G.; Brian, M.; Conti, F.; Rossi, D.; Raffo, L.; Benini, L. NEURAghe: Exploiting CPU-FPGA Synergies for Efficient and Flexible CNN Inference Acceleration on Zynq SoCs. ACM Trans. Reconfig. Technol. Syst.
**2018**, 11, 18:1–18:24. [Google Scholar] [CrossRef] [Green Version] - Guo, K.; Sui, L.; Qiu, J.; Yu, J.; Wang, J.; Yao, S.; Han, S.; Wang, Y.; Yang, H. Angel-Eye: A complete design flow for mapping CNN onto embedded FPGA. IEEE Trans. Comput. Aided Des. Integr. Circuits Syst.
**2017**, 1. [Google Scholar] [CrossRef] - Guo, K.; Zeng, S.; Yu, J.; Wang, Y.; Yang, H. A Survey of FPGA-Based Neural Network Accelerator. arXiv
**2018**, arXiv:cs.AR/1712.08934. [Google Scholar] - Zhang, M.; Li, L.; Wang, H.; Liu, Y.; Qin, H.; Zhao, W. Optimized Compression for Implementing Convolutional Neural Networks on FPGA. Electronics
**2019**, 8, 295. [Google Scholar] [CrossRef] [Green Version] - Sun, W.; Zeng, H.; Yang, Y.E.; Prasanna, V.K. Throughput-Optimized Frequency Domain CNN with Fixed-Point Quantization on FPGA. In Proceedings of the International Conference on ReConFigurable Computing and FPGAs, ReConFig 2018, Cancun, Mexico, 3–5 December 2018; Andrews, D., Cumplido, R., Feregrino, C., Stroobandt, D., Eds.; IEEE: Piscataway, NJ, USA, 2018; pp. 1–8. [Google Scholar]
- Gysel, P.; Pimentel, J.J.; Motamedi, M.; Ghiasi, S. Ristretto: A Framework for Empirical Study of Resource-Efficient Inference in Convolutional Neural Networks. IEEE Trans. Neural Netw. Learn. Syst.
**2018**, 29, 5784–5789. [Google Scholar] [CrossRef] [PubMed] - Peres, T.; Gonçalves, A.; Véstias, M.P. Faster Convolutional Neural Networks in Low Density FPGAs Using Block Pruning. In Lecture Notes in Computer Science, Proceedings of the Applied Reconfigurable Computing—15th International Symposium, ARC 2019, Darmstadt, Germany, 9–11 April 2019; Hochberger, C., Nelson, B., Koch, A., Woods, R.F., Diniz, P.C., Eds.; Springer: Cham, Switzerland, 2019; Volume 11444, pp. 402–416. [Google Scholar]
- Cheng, Y.; Wang, D.; Zhou, P.; Zhang, T. A Survey of Model Compression and Acceleration for Deep Neural Networks. arXiv
**2017**, arXiv:1710.09282. [Google Scholar] - Xu, J.; Liu, Z.; Jiang, J.; Dou, Y.; Li, S. CaFPGA: An automatic generation model for CNN accelerator. Microprocess. Microsyst.
**2018**, 60, 196–206. [Google Scholar] [CrossRef] - Xiao, Q.; Liang, Y. Fune: An FPGA Tuning Framework for CNN Acceleration. IEEE Des. Test
**2020**, 37, 46–55. [Google Scholar] [CrossRef] - Hah, T.K.; Liew, Y.T.; Ong, J. Low Precision Constant Parameter CNN on FPGA. arXiv
**2019**, arXiv:1901.04969. [Google Scholar] - Xiao, Q.; Liang, Y.; Lu, L.; Yan, S.; Tai, Y. Exploring Heterogeneous Algorithms for Accelerating Deep Convolutional Neural Networks on FPGAs. In Proceedings of the 54th Annual Design Automation Conference, DAC 2017, Austin, TX, USA, 18–22 June 2017; pp. 62:1–62:6. [Google Scholar]

**Figure 1.**A block scheme of the proposed strategy: (

**A**) a linear convolution, (

**B**) the proposed non-linear (input-dependent) convolution.

**Table 1.**A comparison among some similar simple networks. Memory (Mem) and Floating-Point Operations (FLOPs) have been estimated for the processing of an 800 × 600 image.

Net Name | Type | Layers | Architecture | Features | Param. | Mem | FLOPs |
---|---|---|---|---|---|---|---|

LINNet S | Linear | 2 | 1-16-4 | 16 | 144 | 8 M | 69 M |

HUGENet S | Linear | 2 | 1-128-4 | 128 | 1152 | 61 M | 550 M |

NLCN S | Non-linear | 2 | 1-16-4 | 16 | 1296 | 8 M | 626 M |

VoltNet S | Non-linear | 2 | 1-16-4 | 16 | 1296 | 8 M | 626 M |

LINNet M | Linear | 3 | 1-4-16-4 | 16 | 612 | 10 M | 293 M |

HUGENet M | Linear | 3 | 1-16-128-4 | 128 | 18,576 | 64 M | 8.9 G |

NLCN M | Non-linear | 3 | 1-4-16-4 | 16 | 21,060 | 10 M | 10 G |

VoltNet M | Non-linear | 3 | 1-4-16-4 | 16 | 21,060 | 10 G | 10 G |

**Table 2.**SSIM and PSNR metrics with denoised images from gray(g) and color(c) LIVE(L) and X-ray dataset(x).

Type | SSIM | ||||
---|---|---|---|---|---|

Noisy | NLCN | BM3D | FFD | G-SC | |

Pois_x | 0.672 | 0.961 | 0.948 | 0.947 | 0.940 |

Gaus_x | 0.169 | 0.895 | 0.852 | 0.858 | 0.396 |

Pois+Gaus_x | 0.520 | 0.950 | 0.688 | 0.942 | 0.916 |

Pois_L_g | 0.701 | 0.899 | 0.853 | 0.907 | 0.870 |

Pois_L_c | 0.789 | 0.883 | 0.837 | 0.893 | 0.894 |

Type | PSNR | ||||

Noisy | NLCN | BM3D | FFD | G-SC | |

Pois_x | 31.00 | 41.47 | 40.01 | 39.99 | 39.52 |

Gaus_x | 20.97 | 36.56 | 33.03 | 33.16 | 29.12 |

Pois+Gaus_x | 27.38 | 39.86 | 21.76 | 38.83 | 39.52 |

Pois_L_g | 27.61 | 33.52 | 30.43 | 32.94 | 32.93 |

Pois_L_c | 30.06 | 32.82 | 29.77 | 32.35 | 33.00 |

Type | SSIM | PSNR | ||||
---|---|---|---|---|---|---|

JPEG | NLCN | DnCNN | JPEG | NLCN | DnCNN | |

LIVE_gray | 0.860 | 0.867 | 0.869 | 29.24 | 32.80 | 29.76 |

LIVE_color | 0.860 | 0.865 | 0.869 | 29.30 | 32.64 | 29.76 |

Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |

© 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Marsi, S.; Bhattacharya, J.; Molina, R.; Ramponi, G.
A Non-Linear Convolution Network for Image Processing. *Electronics* **2021**, *10*, 201.
https://doi.org/10.3390/electronics10020201

**AMA Style**

Marsi S, Bhattacharya J, Molina R, Ramponi G.
A Non-Linear Convolution Network for Image Processing. *Electronics*. 2021; 10(2):201.
https://doi.org/10.3390/electronics10020201

**Chicago/Turabian Style**

Marsi, Stefano, Jhilik Bhattacharya, Romina Molina, and Giovanni Ramponi.
2021. "A Non-Linear Convolution Network for Image Processing" *Electronics* 10, no. 2: 201.
https://doi.org/10.3390/electronics10020201