# Nonuniformity Correction of Single Infrared Images Based on Deep Filter Neural Network

^{1}

^{2}

^{3}

^{*}

## Abstract

**:**

## 1. Introduction

- A nonuniformity correction method based on deep learning is proposed. Combined with the nonuniform noise model, large numbers of simulated infrared images are used to train our network structure, which can effectively correct real nonuniform infrared images.
- In our network structure, the use of the deep network structure improves the learning ability of the network and solves the underfitting problem. Taking the method of using high-frequency information as input to the network effectively eliminates background interference, and using the subtraction structure to distinguish between fixed-mode noise and noise-free images reduces the mapping range during training, which makes the learning process of the deep model easier.

## 2. Related Work

## 3. Our Methods

_{H}and a low-frequency component I

_{L}are separated by using a bilateral filter for a single image I containing nonuniform noise; the high-frequency component I

_{H}contains image structure details and nonuniform noise, which are used as the input for the network model, and through the network model, nonuniform noise can be fitted. Finally, the subtraction structure of the network model will be used to get the corrected image.

#### 3.1. Nonuniformity Model and Training Dataset

#### 3.2. Network Design

#### 3.2.1. High-Frequency Information Extraction

#### 3.2.2. Mapping Learning and Subtraction Structure

_{H}that removes background interference, the network takes I

_{H}as input and outputs nonuniform noise N, and the network parameters are learned through nonlinear mapping. Finally, in the subtraction structure we subtract the estimated nonuniform noise N from the original Y to obtain the final NUC estimated result X. In order to learn more image features, we increase the number of layers in the network. In this step, a shortcut connection is used to avoid the gradient disappearing due to the excessive number of network layers. The shortcut connection neither introduces external parameters nor increases computational complexity. For input i, the output after passing through the 2-layer network is added to the original i to obtain a new output [24]. To increase the nonlinearity, the rectified linear unit (ReLU) is used as an activation function after each convolutional layer, then the feature maps are aggregated to generate the output of nonuniform noise.

_{i}= M(I

_{Hi}),

_{i}= Y

_{i}− N

_{i},

_{l}= max(0,W

_{l}*X

_{l−1}+ B

_{l}),

_{l+1}= max(0,W

_{l+1}*X

_{l}+ B

_{l+1}) + X

_{l−1},

#### 3.3. More Analysis

- In this paper, the subtraction structure method is adopted to make the network learn the difference between the nonuniform infrared image and the clear infrared image during the training process, instead of directly learning the noise-free infrared image; this method can effectively reduce the mapping range. This makes the training process easier, and the subtraction structure can also propagate nondestructive information directly throughout the network, which is very useful for estimating the final normal infrared image.
- An important concept of neural networks is the pooling layer, which usually comes after the convolutional layer. Although the pooling layer can reduce the dimension of the output data, the pooling operation will bring information loss. Therefore, in order to avoid losing part of the information during the training process, all the pooling layers in our network are removed, and at the same time, we used a batch normalization layer to solve the problem of network overfitting and alleviate internal covariate shift [26]. The role of batch normalization is to perform a normalization process on the data. This method can prevent the input distribution of the hidden layer from constantly changing, calculate the mean and variance of the batch data, and normalize the batch data to reduce the input distribution of each hidden layer node to (−1, 1), which reduces the input space and the difficulty of tuning.

## 4. Experimental Results and Analysis

#### 4.1. Parameter Settings

#### 4.2. Training

#### 4.3. Algorithm Comparison and Quality Evaluation

^{T}= [1,−1] is a vertical mask, and ‖•‖

_{1}represents the L1 norm. The smaller the value of ρ, the better the image will be.

#### 4.4. Analysis of Experimental Data

## 5. Conclusions

## Author Contributions

## Funding

## Conflicts of Interest

## References

- Lin, B.; Fang, Y.H. An algorithm of improved BP neural network Non-uniformity correction. Eng. Surv. Mapp.
**2013**, 22, 24–27. [Google Scholar] [CrossRef] - Han, K.L.; He, C.F. A Nonuniformity Correction Algorithm for IRFPAs Based on Two Points and it’s Realization by DSP. Infrared Technol.
**2007**, 9, 541–544. [Google Scholar] [CrossRef] - Scribner, D.A.; Kruer, M.R.; Killiany, J.M. Infrared focal plane array technology. Proc. IEEE
**2002**, 79, 66–85. [Google Scholar] [CrossRef] - Harris, J.G.; Chiang, Y.-M. Non-uniformity correction of infrared image sequences using the constant-statistics constraint. IEEE Trans. Image Process.
**1999**, 8, 1148–1151. [Google Scholar] [CrossRef] [PubMed] - Zuo, C.; Chen, Q.; Gu, G.; Sui, X. Scene-based nonuniformity correction algorithm based on interfram registration. J. Opt. Soc. Am. A
**2011**, 9, 1164–1176. [Google Scholar] [CrossRef] [PubMed] - Tendero, Y.; Landeau, S.; Gilles, J. Non-uniformity correction of infrared images by midway equalization. Image Process.
**2012**, 2, 134–146. [Google Scholar] [CrossRef] - Vera, E.; Meza, P.; Torres, S. Total variation approach for adaptive nonuniformity correction in focal-plane arrays. Opt. Lett.
**2011**, 36, 172–174. [Google Scholar] [CrossRef] [PubMed] - Cao, Y.; Yang, M.Y.; Tisse, C.-L. Effective strip noise removal for Low-textured infrared images Based on 1D guided filtering. IEEE Trans. Circuits Syst. Video Technol.
**2016**, 26, 2176–2188. [Google Scholar] [CrossRef] - Buades, A.; Coll, B.; Morel, J.M. Nonlocal image and movie denoising. Int. J. Comput. Vis.
**2008**, 76, 123–139. [Google Scholar] [CrossRef] - Zhao, J.; Zhou, Q.; Chen, Y.; Liu, T.; Feng, H.; Xu, Z.; Li, Q. Single image stripe nonuniformity correction with gradient-constrained optimization model for infrared focal plane arrays. Opt. Commun.
**2013**, 296, 47–52. [Google Scholar] [CrossRef] - Zhang, D.; Han, X.; Deng, C. Review on the Research and Practice of Deep Learning and Reinforcement Learning in Smart Grids. CSEE J. Power Energy Syst.
**2018**, 4, 362–370. [Google Scholar] [CrossRef] - Qiu, X.; Jiang, T.; Wang, N. Safeguarding multiuser communication using full-duplex jamming and Q-learning algorithm. IET Commun.
**2018**, 12, 1805–1811. [Google Scholar] [CrossRef] - He, K.; Zhang, X.; Ren, S.; Sun, J. Spatial pyramid pooling in deep convolutional networks for visual recognition. Proc. Eur. Conf. Comput. Vis
**2014**, 8691, 346–361. [Google Scholar] [CrossRef] - Ouyang, W.; Luo, P.; Zeng, X.; Qiu, S.; Tian, Y.; Li, H.; Yang, S.; Wang, Z.; Xiong, Y.; Qian, C.; et al. Deepid-Net: Multi-Stage and Deformable Deep Convolutional Neural Networks for Object Detection. Available online: https://arxiv.org/abs/1409.3505 (accessed on 16 October 2018).
- Sun, Y.; Chen, Y.; Wang, X.; Tang, X. Deep learning face representation by joint identification-verification. Proc. Adv. Neural Inf. Process. Syst.
**2014**, 2, 1988–1996. [Google Scholar] - Dong, C.; Chen, C.; He, K.; Tang, X. Learning a deep convolutional network for image super-resolution. ECCV
**2014**, 8692, 184–199. [Google Scholar] - Kuang, X.; Sui, X.; Chen, Q.; Gu, G. Single infrared image stripe noise removal using deep convolutional networks. IEEE Photonic J.
**2017**, 9, 1–13. [Google Scholar] [CrossRef] - Jian, X.; Wang, F.; Guo, Q. Improved single image non-uniformity correction based on compressive. J. Mechan. Electr. Eng.
**2017**, 34, 1491–1495. [Google Scholar] [CrossRef] - Chang, Y.; Yan, L.; Wu, T.; Zhong, S. Remote sensing image stripe noise removal: From image decomposition perspective. IEEE Trans. Geosci. Remote Sens.
**2016**, 54, 7018–7031. [Google Scholar] [CrossRef] - Münch, B.; Trtik, P.; Marone, F.; Stampanoni, M. Stripe and ring artifact removal with combined wavelet-Fourier filtering. Opt. Express
**2009**, 17, 8567–8591. [Google Scholar] [CrossRef] - Boutemedjet, A.; Deng, C.; Zhao, B. Edge-Aware Unidirectional Total Variation Model for Stripe Non-Uniformity Correction. Sensors
**2018**, 18, 1164. [Google Scholar] [CrossRef] [PubMed] - Li, F.; He, X.; Wei, Z.; He, J. Multiframe infrared image super-resolution reconstruction using generative adversarial networks. Infrared Laser Eng.
**2018**, 47, 26–33. [Google Scholar] [CrossRef] - Lei, X.; Gu, G.; Sui, X. Improved temporal high-pass filter nonuniformity correction based on bilateral filter. Laser Infrared
**2012**, 42, 831–836. [Google Scholar] - He, K.; Zhang, X.; Ren, S.; Sun, J. Deep residual learning for image recognition. In Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 27–30 June 2016; pp. 770–778. [Google Scholar]
- Nair, V.; Hinton, G.E. Rectified linear units improve restricted Boltzmann machines. In Proceedings of the 27th International Conference on International Conference on Machine Learning, Haifa, Israe, 21–24 June 2010. [Google Scholar]
- Ioffe, S.; Szegedy, C. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. Available online: http://arxiv.org/abs/1502.03167 (accessed on 16 October 2018).
- Kiefer, J.; Wolfowitz, J. Stochastic estimation of the maximum of a regression function. Ann. Math. Stat.
**1952**, 23, 462–466. [Google Scholar] [CrossRef] - Liu, Y.; Zhu, H.; Zhao, Y. Nonuniformity correction algorithm based on infrared focal plane array readout architecture. Opt. Precis. Eng.
**2008**, 1, 128–133. [Google Scholar] - Pezoa, J.E.; Hayat, M.M.; Torres, S.N.; Rahman, M.A. Multimodel Kalman filtering for adaptive nonuniformity correction in infrared sensors. J. Opt. Soc. Am. A Opt. Image Sci. Vis.
**2003**, 20, 470–480. [Google Scholar] [CrossRef] - Hayat, M.M.; Torres, S.N.; Armstrong, E. Statistical algorithm for nonuniformity correction in focal-plane arrays. Appl. Opt.
**1999**, 38, 772–780. [Google Scholar] [CrossRef] [PubMed]

**Figure 1.**Nonuniform noise image and our result. (

**a**) Nonuniform infrared image and enlarged image; (

**b**) our nonuniformity correction (NUC) results and enlarged image.

**Figure 2.**Training pairs (360 × 512 resolution). (

**a**) Clean infrared image; (

**b**) infrared image with simulated fixed-pattern noise (FPN).

**Figure 4.**Images and pixel histograms. (

**a**) Nonuniform infrared image Y; (

**b**) high-frequency component I

_{H}; (

**c**) pixel histogram of Y; (

**d**) pixel histogram of I

_{H}.

**Figure 5.**Curve graph of training. (

**a**) Our convergence result; (

**b**) comparison of our results with Stripe noise removal convolutional neural network (SNRCNN).

**Figure 6.**Comparison with each algorithm (image resolution 384 × 288): (

**a**) Original1, (

**b**) guided filter (GF), (

**c**) Midway Histogram Equalization (MHE), (

**d**) SNRCNN, (

**e**) Ours, (

**f**) Original2, (

**g**) GF, (

**h**) MHE, (

**i**) SNRCNN, (

**j**) Ours.

**Table 1.**Evaluation parameters of Original1. RMSE, root mean squared error; PSNR, peak signal-to-noise ratio; ρ, image roughness; T, processing time.

Algorithms | RMSE | PSNR (dB) | ρ | T (s) |
---|---|---|---|---|

GF | 47.56 | 14.62 | 0.33 | 0.38 s |

MHE | 11.80 | 26.69 | 0.47 | 182.54 s |

SNRCNN | 5.39 | 33.50 | 0.57 | 1.86 s |

Ours | 0.33 | 57.87 | 0.24 | 0.73 s |

Algorithms | RMSE | PSNR (dB) | ρ | T (s) |
---|---|---|---|---|

GF | 31.13 | 18.30 | 0.39 | 0.42 s |

MHE | 12.05 | 26.51 | 0.56 | 186.86 s |

SNRCNN | 5.28 | 33.66 | 0.65 | 2.16 s |

Ours | 0.37 | 56.83 | 0.28 | 0.75 s |

© 2018 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Jian, X.; Lv, C.; Wang, R.
Nonuniformity Correction of Single Infrared Images Based on Deep Filter Neural Network. *Symmetry* **2018**, *10*, 612.
https://doi.org/10.3390/sym10110612

**AMA Style**

Jian X, Lv C, Wang R.
Nonuniformity Correction of Single Infrared Images Based on Deep Filter Neural Network. *Symmetry*. 2018; 10(11):612.
https://doi.org/10.3390/sym10110612

**Chicago/Turabian Style**

Jian, Xianzhong, Chen Lv, and Ruzhi Wang.
2018. "Nonuniformity Correction of Single Infrared Images Based on Deep Filter Neural Network" *Symmetry* 10, no. 11: 612.
https://doi.org/10.3390/sym10110612