# Temporal-Spatial Nonlinear Filtering for Infrared Focal Plane Array Stripe Nonuniformity Correction

^{1}

^{2}

^{*}

## Abstract

**:**

## 1. Introduction

- The proposed NUC algorithm accurately determines the nonuniformity information and efficiently removes the corresponding nonuniformity under the guidance of the estimated nonuniformity label. Due to the nonlinear filtering used in the temporal domain, our method requires fewer sequential frames in a video to realize more accurate correction results. In addition, it does not have the problem of slow convergence and ghosting artifacts.
- Based on the observation that a weighted guided image filter can be used as a satisfactory nonuniformity estimate in the spatial domain, a novel global weight map sensitive to stripe noise is introduced into the guided image filter to improve its efficiency in suppressing stripe noise and preserving edge information.
- Compared with the single-frame-based NUC methods, our proposed method makes full use of the temporal characteristic of the nonuniformity to substantially improve the nonuniformity estimation accuracy. Consequently, the degradation of the corrected image is greatly reduced.

## 2. Related Works

#### 2.1. Scene-Based Nonuniformity Correction

#### 2.2. Nonuniformity Correction Formation

## 3. Weighted Guided Image Filtering and Global Weight Map

#### 3.1. Analysis of the Global Weight Map

#### 3.2. Kernel Function

## 4. Proposed Method

#### 4.1. Spatial-Domain Nonuniformity Estimation via Weighted Guided Image Filtering

#### 4.2. Temporal-Domain Nonuiformity Correction Via a Nonlinear Diffusion Equation

## 5. Experiment and Analysis

#### 5.1. Objective NUC Quality Metrics

_{1}= [1, −1] is a horizontal mask, h

_{2}= [1, −1]

^{T}is a vertical mask, and ${\Vert \u2022\Vert}_{1}$ denotes the L

_{1}-norm. Similar to nonuniformity U, a smaller value of $\rho $ is expected in the corrected image.

#### 5.2. Implementation Details

#### 5.3. Experiment Results and Discussion

#### 5.3.1. Experiment 1

#### 5.3.2. Experiment 2

## 6. Conclusions

## Author Contributions

## Funding

## Conflicts of Interest

## References

- Scribner, D.A.; Kruer, M.R.; Killiany, J.M. Infrared focal plane array technology. Proc. IEEE
**1991**, 79, 66–85. [Google Scholar] [CrossRef] - Friedenberg, A.; Goldblatt, I. Nonuniformity two-point linear correction errors in infrared focal plane arrays. Opt. Eng.
**1998**, 37, 1251–1253. [Google Scholar] [CrossRef] - Sungho, K. Two-point correction and minimum filter-based nonuniformity correction for scan-based aerial infrared cameras. Opt. Eng.
**2012**, 51, 106401. [Google Scholar] - Scribner, D.A. Nonuniformity correction for staring IR focal plane arrays using scene-based techniques. In Proceedings of the International Society for Optics and Photonics (SPIE), Washington, DC, USA, 1 September 1990; pp. 225–233. [Google Scholar]
- Scribner, D.A. Adaptive nonuniformity correction for IR focal-plane arrays using neural networks. In Proceedings of the International Society for Optics and Photonics (SPIE), San Diego, CA, USA, 1 November 1991; pp. 100–109. [Google Scholar]
- Guan, J.; Lai, R.; Xiong, A. Wavelet Deep Neural Network for Stripe Noise Removal. IEEE Access
**2019**, 7, 44544–44554. [Google Scholar] [CrossRef] - Liu, C.W.; Sui, X.B.; Liu, Y. FPN estimation based nonuniformity correction for infrared imaging system. Infrared Phys. Technol.
**2019**, 96, 22–29. [Google Scholar] [CrossRef] - Lai, R.; Guan, J.; Yang, Y. Spatiotemporal Adaptive Nonuniformity Correction Based on BTV Regularization. IEEE Access
**2019**, 7, 753–762. [Google Scholar] [CrossRef] - Jian, X.Z.; Lu, R.Z.; Guo, Q.; Wang, G.P. Single image non-uniformity correction using compressive sensing. Infrared Phys. Technol.
**2016**, 76, 360–364. [Google Scholar] [CrossRef] - Harris, J.G.; Chiang, Y.M. Nonuniformity correction of infrared image sequences using the constant-statistics constraint. IEEE Trans. Image Process.
**1999**, 8, 1148–1151. [Google Scholar] [CrossRef] - Qian, W.; Chen, Q.; Gu, G. Space low-pass and temporal high-pass nonuniformity correction algorithm. Opt. Rev.
**2010**, 17, 24–29. [Google Scholar] [CrossRef] - Zuo, C.; Chen, Q.; Gu, G.; Qian, W. New Temporal High-Pass Filter Nonuniformity Correction Based on Bilateral Filter. Opt. Rev.
**2011**, 18, 197–202. [Google Scholar] [CrossRef] - Li, Z.; Shen, T.; Lou, S. Scene-based nonuniformity correction based on bilateral filter with reduced ghosting. Infrared Phys. Technol.
**2016**, 77, 360–365. [Google Scholar] [CrossRef] - Zhang, T.X.; et al. PDE-based deghosting algorithm for correction of non-uniformity in infrared focal plane array. J. Infrared Millim. Waves
**2012**, 31, 177–182. [Google Scholar] [CrossRef] - Lai, R.; Yue, G.; Zhang, G. Total Variation Based Neural Network Regression for Nonuniformity Correction of Infrared Images. Symmetry
**2018**, 10, 157. [Google Scholar] [CrossRef] - Hardie, R.C.; Baxley, F.; Brys, B.; Hytla, P. Scene-Based Nonuniformity Correction with Reduced Ghosting Using a Gated LMS Algorithm. Opt. Express
**2009**, 17, 14918–14933. [Google Scholar] [CrossRef] - Zhou, D.; Wang, D.; Huo, L.; Liu, R.; Jia, P. Scene-based nonuniformity correction for airborne point target detection systems. Opt. Express
**2017**, 25, 14210–14226. [Google Scholar] [CrossRef] - Hardie, R.C.; Hayat, M.M.; Armstrong, E.; Yasuda, B. Armstrong. Scene-Based Nonuniformity Correction with Video Sequences and Registration. Appl. Opt.
**2000**, 39, 1241–1250. [Google Scholar] [CrossRef] - Zeng, J.; Sui, X.; Gao, H. Adaptive Image-Registration-Based Nonuniformity Correction Algorithm with Ghost Artifacts Eliminating for Infrared Focal Plane Arrays. IEEE Photonic J.
**2015**, 7, 1–16. [Google Scholar] [CrossRef] - Rong, S.H.; Zhou, H.X.; Qin, H.L. Nonuniformity correction for an infrared focal plane array based on diamond search block matching. JOSA A
**2016**, 33, 938–946. [Google Scholar] - Tendero, Y.; Landeau, S.; Gilles, J. Non-Uniformity Correction of Infrared Images by Midway Equalization. Image Process. Line
**2012**, 2, 134–146. [Google Scholar] [CrossRef] - Kuang, X.; Sui, X.; Chen, Q.; Gu, G. Single infrared image stripe noise removal using deep convolutional networks. IEEE Photonics J.
**2017**, 9, 1–13. [Google Scholar] [CrossRef] - He, Z.; Cao, Y.; Dong, Y.; Yang, J.; Cao, Y.; Tisse, C.L. Single-image-based nonuniformity correction of uncooled long-wave infrared detectors: a deep-learning approach. Appl. Opt.
**2018**, 57, D155–D164. [Google Scholar] [CrossRef] - Zeng, Q.; Qin, H.; Yan, X.; Yang, S.; Yang, T. Single Infrared Image-Based Stripe Nonuniformity Correction via a Two-Stage Filtering Method. Sensors
**2018**, 18, 4299. [Google Scholar] [CrossRef] - Black, W.T.; Tyo, J.S. Feedback-integrated scene cancellation scene-based nonuniformity correction algorithm. J. Electron. Imaging
**2014**, 23, 023005. [Google Scholar] [CrossRef] - Li, Z.; Zheng, J.; Zhu, Z.; Yao, W.; Wu, S. Weighted Guided Image Filtering. IEEE Trans. Image Process.
**2015**, 24, 120–129. [Google Scholar] - He, K.; Sun, J.; Tang, X. Guided Image Filtering. IEEE Trans. Pattern Anal. Mach. Intell.
**2013**, 35, 1397–1409. [Google Scholar] [CrossRef] - Li, Z.; Zheng, J. Edge-preserving decomposition-based single image haze removal. IEEE Trans. Image Process.
**2015**, 24, 5432–5441. [Google Scholar] [CrossRef] - Li, Z.; Zheng, J. Detail-Enhanced Multi-Scale Exposure Fusion. IEEE Trans. Image Process.
**2017**, 26, 1243–1252. [Google Scholar] [CrossRef] - Kou, F.; Chen, W.; Wen, C.; Li, Z. Gradient Domain Guided Image Filtering. IEEE Trans. Image Process.
**2015**, 24, 4528–4539. [Google Scholar] [CrossRef] - Zhao, J.; Zhou, Q.; Chen, Y.; Liu, T.; Feng, H.; Xu, Z.; Li, Q. Single image stripe nonuniformity correction with gradient-constrained optimization model for infrared focal plane arrays. Opt. Commun.
**2013**, 296, 47–52. [Google Scholar] [CrossRef]

**Figure 1.**Example of nonuniformity with the two-point method (this image is of size 320 × 240 and was obtained by an uncooled infrared detector): (

**a**) The source image, (

**b**) the corrected image that was obtained via the two-point method, and (

**c**) the corrected result that was obtained via the two-point method after 0.5 hours.

**Figure 2.**Gradient map and weighted map. (

**a1–a3**) IR images with stripe nonuniformity. (

**b1–b3**) Gradient maps ${G}_{t}$ of the original images. (

**c1–c3**) 3D gradient maps of the original images. (

**d1–d3**) Weighted maps of the original images that were obtained using the kernel function.

**Figure 4.**Results that were obtained by various filters. (

**a**) The original image, (

**b**) WGIF using our proposed weight map, (

**c**) WGIF using the Sobel gradient operator weight (

**d**) WGIF using the weight in [26] (

**e**) GIF (

**f**) GGIF.

**Figure 6.**Results of stripe nonuniformity estimation. (

**a1–a2**) The original image, (

**b1–b2**) the ideal image, and (

**c1–c2**) the stripe nonuniformity image.

**Figure 7.**Temporal profile of the estimated nonuniformity. (

**a**) The estimated stripe nonuniformity sequence and (

**b**) the temporal profile of two pixels in sequence.

**Figure 8.**Results of nonlinear diffusion equation filtering. (

**a**) A pixel containing the nonuniformity and (

**b**) a pixel containing the moving scene.

**Figure 9.**Four raw IR image sequences for testing. (

**a**) A walking girl, (

**b**) a moving small target against a sky background, (

**c**) a plant, and (

**d**) windows.

**Figure 10.**NUC results that were obtained using various weights. (

**a1–a4**) The original image, (

**b1–b4**) the proposed weight, (

**c1–c4**) the weight of [26], and (

**d1–d4**) the Sobel weight.

**Figure 12.**NUC results of five methods. (

**a1–a4**) The original image, (

**b1–b4**) the TVRNN NN result, (

**c1–c4**) the BF-THPF result, (

**d1–d4**) the MIRE result, (

**e1–e4**) the CNN result, and (

**f1–f4**) the result of the proposed method.

**Figure 14.**Curves of roughness and nonuniformity for Sequence 1 and Sequence 2. (

**a1,a2**) Nonuniformity and (

**b1,b2**) roughness.

Method | Parameter Settings |
---|---|

BTHPF-NUC | The size of the filter window: D = 4; the two standard deviation parameters: σ_{d} = 7 and σ_{r} = 30; the time constant: T = 3 |

TVRNN NN-NUC | The spatial average kernel size: 9 × 9; iterative step: $\mu =2\times {10}^{-7}$ |

MIRE NUC | Regulation parameter: s = 1; the window size: 8 × s. |

CNN NUC | Trained CNN in the literature [23] |

Proposed method | w_{d} = 5, σ_{1} = 0.003, σ_{2} = 10, r = 20, α = −0.8, and m = 10 |

Method | The 100th Frame of Seq. 1 | The 50th Frame of Seq. 2 | The 50th Frame of Seq. 3 | The 200th Frame of Seq. 4 | ||||
---|---|---|---|---|---|---|---|---|

U/% | ρ/% | U/% | ρ/% | U/% | ρ/% | U/% | ρ/% | |

Original image | 11.12 | 16.94 | 2.36 | 3.88 | 5.66 | 5.34 | 8.48 | 10.61 |

BTHPF-NUC | 6.95 | 8.52 | 1.11 | 1.33 | 4.11 | 3.73 | 6.06 | 5.61 |

TVRNN NN-NUC | 10.16 | 15.47 | 1.79 | 2.85 | 5.47 | 5.08 | 7.99 | 9.05 |

MIRE NUC | 10.34 | 16.56 | 2.11 | 3.68 | 5.52 | 5.41 | 7.87 | 10.37 |

CNN NUC | 11.47 | 17.62 | 2.25 | 3.87 | 5.58 | 5.33 | 8.51 | 10.83 |

Proposed method | 6.53 | 3.61 | 0.91 | 0.56 | 3.72 | 1.55 | 5.91 | 1.66 |

© 2019 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Li, J.; Qin, H.; Yan, X.; Zeng, Q.; Yang, T.
Temporal-Spatial Nonlinear Filtering for Infrared Focal Plane Array Stripe Nonuniformity Correction. *Symmetry* **2019**, *11*, 673.
https://doi.org/10.3390/sym11050673

**AMA Style**

Li J, Qin H, Yan X, Zeng Q, Yang T.
Temporal-Spatial Nonlinear Filtering for Infrared Focal Plane Array Stripe Nonuniformity Correction. *Symmetry*. 2019; 11(5):673.
https://doi.org/10.3390/sym11050673

**Chicago/Turabian Style**

Li, Jia, Hanlin Qin, Xiang Yan, Qingjie Zeng, and Tingwu Yang.
2019. "Temporal-Spatial Nonlinear Filtering for Infrared Focal Plane Array Stripe Nonuniformity Correction" *Symmetry* 11, no. 5: 673.
https://doi.org/10.3390/sym11050673