Infrared Image Small-Target Detection Based on Improved FCOS and Spatio-Temporal Features
Abstract
:1. Introduction
- As the infrared small-target detection does not require the high-level semantics of the network, we lighten the FCOS detection network and select only the low-level infrared small target features.
- In order to further improve the response of small targets, the traditional maximum filtering method is added, which helps to improve the detection performance of small infrared targets.
- For the noise points in the moving infrared image, we use the image sequence as the input of the network to add time domain features, establish the connection between the images, and further eliminate the static noise points in the image.
- We conducted a comparative experiment on the published infrared small target dataset to prove the detection performance of the method in this paper. The results show that, compared with other infrared small target detection methods, the improved FCOS method proposed in this paper has better performance for small-target detection and has better real-time performance.
2. Related Work
2.1. Small-Target Detection Methods Basd on Deep Learning
- The methods based on multiscale feature learning, whose main idea is to learn different scale objects separately, which mainly solves the problem of the small target itself with few discriminative features. This type of method is represented by FPN, and its main idea is to integrate the underlying spatial information and high-level semantic information to enhance the target characteristics [17].
- The methods based on the receptive field, whose representative is the Trident Network. The idea is that small targets require smaller receptive fields [18], and large targets need larger receptive fields, and then dilated convolutions with different dilation rates are used to form different feelings that are responsible for detecting three branches of different scale objects.
- The GAN-based methods use GAN to generate high-resolution images or high-resolution features. For example, in [19], the trained detector is used to obtain the subimage containing the target, and then the generator is used to generate the corresponding high-definition image, discriminator is responsible for judging whether the generated image is real or fake, and at the same time acts as a detector to predict the category and location of the target.
- The context-based methods use the relationship between the environment information of the small target and other easy-to-detect targets to assist the detection of small targets, such as Relation Network [20] implicitly modeling the relationship between two targets through Transformer, and use this relationship to strengthen the characteristics of each object.
- The methods based on the dataset itself. For example, in Stitcher [21], the proportion of the loss of the small target in the total loss is used as the feedback signal. When the proportion is less than a certain threshold, the four pictures are combined into one picture as the input for the next iteration, which is equivalent to increasing the number of small targets. Augmentation [22] solves this problem directly by simply copying and pasting.
2.2. Infrared Small-Target Detection Methods
2.3. FCOS Target Detection Algorithm and Its Limitation for Infrared Small-Target Detection
- As the small infrared target occupies very few pixels compared to the whole image, and the architecture of the FCOS algorithm is ResNet+FPN, after multiple downsampling, the feature area occupied by the small target is extremely small, and it is even impossible to perceive learning. Therefore, it is more appropriate to use some high-resolution feature maps for learning, which can strengthen the neural network’s perception of small target areas.
- Balance of positive and negative samples. In the infrared image, the small infrared target occupies fewer feature points, so all the pixels contained in the bbox of the target should be used as positive sample points for training to increase the number of positive samples.
- Noise points in the infrared image have a greater impact on the detection of small infrared targets, but the small infrared targets can be separated according to the characteristics of the small infrared targets in the image sequence. Therefore, it is possible to add the temporal characteristics of the image in the neural network.
3. Infrared Small-Target Detection Based on Improved FCOS
3.1. Image Preprocessing
3.2. Improved FCOS Network
3.3. Adding Spatio-Temporal Features
4. Experiment and Analysis
4.1. Experimental Dataset
- Published dataset of infrared image sequences
- Published single-frame infrared small target image dataset
4.2. Evaluation Indicators and Experimental Settings
4.3. Comparison of Preprocessing Methods
4.4. Image Sequence Length Comparison
4.5. Compartive Experiments of Different Measures
4.6. Comparison of the Method in This Paper and Other Methods
5. Conclusions
Author Contributions
Funding
Conflicts of Interest
References
- Tom, V.T.; Peli, T.; Leung, M.; Bondaryk, J.E. Morphology-based algorithm for point target detection in infrared backgrounds. In Proceedings of the Signal and Data Processing of Small Targets 1993, Orlando, FL, USA, 11–16 April 1993; Drummond, O.E., Ed.; International Society for Optics and Photonics, SPIE: Bellingham, WA, USA, 1993; Volume 1954, pp. 2–11. [Google Scholar] [CrossRef]
- Deshpande, S.D.; Er, M.H.; Venkateswarlu, R.; Chan, P. Max-mean and max-median filters for detection of small targets. In Proceedings of the Signal and Data Processing of Small Targets 1999, Denver, CO, USA, 18–23 July 1999; Drummond, O.E., Ed.; International Society for Optics and Photonics, SPIE: Bellingham, WA, USA, 1999; Volume 3809, pp. 74–83. [Google Scholar] [CrossRef]
- Bae, T.W.; Zhang, F.; Kweon, I.S. Edge directional 2D LMS filter for infrared small-target detection. Infrared Phys. Technol. 2012, 55, 137–145. [Google Scholar] [CrossRef]
- Gao, C.; Meng, D.; Yang, Y.; Wang, Y.; Zhou, X.; Hauptmann, A.G. Infrared Patch-Image Model for Small-Target Detection in a Single Image. IEEE Trans. Image Process. 2013, 22, 4996–5009. [Google Scholar] [CrossRef] [PubMed]
- Girshick, R.; Donahue, J.; Darrell, T.; Malik, J. Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Columbus, OH, USA, 23–28 June 2014. [Google Scholar]
- Girshick, R. Fast R-CNN. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), Santiago, Chile, 7–13 December 2015. [Google Scholar]
- Liu, W.; Anguelov, D.; Erhan, D.; Szegedy, C.; Reed, S.; Fu, C.Y.; Berg, A.C. Ssd: Single shot multibox detector. In Proceedings of the European Conference on Computer Vision, Amsterdam, The Netherlands, 11–14 October 2016; Springer: Cham, Switzerland, 2016; pp. 21–37. [Google Scholar]
- Redmon, J.; Farhadi, A. YOLOv3: An Incremental Improvement. arXiv 2018, arXiv:1804.02767. [Google Scholar]
- Liu, S.; Huang, D.; Wang, A. Receptive Field Block Net for Accurate and Fast Object Detection. In Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany, 8–14 September 2018. [Google Scholar]
- Zhang, S.; Wen, L.; Bian, X.; Lei, Z.; Li, S.Z. Single-Shot Refinement Neural Network for Object Detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA, 18–23 June 2018. [Google Scholar]
- Shi, M.; Wang, H. Infrared Dim and Small-Target Detection Based on Denoising Autoencoder Network. Mob. Netw. Appl. 2020, 25, 1469–1483. [Google Scholar] [CrossRef]
- Zhao, M.; Cheng, L.; Yang, X.; Feng, P.; Liu, L.; Wu, N. TBC-Net: A real-time detector for infrared small-target detection using semantic constraint. arXiv 2020, arXiv:2001.05852. [Google Scholar]
- Ju, M.; Luo, J.; Liu, G.; Luo, H. ISTDet: An efficient end-to-end neural network for infrared small target detection. Infrared Phys. Technol. 2021, 114, 103659. [Google Scholar] [CrossRef]
- Wang, H.; Zhou, L.; Wang, L. Miss detection vs. false alarm: Adversarial learning for small object segmentation in infrared images. In Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Korea, 27 October–2 November 2019; pp. 8509–8518. [Google Scholar]
- Zhao, B.; Wang, C.; Fu, Q.; Han, Z. A novel pattern for infrared small-target detection with generative adversarial network. IEEE Trans. Geosci. Remote Sens. 2020, 59, 4481–4492. [Google Scholar] [CrossRef]
- Tian, Z.; Shen, C.; Chen, H.; He, T. Fcos: Fully convolutional one-stage object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Korea, 27 October–2 November 2019; pp. 9627–9636. [Google Scholar]
- Lin, T.Y.; Dollár, P.; Girshick, R.; He, K.; Hariharan, B.; Belongie, S. Feature pyramid networks for object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, 21–26 July 2017; pp. 2117–2125. [Google Scholar]
- Li, Y.; Chen, Y.; Wang, N.; Zhang, Z. Scale-aware trident networks for object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Korea, 27 October–2 November 2019; pp. 6054–6063. [Google Scholar]
- Bai, Y.; Zhang, Y.; Ding, M.; Ghanem, B. Sod-mtgan: Small object detection via multi-task generative adversarial network. In Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany, 8–14 September 2018; pp. 206–221. [Google Scholar]
- Hu, H.; Gu, J.; Zhang, Z.; Dai, J.; Wei, Y. Relation networks for object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18–23 June 2018; pp. 3588–3597. [Google Scholar]
- Chen, Y.; Zhang, P.; Li, Z.; Li, Y.; Zhang, X.; Meng, G.; Xiang, S.; Sun, J.; Jia, J. Stitcher: Feedback-driven data provider for object detection. arXiv 2020, arXiv:2004.12432. [Google Scholar]
- Kisantal, M.; Wojna, Z.; Murawski, J.; Naruniec, J.; Cho, K. Augmentation for small object detection. arXiv 2019, arXiv:1902.07296. [Google Scholar]
- Singh, B.; Davis, L.S. An analysis of scale invariance in object detection snip. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18–23 June 2018; pp. 3578–3587. [Google Scholar]
- Singh, B.; Najibi, M.; Davis, L.S. Sniper: Efficient multi-scale training. arXiv 2018, arXiv:1805.09300. [Google Scholar]
- Hui, B.; Song, Z.; Fan, H.; Zhong, P.; Hu, W.; Zhang, X.; Ling, J.; Su, H.; Jin, W.; Zhang, Y.; et al. A Dataset for Infrared Detection and Tracking of Dim-Small Aircraft Target under Ground/Air Background; Science Data Bank: Beijing, China, 2019; (In Chinese). [Google Scholar] [CrossRef]
- Dai, Y.; Wu, Y.; Zhou, F.; Barnard, K. Asymmetric contextual modulation for infrared small-target detection. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, Waikoloa, HI, USA, 3–8 January 2021; pp. 950–959. [Google Scholar]
- Dai, Y.; Wu, Y.; Zhou, F.; Barnard, K. Attentional Local Contrast Networks for Infrared Small Target Detection. IEEE Trans. Geosci. Remote Sens. 2021, 59, 9813–9824. [Google Scholar] [CrossRef]
- Huang, S.; Peng, Z.; Wang, Z.; Wang, X.; Li, M. Infrared Small-Target Detection by Density Peaks Searching and Maximum-Gray Region Growing. IEEE Geosci. Remote Sens. Lett. 2019, 16, 1919–1923. [Google Scholar] [CrossRef]
Sequence1 | A Single-Frame Infrared Dataset | |||
---|---|---|---|---|
Precision | Recall | Precision | Recall | |
Top-hat | 0.610 | 0.818 | 0.775 | 0.964 |
Mean-filter | 0.151 | 0.179 | 0.90 | 0.96 |
Enhance-contrast | 0.947 | 0.955 | 0.958 | 0.990 |
Gradient | 0.973 | 0.992 | 0.946 | 0.982 |
MPCM | 0.979 | 0.992 | 0.880 | 0.951 |
Maximum | 0.984 | 0.992 | 0.973 | 0.990 |
Length | Sequence1 | Sequence2 | ||||||||
---|---|---|---|---|---|---|---|---|---|---|
1 | 2 | 3 | 5 | 7 | 1 | 2 | 3 | 5 | 7 | |
precision | 0.982 | 0.980 | 0.984 | 0.973 | 0.960 | 0.989 | 0.989 | 0.989 | 0.988 | 0.979 |
recall | 0.990 | 0.992 | 0.992 | 0.982 | 0.977 | 0.990 | 0.993 | 0.996 | 0.992 | 0.989 |
FPS | 32.9 | 31.2 | 36.2 | 28.9 | 22.7 | 58.6 | 43.6 | 41.2 | 24.1 | 15.0 |
Sequence1 | A Single-Frame Infrared Dataset | |||||
---|---|---|---|---|---|---|
Precision | Recall | FPS | Precision | Recall | FPS | |
Improved FCOS | 0.967 | 0.987 | 30.5 | 0.966 | 0.986 | 25.5 |
Improved FCOS + traditional Filtering | 0.982 | 0.990 | 32.9 | 0.972 | 0.992 | 29.0 |
Improved FCOS + time domain features | 0.978 | 0.990 | 29.9 | - | - | - |
Improved FCOS + time domain features + traditional Filtering | 0.984 | 0.992 | 36.2 | - | - | - |
Detector | YOLO v3 | RFBNet | RefineNet | ALCNet | Density Peaks Searching | FCOS | Proposed | Proposed * |
---|---|---|---|---|---|---|---|---|
precision | 80.33 | 83.74 | 85.17 | 88.34 | 56.31 | 0 | 98.40 | 98.30 |
FPS (GPU) | 35.4 | 34.78 | 27.16 | 7.69 | - | 22.72 | 36.2 | 35.5 |
FPS (CPU) | 1.39 | 17.6 | 5.4 | 2.64 | 12.56 | 0.5 | 19.0 | 16.0 |
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |
© 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Yao, S.; Zhu, Q.; Zhang, T.; Cui, W.; Yan, P. Infrared Image Small-Target Detection Based on Improved FCOS and Spatio-Temporal Features. Electronics 2022, 11, 933. https://doi.org/10.3390/electronics11060933
Yao S, Zhu Q, Zhang T, Cui W, Yan P. Infrared Image Small-Target Detection Based on Improved FCOS and Spatio-Temporal Features. Electronics. 2022; 11(6):933. https://doi.org/10.3390/electronics11060933
Chicago/Turabian StyleYao, Shengbo, Qiuyu Zhu, Tao Zhang, Wennan Cui, and Peimin Yan. 2022. "Infrared Image Small-Target Detection Based on Improved FCOS and Spatio-Temporal Features" Electronics 11, no. 6: 933. https://doi.org/10.3390/electronics11060933
APA StyleYao, S., Zhu, Q., Zhang, T., Cui, W., & Yan, P. (2022). Infrared Image Small-Target Detection Based on Improved FCOS and Spatio-Temporal Features. Electronics, 11(6), 933. https://doi.org/10.3390/electronics11060933