Dense Out-of-Distribution Detection by Robust Learning on Synthetic Negative Data
Abstract
:1. Introduction
2. Related Work
2.1. Anomaly Detection
2.2. Classification in the Presence of Outliers
2.3. Open-Set Recognition
2.4. Generative Models for Synthetic Negative Data
3. Dense OOD Detection with NFlowJS
3.1. Training with Synthetic Negative Data
3.2. Loss in Synthetic Negative Pixels
3.3. Outlier-Aware Inference with Divergence-Based Scoring
3.4. Coverage-Oriented Generation of Synthetic Negatives
4. Experimental Setup
4.1. Benchmarks and Datasets
4.2. Metrics
4.3. Implementation Details
5. Experimental Evaluation
5.1. Dense Out-of-Distribution Detection in Road-Driving Scenes
Method | SegmentMeIfYouCan [11] | Fishyscapes [12] | |||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Aux | Img | Anomalies | Obstacles | LAF-nK | FS LAF | FS Static | CS val | ||||||
Data | Rsyn. | AP | AP | AP | AP | AP | |||||||
SynBoost [19] | ✓ | ✓ | 56.4 | 61.9 | 71.3 | 3.2 | 81.7 | 4.6 | 43.2 | 15.8 | 72.6 | 18.8 | 81.4 |
Prior Entropy [92] | ✓ | ✗ | - | - | - | - | - | - | 34.3 | 47.4 | 31.3 | 84.6 | 70.5 |
OOD Head [21] | ✓ | ✗ | - | - | - | - | - | - | 31.3 | 19.0 | 96.8 | 0.3 | 79.6 |
Void Classifier [12] | ✓ | ✗ | 36.6 | 63.5 | 10.4 | 41.5 | 4.8 | 47.0 | 10.3 | 22.1 | 45.0 | 19.4 | 70.4 |
Image Resyn. [14] | ✗ | ✓ | 52.3 | 25.9 | 37.7 | 4.7 | 57.1 | 8.8 | 5.7 | 48.1 | 29.6 | 27.1 | 81.4 |
Road Inpaint. [20] | ✗ | ✓ | - | - | 54.1 | 47.1 | 82.9 | 35.8 | - | - | - | - | - |
Max softmax [40] | ✗ | ✗ | 28.0 | 72.1 | 15.7 | 16.6 | 30.1 | 33.2 | 1.8 | 44.9 | 12.9 | 39.8 | 80.3 |
MC Dropout [18] | ✗ | ✗ | 28.9 | 69.5 | 4.9 | 50.3 | 36.8 | 35.6 | - | - | - | - | - |
ODIN [32] | ✗ | ✗ | 33.1 | 71.7 | 22.1 | 15.3 | 52.9 | 30.0 | - | - | - | - | - |
SML [91] | ✗ | ✗ | - | - | - | - | - | - | - | 31.7 | 21.9 | 52.1 | 20.5 |
Embed. Dens. [12] | ✗ | ✗ | 37.5 | 70.8 | 0.8 | 46.4 | 61.7 | 10.4 | 4.3 | 47.2 | 62.1 | 17.4 | 80.3 |
JSRNet [15] | ✗ | ✗ | 33.6 | 43.9 | 28.1 | 28.9 | 74.2 | 6.6 | - | - | - | - | - |
NFlowJS (ours) | ✗ | ✗ | 56.9 | 34.7 | 85.5 | 0.4 | 89.3 | 0.7 | 39.4 | 9.0 | 52.1 | 15.4 | 77.4 |
5.2. Dense Out-of-Distribution Detection in Remote Sensing
5.3. Sensitivity of OOD Detection to Depth
5.4. Inference Speed
5.5. Visualization of Synthetic Outliers
5.6. Synthetic Negatives and Separation in the Feature Space
6. Ablations
6.1. Impacts of the Loss Function and OOD Score
6.2. Impact of the Choice of Generative Model
6.3. Impact of Pre-Training
6.4. Impact of Temperature Scaling
7. Conclusions
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Conflicts of Interest
References
- Everingham, M.; Gool, L.V.; Williams, C.K.I.; Winn, J.M.; Zisserman, A. The Pascal Visual Object Classes (VOC) Challenge. Int. J. Comput. Vis. 2010, 88, 303–338. [Google Scholar] [CrossRef]
- Farabet, C.; Couprie, C.; Najman, L.; LeCun, Y. Learning Hierarchical Features for Scene Labeling. IEEE Trans. Pattern Anal. Mach. Intell. 2013, 35, 1915–1929. [Google Scholar] [CrossRef] [PubMed]
- Cheng, B.; Collins, M.D.; Zhu, Y.; Liu, T.; Huang, T.S.; Adam, H.; Chen, L. Panoptic-DeepLab: A Simple, Strong, and Fast Baseline for Bottom-Up Panoptic Segmentation. In Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA, 13–19 June 2020; pp. 12472–12482. [Google Scholar] [CrossRef]
- Godard, C.; Aodha, O.M.; Firman, M.; Brostow, G.J. Digging Into Self-Supervised Monocular Depth Estimation. In Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision, Seoul, Republic of Korea, 27 October–2 November 2019; pp. 3827–3837. [Google Scholar]
- Luc, P.; Couprie, C.; LeCun, Y.; Verbeek, J. Predicting Future Instance Segmentation by Forecasting Convolutional Features. In Proceedings of the 15th European Conference on Computer Vision, ECCV, Munich, Germany, 8–14 September 2018. [Google Scholar]
- Uhlemeyer, S.; Rottmann, M.; Gottschalk, H. Towards Unsupervised Open World Semantic Segmentation. In Proceedings of the the 38th Conference on Uncertainty in Artificial Intelligence, Eindhoven, The Netherlands, 1–5 August 2022. [Google Scholar]
- Orsic, M.; Segvic, S. Efficient semantic segmentation with pyramidal fusion. Pattern Recognit. 2021, 110, 107611. [Google Scholar] [CrossRef]
- Guo, C.; Pleiss, G.; Sun, Y.; Weinberger, K.Q. On Calibration of Modern Neural Networks. In Proceedings of the 34th International Conference on Machine Learning, ICML, Sydney, Australia, 6–11 August 2017. [Google Scholar]
- Zendel, O.; Honauer, K.; Murschitz, M.; Steininger, D.; Dominguez, G.F. WildDash—Creating Hazard-Aware Benchmarks. In Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany, 8–14 September 2018. [Google Scholar]
- Sakaridis, C.; Dai, D.; Van Gool, L. ACDC: The Adverse Conditions Dataset with Correspondences for Semantic Driving Scene Understanding. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Online, 11–17 October 2021. [Google Scholar]
- Chan, R.; Lis, K.; Uhlemeyer, S.; Blum, H.; Honari, S.; Siegwart, R.; Salzmann, M.; Fua, P.; Rottmann, M. SegmentMeIfYouCan: A Benchmark for Anomaly Segmentation. arXiv 2021, arXiv:2104.14812. [Google Scholar]
- Blum, H.; Sarlin, P.E.; Nieto, J.; Siegwart, R.; Cadena, C. The Fishyscapes Benchmark: Measuring Blind Spots in Semantic Segmentation. Int. J. Comput. Vis. 2021, 129, 3119–3135. [Google Scholar] [CrossRef]
- Bevandić, P.; Krešo, I.; Oršić, M.; Šegvić, S. Dense open-set recognition based on training with noisy negative images. Image Vis. Comput. 2022, 124, 104490. [Google Scholar] [CrossRef]
- Lis, K.; Nakka, K.K.; Fua, P.; Salzmann, M. Detecting the Unexpected via Image Resynthesis. In Proceedings of the International Conference on Computer Vision, ICCV, Seoul, Republic of Korea, 27 October–2 November 2019. [Google Scholar]
- Vojir, T.; Šipka, T.; Aljundi, R.; Chumerin, N.; Reino, D.O.; Matas, J. Road Anomaly Detection by Partial Image Reconstruction With Segmentation Coupling. In Proceedings of the International Conference on Computer Vision, ICCV, Montreal, QC, Canada, 10–17 October 2021. [Google Scholar]
- De Carvalho, O.L.F., Jr.; de Carvalho Júnior, O.A.; Silva, C.R.; de Albuquerque, A.O.; Santana, N.C.; Borges, D.L.; Gomes, R.A.T.; Guimarães, R.F. Panoptic Segmentation Meets Remote Sensing. Remote Sens. 2022, 14, 965. [Google Scholar] [CrossRef]
- Da Silva, C.C.V.; Nogueira, K.; Oliveira, H.N.; dos Santos, J.A. Towards Open-Set Semantic Segmentation of Aerial Images. In Proceedings of the 2020 IEEE Latin American GRSS & ISPRS Remote Sensing Conference (LAGIRS), Santiago, Chile, 22–26 March 2020. [Google Scholar]
- Kendall, A.; Gal, Y. What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision? In Proceedings of the Neural Information Processing Systems, Long Beach, CA, USA, 4–9 December 2017. [Google Scholar]
- Biase, G.D.; Blum, H.; Siegwart, R.; Cadena, C. Pixel-Wise Anomaly Detection in Complex Driving Scenes. In Proceedings of the Computer Vision and Pattern Recognition, CVPR, Nashville, TN, USA, 20–25 June 2021. [Google Scholar]
- Lis, K.; Honari, S.; Fua, P.; Salzmann, M. Detecting Road Obstacles by Erasing Them. IEEE Trans. Pattern Anal. Mach. Intell. 2020, 1–11. [Google Scholar] [CrossRef] [PubMed]
- Bevandic, P.; Kreso, I.; Orsic, M.; Segvic, S. Simultaneous Semantic Segmentation and Outlier Detection in Presence of Domain Shift. In Proceedings of the 41st DAGM German Conference, DAGM GCPR, Dortmund, Germany, 10–13 September 2019. [Google Scholar] [CrossRef]
- van Amersfoort, J.; Smith, L.; Jesson, A.; Key, O.; Gal, Y. On Feature Collapse and Deep Kernel Learning for Single Forward Pass Uncertainty. arXiv 2021, arXiv:2102.11409. [Google Scholar]
- Perera, P.; Morariu, V.I.; Jain, R.; Manjunatha, V.; Wigington, C.; Ordonez, V.; Patel, V.M. Generative-Discriminative Feature Representations for Open-Set Recognition. In Proceedings of the Computer Vision and Pattern Recognition, CVPR, Seattle, WA, USA, 13–19 June 2020. [Google Scholar]
- González, C.; Gotkowski, K.; Fuchs, M.; Bucher, A.; Dadras, A.; Fischbach, R.; Kaltenborn, I.J.; Mukhopadhyay, A. Distance-based detection of out-of-distribution silent failures for COVID-19 lung lesion segmentation. Med. Image Anal. 2022, 82, 102596. [Google Scholar] [CrossRef]
- Gawlikowski, J.; Saha, S.; Kruspe, A.M.; Zhu, X.X. Towards Out-of-Distribution Detection for Remote Sensing. In Proceedings of the IEEE International Geoscience and Remote Sensing Symposium, IGARSS, Brussels, Belgium, 11–16 July 2021; pp. 8676–8679. [Google Scholar]
- Lee, K.; Lee, H.; Lee, K.; Shin, J. Training Confidence-calibrated Classifiers for Detecting Out-of-Distribution Samples. In Proceedings of the 6th International Conference on Learning Representations, ICLR.
- Dinh, L.; Sohl-Dickstein, J.; Bengio, S. Density estimation using Real NVP. In Proceedings of the 5th International Conference on Learning Representations, ICLR, Toulon, France, 24–26 April 2017. [Google Scholar]
- Grcić, M.; Bevandić, P.; Šegvić, S. Dense Open-set Recognition with Synthetic Outliers Generated by Real NVP. In Proceedings of the 16th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications, VISIGRAPP, Online, 8–10 February 2021. [Google Scholar]
- Hawkins, D.M. Identification of Outliers; Monographs on Applied Probability and Statistics; Springer: Dordrecht, The Netherlands, 1980. [Google Scholar]
- Ruff, L.; Kauffmann, J.R.; Vandermeulen, R.A.; Montavon, G.; Samek, W.; Kloft, M.; Dietterich, T.G.; Müller, K. A Unifying Review of Deep and Shallow Anomaly Detection. Proc. IEEE 2021, 109, 756–795. [Google Scholar] [CrossRef]
- Zhang, L.H.; Goldstein, M.; Ranganath, R. Understanding Failures in Out-of-Distribution Detection with Deep Generative Models. In Proceedings of the 38th International Conference on Machine Learning, ICML, Online, 18–24 July 2021. [Google Scholar]
- Liang, S.; Li, Y.; Srikant, R. Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks. In Proceedings of the 6th International Conference on Learning Representations, ICLR, Vancouver, BC, Canada, 30 April–3 May 2018. [Google Scholar]
- Hendrycks, D.; Mazeika, M.; Dietterich, T.G. Deep Anomaly Detection with Outlier Exposure. In Proceedings of the 7th International Conference on Learning Representations, ICLR, New Orleans, LO, USA, 6–9 May 2019. [Google Scholar]
- Zhou, K.; Li, J.; Xiao, Y.; Yang, J.; Cheng, J.; Liu, W.; Luo, W.; Liu, J.; Gao, S. Memorizing Structure-Texture Correspondence for Image Anomaly Detection. IEEE Trans. Neural Netw. Learn. Syst. 2022, 33, 2335–2349. [Google Scholar] [CrossRef]
- Yang, Z.; Zhang, T.; Bozchalooi, I.S.; Darve, E. Memory-Augmented Generative Adversarial Networks for Anomaly Detection. IEEE Trans. Neural Netw. Learn. Syst. 2022, 33, 2324–2334. [Google Scholar] [CrossRef]
- Li, W.; Mahadevan, V.; Vasconcelos, N. Anomaly Detection and Localization in Crowded Scenes. IEEE Trans. Pattern Anal. Mach. Intell. 2014, 36, 18–32. [Google Scholar] [PubMed]
- Bergmann, P.; Batzner, K.; Fauser, M.; Sattlegger, D.; Steger, C. The MVTec Anomaly Detection Dataset: A Comprehensive Real-World Dataset for Unsupervised Anomaly Detection. Int. J. Comput. Vis. 2021, 129, 1038–1059. [Google Scholar] [CrossRef]
- Wang, X.; Che, Z.; Jiang, B.; Xiao, N.; Yang, K.; Tang, J.; Ye, J.; Wang, J.; Qi, Q. Robust Unsupervised Video Anomaly Detection by Multipath Frame Prediction. IEEE Trans. Neural Netw. Learn. Syst. 2022, 33, 2301–2312. [Google Scholar] [CrossRef]
- Massoli, F.V.; Falchi, F.; Kantarci, A.; Akti, S.; Ekenel, H.K.; Amato, G. MOCCA: Multilayer One-Class Classification for Anomaly Detection. IEEE Trans. Neural Netw. Learn. Syst. 2022, 33, 2313–2323. [Google Scholar] [CrossRef]
- Hendrycks, D.; Gimpel, K. A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks. In Proceedings of the 5th International Conference on Learning Representations, ICLR, Toulon, France, 24–26 April 2017. [Google Scholar]
- Macêdo, D.; Ren, T.I.; Zanchettin, C.; Oliveira, A.L.I.; Ludermir, T.B. Entropic Out-of-Distribution Detection: Seamless Detection of Unknown Examples. IEEE Trans. Neural Netw. Learn. Syst. 2022, 33, 2350–2364. [Google Scholar] [CrossRef] [PubMed]
- Dhamija, A.R.; Günther, M.; Boult, T.E. Reducing Network Agnostophobia. In Proceedings of the Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, Montréal, QC, Canada, 3–8 December 2018. [Google Scholar]
- Lucas, T.; Shmelkov, K.; Alahari, K.; Schmid, C.; Verbeek, J. Adaptive Density Estimation for Generative Models. In Proceedings of the Neural Information Processing Systems, Vancouver, BC, Canada, 8–14 December 2019. [Google Scholar]
- Du, X.; Wang, Z.; Cai, M.; Li, Y. VOS: Learning What You Don’t Know by Virtual Outlier Synthesis. In Proceedings of the Tenth International Conference on Learning Representations, ICLR 2022, Online, 25–29 April 2022. [Google Scholar]
- Zhao, Z.; Cao, L.; Lin, K. Revealing the Distributional Vulnerability of Discriminators by Implicit Generators. IEEE Trans. Pattern Anal. Mach. Intell. 2023, 45, 8888–8901. [Google Scholar] [CrossRef]
- Kumar, N.; Segvic, S.; Eslami, A.; Gumhold, S. Normalizing Flow based Feature Synthesis for Outlier-Aware Object Detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR, Vancouver, BC, Canada, 18–22 June 2023. [Google Scholar]
- Blum, H.; Sarlin, P.; Nieto, J.I.; Siegwart, R.; Cadena, C. Fishyscapes: A Benchmark for Safe Semantic Segmentation in Autonomous Driving. In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, Seoul, Republic of Korea, 27–28 October 2019. [Google Scholar]
- Du, X.; Wang, X.; Gozum, G.; Li, Y. Unknown-Aware Object Detection: Learning What You Don’t Know from Videos in the Wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, 18–24 June 2022; pp. 13668–13678. [Google Scholar]
- Riedlinger, T.; Rottmann, M.; Schubert, M.; Gottschalk, H. Gradient-Based Quantification of Epistemic Uncertainty for Deep Object Detectors. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, WACV 2023, Waikoloa, HI, USA, 2–7 January 2023; pp. 3910–3920. [Google Scholar]
- Xia, Y.; Zhang, Y.; Liu, F.; Shen, W.; Yuille, A.L. Synthesize Then Compare: Detecting Failures and Anomalies for Semantic Segmentation. In Proceedings of the 16th European Conference on Computer Vision, ECCV, Online, 23–28 August 2020. [Google Scholar]
- Chan, R.; Rottmann, M.; Gottschalk, H. Entropy Maximization and Meta Classification for Out-of-Distribution Detection in Semantic Segmentation. In Proceedings of the International Conference on Computer Vision, ICCV, Montreal, QC, Canada, 10–17 October 2021. [Google Scholar]
- Scheirer, W.J.; de Rezende Rocha, A.; Sapkota, A.; Boult, T.E. Toward Open Set Recognition. IEEE Trans. Pattern Anal. Mach. Intell. 2013, 35, 1757–1772. [Google Scholar] [CrossRef]
- Bendale, A.; Boult, T.E. Towards Open Set Deep Networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR, Las Vegas, NV, USA, 27–30 June 2016. [Google Scholar]
- Zhang, H.; Li, A.; Guo, J.; Guo, Y. Hybrid Models for Open Set Recognition. In Proceedings of the 16th European Conference on Computer Vision ECCV, Online, 23–28 August 2020. [Google Scholar]
- Oliveira, H.; Silva, C.; Machado, G.L.S.; Nogueira, K.; dos Santos, J.A. Fully convolutional open set segmentation. Mach. Learn. 2021, 112, 1733–1784. [Google Scholar] [CrossRef]
- Scheirer, W.J.; Jain, L.P.; Boult, T.E. Probability Models for Open Set Recognition. IEEE Trans. Pattern Anal. Mach. Intell. 2014, 36, 2317–2324. [Google Scholar] [CrossRef]
- Cen, J.; Yun, P.; Cai, J.; Wang, M.Y.; Liu, M. Deep Metric Learning for Open World Semantic Segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, BC, Canada, 11–17 October 2021; pp. 15313–15322. [Google Scholar]
- Chen, G.; Peng, P.; Wang, X.; Tian, Y. Adversarial Reciprocal Points Learning for Open Set Recognition. IEEE Trans. Pattern Anal. Mach. Intell. 2022, 44, 8065–8081. [Google Scholar] [CrossRef] [PubMed]
- Hendrycks, D.; Basart, S.; Mazeika, M.; Mostajabi, M.; Steinhardt, J.; Song, D. Scaling out-of-distribution detection for real-world settings. arXiv 2019, arXiv:1911.11132. [Google Scholar]
- Vaze, S.; Han, K.; Vedaldi, A.; Zisserman, A. Open-Set Recognition: A Good Closed-Set Classifier is All You Need. In Proceedings of the the Tenth International Conference on Learning Representations, ICLR 2022, Online, 25–29 April 2022. [Google Scholar]
- Neal, L.; Olson, M.L.; Fern, X.Z.; Wong, W.; Li, F. Open Set Learning with Counterfactual Images. In Proceedings of the European Conference on Computer Vision, Munich, Germany, 8–14 September 2018. [Google Scholar]
- Kong, S.; Ramanan, D. OpenGAN: Open-Set Recognition Via Open Data Generation. In Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision (ICCV), Montreal, QC, Canada, 10–17 October 2021. [Google Scholar] [CrossRef]
- Boult, T.E.; Cruz, S.; Dhamija, A.R.; Günther, M.; Henrydoss, J.; Scheirer, W.J. Learning and the Unknown: Surveying Steps toward Open World Recognition. In Proceedings of the AAAI Conference on Artificial Intelligence, Honolulu, HI, USA, 27 January–1 February 2019. [Google Scholar]
- Geng, C.; Huang, S.; Chen, S. Recent Advances in Open Set Recognition: A Survey. IEEE Trans. Pattern Anal. Mach. Intell. 2021, 43, 3614–3631. [Google Scholar] [CrossRef] [PubMed]
- Brilhadoz, A.; Gutoski, M.; Lazzaretti, A.E.; Lopes, H.S. A Comparative Study for Open Set Semantic Segmentation Methods. In Proceedings of the XV Congresso Brasileiro de Inteligência Computacional, Joinville, Brasil, 3–6 de October 2021. [Google Scholar]
- Michieli, U.; Zanuttigh, P. Knowledge distillation for incremental learning in semantic segmentation. Comput. Vis. Image Underst. 2021, 205, 103167. [Google Scholar] [CrossRef]
- Yu, Y.; Ji, Z.; Guo, J.; Pang, Y. Transductive Zero-Shot Learning with Adaptive Structural Embedding. IEEE Trans. Neural Netw. Learn. Syst. 2018, 29, 4116–4127. [Google Scholar] [CrossRef] [PubMed]
- Shaban, A.; Bansal, S.; Liu, Z.; Essa, I.; Boots, B. One-Shot Learning for Semantic Segmentation. In Proceedings of the British Machine Vision Conference, BMVC, London, UK, 4–7 September 2017. [Google Scholar]
- Lu, J.; Jin, S.; Liang, J.; Zhang, C. Robust Few-Shot Learning for User-Provided Data. IEEE Trans. Neural Netw. Learn. Syst. 2021, 32, 1433–1447. [Google Scholar] [CrossRef] [PubMed]
- Salakhutdinov, R.; Hinton, G. Deep Boltzmann Machines. In Proceedings of the Twelth International Conference on Artificial Intelligence and Statistics, Clearwater Beach, FL, USA, 16–18 April 2009. [Google Scholar]
- Van Oord, A.; Kalchbrenner, N.; Kavukcuoglu, K. Pixel recurrent neural networks. In Proceedings of the International Conference on Machine Learning, New York, NY, USA, 20–22 June 2016. [Google Scholar]
- Kingma, D.P.; Welling, M. Auto-Encoding Variational Bayes. In Proceedings of the International Conference on Learning Representations, ICLR, Banff, AB, Canada, 14–16 April 2014. [Google Scholar]
- Vahdat, A.; Kautz, J. NVAE: A Deep Hierarchical Variational Autoencoder. In Proceedings of the Neural Information Processing Systems, Online, 6–12 December 2020. [Google Scholar]
- Goodfellow, I.J.; Pouget-Abadie, J.; Mirza, M.; Xu, B.; Warde-Farley, D.; Ozair, S.; Courville, A.C.; Bengio, Y. Generative adversarial networks. Commun. ACM 2020, 63, 139–144. [Google Scholar] [CrossRef]
- Grcić, M.; Grubišić, I.; Šegvić, S. Densely connected normalizing flows. In Proceedings of the Neural Information Processing Systems, Online, 6–14 December 2021. [Google Scholar]
- Kingma, D.P.; Dhariwal, P. Glow: Generative Flow with Invertible 1 × 1 Convolutions. In Proceedings of the Neural Information Processing Systems, Montreal, QC, Canada, 3–8 December 2018. [Google Scholar]
- Metz, L.; Poole, B.; Pfau, D.; Sohl-Dickstein, J. Unrolled Generative Adversarial Networks. In Proceedings of the 5th International Conference on Learning Representations, Toulon, France, 24–26 April 2017. [Google Scholar]
- Maag, K.; Chan, R.; Uhlemeyer, S.; Kowol, K.; Gottschalk, H. Two Video Data Sets for Tracking and Retrieval of Out of Distribution Objects. In Proceedings of the Asian Conference on Computer Vision, Macau, China, 4–8 December 2022. [Google Scholar]
- Bevandic, P.; Kreso, I.; Orsic, M.; Segvic, S. Discriminative out-of-distribution detection for semantic segmentation. arXiv 2018, arXiv:1808.07703. [Google Scholar]
- Pinggera, P.; Ramos, S.; Gehrig, S.; Franke, U.; Rother, C.; Mester, R. Lost and Found: Detecting small road hazards for self-driving vehicles. In Proceedings of the International Conference on Intelligent Robots and Systems, IROS, Daejeon, Republic of Korea, 9–14 October 2016. [Google Scholar]
- Grcic, M.; Bevandic, P.; Segvic, S. DenseHybrid: Hybrid Anomaly Detection for Dense Open-set Recognition. In Proceedings of the European Conference on Computer Vision, Tel Aviv, Israel, 23–27 October 2022. [Google Scholar]
- Vaze, S.; Han, K.; Vedaldi, A.; Zisserman, A. The Semantic Shift Benchmark. In Proceedings of the ICML 2022 Shift Happens Workshop, Baltimore, MD, USA, 22 July 2022. [Google Scholar]
- Zhu, X.X.; Hu, J.; Qiu, C.; Shi, Y.; Kang, J.; Mou, L.; Bagheri, H.; Häberle, M.; Hua, Y.; Huang, R.; et al. So2Sat LCZ42: A Benchmark Dataset for Global Local Climate Zones Classification. arXiv 2019, arXiv:1912.12171. [Google Scholar]
- Davis, J.; Goadrich, M. The relationship between Precision-Recall and ROC curves. In Proceedings of the International Conference on Machine Learning (ICML 2006), Pittsburgh, PA, USA, 25–29 June 2006. [Google Scholar]
- Kreso, I.; Krapac, J.; Segvic, S. Efficient Ladder-Style DenseNets for Semantic Segmentation of Large Images. IEEE Trans. Intell. Transp. Syst. 2021, 22, 4951–4961. [Google Scholar] [CrossRef]
- Cordts, M.; Omran, M.; Ramos, S.; Rehfeld, T.; Enzweiler, M.; Benenson, R.; Franke, U.; Roth, S.; Schiele, B. The Cityscapes Dataset for Semantic Urban Scene Understanding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR, Las Vegas, NV, USA, 27–30 June 2016. [Google Scholar]
- Neuhold, G.; Ollmann, T.; Bulò, S.R.; Kontschieder, P. The Mapillary Vistas Dataset for Semantic Understanding of Street Scenes. In Proceedings of the IEEE International Conference on Computer Vision, ICCV, Venice, Italy, 22–29 October 2017. [Google Scholar]
- Bulò, S.R.; Porzi, L.; Kontschieder, P. In-Place Activated BatchNorm for Memory-Optimized Training of DNNs. In Proceedings of the Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18–23 June 2018. [Google Scholar]
- Barron, J.T. A General and Adaptive Robust Loss Function. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA, 15–20 June 2019; pp. 4331–4339. [Google Scholar]
- Zhu, Y.; Sapra, K.; Reda, F.A.; Shih, K.J.; Newsam, S.D.; Tao, A.; Catanzaro, B. Improving Semantic Segmentation via Video Propagation and Label Relaxation. In Proceedings of the Conference on Computer Vision and Pattern Recognition, CVPR, Long Beach, CA, USA, 15–20 June 2019. [Google Scholar]
- Jung, S.; Lee, J.; Gwak, D.; Choi, S.; Choo, J. Standardized Max Logits: A Simple yet Effective Approach for Identifying Unexpected Road Obstacles in Urban-Scene Segmentation. In Proceedings of the International Conference on Computer Vision, ICCV, Online, 11–17 October 2021. [Google Scholar]
- Malinin, A.; Gales, M.J.F. Predictive Uncertainty Estimation via Prior Networks. In Proceedings of the Annual Conference on Neural Information Processing Systems, Montréal, QC, Canada, 2–8 December 2018. [Google Scholar]
- Liu, W.; Wang, X.; Owens, J.D.; Li, Y. Energy-based Out-of-distribution Detection. In Proceedings of the Thirty-Fourth Annual Conference on Neural Information Processing Systems NeurIPS, Online, 6–12 December 2020. [Google Scholar]
- Franchi, G.; Bursuc, A.; Aldea, E.; Dubuisson, S.; Bloch, I. TRADI: Tracking Deep Neural Network Weight Distributions. In Proceedings of the 16th European Conference on Computer Vision, ECCV, Online, 23–28 August 2020. [Google Scholar]
- Franchi, G.; Bursuc, A.; Aldea, E.; Dubuisson, S.; Bloch, I. One Versus all for deep Neural Network Incertitude (OVNNI) quantification. arXiv 2020, arXiv:2006.00954. [Google Scholar]
- Sun, Y.; Guo, C.; Li, Y. ReAct: Out-of-distribution Detection With Rectified Activations. In Proceedings of the Thirty-Fifth Annual Conference on Neural Information Processing Systems NeurIPS, Online, 6–14 December 2021. [Google Scholar]
- Stefano, C.D.; Sansone, C.; Vento, M. To reject or not to reject: That is the question-an answer in case of neural classifiers. IEEE Trans. Syst. Man Cybern. Part C 2000, 30, 84–94. [Google Scholar] [CrossRef]
Method | Aux Data | AP ↑ | ↓ | AUROC ↑ |
---|---|---|---|---|
OOD head [21] | ✓ | 34.9 ± 6.8 | 40.9 ± 3.9 | 88.8 ± 1.6 |
Max-softmax [21] | ✓ | 33.8 ± 5.1 | 35.5 ± 3.4 | 91.1 ± 1.0 |
Void Classifier [12] | ✓ | 25.6 ± 5.5 | 44.2 ± 4.7 | 87.7 ± 1.7 |
MC Dropout [18] | ✗ | 9.7 ± 1.2 | 41.1 ± 3.7 | 86.0 ± 1.2 |
ODIN [32] | ✗ | 6.0 ± 0.5 | 53.7 ± 7.0 | 79.9 ± 1.5 |
Max-softmax [40] | ✗ | 5.0 ± 0.5 | 48.8 ± 4.7 | 78.7 ± 1.5 |
NFlowJS (ours) | ✗ | 30.2 ± 4.1 | 32.3 ± 5.9 | 92.3 ± 1.3 |
Method | Aux. | Anomaly Det. | Closed | Open | ||
---|---|---|---|---|---|---|
Data | AP | o- | ||||
SynthCP [50] | ✗ | 9.3 | 28.4 | - | - | - |
TRADI [94] | ✗ | 7.2 | 25.3 | - | - | - |
OVNNI [95] | ✗ | 12.6 | 22.2 | 54.6 | - | - |
Synthetic Outliers + Entropy [28] | ✗ | 12.7 | 25.2 | 59.7 | - | - |
Deep Metric Learning [57] | ✗ | 14.7 | 17.3 | - | - | - |
Max-softmax [40] | ✗ | 7.5 | 27.9 | 65.0 | 32.4 | 35.1 |
Max logit [59] | ✗ | 11.6 | 22.5 | 65.0 | 38.0 | 41.2 |
ODIN [32] | ✗ | 7.0 | 28.7 | 65.0 | - | 28.8 |
ReAct [96] | ✗ | 10.9 | 21.2 | 62.7 | 31.8 | 34.0 |
Energy [93] | ✓ | 12.9 | 18.2 | 63.3 | 39.6 | 42.7 |
Max-softmax + Outlier Exposure [33] | ✓ | 14.6 | 17.7 | 61.7 | 40.8 | 43.8 |
Outlier Head [21] | ✓ | 19.7 | 56.2 | 66.6 | - | 33.9 |
Outlier Head*Max Softmax [13] | ✓ | 18.8 | 30.9 | 66.6 | - | 43.6 |
NFlowJS (ours) | ✗ | 22.2 | 16.2 | 65.0 | 41.6 | 44.9 |
Method | Aux. Data | AP | AUROC | |
---|---|---|---|---|
Max-softmax [40] | ✗ | 35.1 | 13.5 | 96.4 |
Max-softmax + Outlier Exposure [33] | ✓ | 32.2 | 9.6 | 97.0 |
Energy [93] | ✓ | 38.1 | 11.0 | 96.8 |
GAN negatives [26] | ✗ | 31.7 | 9.2 | 96.9 |
MOP [17] | ✗ | 24.5 | 10.9 | 96.0 |
Dirichlet Prior Network [25] | ✓ | 27.3 | 9.1 | 97.1 |
JSDiv (ours) | ✓ | 38.4 | 12.5 | 96.5 |
NFlowJS (ours) | ✗ | 44.1 | 8.8 | 97.8 |
Range (in Meters) | NFlowJS | MSP | ML | SynBoost | OOD-Head | ODIN |
---|---|---|---|---|---|---|
5–10 | 0.7 | 16.6 | 4.7 | 0.2 | 7.9 | 10.9 |
10–15 | 1.2 | 18.0 | 7.3 | 17.7 | 10.6 | 9.0 |
15–20 | 0.8 | 19.3 | 5.9 | 25.0 | 16.9 | 11.1 |
20–25 | 1.1 | 23.2 | 5.8 | 23.3 | 23.6 | 13.4 |
25–30 | 1.8 | 28.0 | 7.1 | 18.8 | 26.7 | 16.6 |
30–35 | 2.7 | 32.6 | 7.6 | 27.4 | 30.8 | 22.6 |
35–40 | 3.5 | 37.9 | 10.1 | 25.4 | 36.8 | 25.9 |
40–45 | 5.6 | 41.4 | 13.2 | 25.8 | 42.2 | 30.3 |
45–50 | 8.8 | 46.3 | 15.8 | 29.9 | 52.0 | 37.9 |
Method | Resynth. | Infer. Time (ms) | FPS |
---|---|---|---|
SynthCP [50] | ✓ | 146.9 | 6.8 |
SynBoost [19] | ✓ | 1055.5 | <1 |
LDN-121 (Base) [85] | ✗ | 46.5 | 21.5 |
Base + ODIN [32] | ✗ | +149.1 | 5.11 |
Base + MC = 2 Dropout [18] | ✗ | +45.8 | 10.83 |
Base + NFlowJS (ours) | ✗ | +7.8 | 18.4 |
Loss | AnomalyTrack-val | ObstacleTrack-val | |||
---|---|---|---|---|---|
AP | AP | ||||
KL | MSP | 57.5 ± 0.7 | 29.0 ± 1.7 | 95.1 ± 0.2 | 0.2 ± 0.1 |
KL | KL | 55.7 ± 0.4 | 26.3 ± 1.3 | 94.3 ± 0.2 | 0.1 ± 0.0 |
RKL | RKL | 57.0 ± 0.3 | 28.9 ± 0.3 | 94.4 ± 0.1 | 0.3 ± 0.0 |
JSD | MSP | 63.0 ± 0.5 | 22.8 ± 0.7 | 96.1 ± 0.2 | 0.2 ± 0.0 |
JSD | JSD | 63.3 ± 0.6 | 19.8 ± 0.8 | 95.8 ± 0.2 | 0.1 ± 0.0 |
Generator | AnomalyTrack-val | ObstacleTrack-val | ||
---|---|---|---|---|
AP | AP | |||
GMM-VOS | 56.7 ± 0.2 | 28.0 ± 0.4 | 81.8 ± 0.5 | 3.9 ± 0.2 |
GAN | 56.1 ± 0.4 | 26.1 ± 0.6 | 80.8 ± 0.4 | 3.6 ± 0.1 |
NFlow | 61.4 ± 0.8 | 21.7 ± 1.3 | 94.9 ± 0.1 | 0.1 ± 0.1 |
Cls. | Flow | AnomalyTrack-val | ObstacleTrack-val | ||
---|---|---|---|---|---|
AP | AP | ||||
✗ | ✗ | 56.9 ± 1.2 | 27.8 ± 2.1 | 90.5 ± 0.3 | 1.0 ± 0.1 |
✓ | ✗ | 61.4 ± 0.8 | 21.7 ± 1.3 | 94.9 ± 0.1 | 0.1 ± 0.1 |
✓ | ✓ | 63.3 ± 0.6 | 19.8 ± 0.8 | 95.8 ± 0.2 | 0.1 ± 0.0 |
Temp. | AnomalyTrack-val | ObstacleTrack-val | ||
---|---|---|---|---|
AP | AP | |||
T = 1 | 59.7 ± 0.5 | 40.0 ± 0.8 | 92.6 ± 0.3 | 1.1 ± 0.1 |
T = 1.5 | 62.7 ± 0.6 | 23.7 ± 0.9 | 95.3 ± 0.2 | 0.2 ± 0.0 |
T = 2 | 63.3 ± 0.6 | 19.8 ± 0.8 | 95.8 ± 0.2 | 0.1 ± 0.0 |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2024 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Grcić, M.; Bevandić, P.; Kalafatić, Z.; Šegvić, S. Dense Out-of-Distribution Detection by Robust Learning on Synthetic Negative Data. Sensors 2024, 24, 1248. https://doi.org/10.3390/s24041248
Grcić M, Bevandić P, Kalafatić Z, Šegvić S. Dense Out-of-Distribution Detection by Robust Learning on Synthetic Negative Data. Sensors. 2024; 24(4):1248. https://doi.org/10.3390/s24041248
Chicago/Turabian StyleGrcić, Matej, Petra Bevandić, Zoran Kalafatić, and Siniša Šegvić. 2024. "Dense Out-of-Distribution Detection by Robust Learning on Synthetic Negative Data" Sensors 24, no. 4: 1248. https://doi.org/10.3390/s24041248
APA StyleGrcić, M., Bevandić, P., Kalafatić, Z., & Šegvić, S. (2024). Dense Out-of-Distribution Detection by Robust Learning on Synthetic Negative Data. Sensors, 24(4), 1248. https://doi.org/10.3390/s24041248