A Comparative Analysis of Object Detection Metrics with a Companion Open-Source Toolkit

Padilla, Rafael; Passos, Wesley L.; Dias, Thadeu L. B.; Netto, Sergio L.; da Silva, Eduardo A. B.

doi:10.3390/electronics10030279

Submit to this Journal Review for this Journal Edit a Special Issue

Article Menu

...

Open AccessArticle

A Comparative Analysis of Object Detection Metrics with a Companion Open-Source Toolkit

and

Eduardo A. B. da Silva

Electrical Engineering Program/Alberto Luiz Coimbra Institute for Post-Graduation and Research in Engineering (PEE/COPPE), PO Box 68504, Rio de Janeiro 21941-972, RJ, Brazil

Author to whom correspondence should be addressed.

Academic Editor: Tomasz Trzcinski

Electronics 2021, 10(3), 279; https://doi.org/10.3390/electronics10030279

Received: 25 December 2020 / Revised: 15 January 2021 / Accepted: 20 January 2021 / Published: 25 January 2021

(This article belongs to the Special Issue Deep Learning Based Object Detection)

View Full-Text Download PDF

Browse Figures

Citation Export

Abstract

Recent outstanding results of supervised object detection in competitions and challenges are often associated with specific metrics and datasets. The evaluation of such methods applied in different contexts have increased the demand for annotated datasets. Annotation tools represent the location and size of objects in distinct formats, leading to a lack of consensus on the representation. Such a scenario often complicates the comparison of object detection methods. This work alleviates this problem along the following lines: (i) It provides an overview of the most relevant evaluation methods used in object detection competitions, highlighting their peculiarities, differences, and advantages; (ii) it examines the most used annotation formats, showing how different implementations may influence the assessment results; and (iii) it provides a novel open-source toolkit supporting different annotation formats and 15 performance metrics, making it easy for researchers to evaluate the performance of their detection algorithms in most known datasets. In addition, this work proposes a new metric, also included in the toolkit, for evaluating object detection in videos that is based on the spatio-temporal overlap between the ground-truth and detected bounding boxes. View Full-Text

Keywords: object-detection metrics; precision; recall; evaluation; automatic assessment; bounding boxes

▼ Show Figures

Figure 1

This is an open access article distributed under the Creative Commons Attribution License which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited

Share and Cite

MDPI and ACS Style

Padilla, R.; Passos, W.L.; Dias, T.L.B.; Netto, S.L.; da Silva, E.A.B. A Comparative Analysis of Object Detection Metrics with a Companion Open-Source Toolkit. Electronics 2021, 10, 279. https://doi.org/10.3390/electronics10030279

AMA Style

Padilla R, Passos WL, Dias TLB, Netto SL, da Silva EAB. A Comparative Analysis of Object Detection Metrics with a Companion Open-Source Toolkit. Electronics. 2021; 10(3):279. https://doi.org/10.3390/electronics10030279

Chicago/Turabian Style

Padilla, Rafael, Wesley L. Passos, Thadeu L.B. Dias, Sergio L. Netto, and Eduardo A.B. da Silva. 2021. "A Comparative Analysis of Object Detection Metrics with a Companion Open-Source Toolkit" Electronics 10, no. 3: 279. https://doi.org/10.3390/electronics10030279

Find Other Styles

Note that from the first issue of 2016, MDPI journals use article numbers instead of page numbers. See further details here.

Article Metrics

Abstract views Pdf views Html views

Article Access Map by Country/Region

Author Biographies

Rafael Padilla

Rafael Padilla was born in Mogi das Cruzes, São Paulo, Brazil in 1984. He received his Electrical Engineering degree from the Universidade Federal do Amazonas (UFAM) in 2007, and a M.Sc. degree in Electrical Engineering from the Universidade Federal do Amazonas (UFAM) in 2012. He is currently pursuing a D.Sc. in Electrical Engineering from the Universidade Federal do Rio de Janeiro (COPPE/UFRJ). Former evangelist of Nokia and Microsoft technologies, he has professional experience in industrial engineering, test engineering, mobile, and system development. He is a visiting lecturer at universities teaching topics related to programming and mobile development. His research fields include computer vision, image processing, machine learning, and digital signal processing.

Wesley L. Passos

Wesley L. Passos was born in Nilópolis, Rio de Janeiro, Brazil in 1991. He is currently pursuing his D.Sc. in Electrical Engineering from the Universidade Federal do Rio de Janeiro (COPPE/UFRJ). He received his M.Sc. in Electrical Engineering from the Universidade Federal do Rio de Janeiro (COPPE/UFRJ) in 2019, and a B.Sc. in Control and Automation Engineering from Centro Federal de Educação Tecnologica (CEFET/RJ) in 2016. He won the Google Latin America Research Awards (LARA) in 2019 and 2020. His research interests include computer vision, machine learning, and big data.

Thadeu L. B. Dias

Thadeu Luiz Barbosa Dias was born in Rio de Janeiro, Brazil, in 1994. He has a B.Sc. degree in Electronics and Computer Engineering from the Universidade Federal do Rio de Janeiro (POLI/UFRJ) and a M.Sc. degree in Electrical Engineering at COPPE (PEE/COPPE/UFRJ) in 2020. His research interests include digital signal processing, image processing, machine learning, and information geometry.

Sergio L. Netto

Sergio L. Netto was born in Rio de Janeiro, Brazil. He received his B.Sc. (cum laude) degree from the Federal University of Rio de Janeiro (UFRJ), Brazil, in 1991, the M.Sc. degree from COPPE/UFRJ in 1992, and his Ph.D. degree from the University of Victoria, BC, Canada, in 1996, all in electrical engineering. Since 1997, he has been with the Department of Electronics and Computer Engineering, Poli/UFRJ, and since 1998, he has been with the Program of Electrical Engineering, COPPE/UFRJ. He is the co-author (with P. S. R. Diniz and E. A. B. da Silva) of Digital Signal Processing: System Analysis and Design (Cambridge University Press, second edition, 2010). His research and teaching interests lie in areas of digital signal processing, adaptive filtering, speech processing, information theory, computer vision, and machine learning.

Eduardo A. B. da Silva

Eduardo A. B. da Silva was born in Rio de Janeiro, Brazil. He received his Electronics Engineering degree from the Instituto Militar de Engenharia (IME), Brazil, in 1984, a M.Sc. degree in Electrical Engineering from the Universidade Federal do Rio de Janeiro (COPPE/UFRJ) in 1990, and a Ph.D. degree in Electronics from the University of Essex, England, in 1995. He has been a professor at Universidade Federal do Rio de Janeiro since 1989. He is the co-author of the book “Digital Signal Processing: System Analysis and Design”, published by Cambridge University Press. He has published more than 70 papers in international journals. His research interests lie in the fields of signal and image processing, signal compression, 3D videos, computer vision, light fields, and machine learning, together with its applications to telecommunications and the oil and gas industry. He is the co-editor of the future standard ISO/IEC CD 21794-2, JPEG Pleno Plenoptic Image Coding System, and is currently the Requirements Vice Chair of JPEG.

Article Menu

A Comparative Analysis of Object Detection Metrics with a Companion Open-Source Toolkit

Abstract

Share and Cite

Article Metrics

Article Access Statistics

Article Access Map by Country/Region

Author Biographies

Further Information

Guidelines

MDPI Initiatives

Follow MDPI