Next Article in Journal
Possibility of Using Fermented Curly Kale Juice to Manufacture Feta-Type Cheese
Previous Article in Journal
A Palm-Jacobaeus Loss Formula for Multi-Service Systems with Separated Resources
Article

The Elephant in the Machine: Proposing a New Metric of Data Reliability and its Application to a Medical Case to Assess Classification Reliability

1
Department of Informatics, Sistemics and Communication (DISCo), University of Milano-Bicocca, 20126 Milano, Italy
2
IRCCS Istituto Ortopedico Galeazzi, 20161 Milano, Italy
3
Department of Biomedicine, Neuroscience and Advanced Diagnostics (BIND), University of Palermo, 90133 Palermo, Italy
4
Unit of Radiology, Clinical Institutes Zucchi, 20900 Monza, Italy
5
Diagnostic Imaging Department, Pineta Grande Hospital, 81030 Castel Volturno, Italy
6
Department of Biomedical Sciences for Health, Università degli Studi di Milano, 20122 Milano, Italy
7
Department of Radiology, Ospedale Evangelico Internazionale Genova, 16122 Genova, Italy
8
Department of Medicine and Health Sciences, University of Molise, 86100 Campobasso, Italy
9
Varelli Institute, 80126 Naples, Italy
*
Author to whom correspondence should be addressed.
These authors contributed equally to this work.
Appl. Sci. 2020, 10(11), 4014; https://doi.org/10.3390/app10114014
Received: 29 April 2020 / Revised: 30 May 2020 / Accepted: 4 June 2020 / Published: 10 June 2020
(This article belongs to the Section Computing and Artificial Intelligence)
In this paper, we present and discuss a novel reliability metric to quantify the extent a ground truth, generated in multi-rater settings, as a reliable basis for the training and validation of machine learning predictive models. To define this metric, three dimensions are taken into account: agreement (that is, how much a group of raters mutually agree on a single case); confidence (that is, how much a rater is certain of each rating expressed); and competence (that is, how accurate a rater is). Therefore, this metric produces a reliability score weighted for the raters’ confidence and competence, but it only requires the former information to be actually collected, as the latter can be obtained by the ratings themselves, if no further information is available. We found that our proposal was both more conservative and robust to known paradoxes than other existing agreement measures, by virtue of a more articulated notion of the agreement due to chance, which was based on an empirical estimation of the reliability of the single raters involved. We discuss the above metric within a realistic annotation task that involved 13 expert radiologists in labeling the MRNet dataset. We also provide a nomogram by which to assess the actual accuracy of a classification model, given the reliability of its ground truth. In this respect, we also make the point that theoretical estimates of model performance are consistently overestimated if ground truth reliability is not properly taken into account. View Full-Text
Keywords: inter-rater agreement; reliability; ground truth; machine learning; MRNet; knee; magnetic resonance imaging inter-rater agreement; reliability; ground truth; machine learning; MRNet; knee; magnetic resonance imaging
Show Figures

Figure 1

MDPI and ACS Style

Cabitza, F.; Campagner, A.; Albano, D.; Aliprandi, A.; Bruno, A.; Chianca, V.; Corazza, A.; Di Pietto, F.; Gambino, A.; Gitto, S.; Messina, C.; Orlandi, D.; Pedone, L.; Zappia, M.; Sconfienza, L.M. The Elephant in the Machine: Proposing a New Metric of Data Reliability and its Application to a Medical Case to Assess Classification Reliability. Appl. Sci. 2020, 10, 4014. https://doi.org/10.3390/app10114014

AMA Style

Cabitza F, Campagner A, Albano D, Aliprandi A, Bruno A, Chianca V, Corazza A, Di Pietto F, Gambino A, Gitto S, Messina C, Orlandi D, Pedone L, Zappia M, Sconfienza LM. The Elephant in the Machine: Proposing a New Metric of Data Reliability and its Application to a Medical Case to Assess Classification Reliability. Applied Sciences. 2020; 10(11):4014. https://doi.org/10.3390/app10114014

Chicago/Turabian Style

Cabitza, Federico, Andrea Campagner, Domenico Albano, Alberto Aliprandi, Alberto Bruno, Vito Chianca, Angelo Corazza, Francesco Di Pietto, Angelo Gambino, Salvatore Gitto, Carmelo Messina, Davide Orlandi, Luigi Pedone, Marcello Zappia, and Luca M. Sconfienza. 2020. "The Elephant in the Machine: Proposing a New Metric of Data Reliability and its Application to a Medical Case to Assess Classification Reliability" Applied Sciences 10, no. 11: 4014. https://doi.org/10.3390/app10114014

Find Other Styles
Note that from the first issue of 2016, MDPI journals use article numbers instead of page numbers. See further details here.

Article Access Map by Country/Region

1
Back to TopTop