Next Article in Journal
An Experimental Multi-Target Tracking of AM Radio-Based Passive Bistatic Radar System via Multi-Static Doppler Shifts
Previous Article in Journal
Estimation with Uncertainty via Conditional Generative Adversarial Networks
Article

A Systematic Approach for Evaluating Artificial Intelligence Models in Industrial Settings

1
Interdisciplinary Centre for Security, Reliability and Trust, University of Luxembourg, 6 Rue Richard Coudenhove-Kalergi, L-1359 Luxembourg, Luxembourg
2
Cebi Luxembourg S.A, 30 rue J.F. Kennedy, L-7327 Steinsel, Luxembourg
*
Author to whom correspondence should be addressed.
Sensors 2021, 21(18), 6195; https://doi.org/10.3390/s21186195
Received: 25 August 2021 / Revised: 10 September 2021 / Accepted: 11 September 2021 / Published: 15 September 2021
(This article belongs to the Section Intelligent Sensors)
Artificial Intelligence (AI) is one of the hottest topics in our society, especially when it comes to solving data-analysis problems. Industry are conducting their digital shifts, and AI is becoming a cornerstone technology for making decisions out of the huge amount of (sensors-based) data available in the production floor. However, such technology may be disappointing when deployed in real conditions. Despite good theoretical performances and high accuracy when trained and tested in isolation, a Machine-Learning (M-L) model may provide degraded performances in real conditions. One reason may be fragility in treating properly unexpected or perturbed data. The objective of the paper is therefore to study the robustness of seven M-L and Deep-Learning (D-L) algorithms, when classifying univariate time-series under perturbations. A systematic approach is proposed for artificially injecting perturbations in the data and for evaluating the robustness of the models. This approach focuses on two perturbations that are likely to happen during data collection. Our experimental study, conducted on twenty sensors’ datasets from the public University of California Riverside (UCR) repository, shows a great disparity of the models’ robustness under data quality degradation. Those results are used to analyse whether the impact of such robustness can be predictable—thanks to decision trees—which would prevent us from testing all perturbations scenarios. Our study shows that building such a predictor is not straightforward and suggests that such a systematic approach needs to be used for evaluating AI models’ robustness. View Full-Text
Keywords: time series classification; artificial intelligence robustness; industrial internet of things; adversarial time series classification; artificial intelligence robustness; industrial internet of things; adversarial
Show Figures

Figure 1

MDPI and ACS Style

Benedick, P.-L.; Robert, J.; Le Traon, Y. A Systematic Approach for Evaluating Artificial Intelligence Models in Industrial Settings. Sensors 2021, 21, 6195. https://doi.org/10.3390/s21186195

AMA Style

Benedick P-L, Robert J, Le Traon Y. A Systematic Approach for Evaluating Artificial Intelligence Models in Industrial Settings. Sensors. 2021; 21(18):6195. https://doi.org/10.3390/s21186195

Chicago/Turabian Style

Benedick, Paul-Lou, Jérémy Robert, and Yves Le Traon. 2021. "A Systematic Approach for Evaluating Artificial Intelligence Models in Industrial Settings" Sensors 21, no. 18: 6195. https://doi.org/10.3390/s21186195

Find Other Styles
Note that from the first issue of 2016, MDPI journals use article numbers instead of page numbers. See further details here.

Article Access Map by Country/Region

1
Back to TopTop