Next Article in Journal
A Robust Indoor Positioning Method based on Bluetooth Low Energy with Separate Channel Information
Previous Article in Journal
Transparent Film-Type Vibrotactile Actuator Array and Its Haptic Rendering Using Beat Phenomenon
Open AccessArticle

Practical Considerations for Accuracy Evaluation in Sensor-Based Machine Learning and Deep Learning

Department of Electrical and Computer Engineering, Dalhousie University, Halifax, NS B3H 4R2, Canada
*
Author to whom correspondence should be addressed.
Sensors 2019, 19(16), 3491; https://doi.org/10.3390/s19163491
Received: 4 July 2019 / Revised: 5 August 2019 / Accepted: 7 August 2019 / Published: 9 August 2019
(This article belongs to the Section Intelligent Sensors)
  |  
PDF [1847 KB, uploaded 9 August 2019]
  |  

Abstract

Accuracy evaluation in machine learning is based on the split of data into a training set and a test set. This critical step is applied to develop machine learning models including models based on sensor data. For sensor-based problems, comparing the accuracy of machine learning models using the train/test split provides only a baseline comparison in ideal situations. Such comparisons won’t consider practical production problems that can impact the inference accuracy such as the sensors’ thermal noise, performance with lower inference quantization, and tolerance to sensor failure. Therefore, this paper proposes a set of practical tests that can be applied when comparing the accuracy of machine learning models for sensor-based problems. First, the impact of the sensors’ thermal noise on the models’ inference accuracy was simulated. Machine learning algorithms have different levels of error resilience to thermal noise, as will be presented. Second, the models’ accuracy using lower inference quantization was compared. Lowering inference quantization leads to lowering the analog-to-digital converter (ADC) resolution which is cost-effective in embedded designs. Moreover, in custom designs, analog-to-digital converters’ (ADCs) effective number of bits (ENOB) is usually lower than the ideal number of bits due to various design factors. Therefore, it is practical to compare models’ accuracy using lower inference quantization. Third, the models’ accuracy tolerance to sensor failure was evaluated and compared. For this study, University of California Irvine (UCI) ‘Daily and Sports Activities’ dataset was used to present these practical tests and their impact on model selection. View Full-Text
Keywords: ADC; deep learning; edge artificial intelligence (AI); ENOB; machine learning; low power; low quantization; sensor failure; sensor fusion; thermal noise ADC; deep learning; edge artificial intelligence (AI); ENOB; machine learning; low power; low quantization; sensor failure; sensor fusion; thermal noise
Figures

Graphical abstract

This is an open access article distributed under the Creative Commons Attribution License which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited (CC BY 4.0).
SciFeed

Share & Cite This Article

MDPI and ACS Style

Hammad, I.; El-Sankary, K. Practical Considerations for Accuracy Evaluation in Sensor-Based Machine Learning and Deep Learning. Sensors 2019, 19, 3491.

Show more citation formats Show less citations formats

Note that from the first issue of 2016, MDPI journals use article numbers instead of page numbers. See further details here.

Related Articles

Article Metrics

Article Access Statistics

1

Comments

[Return to top]
Sensors EISSN 1424-8220 Published by MDPI AG, Basel, Switzerland RSS E-Mail Table of Contents Alert
Back to Top