Next Article in Journal
The Blood Cytokine Profile of Young People with Early Ischemic Heart Disease Comorbid with Abdominal Obesity
Next Article in Special Issue
Reframing Psychiatry for Precision Medicine
Previous Article in Journal
Current Biomarkers for Alzheimer’s Disease: From CSF to Blood
Previous Article in Special Issue
The Autonomic Nervous System Differentiates between Levels of Motor Intent and End Effector
Open AccessArticle

Precision Telemedicine through Crowdsourced Machine Learning: Testing Variability of Crowd Workers for Video-Based Autism Feature Recognition

1
Department of Bioengineering, Stanford University, 443 Via Ortega, Stanford, CA 94305, USA
2
Department of Pediatrics (Systems Medicine), Stanford University, 1265 Welch Rd., Stanford, CA 94305, USA
3
Department of Biomedical Data Science, Stanford University, 1265 Welch Rd., Stanford, CA 94305, USA
4
Department of Neuroscience, Stanford University, 213 Quarry Rd., Stanford, CA 94305, USA
5
Department of Computer Science, Stanford University, 353 Jane Stanford Way, Stanford, CA 94305, USA
6
School of Education, Stanford University, 485 Lasuen Mall, Stanford, CA 94305, USA
*
Author to whom correspondence should be addressed.
J. Pers. Med. 2020, 10(3), 86; https://doi.org/10.3390/jpm10030086
Received: 29 June 2020 / Revised: 9 August 2020 / Accepted: 10 August 2020 / Published: 13 August 2020
Mobilized telemedicine is becoming a key, and even necessary, facet of both precision health and precision medicine. In this study, we evaluate the capability and potential of a crowd of virtual workers—defined as vetted members of popular crowdsourcing platforms—to aid in the task of diagnosing autism. We evaluate workers when crowdsourcing the task of providing categorical ordinal behavioral ratings to unstructured public YouTube videos of children with autism and neurotypical controls. To evaluate emerging patterns that are consistent across independent crowds, we target workers from distinct geographic loci on two crowdsourcing platforms: an international group of workers on Amazon Mechanical Turk (MTurk) (N = 15) and Microworkers from Bangladesh (N = 56), Kenya (N = 23), and the Philippines (N = 25). We feed worker responses as input to a validated diagnostic machine learning classifier trained on clinician-filled electronic health records. We find that regardless of crowd platform or targeted country, workers vary in the average confidence of the correct diagnosis predicted by the classifier. The best worker responses produce a mean probability of the correct class above 80% and over one standard deviation above 50%, accuracy and variability on par with experts according to prior studies. There is a weak correlation between mean time spent on task and mean performance (r = 0.358, p = 0.005). These results demonstrate that while the crowd can produce accurate diagnoses, there are intrinsic differences in crowdworker ability to rate behavioral features. We propose a novel strategy for recruitment of crowdsourced workers to ensure high quality diagnostic evaluations of autism, and potentially many other pediatric behavioral health conditions. Our approach represents a viable step in the direction of crowd-based approaches for more scalable and affordable precision medicine. View Full-Text
Keywords: crowdsourcing; machine learning; diagnostics; telemedicine; autism; pediatrics crowdsourcing; machine learning; diagnostics; telemedicine; autism; pediatrics
Show Figures

Figure 1

MDPI and ACS Style

Washington, P.; Leblanc, E.; Dunlap, K.; Penev, Y.; Kline, A.; Paskov, K.; Sun, M.W.; Chrisman, B.; Stockham, N.; Varma, M.; Voss, C.; Haber, N.; Wall, D.P. Precision Telemedicine through Crowdsourced Machine Learning: Testing Variability of Crowd Workers for Video-Based Autism Feature Recognition. J. Pers. Med. 2020, 10, 86. https://doi.org/10.3390/jpm10030086

AMA Style

Washington P, Leblanc E, Dunlap K, Penev Y, Kline A, Paskov K, Sun MW, Chrisman B, Stockham N, Varma M, Voss C, Haber N, Wall DP. Precision Telemedicine through Crowdsourced Machine Learning: Testing Variability of Crowd Workers for Video-Based Autism Feature Recognition. Journal of Personalized Medicine. 2020; 10(3):86. https://doi.org/10.3390/jpm10030086

Chicago/Turabian Style

Washington, Peter; Leblanc, Emilie; Dunlap, Kaitlyn; Penev, Yordan; Kline, Aaron; Paskov, Kelley; Sun, Min W.; Chrisman, Brianna; Stockham, Nathaniel; Varma, Maya; Voss, Catalin; Haber, Nick; Wall, Dennis P. 2020. "Precision Telemedicine through Crowdsourced Machine Learning: Testing Variability of Crowd Workers for Video-Based Autism Feature Recognition" J. Pers. Med. 10, no. 3: 86. https://doi.org/10.3390/jpm10030086

Find Other Styles
Note that from the first issue of 2016, MDPI journals use article numbers instead of page numbers. See further details here.

Article Access Map by Country/Region

1
Search more from Scilit
 
Search
Back to TopTop