Next Article in Journal
Analysis of the Multi-Steps Package (MSP) for Series-Connected SiC-MOSFETs
Previous Article in Journal
Optical Camera Communications: Principles, Modulations, Potential and Challenges
Article

Performance Prediction for Convolutional Neural Network on Spark Cluster

Department of Computer Science and Engineering, Korea University, Seoul 02841, Korea
*
Author to whom correspondence should be addressed.
Electronics 2020, 9(9), 1340; https://doi.org/10.3390/electronics9091340
Received: 22 July 2020 / Revised: 11 August 2020 / Accepted: 15 August 2020 / Published: 19 August 2020
(This article belongs to the Section Computer Science & Engineering)
Applications with large-scale data are processed on a distributed system, such as Spark, as they are data- and computation-intensive. Predicting the performance of such applications is difficult, because they are influenced by various aspects of configurations from the distributed framework level to the application level. In this paper, we propose a completion time prediction model based on machine learning for the representative deep learning model convolutional neural network (CNN) by analyzing the effects of data, task, and resource characteristics on performance when executing the model in Spark cluster. To reduce the time utilized in collecting the data for training the model, we consider the causal relationship between the model features and the completion time based on Spark CNN’s distributed data-parallel model. The model features include the configurations of the Data Center OS Mesos environment, configurations of Apache Spark, and configurations of the CNN model. By applying the proposed model to famous CNN implementations, we achieved 99.98% prediction accuracy about estimating the job completion time. In addition to the downscale search area for the model features, we leverage extrapolation, which significantly reduces the model build time at most to 89% with even better prediction accuracy in comparison to the actual work. View Full-Text
Keywords: convolutional neural network; feature engineering; machine learning; performance prediction convolutional neural network; feature engineering; machine learning; performance prediction
Show Figures

Figure 1

MDPI and ACS Style

Myung, R.; Yu, H. Performance Prediction for Convolutional Neural Network on Spark Cluster. Electronics 2020, 9, 1340. https://doi.org/10.3390/electronics9091340

AMA Style

Myung R, Yu H. Performance Prediction for Convolutional Neural Network on Spark Cluster. Electronics. 2020; 9(9):1340. https://doi.org/10.3390/electronics9091340

Chicago/Turabian Style

Myung, Rohyoung; Yu, Heonchang. 2020. "Performance Prediction for Convolutional Neural Network on Spark Cluster" Electronics 9, no. 9: 1340. https://doi.org/10.3390/electronics9091340

Find Other Styles
Note that from the first issue of 2016, MDPI journals use article numbers instead of page numbers. See further details here.

Article Access Map by Country/Region

1
Search more from Scilit
 
Search
Back to TopTop