Detailed Information

Cited 0 time in webofscience Cited 0 time in scopus
Metadata Downloads

Performance Prediction for Convolutional Neural Network on Spark Cluster

Authors
Myung, RohyoungYu, Heonchang
Issue Date
9월-2020
Publisher
MDPI
Keywords
convolutional neural network; feature engineering; machine learning; performance prediction
Citation
ELECTRONICS, v.9, no.9
Indexed
SCIE
SCOPUS
Journal Title
ELECTRONICS
Volume
9
Number
9
URI
https://scholar.korea.ac.kr/handle/2021.sw.korea/53232
DOI
10.3390/electronics9091340
ISSN
2079-9292
Abstract
Applications with large-scale data are processed on a distributed system, such as Spark, as they are data- and computation-intensive. Predicting the performance of such applications is difficult, because they are influenced by various aspects of configurations from the distributed framework level to the application level. In this paper, we propose a completion time prediction model based on machine learning for the representative deep learning model convolutional neural network (CNN) by analyzing the effects of data, task, and resource characteristics on performance when executing the model in Spark cluster. To reduce the time utilized in collecting the data for training the model, we consider the causal relationship between the model features and the completion time based on Spark CNN's distributed data-parallel model. The model features include the configurations of the Data Center OS Mesos environment, configurations of Apache Spark, and configurations of the CNN model. By applying the proposed model to famous CNN implementations, we achieved 99.98% prediction accuracy about estimating the job completion time. In addition to the downscale search area for the model features, we leverage extrapolation, which significantly reduces the model build time at most to 89% with even better prediction accuracy in comparison to the actual work.
Files in This Item
There are no files associated with this item.
Appears in
Collections
Graduate School > Department of Computer Science and Engineering > 1. Journal Articles

qrcode

Items in ScholarWorks are protected by copyright, with all rights reserved, unless otherwise indicated.

Related Researcher

Researcher YU, Heon chang photo

YU, Heon chang
컴퓨터학과
Read more

Altmetrics

Total Views & Downloads

BROWSE