DIVE-C: Distributed-parallel Virtual Environment on Cloud computing platform

Abstract

In social media services and social network services, it is necessary to collect, analyze and process their big data with low maintenance cost. Therefore, distributed-parallel data processing on cloud platform is getting spotlight as useful solution for them. In this paper, we present a new architecture of DIVE-C: DIstributed-parallel Virtual Environment on Cloud computing platform for distributed parallel data processing applications which offers a transparent virtual computing environment in order to provide a way easy to launch user's distributed parallel applications. It hides the complexity of the cloud, and helps users to focus on their new applications and core services. DIVE-C uses agent-based resource management scheme to configure VM resources and application deployment for offering various distributed-parallel application models. VM resources are automatically provided by unified cloud management layer. Furthermore, an easy-to-use web interface of DIVE-C offers convenience to users. We implemented a prototype of DIVE-C, and its experiment results show the competitive performance of DIVE-C for dynamic resource and virtual computing environment provisioning for various data processing models. © 2013 SERSC.