TY - GEN
T1 - STEAMEngine
T2 - 18th International Conference on High Performance Computing, HiPC 2011
AU - Cardosa, Michael
AU - Narang, Piyush
AU - Chandra, Abhishek
AU - Pucha, Himabindu
AU - Singh, Aameek
PY - 2011
Y1 - 2011
N2 - MapReduce has gained in popularity as a distributed data analysis paradigm, particularly in the cloud, where MapReduce jobs are run on virtual clusters. The provisioning of MapReduce jobs in the cloud is an important problem for optimizing several user as well as provider-side metrics, such as runtime, cost, throughput, energy, and load. In this paper, we present an intelligent provisioning framework called STEAMEngine that consists of provisioning algorithms to optimize these metrics through a set of common building blocks. These building blocks enable spatio-temporal tradeoffs unique to MapReduce provisioning: along with their resource requirements (spatial component), a MapReduce job runtime (temporal component) is a critical element for any provisioning algorithm. We also describe tw o novel provisioning algorithms a user-driven performance optimization and a provider-driven energy optimization that leverage these building blocks. Our experimental results based on an Amazon EC2 cluster and a local Xen/Hadoop cluster show the benefits of STEAMEngine through improvements in performance and energy via the use of these algorithms and building blocks.
AB - MapReduce has gained in popularity as a distributed data analysis paradigm, particularly in the cloud, where MapReduce jobs are run on virtual clusters. The provisioning of MapReduce jobs in the cloud is an important problem for optimizing several user as well as provider-side metrics, such as runtime, cost, throughput, energy, and load. In this paper, we present an intelligent provisioning framework called STEAMEngine that consists of provisioning algorithms to optimize these metrics through a set of common building blocks. These building blocks enable spatio-temporal tradeoffs unique to MapReduce provisioning: along with their resource requirements (spatial component), a MapReduce job runtime (temporal component) is a critical element for any provisioning algorithm. We also describe tw o novel provisioning algorithms a user-driven performance optimization and a provider-driven energy optimization that leverage these building blocks. Our experimental results based on an Amazon EC2 cluster and a local Xen/Hadoop cluster show the benefits of STEAMEngine through improvements in performance and energy via the use of these algorithms and building blocks.
UR - http://www.scopus.com/inward/record.url?scp=84863411965&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=84863411965&partnerID=8YFLogxK
U2 - 10.1109/HiPC.2011.6152649
DO - 10.1109/HiPC.2011.6152649
M3 - Conference contribution
AN - SCOPUS:84863411965
SN - 9781457719516
T3 - 18th International Conference on High Performance Computing, HiPC 2011
BT - 18th International Conference on High Performance Computing, HiPC 2011
PB - IEEE Computer Society
Y2 - 18 December 2011 through 21 December 2011
ER -