Elastic job bundling: An adaptive resource request strategy for large-scale parallel applications

Feng Liu, Jon B. Weissman

Research output: Chapter in Book/Report/Conference proceedingConference contribution

10 Scopus citations

Abstract

In today's batch queue HPC cluster systems, the user submits a job requesting a fixed number of processors. The system will not start the job until all of the requested resources become available simultaneously. When cluster workload is high, large sized jobs will experience long waiting time due to this policy. In this paper, we propose a new approach that dynamically decomposes a large job into smaller ones to reduce waiting time, and lets the application expand across multiple subjobs while continuously achieving progress. This approach has three benefits: (i) application turnaround time is reduced, (ii) system fragmentation is diminished, and (iii) fairness is promoted. Our approach does not depend on job queue time prediction but exploits available backfill opportunities. Simulation results have shown that our approach can reduce application mean turnaround time by up to 48%.

Original languageEnglish (US)
Title of host publicationProceedings of SC 2015
Subtitle of host publicationThe International Conference for High Performance Computing, Networking, Storage and Analysis
PublisherIEEE Computer Society
ISBN (Electronic)9781450337236
DOIs
StatePublished - Nov 15 2015
EventInternational Conference for High Performance Computing, Networking, Storage and Analysis, SC 2015 - Austin, United States
Duration: Nov 15 2015Nov 20 2015

Publication series

NameInternational Conference for High Performance Computing, Networking, Storage and Analysis, SC
Volume15-20-November-2015
ISSN (Print)2167-4329
ISSN (Electronic)2167-4337

Other

OtherInternational Conference for High Performance Computing, Networking, Storage and Analysis, SC 2015
CountryUnited States
CityAustin
Period11/15/1511/20/15

Keywords

  • HPC
  • elasticity
  • parallel job scheduling

Fingerprint Dive into the research topics of 'Elastic job bundling: An adaptive resource request strategy for large-scale parallel applications'. Together they form a unique fingerprint.

Cite this