In the era of rapid experimental expansion data analysis needs are rapidly outpacing the capabilities of small institutional clusters and looking to integrate HPC resources into their workflow. We propose one way of reconciling on-demand needs of experimental analytics with the batch managed HPC resources within a system that dynamically moves nodes between an on-demand cluster configured with cloud technology (OpenStack) and a traditional HPC cluster managed by a batch scheduler (Torque). We evaluate this system experimentally both in the context of real-life traces representing two years of a specific institutional need, and via experiments in the context of synthetic traces that capture generalized characteristics of potential batch and on-demand workloads. Our results for the real-life scenario show that our approach could reduce the current investment in on-demand infrastructure by 82% while at the same time improving the mean batch wait time almost by an order of magnitude (8x).
|Original language||English (US)|
|Title of host publication||Proceedings - International Conference for High Performance Computing, Networking, Storage, and Analysis, SC 2018|
|Publisher||Institute of Electrical and Electronics Engineers Inc.|
|Number of pages||11|
|State||Published - Mar 11 2019|
|Event||2018 International Conference for High Performance Computing, Networking, Storage, and Analysis, SC 2018 - Dallas, United States|
Duration: Nov 11 2018 → Nov 16 2018
|Name||Proceedings - International Conference for High Performance Computing, Networking, Storage, and Analysis, SC 2018|
|Conference||2018 International Conference for High Performance Computing, Networking, Storage, and Analysis, SC 2018|
|Period||11/11/18 → 11/16/18|
Bibliographical noteFunding Information:
This material is based upon work supported by the U.S. Department of Energy, under the DOE-LAB-14-1003 and the NSF under the NSF-1443080 award. Results presented in this paper were obtained using the Chameleon testbed supported by the National Science Foundation.
© 2018 IEEE.
- Computers and information processing
- Distributed computing
- Grid computing