A statistical evaluation of the impact of parameter selection on storage system benchmarks

Nohhyun Park, Weijun Xiao, Kyubaik Choi, David J. Lilja

Research output: Chapter in Book/Report/Conference proceedingConference contribution

6 Scopus citations

Abstract

The performance evaluation of storage systems is a difficult task due to lack of representative workloads. Storage benchmark tools test a single aspect such as random read performance at a time. While this approach does allow us to gain useful insights to the system behavior, current practices are largely inadequate due to large benchmark input parameter space. For example, a benchmark with 10 parameters require at least 1024 experiments if conducted exhaustively even if we only test extreme values of each parameter. Furthermore, there is no way to tell if the benchmark being used is enough to test all realistic scenarios. As a result, researchers and developers rely on multiple benchmarks with ad hoc input parameters. We propose a method to quickly identify input parameters that have high effect on the performance metric of interest. We also show that using multiple benchmarks is unnecessary at times and a good benchmark can cover all operational space by providing a control over key parameters that affect the performance metric being measured.

Original languageEnglish (US)
Title of host publication2011 IEEE 7th International Workshop on Storage Network Architecture and Parallel I/Os, SNAPI 2011
PublisherIEEE Computer Society
ISBN (Print)9781457704284
StatePublished - Dec 1 2011
Event2011 IEEE 7th International Workshop on Storage Network Architecture and Parallel I/Os, SNAPI 2011 - Denver, CO, United States
Duration: May 25 2011May 25 2011

Other

Other2011 IEEE 7th International Workshop on Storage Network Architecture and Parallel I/Os, SNAPI 2011
CountryUnited States
CityDenver, CO
Period5/25/115/25/11

Fingerprint Dive into the research topics of 'A statistical evaluation of the impact of parameter selection on storage system benchmarks'. Together they form a unique fingerprint.

Cite this