The exigency of benchmark and compiler drift: Designing tomorrow's processors with yesterday's tools

Joshua J. Yi, Hans Vandierendonck, Lieven Eeckhout, David J. Lilja

Research output: Chapter in Book/Report/Conference proceedingConference contribution

10 Scopus citations

Abstract

Due to the amount of time required to design a new processor, one set of benchmark programs may be used during the design phase while another may be the standard when the design is finally delivered. Using one benchmark suite to design a processor while using a different, presumably more current, suite to evaluate its ultimate performance may lead to sub-optimal design decisions if there are large differences between the characteristics of the two suites and their respective compilers. We call this changes across time "drift". To evaluate the impact of using yesterday's benchmark and compiler technology to design tomorrow's processors, we compare common benchmarks from the SPEC 95 and SPEC 2000 benchmark suites. Our results yield three key conclusions. First, we show that the amount of drift, for common programs in successive SPEC benchmark suites, is significant. In SPEC 2000, the main memory access time is a far more significant performance bottleneck than in SPEC 95, while less significant SPEC 2000 performance bottlenecks include the L2 cache latency, the L1 I-cache size, and the number of reorder buffer entries. Second, using two different statistical techniques, we show that compiler drift is not as significant as benchmark drift. Third, we show that benchmark and compiler drift can have a significant impact on the final design decisions. Specifically, we use a one-parameter-at-a-time optimization algorithm to design two different year-2000 processors, one optimized for SPEC 95 and the other optimized for SPEC 2000, using the energy-delay product (EDP) as the optimization criterion. The results show that using SPEC 95 to design a year-2000 processor results in an 18.5% larger EDP and a 20.8% higher CPI than using the SPEC 2000 benchmarks to design the corresponding processor. Finally, we make a few recommendations to help computer architects minimize the effects of benchmark and compiler drift.

Original languageEnglish (US)
Title of host publicationProceedings of the 20th Annual International Conference on Supercomputing, ICS 2006
Pages75-86
Number of pages12
DOIs
StatePublished - Dec 1 2006
Event20th Annual International Conference on Supercomputing, ICS 2006 - Cairns, Queensland, Australia
Duration: Jun 28 2006Jul 1 2006

Publication series

NameProceedings of the International Conference on Supercomputing

Conference

Conference20th Annual International Conference on Supercomputing, ICS 2006
Country/TerritoryAustralia
CityCairns, Queensland
Period6/28/067/1/06

Keywords

  • Benchmark drift
  • Compiler drift
  • Microprocessor design

Fingerprint

Dive into the research topics of 'The exigency of benchmark and compiler drift: Designing tomorrow's processors with yesterday's tools'. Together they form a unique fingerprint.

Cite this