SPEC does not certify results

Nothing is more fun than arguing with BM Seer. He usually helps me more than anyone in keeping everyone at Sun in compliance with SPEC’s fair use rules. But in a recent posting on SPECweb2005 for Sun SPARC Enterprise T5220 he refers to SPEC published results as "Certified." Actually as the official SPEC disclaimer spells out, "the contents of any SPEC reporting page are the
submitter’s responsibility. SPEC makes no warranties about the accuracy
or veracity of this data."

Most SPEC benchmark results can be used without SPEC review. They must comply with all the run and reporting rules, including the requirement for a full disclosure report. And their rules compliance can be challenged on the basis of the details in that report. There is a real value to readers, and hence to vendors, of publishing a result at spec.org. Such results are peer reviewed by other SPEC committee members including competitors, prior to publication. If a result is found to be not in compliance with the run rules it is not published, and the result cannot be used elsewhere either. However, passing this review is not a guarantee or certification that the result is accurate.

Instead of a paid independent audit process, SPEC relies on full disclosure and peer review to increase confidence in the reliability of results. From the details in the full disclosure report anyone should be able to reproduce the performance experiment and obtain substantially the same results. From time to time competitors will conduct such replication experiments on each others’ systems, and if they cannot get the same number they bring it to SPEC to either get some details of the test configuration that were erroneously left out of the full disclosure report, or to have the published result marked non-compliant. By this method SPEC dramatically lowers the cost of benchmarking, making it possible to have the thousands of results posted on spec.org, while keeping vendors honest by the fear of exposure and humiliation.