SPECtacular awards & new web performance/energy benchmark

The last of the 2009 SPECtacular
awards. SPECweb2005 is the
industry standard performance metric for web servers, and today it is
joined by SPECweb2009, the
industry standard performance and energy metric for web servers. The
benchmark includes a banking workload (all SSL), a support workload
(no SSL), and an ecommerce workload (mixed). This is the first
application of the SPECpower
methodology to potentially large system under test
configurations. In the initial
benchmark results you can see one system with and one without
external storage, and the test report lets you see the power
consumption of just the server, of the storage, and of the entire
configuration at various utilization levels. The entire committee did
a fantastic job with this benchmark. As always, I won’t list anyone’s
name without permission. (But give me the okay and I’ll update this
posting!) SPEC recognizes:

Gary
Frost (AMD) who
stepped in to fill a key developer role in an emergency with the
release clock ticking. He took over the control code after a sudden
reassignment, and frankly we handed him quite an undocumented mess.
Gary was up to the challenge and produced the finished code.

Another engineer from AMD
had primary responsibility for the reporting page generator. You
often can’t know exactly what information ought to go into a full
disclosure report (FDR) until you see it. Nor how you want it
organized and arranged. Nor what data integrity cross checks need be
present to avoid errors. So the committee changed requirements often
during development. But no matter how many requirements were placed
on him, he turned around with the needed code within a week!

An engineer from Fujitsu
Technology Solutions became the de facto quality assurance
office because of his thorough and methodical testing practices. If
there are a hundred ways software in general can go wrong, then there
are a thousand ways benchmark software can go wrong, as by its nature
it runs on systems stressed to the limit. When SPEC benchmark
software just works that is largely due to people like this engineer
who forsee, test, and diagnose every possible failure unanticipated
by the authors.

And, if you’d like to see all of the
SPECtacular awards, then follow
the tags!

Alan Adamson is SPECtacular

Another
SPECtacular award from the SPEC annual meeting: Alan
Adamson retired from IBM
where he had been their primary SPEC Representative, held a number of
different elective positions in SPEC, and earned deep respect and
trust from his colleagues. Coming from the IBM
Toronto Software Lab, Alan was a natural to lead SPEC’s Java
committee. Having put that very large committee in smooth running
order, Alan was elected secretary to the Power committee helping it
to produce the first industry standard power performance benchmark.
Meanwhile he led the OSG
steering committee which coordinates activities of all the SPEC OSG
committees.

Alan genuinely cares about the
well-being of SPEC and the people involved. He demonstrates
incredible thoughtfulness and effectiveness in thinking about SPEC’s
benchmark development. He fosters the fun and friendly SPEC culture
where there is always time to share a joke or a funny story if
appropriate. At the same time he creates space for candid discussions
of serious matter. Alan’s leadership and personal effort has been a
big contributor to the success of SPEC.

Alan continues to hold one position in
SPEC, as a director, because members of the board of directors are
elected as individuals, not as companies. Alan serves as a general
chair of the 2010 WOSP/SIPEWInternational Conference on Performance Engineering, a joint
conference of SPEC and ACM which
brings together top academic researchers and industry practitioners
in performance engineering.

You can follow Alan on his blog,
for interesting insights on art, technology, politics, and life -
where he is just as opinionated as ever, just as modest as ever, just
as intolerant of stupidity, and just as tolerant of the people
involved – even when we are opinionated, immodest, and stupid at
times. For all his hard work in SPEC I can think of nobody more
deserving of a relaxing retirement than Alan, and nobody whom we will
miss more than him!

Klaus-Dieter Lange is SPECtacular

Another SPECtacular award from the SPEC annual meeting: Klaus Lange (HP) has become a valuable conduit across different levels of the organization and across benchmark subcommittees, by virtue of becoming indispensable in all of them. Though Klaus is an experienced "SPEC hand" he never forgot what he faced as a newcomer, and took it on himself to organize a new member orientation program to help new institutions integrate into SPEC more easily and effectively. As chair of the SPECpower committee Klaus delivered the industry’s first energy efficiency benchmark, and leads the committee in aiding other groups as they add energy metrics to a wide range of benchmarks. These groups include many SPEC committees as well as other industry consortia. As HP’s representative on the OSG steering committee Klaus has earned respect for his opinions with his diligence and fair mindedness. As a member of the Board of Directors he is often the first to step up to volunteer for important projects, as well as exercising sound judgment in conducting SPEC’s business operations.

John Henning is SPECtacular

Another SPECtacular award from the SPEC
annual meeting: John
Henning of Sun Microsystems is
secretary of the Open Systems Group steering committee. John has been
the driving force behind improvements to our policy
document. This is crucial to efficient operation of the
organization, especially as so many new organizations have joined
SPEC and so many new participants have joined into the work even from
long time SPEC member companies. John is also the one who reminds all
of us to pause in our lecturing and really listen to our adversaries,
the dissident minority voice. Sometimes they have a point that is
valuable to the task at hand, if we only recognize it, and thereby harness all of the energy and creativity of the group.

David Morse is SPECtacular

Another SPECtacular award from the SPEC annual meeting: David Morse (Dell) served as vice-chair and now chair of the Open Systems Group steering committee, his effective organization and leadership of a rather fractious bunch, with successful release of many benchmarks, and formalization of rules and
procedures to put everyone on an even footing with the "good old boys" and reduce risk and uncertainty in members’ use of the benchmarks. Another example of his dedication is his implementation of bookmarkable search extensions to benchmark result queries on spec.org. David is equally comfortable and competent in the most complicated leadership roles and in the most difficult and detailed technical roles.

Paula Smith is SPECtacular

Another
SPECtacular award from the SPEC annual meeting: Paula Smith
(VMware) was honored for her
tireless, competent and patient work managing the SPEC office and the
people there. Paula consistently exhibits what make SPEC an unique
place. The attention and enthusiasm she brings to her volunteer work
make her a pleasure to interact with. She goes above and beyond in
everything she does, and is often able to turn emergencies into
opportunities. Most impressive is how she maintains this over time
and in every interaction, despite many competing pressures for her
attention. Beyond this management work, she also manages to handle
the organizational and technical work of chairing the Virtualization
committee, and of course her day job at VMware.

Cloyce Spradling is SPECtacular

Another SPECtacular award from the
SPEC annual meeting: Cloyce Spradling (Sun
Microsystems) was honored for continued timely support of SPEC
CPU, HPG and editorial tools. The key factor is his timeliness, in
that he responds to unplanned, asynchronous requests, if not with a
solution then at least with a map to help people find their way out
of the woods. And that’s on top of his day job at Sun.

SPEC awards, power performance

More 2009
SPECtacular awards. The SPECpower committee has been busy. They
released version 1.10 of
the SPECpower_ssj2008 benchmark as a no-cost upgrade to existing
licensees. It adds support for measurement of multi-node (blade)
servers, improves usability, and adds a graphical display of power
data during benchmark execution. Review and publication of benchmark
results continues apace, with a spirited competition for first place,
and with ever more power
analyzers accepted for testing, and more test labs qualified for
independent publication. They have also been assisting several other
benchmark committees inside SPEC, and other industry
standard benchmark organizations, to implement energy measurement for
their benchmarks. SPECpower is more than just a benchmark; it is a
methodology,
and the methodology is modified and expanded as necessary over time
to accommodate energy measurements for all the different workloads
which are relevant to the real world in those market segments. In
alphabetical order SPEC recognizes:

• Chris
  Boire (Sun
  Microsystems) – As release manager he coordinated and
  integrated development activities to keep the deliverables on
  schedule.

• David
  Schmidt (HP)
  – He created stand-alone and network integrated tools for
  automated results checking to help insure that results submissions
  are correct and complete.

• Greg Darnell (Dell)
  – Author of the PTDaemon, he helped many other groups get started
  measuring power for their benchmarks. He helps out with whatever
  needs to be done, technical or organizational.

• Hansfried
  Block (Fujitsu
  Technology Solutions) – He automated the process of determining
  power analyzer precision, handled the acceptance of several new
  power analyzers, and was instrumental in getting multi-channel
  analyzers accepted.

• Harry
  Li (Intel)
  – He was primary developer of the Visual Activity Monitor, giving
  an unique view of the system’s activity.

• Jeremy
  Arnold (IBM)
  – If I tried to recount all the accomplishments Jeremy was cited
  for I’d probably run into some internal blog size limit. Suffice it
  to say he is a primary developer on many parts of the code, who
  never turns down a plea for help, and who is never satisfied until
  the entire benchmark package is right.

• Karl
  Huppler (IBM)
  – As primary author/editor of the Power and Performance
  Methodology, he organized the document to capture deep technical
  consensus in the committee, and made it readable and understandable
  for people new to the field.

• Matthew
  Galloway (HP)
  – He designed the control software to drive multiple JVMs,
  enabling multi node (blade) testing.

• An engineer (AMD)
  – Who created and maintained much of the web content explaining
  the benchmark and methodology to the public.

SPEC awards, virtualization

More 2009 SPECtacular awards. SPEC’s
forthcoming virtualization
benchmark will provide meaningful metrics of hardware and
software performance in data center consolidation. As complex as this
benchmark is, running several different benchmarks together in
virtual machines on a host system under test, the code is only half
the story. As with all benchmarks the workload is vital, to represent
realistic usage scenario(s) so that performance improvements made on
the benchmark will also benefit real world users. And the run rules
are vital, needing to accommodate technology improvements over the
lifetime of the benchmark, while precluding unrepresentative
optimizations exploiting rule loopholes. (Or what the layman might
call “cheating”) There is spirited debate from companies
representing rather diverse user communities, all with an interest in
seeing that their customers’ needs are addressed by the benchmark. In
the end when this group of top engineers reaches a consensus you know
they’ve come up with a benchmark that is as rock solid as is possible
to make. From among this great
team of partners and competitors, three were singled out for
SPECtacular awards:

Andrew
Bond of HP always steps forward when
a person is needed to test new code, features, parameter tuning. He
performed many experiments whose results showed the committee the
sensitivity of the benchmark to various parameters, sizes, and
configuration options, so that the right choices could be made for
fair benchmark comparisons. He also created scripts to set up and
configure new guest VMs for each workload.

Chris
Floyd of IBM improved and tailored
the mail server and application server workloads for the new
benchmark. He’s revamped these workloads several times to improve
the I/O profiles and add burstiness to the application server
transaction injection. He helps the other developers at regular
on-line coding sessions, explaining new features, and resolving
problems. He even helps out when on vacation.

Greg
Kopczynski of VMware
developed a (necessarily) complex and feature extensive
harness for the benchmark. He responds to countless pleas for help,
assistance, debugging, etc., in true SPEC fashion without asking
whether the help is for a partner or a competitor. He added
burstiness to the web server workload. And he integrates new code and
changes from all the developers for each development kit revision.

Thanks for your great efforts!

SPEC awards, graphics

More 2009
SPECtacular awards. Sometimes even success doesn’t succeed, at first.
SPEC developed a workstation
energy consumption benchmark, and a lot of people worked extra
hard to deliver it in time for EPA to consider using it in the Energy
Star program which is being extended beyond PC’s to also include
workstations, servers, thin clients, and storage. Although EPA
decided not to use our test for the workstation program at this time,
the work is still important and I am confident it will be used in
some way. A graphics processor can easily use
more energy than a CPU, especially a high performance accelerated
3D processor. For their exceptional work in producing this benchmark
I thank David Reiner of AMD,
Joerg Grosshennig of Fujitsu
Technology Solutions, Paul Besl of Intel,
and an engineer from NVIDIA.