bdg@tetons.UUCP (Blaine Gaither) (09/29/90)
The November Issue of ACM Sigmetrics Performance Evaluation Review
will feature an editorial on Commercial Benchmark Suites.
Of particular interest are the Neal Nelson benchmark, the
UnixWorld/NealNelson benchmark, AIM benchmarks or any others that our
readership wishes to discuss. Please address your comments to the
editor.
The editorial was already partly complete before the UnixWorld article
(discussed in the August issue of PER) was published. The UnixWorld
article seemed particularly irresponsible. The damage caused by a
possibly flawed benchmark is not limited to misleading the benchmark's
clients, but also tends to cause redirection of vendor resources into
areas which may not give maximum gain to the customers.
Even IBM has scarce resources. At this stage in the RS/6000
development, is it constructive to "tune" awk as opposed to other
system characteristics?
From the August issue:
Coming Editorial - Call for letters-to-the-editor
AWK THE NEAL NELSON BENCHMARK
OR
UNIXWORLD'S SHAME
By: Blaine Gaither
The September 1990 issue of UnixWorld just crossed my desk.
On page 13 in a section titled "INDUSTRY NEWS" there is an
article titled "BIG BLUE'S RS/6000 ISN'T WHITE LIGHTNING".
In this article it is stated that recent UnixWorld/Neal
Nelson & Associates benchmarking of the RS/6000 POWERserver
320 found, among other things, that the RS/6000 was about
20% slower than the Hewlett-Packard Vectra 486 in floating-
point and integer calculations.
The article goes on to quote Nelson(sic) as offering two
possible explanations for the poor floating-point results:
1) IBM could have a slow implementation of awk, which is
used in the floating-point tests; or 2) The IBM system's
floating-point speed is less than they expected.
I have another possible explanation: Could the
UnixWorld/Neal Nelson benchmark have a poor experimental
design and give erroneous results? Should the title of the
article have been: "The IBM RS/6000 Shows the UnixWorld/Neal
Nelson Benchmark to be Flawed"?
For those of you who aren't familiar with awk, it is a
utility for pattern scanning and processing. It is often
implemented as an interpreter and it does its internal
arithmetic in floating-point.
Criticism of the Neal Nelson & Associates benchmarking
methodology has been floating around the performance community
for years. The next issue of PER will feature an editorial
on this and other commercial benchmarking suites. If you have
comments you wish to publish on these topics please send a
letter to the editor by November 7.
Standard disclaimers. These are my opinions, not my employers or the
ACM's. Should any opinions or data published as fact prove incorrect
I will gladly publish a correction.
Blaine Gaither
Amdahl Corporation
143 No. 2nd East St., Rexburg, Idaho 83440-1619
UUCP:{ames,decwrl,sun,uunet}!amdahl!tetons!bdg (208) 356-8915
INTERNET: bdg@tetons.idaho.amdahl.com