era@scdpyr.ucar.edu (Ed Arnold) (09/13/88)
Several months ago, I posted an inquiry for information on benchmarking
packages; the information received has been condensed into this posting.
In some cases, it wasn't possible to follow a lead as far as an address
and phone number. If anyone out there has any further leads on any of the
following material, please notify me; I will respost this item if further
information comes to light.
CAVEAT: there may be inaccuracies, or plain expressions of opinion,
in this information. In particular, if you are a vendor and are
insulted by something you read here, there is no point in calling
me to chew me out! If you care to e-mail me more correct or complete
information on your product, then I will repost a correction of this
article.
ACQUISITION SUPPORT
I've come across one company which claims to offer a "total acquisition
support" service, i.e. RFP writing and benchmarking independent of any
vendor. I suspect there are others who do this sort of thing, however,
I have not yet run across them.
o A&T Systems, Silver Spring MD; Marcus Rogerson, 301-384-1425
o Vendors: almost all vendors of any size will have someone who is
a specialist in performance analysis. They will have run one or
more "standard" benchmarking suites on their equipment, and will be
happy to provide you with this information. Whether you choose to
believe it is, of course, up to you; if you take this approach, be
sure you understand how settable parameters were likely to have
affected the result, and whether it is possible that some other
settings might have resulted in a complete turnaround of the results
with regard to comparing two systems. (Easier said than done. :-))
VALIDATION SUITES
o SVVS: the System V Validation Suite is available directly from AT&T
technology licensing in North Carolina, 800-828-unix. However, you
must be a S/V source licensee to obtain/use this.
o POSIX: there is a company working with the National Bureau of Standards
to construct a validation suite for POSIX. I haven't yet found out
who that is.
o Perennial: Perennial Inc. (4677 Old Ironsides Drive #450, Santa Clara, CA,
95054; 408-727-2255) has available validation suites for System V.x
and BSD 4.x, for C compilers, and some performance benchmarking stuff.
o BRL: Doug Gwyn of BRL (gwyn@brl.arpa) has a small suite of System V.2
verification and load testing programs, built for an Army procurement,
which is available for free.
o Certification thru AT&T or CSRG: vendors who claim to be compliant with
System V.x should have a certificate on file with the AT&T compliance
testing group. In addition, one reference on the net claims that the
Berkeley CSRG has a similar program for BSD4.x, although I'm not
personally aware this is so. Anybody know?
BENCHMARKING SUITES
The following benchmark suites are available. In some cases,
I haven't yet determined how to obtain the item.
o Monash University (MUSBUS) benchmark suite, which has been posted to
Usenet in comp.sources.unix. This suite was authored by Ken J. McDonnell,
now with Pyramid Technology: kenj@pyrnova.pyramid.com. It may be
obtained from uunet.uu.net via anonymous ftp, or via the archive server
at uunet!netlib. The 5.0 version is in comp.sources.unix/volume11/musbus;
the patch for version 5.2 is in comp.sources.unix/volume12/musbus5.2.
In personal correspondence with the author, he stated that MUSBUS does
a better job than some of the older suites, particularly with regard to
imposing a workload of real programs and finding a system's point of
resource depletion.
o Purdue benchmark suite, from Purdue University. Although a correspondent
claimed there is such a thing, I have not been able to obtain a copy
of it, or a firm reference.
o BRLCAD, available free of charge from Ballistics Research Lab.
Info on this may be obtained on request from cad@brl.arpa.
o ditroff: one correspondent claims that multiple copies of ditroff running
are, in his experience, as good a way of any at testing cpu and i/o
performance.
o ENEA suite: probably available for free from Mats Josefsson of Enea Data,
Taeby, Sweden. Mail uunet!enea!mats to find out about this.
o Linpack, Dhrystone, Whetstone: these traditional benchmarking tools
are available from the archive server, netlib@anl-mcs.arpa (research!netlib).
It is my understanding that these tools aren't very good at measuring
the performance of large, multi-processor systems; they are best for
measuring traditional single-CPU architectures, or for those who are
more concerned with flat-out floating point performance.
To find out more about what's available from netlib, mail to the
address given, with a message body that contains "send index".
o AIM: AIM Technology, 3350 West Bayshore Road #203, Palo Alto CA 94303;
415-856-8649 makes available a package called the AIM Benchmarks,
which has been around for a number of years. I believe the package
is now referred to as "AIM 3".
o We at NCAR have a home-brew suite that we built in 1985; this was to
support an acquisition effort that ended in the purchase of two
Pyramid 90x machines. If you want a copy of this, mail me
(era@ncar.ucar.edu, ncar!era).
o University of Texas: Michael Cerda at U. Texas (cerda@emx.utexas.edu)
supposedly has a good suite of benchmarks called "sequent.sh".
I have not yet seen this, however.
o Workstation Laboratories: it is my understanding that this company offers
a product similar to AIM, tailored to the workstation market. However,
I have not yet determined an address or telephone number.
o The Usenet management/archive node, uunet.uu.net, may have items for
benchmarking, other than MUSBUS mentioned above. Interested users
may use anonymous ftp, or mail to uunet!netlib ("send index"), to
investigate this.
o There is a benchmark named "TP-1", designed to simulate on-line
transaction processing in banking, database access, and similar
environments. This benchmark is typically used on big systems,
such as Tandem networks, 3090s, and the like. Apparently, TP-1
is rather sensitive to the parameters with which it is setup.
I have not yet determined if/where this suite may be obtained.
SYSTEM PERFORMANCE EVALUATIONS
There are several companies/organizations which provide information
from running their particular benchmark suite, on "typical" configurations
hardware from many different vendors.
o BRLCAD includes stats for many machines, from small Suns to Cray.
o AIM: (see address above) AIM makes available "Unix Performance Reports",
the results of the AIM Benchmarks run on a large number of different
machines, such as Sun, Pyramid, and Sequent. The data provided is
broken down into categories of subsystem and application performance.
Three of their reports can currently be obtained for free; each
additional machine report is $25.
o Neal Nelson Associates: Chicago, 312-332-3242. Apparently, their suite
is sometimes referred to as the "Neal Nelson Business Benchmark."
They offer source leasing of their suite, canned performance reports
(one machine against another, for $96), and custom testing. In a
conversation with Karen McBride of this company, she claimed that they
have worked with a number of government agencies in developing averaged
numbers (from their performance reports) for inclusion in RFPs, and will
provide references into those agencies on request.
I have read that the comments above about Linpack & friends are also
applicable to the Nelson suite, i.e., it is better suited to measuring
traditional single-CPU architectures, than large multi-processor systems.
However, I have no personal knowledge of whether or not this is so.
--------
Ed Arnold * NCAR (Nat'l Center for Atmospheric Research) * Mesa Lab
PO Box 3000 * Boulder, CO 80307-3000 * 303-497-1253
era@ncar.ucar.edu [128.117.64.4] * {ames,gatech,noao,...}!ncar!era