eugene@nas.nasa.gov (Eugene N. Miya) (06/02/91)
[WARNING ROCKY ROAD AHEAD -- UNDER CONSTRUCTION reduce MFLOPS ahead.]
You will typically ask, and I (amelia) must try to answer
1) where can you pick up source code?
2) where can you get results?
This list can start to answer 1). 2) is a more difficult
proposition, and I can't begin to describe the problems. A few sources
in 1) can answer 2). As you will see the problem isn't obtaining
information, it's sorting it out, and learning something meaningful.
PERFECT
(post-facto acronynm:
PERFormance Evaluation for Cost-effective Transformations)
Contact: J. Larson (jlarson@csrd.uiuc.edu)
Center for Supercomputing Research and Development
305 Talbot Laboratory
104 South Wright Street
Urbana, Illinois 61801
NISTLIB/NBSLIB
nistlib@cmr.ncsl.nist.gov
send index
includes LLNL loops (aka Livermore Loops, Livermore Fortrasn Kernels (LFK),
NAS kernels, Los Alamos benchmarks, parcbench, whetstone, dhrystone,
mendez, and many others
no results
NETLIB benchmark index (Linpack benchmark)
netlib@ornl.gov
send index from benchmark
includes linpack (100x100,300x300,1000x1000)
SPEC
[Systems Performance Evaluation Cooperative]
c/o Franson & Hagerty
181 Metro Drive
Suite 300
San Jose, CA 95110
Phone: 408-453-5220
Fax: 408-453-8723
SPEC membership costs:
Initiation $10,000
Annual Dues $ 5,000
SPEC Associate:
Initiation $2,500
Annual Dues $1,000
To qualify as a SPEC associate, you must be an accredited educational
institution or a non-profit organization. An associate has no voting
privileges. An associate will receive the newsletter and the benchmark
tapes as they are available. In addition, an associate will have early
access to benchmarks under development so that an associate may act in an
advisory capacity to SPEC.
The SPEC tape still costs $699 which includes the cost of a 1 year
subscription to the SPEC newsletter. The tape by itself costs $300.
There are no discounts for the SPEC tape/newsletter.
Performance mailing list: (oriented toward performance analysis
using quantitative modeling (Mean-Value Analysis [MVA], etc.)
performmance@vuse.vanderbilt.edu
ACM/SIGMETRICS (Perforance Evaluation Review)
Other benchmarks:
iostone:
park@iris.ucdavis.edu
iocall:
I have two versions. (Will substitute developer net address soon)
iobench:
bonnie:
I have a version of this as well.
slalom:
Archive-directory: tantalus.al.iastate.edu:/pub/Slalom/ [129.186.200.15]
%A J. Gustafson
%A D. Rover
%A S. Elbert
%A M. Carter
%T SLALOM: The First Scalable Supercomputer Benchmark
%J Supercomputing Review
%D November 1990
%P 56-61
%A John Gustafson
%A Diane Rover
%A Stephen Elbert
%A Michael Carter
%T The Design of a Scalable, Fixed-Time Computer Benchmark
%R IS-5049/UC-32
%I Ames Lab, Iowa State University
%D 1990
Tantalus.al.iastate.edu has the SLALOM sources in various languages and other
things including the latest reported results in
pub/Slalom/Reports/CurrentReport
smith:
smith@berkeley.edu
OLTP (On-Line Transaction Processing):
Jim Gray, Tandem
Serlin?
Gibson mix:
IBM
NCR benchmark:
Have a copy (pio) might not be up to date.
nhfstone/NFSstone: (I have a copy of this.)
To: nhfsstone-request@legato.com
Subject: send unsupported nhfsstone
tcp/ip benchmarks:
ttcp
Archive: sgi.com:/sgi/src/ttcp.shar [192.48.153.1]
(TTCP is the most common TCP benchmark. Most commonly used as well as most
commonly abused and perverted to generate marketing numbers, and most
commonly enhanced to do things the enhancer considers important. Look
for the authorative source on brl.mil and more or less true copies many
other places, including sgi.com.)
Rhealstone:
real time
Lhynestone:
graphics (TBD, work suspended)
MUSBUS (current version is 5.0 with patches to 5.2):
uunet: comp.source.unix
volume11/musbus, patches to 5.2 on volume12/musbus5.2
gbench: (X graphics benchmark)
uunet: comp.source.unix
volume 15
Don't forget the most commonly used performance monitor: x11perf;
sources are in uunet.uu.net:/X/contrib/x11perf.tar.Z and others.
Also, I can give you the VGX benchmark I just obtained from SGI if you
didn't get it from comp.benchmarks.
tbench:
uunet: comp.source.misc
plum-benchmarks:
uunet: comp.source.unix
volume 20
Byte benchmarks: (If not below, I have them, also integrated into MUSBUS[?}).
me.utoronto.ca
pub/byteunix.tar.Z
Gabriel: (LISP)
See his book on the subject MIT press. [I'll fill in the entry,
I have the code.]
SEI Ada benchmarks:
The Software Engineering Institute of the CMU/DARPA has a set
of Ada benchmarks. Contact them. They may require special
permission.
GPC
Graphics Performance Characterization {group|committee}
contact Bob Willis at NCGA (703-698-9600)
article on PLB in July 1989 Computer Graphics World.
GPC is the group and PLB (Picture Level Benchmarks) is the program.
There have been updates on the status of PLB in recent issues of
Computer Graphics World.
Also, there is the ``Standard Procedural Database'', or SPD, mentioned
as a set of standard models for evaluating rendering performance.
This was mentioned initially in IEEE Computer Graphics & Applications,
November 1987. It is being distributed by netlib.
RhosettaStone:
Speech synthesis and recognition benchmark:
e.g., How to recognize speech.
How to wreck a nice beach.
My sister machine: eos.arc.nasa.gov.
COMMERCIALLY AVAILABLE BENCHMARKS (i.e. mucho $$)
The following not only cost some money, but people who pay for them
are not allowed to report results in some cases. Caveat emptor.
Mention here does NOT constitute an endorsement.
AIM Technology Benchmark (Dronek and beyond)
[Actually some interesting stuff in one or two versions.]
Neal Nelson Business Benchmark
35 East Wacker Dr.
Suite 150
Chicago, IL 60601-1462
312-332-3242
I was informed this was blasted in PER (SIGMETRICS).
Reports are not open (copyrights).
Khornerstone (The Lab Report)
IF you are thinking of writing benchmarks, you need a good name.
ARS/Workstation Laboratories
4324 N. Beltline, Suite C211
Irving, TX 75038
(214)-570-7100 [Fax (214)-570-4201]
$995 at this time.
Reports are not open (copyrights).
Seriously, append the year to the end of your name like name90.
It sets a useful point for revision control. You can learn about
other things, but that's later.
NO SINGLE-FIGURE-OF-MERITs.
Those who can build hardware.
Those who can't,benchmark.
^ A
s / \ r
m / \ c
h / \ h
t / \ i
i / \ t
r / \ e
o / \ c
g / \ t
l / \ u
A / \ r
<_____________________> e
Language