rnovak@mips.COM (Robert E. Novak) (01/27/90)
From: rnovak@mips.COM (Robert E. Novak)
As promised here is a list of criteria to look at when considering a
program to be submitted to SPEC for consideration as an application
benchmark. This is not intended to be an exhaustive list, in fact it is
only this author's opinions. This list has not been 'blessed' by the SPEC
steering committee. If your program does not meet on all criteria, it
doesn't mean an automatic rejection, it just depends on how much work we
need to do to make it acceptable and how appealing your application is to
SPEC. After your program has been submitted as a candidate program, the
Steering Committee will need to find a sponsor (a Steering Committee
member) and a project leader (who will actually do the work).
Submit Permission forms and/or programs to one of the following:
rnovak@mips.com, shanley@cup.portal.com. We will reply ASAP (remember that
next week is UniForum). Include e-mail, U.S. Mail telephone and FAX
numbers as applicable or possible. To request a more information on SPEC,
call 1-415-792-2901.
Criteria
1. The program should be an application-oriented program,
i.e. a program that is frequently used by an active
user community.
2. Each program must be a key component in a long-range
goal to test overall processor/system performance.
SPEC wants to exhaustively exercise not only the CPU
and FPU, but also the Disk I/O, Tape I/O and Network
I/O as well as the operating system scheduler.
3. Each program must be a representative component in a
wide-range of applications. SPEC wants representative
programs from scientific (CAD, ECAD, MCAD, Seismic,
Nuclear Physics, Astronomy, etc.), Technical (CASE,
compilers, CPU simulators, debuggers, software
development aids, etc.) and commercial applications
(Accounts Receivable, MRP, ISAM files, RDBMS based MIS
applications, flight scheduling programs, distribution
routing (trucking, railroads), etc.).
4. Each program should be large, in terms of total
program size, size of most compute-intensive block and
in terms of the total working set size of both the
instruction set and the data reference set.
All too many performance tests have squeezed into on
board CPU caches, which may not be representative of
actual applications.
5. The programs should be long-running programs (greater
than 60 seconds of execution time on a 10 mip CPU).
Programs that use less CPU time than that will result
in measurements that are below the resolution of
system timer programs. In addition, these
applications for performance testing will hopefully
have a lifetime that exceed 18 months of meaningful
duty.
6. Whenever possible, the program should be public domain
or it should be made publicly available under the SPEC
license umbrella. This is the part that makes the
SPEC job the hardest. A software developer must be
willing to allow the competition to examine at least
some part of their source code in order to make the
code available to SPEC.
7. The program must be easily portable to many machines.
A program that is developed for the UNIX (R)
environment is usually the simplest to port to most
platforms.
8. The program's output must be mechanically checkable
for correctness. The benchmarks are useless unless we
can verify that they produce identical results.
9. The programs' source code should conform to existing
language standards for the implementation language.
In attempting to move the SPEC benchmarks from 32 bit
platforms to 64 bit platforms, SPEC has discovered
that this was the greatest sin in Release 1.0.
--
Robert E. Novak MIPS Computer Systems, Inc.
{ames,decwrl,pyramid}!mips!rnovak 928 E. Arques Ave. Sunnyvale, CA 94086
rnovak@mips.COM (rnovak%mips.COM@ames.arc.nasa.gov) +1 408 991-0402
Volume-Number: Volume 18, Number 38