cprice@mips.com (Charlie Price) (06/27/91)
In article <1991Jun25.201134.1988@arcturus.uucp> michael@arcturus.uucp (Michael;Wayne D.) writes: >Sorry if this is a FAQ, but how I do find out more information on a >benchmark called "specmarks". > >Is it possible to obtain the source?, and if so, where can I gey it >from? SPEC, the System Performance Evaluation Cooperative, is a non-profit corporation formed to "establish, maintain and endorse a standardized set of relevant benchmarks that can be applied to the newest generation of high-performance computers". The members of SPEC (at current annual dues of $5000) are Arix, AT&T, Bull S.A., Compaq, CDC, DG, DEC, DuPont, Fujitsu, HP, IBM, Intel, Integraph, MIPS, Motorola, NCR, Prime, Siemens, Silicon Graphics, Solbourne, Stardent, Sun, Unisys. SPEC basically does 2 things: 1) Puts together suites of benchmarks that are generally available in source form. These benchmarks are intended to measure something meaningful and are extensively tested for portability before release. There are strict rules on how these benchmarks must be run and how results must be reported for the trademarked results (SPECmark, SPECint, and SPECfp for the current CPU benchmarks). 2) Publishes SPEC Benchmark results in a quarterly newsletter. There are currently two suites of benchmarks. 1) SPEC Benchmark suite 1.2. This is a suite of 10 CPU (and memory system) intensive benchmarks intended to measure "engine horsepower" on some applications of realistic size and content. The results are expressed as the ratio of the wall clock time to execute the benchmark compared to a (fixed) "SPEC reference time" (which was chosen early-on as the execution time on a VAX 11/780). The results are reported as the geometric mean of the individual ratios: SPECmark -- all 10 SPECint -- the 4 benchmarks that are integer-operation intensive SPECfp -- the 6 benchmarks that are floating-point intensive (in this case that means > 1% of executed instr are FP instr). A separate measure known as SPECthruput may be reported for multiprocessor systems. This attempts to measure the actual CPU/memory performance available from the multiple processors. In effect, it shows up any benchmark slowdown due to resource contention in the MP system. 2) SPEC SDM 1.0 (Systems Development Multitasking) (announced 14-May-91). This consists of two benchmarks that present a multiprocess load to the system. The results are more graphical than the CPU benchmark and are not as easily reduced to single numbers, but the marketing ad numbers used will be the two (different) peak throughput numbers measured in scripts/hour. The benchmark sources are generally available -- but not free. SPEC is charging separately for the two benchmark suites. The cost of the source tapes is intended to support the administrative costs of the corporation -- making tapes, answering questions about the benchmarks, and so on. Call SPEC for the current numbers, but as of early 91 they were: $300 - 1.2 SPEC Benchmark source tape (QIC 24) $1,450 - 1.0 SPEC SDM (Systems Development Multitasking) source tape (QIC 24). $399 - SPEC Newsletter for 1 year (4 issues) There is a reduced price for a combined tape/newsletter order. From the Winter 91 (Vol 3, Issue 1) SPEC Newsletter: SPEC is a trademark of the Standard Performance Evaluation Cooperative. SPEC is a non-profit corporation formed to establish, maintain and endorse a standardized set of relevant benchmarks that can be applied to the newest generation of high-performance computers. The founders of this organization believe that the user community will benefit greatly from an objective series of applications-oriented tests, which can serve as common reference points and be considered during the evaluation process. While no one benchmark can fully characterize overall system performance, the results of a variety pf realistic benchmarks can give valuable insight into expected real performance. SPEC Benchmark Suite Tape/Newsletter Subscriptions Release 1 of the SPEC Benchmark Suite is available in QIC 24 tape format for $300. A one-year subscription (4 issues) to the "SPEC Newsletter" is $399. A combination order (both the tape and a newsletter subscription) is $699. For general information and to place tape/newsletter orders call (408) 453-5220. The cooperative's mailing address is: SPEC c/o Franson & Hagerty 181 Metro Drive, Suite 300 San Jose, CA 95110 TEL: (408) 453-5220 FAX: (408) 453-8723 -- Charlie Price cprice@mips.mips.com (408) 720-1700 MIPS Computer Systems / 928 Arques Ave. MS 1-03 / Sunnyvale, CA 94088-3650