mash@mips.UUCP (John Mashey) (12/22/87)
In article <28200078@ccvaxa> aglew@ccvaxa.UUCP writes: >John Mashey mentioned "understanding DEC's performance methodology" >in one of his recent postings. Can anyone explain DEC's performance >methodology to me, or give me references? I'm still new at this >sort of thing. Maybe someone from DEC will post something. I have no idea how they arrive at the numbers that end up appearing in the glossies. However, I have seen several publications that offer some ideas: [1] Don McInnis, Bob Kusik, Dileep Bhandarkar, "VAX 8800 System Overview", IEEE CH2409-01/87 [which must be Proc. Symp. on Computer Architecture, I think], 316-321. [2] DEC Workstation Marketing, "VAXstation 3200 Performance Analysis", Sept 9, 1987. In [1], the 8700/8800 implementation si described. There is a short performance section, which gives some benchmark results, and also characterizes an 8700 versus an 11/780, with a chart as follows: NUMBER OF BENCHMARKS = 99 MINIMUM = 2.91 MAXIMUM = 7.70 MEAN = 5.01 GEOMETRIC MEAN = 4.92 HARMONIC MEAN = 4.84 (more statistical data) 3.0 - 3.4 4.0 % FFFD 3.4 - 3.8 6.1 % FFFFFF 3.8 - 4.2 7.1 % IFFFFDL 4.2 - 4.6 19.2 % IIFFFFFFFFFDDDDDDLL 4.6 - 5.0 11.1 % CIIIFFLLLLL 5.0 - 5.4 15.2 % FFFFDDDDDDLLLL 5.4 - 5.8 23.2 % CIFFFDDDDDDDDDDDLLLLLL 5.8 - 6.2 6.1 % CCFDLL 6.2 - 6.6 3.0 % DDL 6.6 - 7.0 2.0 % DD 7.0 - 7.4 3.0 % DLL Legend: I Fortran Integer F SP Floating Point D DP FLoating Point L LISP (Integer and Real) C COBOL Additional specific benchmark numbers are given, including oens that show the influence of program size on performance (Jacobi, Linpack, Matrices), VAX 8800 multi-stream thruput relative to single-cpu, etc. I'd like more info than what is in the paper, however, the presentation has some useful points: 1) It is clear that the performance ratio versus a 780 is not a single number for all benchmarks. 2) You can get some idea of the spread for different kinds of benchmarks. (I could wish for C benchmarks as well. I suspect they'd run a little higher, but that's just a guess.) It is interesting to see that the 8700 is much better on DP floating-point than on SP, relative to the 780. 3) The geometric mean is included, indicating that the people involved seem to understand why that's relevant. 4) It's not at all clear how the final "11/780 rating" is obtained, i.e., the official number may be computed from some other set of benchmarks, and these are just offered as backup material, or it may be viewed that some subset of this set is the official set, or, perhaps someone picks the numbers out of there. However, it is useful to note that the "5-6"X rating usually ascribed to the 8700 is at least possible, and they certainly do not call it 7.4X. Item [2] is a competitive analysis of the VS3200 with Sun-3/260, Sun-4/260, and Apollo DN4000. A lot of data is presented, using histograms logically similar to that above. Cost/performance numbers are also computed [you can guess who wins]. Often, 40-60 data points are shown for each system, including Sun-3/260s with/without FPA (the distribution shift is rather dramatic). The benchmarks included a bunch of ECAD, MCAD, Earth Resources, and other real applications. Unfortunately, they didn't get every benchmark for every machine [which is very hard for 61 benchmarks, they clearly were trying pretty hard.] This is a slight flaw, as it is quite possible for missing datapoints to skew the results, i.e., if you happen to have a bunch of benchmarks for one machine that happen to be "easy" ones, but those are missing from the others. (I don't believe this happened here, but it can be a worry). Anyway, again, I don't know exactly what they dp, but as shown in such publications, DEC ratings include large nubmres of benchmarks, and they include floating point. -- -john mashey DISCLAIMER: <generic disclaimer, I speak for me only, etc> UUCP: {ames,decwrl,prls,pyramid}!mips!mash OR mash@mips.com DDD: 408-991-0253 or 408-720-1700, x253 USPS: MIPS Computer Systems, 930 E. Arques, Sunnyvale, CA 94086