mark@mips.UUCP (Mark G. Johnson) (09/15/87)
We finally located a VMS-VAX 11/780 with SPICE available, so we ran
the three public domain SPICE benchmark circuits. Hopefully this
will let the VAX show off its FP performance, as the VMS-fortran
compiler generates better code than the Unix f77 compiler does.
A companion article includes the input circuit files ("decks").
========================== TABLE 1. DIGSR circuit ==========================
Berkeley-2G6
MACHINE seconds Vaxes OS, compiler, notes
------- ------ ----- -------------------------
VAX 11/780 1354.0 0.60 4.3BSD, BerkF77 V2.0
Microvax-II 993.5 0.81 Ultrix 1.1, fortrel
SUN 3/160 901.9 0.90 SunOS 3.2 f77 -O -f68881
VAX 11/780 848.0 0.95 4.3BSD, fortrel
VAX 11/780 808.1 1.0 VMS 4.4, fortran/opt
SUN 3/260 744.8 1.1 SunOS 3.2 f77 -O -f68881
SUN 3/160 506.5 1.6 SunOS 3.2 f77 -O -ffpa
SUN 3/260 361.2 2.2 SunOS 3.2 f77 -O -ffpa
SUN 4/260 225.9 3.6 SunOS 4-beta2 f77 -O3 -Qoption as -Ff0
MIPS M/800 136.5 5.9 UMIPS-BSD V2.01, f77 V1.21
AMDAHL 470 V/7 125.5 6.4 VMSP-CMS 4.0, FORTVS 4.1
MIPS M/1000 114.3 7.1 UMIPS-BSD V2.01, f77 V1.21
FPS 20/64 48.0 16.8 VSPICE (2g6 derivative)
========================= TABLE 2. BIPOLE circuit ==========================
Berkeley-2G6
MACHINE seconds Vaxes OS, compiler, notes
------- ------ ----- -------------------------
VAX 11/780 439.6 0.68 4.3BSD, BerkF77 V2.0
Microvax-II 394.3 0.76 Ultrix 1.1, fortrel
VAX 11/780 312.6 0.96 4.3BSD, fortrel
VAX 11/780 299.1 1.0 VMS 4.4, fortran/opt
SUN 3/160 285.1 1.0 SunOS 3.2 f77 -O -f68881
SUN 3/260 221.7 1.3 SunOS 3.2 f77 -O -f68881
SUN 3/160 170.0 1.8 SunOS 3.2 f77 -O -ffpa
SUN 3/260 112.0 2.7 SunOS 3.2 f77 -O -ffpa
SUN 4/260 63.7 4.7 SunOS 4-beta2 f77 -O3 -Qoption as -Ff0
MIPS M/800 42.6 7.0 UMIPS-BSD V2.01, f77 V1.21
AMDAHL 470 V/7 39.5 7.6 VMSP-CMS 4.0, FORTVS 4.1
MIPS M/1000 35.4 8.4 UMIPS-BSD V2.01, f77 V1.21
FPS 20/64 12.5 23.9 VSPICE (2g6 derivative)
========================= TABLE 3. TORONTO circuit =========================
Berkeley-2G6
MACHINE seconds Vaxes OS, compiler, notes
------- ------ ----- -------------------------
VAX 11/780 460.3 0.63 4.3BSD, BerkF77 V2.0
Microvax-II 366.9 0.80 Ultrix 1.1, fortrel
SUN 3/160 328.6 0.89 SunOS 3.2 f77 -O -f68881
VAX 11/780 302.9 0.96 4.3BSD, fortrel
VAX 11/780 291.7 1.0 VMS 4.4, fortran/opt
SUN 3/260 266.0 1.1 SunOS 3.2 f77 -O -f68881
SUN 3/160 189.1 1.5 SunOS 3.2 f77 -O -ffpa
SUN 3/260 129.4 2.3 SunOS 3.2 f77 -O -ffpa
SUN 4/260 73.4 4.0 SunOS 4-beta2 f77 -O3 -Qoption as -Ff0
MIPS M/800 41.4 7.0 UMIPS-BSD V2.01, f77 V1.21
AMDAHL 470 V/7 39.3 7.4 VMSP-CMS 4.0, FORTVS 4.1
MIPS M/1000 34.5 8.5 UMIPS-BSD V2.01, f77 V1.21
FPS 20/64 17.5 16.7 VSPICE (2g6 derivative)
############################################################################
Since the first posting, we discovered that we were not running the
standard Berkeley Spice2g6 UNIX distribution on our 780. The standard
distribution includes code for block copy and zero that uses VAX
string instructions. Our copy had that replaced with simple byte copy
loops so as to be portable to our own machines. We went back to the
VAX string instruction version for VAX times so that the results will
correlate with results obtained elsewhere with the standard
distribution. And to be fair, we used the 4.2 bcopy/bzero routines on
other machines that we ran it on (SUNs, MIPSs). The end result: the
780 numbers have improved by 15-20% since the last posting, the MIPS
numbers by 7%. (The 1.21 compiler and 2.1 OS releases also improved
MIPS results.)
We have also added 780 results for spice2g6 compiled with two more
compilers: the fortrel compiler from Lawrence Livermore Labs and the
DEC VMS-fortran compiler. The relative performance column ("Vaxes")
is now normalized to the DEC VMS-fortran compiler.
We would like to thank Aedan Coffey, Joerg Bach, and David Hough for
their contributions to this list.
--
-Mark Johnson *** DISCLAIMER: The opinions above are personal. ***
UUCP: {decvax,ucbvax,ihnp4}!decwrl!mips!mark TEL: 408-720-1700 x208
US mail: MIPS Computer Systems, 930 E. Arques, Sunnyvale, CA 94086yoram@cheshire.columbia.edu (Yoram Eisenstadter) (09/16/87)
In article <677@obiwan.UUCP> mark@mips.UUCP (Mark G. Johnson) writes: >We finally located a VMS-VAX 11/780 with SPICE available, so we ran >the three public domain SPICE benchmark circuits... > > Berkeley-2G6 >MACHINE seconds Vaxes OS, compiler, notes >------- ------ ----- ------------------------- >VAX 11/780 1354.0 0.60 4.3BSD, BerkF77 V2.0 >Microvax-II 993.5 0.81 Ultrix 1.1, fortrel >SUN 3/160 901.9 0.90 SunOS 3.2 f77 -O -f68881 >VAX 11/780 848.0 0.95 4.3BSD, fortrel >VAX 11/780 808.1 1.0 VMS 4.4, fortran/opt >SUN 3/260 744.8 1.1 SunOS 3.2 f77 -O -f68881 >SUN 3/160 506.5 1.6 SunOS 3.2 f77 -O -ffpa >SUN 3/260 361.2 2.2 SunOS 3.2 f77 -O -ffpa >SUN 4/260 225.9 3.6 SunOS 4-beta2 f77 -O3 -Qoption as -Ff0 >MIPS M/800 136.5 5.9 UMIPS-BSD V2.01, f77 V1.21 >AMDAHL 470 V/7 125.5 6.4 VMSP-CMS 4.0, FORTVS 4.1 >MIPS M/1000 114.3 7.1 UMIPS-BSD V2.01, f77 V1.21 >FPS 20/64 48.0 16.8 VSPICE (2g6 derivative) Is anyone besides me bothered by the choice of machines used in this comparison? Note that the 11/780 is a mid-1970's design using mid-1970's technology (they were already being produced in '79 I believe). I think (I'm not so sure on this one) that the Amdahl 470 V/7 is also one of Amdahl's earliest machines. (Does anybody know the vintage of the FPS 20/64? Is it modern?) I'd like to see benchmarks against some modern-day VAXes, e.g., the VAX 8800, and against some modern-day big mainframes, e.g., IBM 3090-class machines. Then, we might get some idea of how little machines like the MIPS and SUN fare against current big timesharing systems. (My guess is that the benchmark will run in under 30 seconds on a 3090, and that the big VAXes will beat the MIPS/1000...) Also, notably absent is the figure for the MicroVAX II running VMS (which is what it was intended to run). What is the "fortrel" compiler? Cheers..Yoram Yoram Eisenstadter | Arpanet: yoram@cheshire.columbia.edu Columbia University | Usenet: ...columbia!cheshire!yoram Dept. of Computer Science | Bitnet: yoram@cucsvm New York, NY 10027 | Phone: (212) 280-8180
mash@mips.UUCP (09/17/87)
In article <5002@columbia.edu> yoram@cheshire.columbia.edu (Yoram Eisenstadter) writes: >In article <677@obiwan.UUCP> mark@mips.UUCP (Mark G. Johnson) writes: >>We finally located a VMS-VAX 11/780 with SPICE available, so we ran >>the three public domain SPICE benchmark circuits... >> Berkeley-2G6 >>MACHINE seconds Vaxes OS, compiler, notes >>------- ------ ----- ------------------------- >>VAX 11/780 1354.0 0.60 4.3BSD, BerkF77 V2.0 >>Microvax-II 993.5 0.81 Ultrix 1.1, fortrel >>SUN 3/160 901.9 0.90 SunOS 3.2 f77 -O -f68881 >>VAX 11/780 848.0 0.95 4.3BSD, fortrel >>VAX 11/780 808.1 1.0 VMS 4.4, fortran/opt >>SUN 3/260 744.8 1.1 SunOS 3.2 f77 -O -f68881 >>SUN 3/160 506.5 1.6 SunOS 3.2 f77 -O -ffpa >>SUN 3/260 361.2 2.2 SunOS 3.2 f77 -O -ffpa >>SUN 4/260 225.9 3.6 SunOS 4-beta2 f77 -O3 -Qoption as -Ff0 >>MIPS M/800 136.5 5.9 UMIPS-BSD V2.01, f77 V1.21 >>AMDAHL 470 V/7 125.5 6.4 VMSP-CMS 4.0, FORTVS 4.1 >>MIPS M/1000 114.3 7.1 UMIPS-BSD V2.01, f77 V1.21 >>FPS 20/64 48.0 16.8 VSPICE (2g6 derivative) >Is anyone besides me bothered by the choice of machines used in >this comparison? If you've followed this whole effort by Mark and co (with good cooperation from Dave Hough in particular & various folks elsewhere), the goal was to get a widely available set of good Spice benchmarks. To this end: Mark proposed some on the net, and asked for more examples. Collected public--domainable versison of ours plus others. Sent them to other knowledgable people for critiquing. Published the input decks, and asked for results. Published the numbers we've measured directly, or that others have been kind enough to send. This is a lot of work, and is not even easy to make happen: Spice breaks compilers on many machines, so even being able to run it says something useful. Everybody would be happy to see more numbers, and we often do run around begging for numbers from other machines. However, Mark does have to spend some time designing chips... We certainly believe in benchmarking as strong as machines as we can lay our hands on, but for some bigger machines we have no choice but to rely on others. Certainly this whole process as been as open as one can imagine, and is still early in sequence of doing it. Of the machines above: the 4.3BSD and MicroVAX our ours, as are the MIPS machines. Sun numbers were dgh's. I don't know where Mark got the others. > >Note that the 11/780 is a mid-1970's design using mid-1970's >technology (they were already being produced in '79 I believe). Yes. Fortunately, it's still a good metric in that it's familiar, and, after a while, you find fairly consistent performance ratios amongst VAXen. > >I think (I'm not so sure on this one) that the Amdahl 470 V/7 is >also one of Amdahl's earliest machines. (Does anybody know the >vintage of the FPS 20/64? Is it modern?) V/7 is old. > >I'd like to see benchmarks against some modern-day VAXes, e.g., >the VAX 8800, and against some modern-day big mainframes, e.g., We would, too! That's why Mark went to the pain of doing all this, since we can't quite afford either of those. >IBM 3090-class machines. Then, we might get some idea of how >little machines like the MIPS and SUN fare against current big >timesharing systems. (My guess is that the benchmark will run in >under 30 seconds on a 3090, and that the big VAXes will beat the >MIPS/1000...) Under 30 seconds: probably right, although maybe just barely. The only 3090 numbers I've got handy are for Hspice case "ST230", and taking the ratios versus both 11/780 and M/1000, you'd get either 28X faster or a bit less than 4X faster, so somewhere between 27 and 47 seconds seems right. I'd guess an 8700 [1 of the 2 cpus in an 8800] would do around 135-150 seconds. M/800s almost always beat 8700s on floating-point crunchers, and an M/1000 is 1.2X an M/800 (scales directly with clock on CPU-bound jobs), so that it is typically 1.4X-1.6X as fast as an 8700 (VMS), (i.e. halfway betweeen an 8700 and 8800 in thruput on these things). This kind of ratio has been consistently found with other programs, such as: Doduc (8700VMS = 5.2X 11/780, M/1000 = 8.5X) Digital Review benchmarks (8700 = 1.469 seconds geometric mean, M/1000 = .98) Livermore Loops (geometric mean, 8700 = 1.0MFlops, M/1000 = 1.65) Various Hspice jobs. > >Also, notably absent is the figure for the MicroVAX II running >VMS (which is what it was intended to run). What is the >"fortrel" compiler? Our MicroVax II runs Ultrix. We'd be glad to hear VMS numbers. Fortrel is a LLNL-derived VAX compiler with pretty reasonable optimization. We use it inhouse, especially if trying to approximate VMS numbers when we can't get them. As can be seen, it's about .95X performance relative to VMS, on the 780, and from othere experience, correlates not too badly. (The 3 11/780 numbers give you an idea of relative compiler performance, obviously important in this case.) Conclusions: 1) There's a pretty strong effort to get as good a set of numbers as possible. At least a few people have been quite cooperative, and nobody is trying to hide anything. 2) Send results to mark if you have some. 3) IF you have no results, but you have spare 8700s or 3090s around, send them to ME. I'll figure out something to do with them. Alternatively, if you have 3090s and 8700s lying around, send them, too. -- -john mashey DISCLAIMER: <generic disclaimer, I speak for me only, etc> UUCP: {decvax,ucbvax,ihnp4}!decwrl!mips!mash OR mash@mips.com DDD: 408-991-0253 or 408-720-1700, x253 USPS: MIPS Computer Systems, 930 E. Arques, Sunnyvale, CA 94086