alan@mn-at1.UUCP (Alan Klietz) (03/12/88)
We all know what a VAX "MIPS" is, right? But have you ever bothered to actually measure it? DEC did. A recent column in one of the Unix trade mags reveals that the Vax 780 = 1 MIPS rule-of-thumb may be grossly overstated. Here are some excerpts from the April 1988 issue of Unix/World's "Inside Edge" column, by Omri Serlin, Recently confirmed data published by DEC staffers proves that the VAX 11/780, accepted by the industry and the press as a 1 MIPS machine, was actually less than half as powerful. The discovery affecets numerous industry comparisons relative to the 780. Over the past several years, both DEC and the industry as a whole seem to have accepted the notion that the VAX 11/780 was, more or less, a 1 MIPS machine. DEC publicly confirmed the fact in December 1985, when it published full-page ads characterizing the VAX 8650 as a '6 MIPS' machine; DEC officially regards the machine as having six times the power of an 11/780. The number of performance claims by other players in the computer field that assume a 1 MIPS rating for the 11/780 is too numerous to count. In October 1987, one of the authors [of the DEC study] presented a paper on the performance of the VAX 8800, a dual-processor machine in which each processor is six times as fast as the 11/780, by DEC's official reckoning. This paper reported on some preliminary measurements that showed the 8800 processor to have a cycle per average instruction (CPAI) rate of 8.4. At cycle time of 45ns, this translates into a MIPS rating of 2.65. [End of quote.] Therefore one VAX 780 "MIPS" is approximately 0.44 "honest-to-god" VAX MIPS? (HTGV MIPS?) Or a 126% overestimate? -- Alan Klietz Minnesota Supercomputer Center (*) 1200 Washington Avenue South Minneapolis, MN 55415 UUCP: alan@mn-at1.k.mn.org Ph: +1 612 626 1836 ARPA: alan@uc.msc.umn.edu (was umn-rei-uc.arpa) (*) An affiliate of the University of Minnesota
dennis@gpu.utcs.toronto.edu (Dennis Ferguson) (03/14/88)
In article <413@mn-at1.UUCP> alan@mn-at1.UUCP (Alan Klietz) writes: >A recent column in one of >the Unix trade mags reveals that the Vax 780 = 1 MIPS rule-of-thumb >may be grossly overstated. [...] >Therefore one VAX 780 "MIPS" is approximately 0.44 "honest-to-god" >VAX MIPS? (HTGV MIPS?) Or a 126% overestimate? This is well known. I suspect you will get more than one reply (beside this one) reiterating the story about how the DEC types benchmarked the 11/780 against a then-current 370 which IBM was calling a 1 MIPS machine, found it to run about the same speed, and so for marketing purposes called the 11/780 a "1 MIPS" computer. Thus the "MIPS" referred to are supposed to be native 370 MIPS, not native Vax MIPS. This, unfortunately, is also not true, at least in my experience. I have found that you can match benchmark results on a 370 and a Vax pretty well by multiplying the IBM-reported "MIPS" number by 1.8 or so (i.e. a 13 MIPS 3090 goes faster than one would otherwise be led to believe). I really think some marketeer at DEC just made up the 1 MIPS number so Vaxes would look better against the IBM 370 competition. The fact that trade magazines are just getting around to realizing this shows what a good idea it was. All of which matters not at all, since if you simply define a 780 to be 1 "MIPS" and measure everything against it, it all works out in the end for many practical purposes anyway. Dennis Ferguson University of Toronto
steves@ncr-sd.SanDiego.NCR.COM (Steve Schlesinger) (03/15/88)
>In article <413@mn-at1.UUCP> alan@mn-at1.UUCP (Alan Klietz) writes: >>A recent column in one of >>the Unix trade mags reveals that the Vax 780 = 1 MIPS rule-of-thumb >>may be grossly overstated. Omri Serlin reported that according to DEC data, A VAX 780 executes about 470 thousand instructions per second for "typical" workloads. In article <1988Mar14.002026.3977@gpu.utcs.toronto.edu> dennis@gpu.utcs.toronto.edu (Dennis Ferguson) writes: > >This is well known. I suspect you will get more than one reply (beside >this one) reiterating the story about how the DEC types benchmarked the >11/780 against a then-current 370 which IBM was calling a 1 MIPS machine, >found it to run about the same speed, and so for marketing purposes called >the 11/780 a "1 MIPS" computer. Thus the "MIPS" referred to are supposed to >be native 370 MIPS, not native Vax MIPS. I doubt if anyone really knows for sure where the axiomatic "VAX 780 = 1 mips" came from. I think it had two (possiblly related) points of origin. 1. The VAX 780 was evaluated as 1.06 mips in the August 2, 1982 Computerworld Annual Survey. I spoke to someone at Computerworld at that time. He admitted the very approximate nature of their published mips ratings. He told me that they took performance claims from vendors, did some reason checking on them and published them with suitable statements about the approximate nature of such data. As far as I know DEC wasn't publicly making claims in 1982, that the 780 was equal in performance to the canonical 1 mip IBM machine, the 370/158 Model 3. (I don't know the instruction rate of the 158-3). 2. When systems other than VAXen entered the Unix marketplace, their performance was characterized relative to a VAX. The VAX was the obvious and convenient unit. Since vendors threw around mips claims like confetti on New Year's Eve, mips was a unit that everyone understood (or so someone thought). So if new Unix box had twice the performance of a VAX 780, it was called a 2 VAX mips box while a 780 was a 1 VAX mips box. > >This, unfortunately, is also not true, at least in my experience. I have >found that you can match benchmark results on a 370 and a Vax pretty well >by multiplying the IBM-reported "MIPS" number by 1.8 or so (i.e. a 13 MIPS >3090 goes faster than one would otherwise be led to believe). I really >think some marketeer at DEC just made up the 1 MIPS number so Vaxes would >look better against the IBM 370 competition. The fact that trade magazines >are just getting around to realizing this shows what a good idea it was. Some people at Amdahl published data last year on benchmarking in the IBM and Unix worlds. They should understand this better than anyone, since they sell and evaluate systems in both performance measurement worlds. My recollection is (someone at Amdahl correct me where necessary) that if an Amdahl system was X IBM mips then the same system would be about 1.5 X times the performance of a VAX 780 (NOT VAX MIPS !!!). >All of which matters not at all, since if you simply define a 780 to be >1 "MIPS" and measure everything against it, it all works out in the end >for many practical purposes anyway. > >Dennis Ferguson >University of Toronto Yes. The Unix performance measurement world does this. Usually we state performance as relative to a VAX 780. Unfortunately, even here there is room for confusion. Different versions of Unix have different compilers and (obviously) kernels which have different performance characteristics. VAXes often perform better with VMS compilers. Which is the "real" VAX to measure against? The Unix marketing world has simplified X times the performance of a VAX 780 to X mips. Steve Schlesinger NCR Corporation Disclaimer: This is my opinion, not that of NCR Corporation.
len@elxsi.UUCP (Len Mills) (03/17/88)
In article <413@mn-at1.UUCP> alan@mn-at1.UUCP (Alan Klietz) writes: >We all know what a VAX "MIPS" is, right? But have you ever >bothered to actually measure it? DEC did. A recent column in one of >the Unix trade mags reveals that the Vax 780 = 1 MIPS rule-of-thumb >may be grossly overstated. > >Therefore one VAX 780 "MIPS" is approximately 0.44 "honest-to-god" >VAX MIPS? (HTGV MIPS?) Or a 126% overestimate? > I offer a tiny bit of (probably well-known) historical perspective. Many moons ago, in the dim ages of the past, ITEL's System Development Division performed some measurements on an IBM 370/158 clone (AS-5) using an event counter driven by "EndOP" (microcode end-of-instruction) which yielded 1.08 MIPS over a 36-hour period. Hourly rates varied between 1.01 and 1.20. Common industry usage rated the 158 at one MIP. It also rated the 158 at 2X a 780. From this usage comes the old rule-of-thumb that an IBM MIP is 2X a VAX MIP. This rule is still in common use among business oriented computer publications. Since this group is primarily interested in scientific usage rather than business usage, perhaps the continued use of VUPS is justified. I don't want all the flamage that would arise from proposing the use of IBM MIPS. >-- >Alan Klietz >Minnesota Supercomputer Center (*) >1200 Washington Avenue South >Minneapolis, MN 55415 UUCP: alan@mn-at1.k.mn.org >Ph: +1 612 626 1836 ARPA: alan@uc.msc.umn.edu (was umn-rei-uc.arpa) > >(*) An affiliate of the University of Minnesota -- Len Mills ... {uunet,ucbvax!sun,lll-lcc!lll-tis,altos86,bridge2}!elxsi!len
jesup@pawl16.pawl.rpi.edu (Randell E. Jesup) (03/20/88)
In article <413@mn-at1.UUCP> alan@mn-at1.UUCP (Alan Klietz) writes:
: In October 1987, one of the authors [of the DEC
: study] presented a paper on the performance of the VAX
: 8800, a dual-processor machine in which each processor
: is six times as fast as the 11/780, by DEC's official
: reckoning. This paper reported on some preliminary
: measurements that showed the 8800 processor to have
: a cycle per average instruction (CPAI) rate of 8.4.
: At cycle time of 45ns, this translates into a MIPS rating
: of 2.65.
Which goes to show that 1 8800 MIPS != 1 11/780 MIPS. Not suprising,
given the different memories, busses, caches, etc between an ancient old
(but ubiquitous) Vax 11/780 and an 8800.
That's why people specify 11/780 Vax MIPS, not VAX MIPS.
// Randell Jesup Lunge Software Development
// Dedicated Amiga Programmer 13 Frear Ave, Troy, NY 12180
\\// beowulf!lunge!jesup@steinmetz.UUCP (518) 272-2942
\/ (uunet!steinmetz!beowulf!lunge!jesup) BIX: rjesup
(-: The Few, The Proud, The Architects of the RPM40 40MIPS CMOS Micro :-)jiml@uwslh.UUCP (James E. Leinweber) (03/22/88)
I can't say what a VUP really is. But most of the current
calculations strike me as worthless, since you can't correctly figure the
instruction issue rate of a pipelined machine by dividing the average
cycles executing an instruction into it's clock rate. And assuming it has
one true clock speed is not necessarily valid either; we just
installed a variable speed clock on our obsolescent Vax 11/750.
Current machines are complex enough (variable instruction times, clock
speeds, cache effects, etc.) that no apriori calculation is going to
get you a MIPS figure. Benchmarks, anyone? :-)
Jim Leinweber jiml@uwslh.uucp jiml%uwslh.uucp@cs.wisc.edu
...!{rutgers, ucbvax, ihnp4, ...}!uwvax!uwslh!jiml
State Laboratory of Hygiene @ Univ. of Wisconsin - Madison; (608) 262-8092
--
Jim Leinweber jiml@uwslh.uucp jiml%uwslh.uucp@cs.wisc.edu
...!{rutgers, ucbvax, ihnp4, ...}!uwvax!uwslh!jiml
State Laboratory of Hygiene @ Univ. of Wisconsin - Madison; (608) 262-8092lamaster@ames.arpa (Hugh LaMaster) (03/22/88)
In article <310@uwslh.UUCP> jiml@uwslh.UUCP (James E. Leinweber) writes: >get you a MIPS figure. Benchmarks, anyone? :-) In fact, most of the so-called "MIPS" figures are based on benchmarks anyway - usually the vendor's idea of the "typical" workload - which is begging the question since users often pick machines based on the ability to handle particular workloads. [VAX 11/780's were good for some things- but not very good at floating point. So, since people didn't use them for floating point, floating point isn't "typical" and isn't weighted as heavily in DEC's internal benchmark suite...] Therefore, since all such performance figures depend on benchmarks, you may as well be explicit about it. I think the Linpack benchmark is an excellent STARTING POINT for measuring floating point performance. But, a lot of people are more interested in Unix kernel performance, and it is still a little unclear as to how to go about doing that. Certainly, some simple benchmarks can tell you something- how fast can you fork and exec - but how about file system performance, network performance, etc. ?
walter@garth.UUCP (Walter Bays) (03/24/88)
In article <6304@ames.arpa> lamaster@ames.arc.nasa.gov.UUCP (Hugh LaMaster) writes: >In fact, most of the so-called "MIPS" figures are based on benchmarks >anyway - usually the vendor's idea of the "typical" workload - which >is begging the question since users often pick machines based on the >ability to handle particular workloads. [...] Yes. Benchmarks should be at least somewhat representative of the real workload the user will run. >Certainly, some simple benchmarks can tell you something- how fast can >you fork and exec - but how about file system performance, network >performance, etc. ? The problem is that computer architectures are becoming too complex for performance to be accurately measured by simple benchmarks. These benchmarks are usually small, with a narrow locality of memory reference, do no I/O, and are run with a single user and a single process. Even if they get the instruction mix right, they can be poor predictors of performance on real workloads which (MAY) have large programs, multiple users, multiple processes per user, I/O, network activity, window management, ... Benchmark suites that test each of these functions individually are an improvement, but what is really needed is a way to test them all concurrently. (Preferably without porting 100 applications and data files and running them for a year :-) (I hope I got this posted right. I'm still learning my way around the net.)
coleman@sask.UUCP (Geoff Coleman @ College of Engineering) (03/24/88)
> > In article <413@mn-at1.UUCP> alan@mn-at1.UUCP (Alan Klietz) writes: > : In October 1987, one of the authors [of the DEC > : study] presented a paper on the performance of the VAX > : 8800, a dual-processor machine in which each processor > : is six times as fast as the 11/780, by DEC's official > : reckoning. This paper reported on some preliminary > : measurements that showed the 8800 processor to have > : a cycle per average instruction (CPAI) rate of 8.4. > : At cycle time of 45ns, this translates into a MIPS rating > : of 2.65. > > Which goes to show that 1 8800 MIPS != 1 11/780 MIPS. Not suprising, > given the different memories, busses, caches, etc between an ancient old > (but ubiquitous) Vax 11/780 and an 8800. For an interesting viewpoint on this see the INSIDE EDGE column in the April 1988 edition of UNIX WORLD. According to it a VAX 11/780 Mip is really 0.47 Mips. Geoff Coleman | BITNET: Coleman@sask College of Engineering | UUCP: {utcsri,ihnp4}!sask!skul!geoff University of Saskatchewan | Compserve: 76515,1513 just a number Saskatoon, Saskatchewan | voice: (306) 966-5415 "Why does a hearse horse snicker, hauling a lawyer away?" - Carl Sandburg This is food for the stupid posting program.
gruber@bgsuvax.UUCP (John Gruber) (03/26/88)
The poster of the referenced article wondered about how many MIPS
a VAX 780 ran.
Much of the computer industry uses MIPS as a metric for measuring computer
processor power; and uses it to compare processors from different families
and different manufacturers. If one wasn't interested in knowing how a
processor from DEC fared compared to one from, say, IBM, the comparison
of a 8600 to a 780 would be sufficient information.
Vendors claim that computing MIPS on machines by counting actual instructions
executed per second is unfair, because their machine architecture has
instructions which do more work than other machine architectures.
This sounded like bull to me; but it's not.
The number of machine instructions compiled to run the Dhrystone 1.1
benchmark on a VAX is considerably less than the number of IBM 360/370
instructions needed _for the same work_. And it's important to measure
work/second rather than something artificial. The Dhrystone 1.1 IBM
instructions appeared rather optimal to my trained eye. I don't have
the expertise to say how optimal the VAX instructions were.
I counted the IBM instructions, since I believe the comparisons to MIPS
started there many years ago and that this is what most of the industry is now
comparing. If you multiply this number by the ratio of the Dhrystones of two
machines, you can develop a IMIPS number how many IBM instructions worth of
Dhrystone work a processor does in a second. By running through the benchmark
compiled by Dhrystone 1.1, including subroutines, calculating how many times
each path is executed, I came up with 593 IBM instructions per Dhrystone 1.1
for the register variant.
My best Dhrystone 1.1 measurement for our 780 was 1628. My calculation is
that this is .965 IMIPS.
The VAX in fact executed a lot fewer instructions to complete the Dhrystones
than the IBM computers did. This seemed to be due to the multiple operand
instructions, with multiple addressing modes, that tended to replace
{load, figure, store result} sets of instructions with just one VAX
instruction. I would imagine the difference in measuring a RISC machine
in it's native MIPS would demonstrate more dramatically the difficulty
of using raw instructions/second measures in comparing different computer
architectures. I don't believe in honest-to-goodness-mips.
Note that the above IMIPS comparison assumes that Dhrystones are typical of
the workload one is interested in running on the machines in question, and
also assumes the level of optimization produced by the compilers used in
the benchmark. Your workload or compilers may vary. My results were with
the 4.3 cc compiler, and a MVS c compiler. Other results:
Computer Best Dhrystone 1.1 Perf. IMIPS
785 2018 1.20 IMIPS
4341-2 2407 1.43 IMIPS
8530 7065 4.19 IMIPS
4381-24 5747 3.41 IMIPS (each processor)
John Gruber gruber@andy.bgsu.edu tut!bgsuvax!gruber
--
John Gruber
University Computer Services UUCP:..!cbosgd!osu-eddie!bgsuvax!gruber
Bowling Green State University CSNET: gruber%bgsu@csnet-relay
Bowling Green, OH 43403-0125rmb384@leah.Albany.Edu (Robert Bownes) (04/03/88)
Randal & co.,
That's why, somewhere in Virginia there's this
HUGE bell jar with an 11/780 running 4.1BSD stuffed in it.....Oh, don't
forget the 16 "Standard users" all happily hacking away at their ADM-3a's.
Bob
--
Bob Bownes, Aka Keptin Comrade Dr Bobwrench III | If I didn't say it, It
bownesrm@beowulf.uucp (518)-482-8798 | must be true.
{steinmetz,brspyr1,sun!sunbow}!beowulf!bownesrm | - me, tonite -