clc5q@shamash.cs.Virginia.EDU (Clark L. Coleman) (10/10/90)
The September 17, 1990 issue of PC WEEK has an advertisement on pages 56-57
from Everex. The ad claims the following Dhrystone benchmarks:
Everex STEP 486/33 : 34,000 Dhrystones (19.4 MIPS)
CRAY-X-MP/48 : 17,857 Dhrystones
IBM 3083 : 16,666 Dhrystones
Is the explanation :
1) The Everex compiler got rid of a worthless inner loop in a stupid
benchmark.
2) The i486-based PC is really faster on this scalar benchmark than
a Cray X-MP/48.
3) They figured out some way to hamper the Cray for the benchmark.
4) They're lying.
5) None of the above.
I realize that Dhrystones are not meaningful for supercomputer applications,
but I thought that the scalar integer MIPS of a Cray would still exceed a
PC. Is there any independent source of Cray performance on this test?
-----------------------------------------------------------------------------
"We cannot talk of freedom unless we have private property." -- Gavriil Popov,
Mayor of Moscow, September 11, 1990. ||| clc5q@virginia.eduzabaly@convex.com (Frank Zabaly) (10/12/90)
In article <1990Oct9.192119.4453@murdoch.acc.Virginia.EDU> clc5q@shamash.cs.Virginia.EDU (Clark L. Coleman) writes: >The September 17, 1990 issue of PC WEEK has an advertisement on pages 56-57 >from Everex. The ad claims the following Dhrystone benchmarks: > > Everex STEP 486/33 : 34,000 Dhrystones (19.4 MIPS) > CRAY-X-MP/48 : 17,857 Dhrystones > IBM 3083 : 16,666 Dhrystones > >Is the explanation : <Various possible explanations omitted> Just thought I would quote from "Computer Architecture: A Quantitative Approach" by Hennessy and Patterson, p.48, reproduced without permission: "While there are rationalizations for use early in the design, there is no current valid rationale for using benchmarks and kernals to evaluate working computer systems ... While kernals, toy benchmarks, and synthetic benchmarks(Whetstone and Dhrystone) were an attempt to make fair comparisions among different machines, use of anything less than real programs after initial design studies is likely to give misleading results and lead the designer astray." They later mention that "Dhrystone has a long history of optimizations that skew its performance" and cite a specific string copy operation which can be optimized by a specific compiler, resulting in a 20-30% improvement in performance. They also mention that over 99% of real-world string copies could not be improved by this optimization. I don't pretend to understand the rationale behind benchmarks, but when I read Mr. Coleman's post, I trust them even less than I did before! >"We cannot talk of freedom unless we have private property." -- Gavriil Popov, >Mayor of Moscow, September 11, 1990. ||| clc5q@virginia.edu You have got that right, Gavriil old boy. Bye Frank E. Zabaly UUCP:{sun,harvard,rutgers,uunet}!zabaly!convex Convex Computer Corp INTERNET :zabaly@convex.com ############################################### ########################### "Something semi-automatic # Caffeine, the wonder drug might just do the trick" # that works and works and --Timbuk3 "Standard White Jesus" # works and works and ... ############################################### ###########################
ds@juniper09.cray.com (David Sielaff) (10/13/90)
In article <1990Oct9.192119.4453@murdoch.acc.Virginia.EDU> clc5q@shamash.cs.Virginia.EDU (Clark L. Coleman) writes: >The September 17, 1990 issue of PC WEEK has an advertisement on pages 56-57 >from Everex. The ad claims the following Dhrystone benchmarks: > > Everex STEP 486/33 : 34,000 Dhrystones (19.4 MIPS) > CRAY-X-MP/48 : 17,857 Dhrystones > IBM 3083 : 16,666 Dhrystones > >Is the explanation : > > 1) The Everex compiler got rid of a worthless inner loop in a stupid > benchmark. > > 2) The i486-based PC is really faster on this scalar benchmark than > a Cray X-MP/48. > > 3) They figured out some way to hamper the Cray for the benchmark. > > 4) They're lying. > > 5) None of the above. > >I realize that Dhrystones are not meaningful for supercomputer applications, >but I thought that the scalar integer MIPS of a Cray would still exceed a >PC. Is there any independent source of Cray performance on this test? I can't claim to be an independent source (I do have a bit of bias in this particular case ;-) ), but I have a comment or two about the Cray numbers. Looking at records I found containing benchmarks of various Cray machine/ Cray C compiler combinations over the years. The most recent benchmark I found is 27042 Dhrystones on a Cray X-MP/48 with C 4.0 (This compiler was released about two years ago). I just ran it myself with our most recent C compiler on a Cray X-MPEA/464 (most letters and numbers, but essentially the same machine as far as this benchmark goes), and got 28079 dhrystones. The benchmark they are quoting is with C 2.0 (about four years old), on an older generation of X-MP machines with a slower clock speed (9.5 ns as opposed to 8.5 ns). Plus the fact that the /4 part of the machine type means that the machine has four CPU's, which is meaningless for this benchmark. All of the timings are for one processor. Plus we haven't been making X-MP's for a couple of years (our current product, the Y-MP has a clock speed of 6 ns, a bit faster). And finally, Cray's are designed for floating point, not integer arithmetic, so a lot of extra conversions from integer format to floating point format (and back) will need to be done. If you've stayed with me for this long, what it comes down to is this: benchmarks, like statistics, say exactly what you want them to. They are not lying. They're just not telling the whole truth. I hope this didn't sound like a whole bunch of marketing drivel, I just wanted to point out ways to make benchmarks look exactly how you want them to, not just on a Cray, but any machine. >----------------------------------------------------------------------------- >"We cannot talk of freedom unless we have private property." -- Gavriil Popov, >Mayor of Moscow, September 11, 1990. ||| clc5q@virginia.edu Dave Sielaff Cray Research, Inc. ds@cray.com uunet!cray!ds
ddt@ccwf.cc.utexas.edu (David Taylor) (10/14/90)
In article <161512.23931@timbuk.cray.com> ds@juniper09.cray.com (David Sielaff) writes: >In article <1990Oct9.192119.4453@murdoch.acc.Virginia.EDU> clc5q@shamash.cs.Virginia.EDU (Clark L. Coleman) writes: >>I realize that Dhrystones are not meaningful for supercomputer applications, >>but I thought that the scalar integer MIPS of a Cray would still exceed a >>PC. Clark, when you compare PC's to Cray's, you compare VW Bug's to Ferrari's. Sure, Ferrari's may not get the best mileage, but there are NO substitutes. You can shop for 386 clones. There are no Cray clones. If you're interested in running the Dhrystone benchmark on a Cray X/MP, you're wasting extremely valuable clock cycles. There ought to be a law against that. Try comparing the Perfect Club benchmarks on a PC to a Cray. I think you'll find the Cray orders of magnitude faster. Pay no heed to these canabis-smoking "PC's-the-speed-of-Cray's" marketers. Pipe dreamers, the lot of 'em. >Plus the fact that the /4 part of the machine >type means that the machine has four CPU's, which is meaningless for this >benchmark. All of the timings are for one processor. [... stuff ...] >And finally, Cray's are designed >for floating point, not integer arithmetic, so a lot of extra conversions >from integer format to floating point format (and back) will need to be >done. Amen! Would you go muddin' (Texas term meaning driving pointlessly for hours in a muddy field) in a Ferrari, or would you use it as a high society status symbol/recreational racing vehicle? >If you've stayed with me for this long, what it comes down to is this: >benchmarks, like statistics, say exactly what you want them to. They are >not lying. They're just not telling the whole truth. Understatement of the year. SPEC is the closest that comes to the "whole truth", and even it is pretty far from the mark (bad pun). >Dave Sielaff >Cray Research, Inc. Listen to this man. He may not be marketing (virtue or vice?), but he is enlightened.. ;) =-ddt->
mash@mips.COM (John Mashey) (10/15/90)
In article <38230@ut-emx.uucp> ddt@ccwf.cc.utexas.edu (David Taylor) writes: >In article <161512.23931@timbuk.cray.com> ds@juniper09.cray.com (David Sielaff) writes: >>In article <1990Oct9.192119.4453@murdoch.acc.Virginia.EDU> clc5q@shamash.cs.Virginia.EDU (Clark L. Coleman) writes: > >>>I realize that Dhrystones are not meaningful for supercomputer applications, >>>but I thought that the scalar integer MIPS of a Cray would still exceed a >>>PC. Dhrystone is not geenrally meaningful these days, period. It is well-known that compiler hacks exist (around the strcpy call), that can boost apparent performance 30-40%, without affecting real programs in any noticable fashion. IT almost always overstates performance relative to a VAX, which is why it is so popular among some computer companies :-) ... >Understatement of the year. SPEC is the closest that comes to the >"whole truth", and even it is pretty far from the mark (bad pun). SPEC as it sits, is hardly close to "the whole truth", and of course didn't intend to be. I'd be very suspicious of anything that claimed to be the whole truth, as opposed to a better approximation to reality. For supercomputers, Perfect Club probably tells you more. -- -john mashey DISCLAIMER: <generic disclaimer, I speak for me only, etc> UUCP: mash@mips.com OR {ames,decwrl,prls,pyramid}!mips!mash DDD: 408-524-7015, 524-8253 or (main number) 408-720-1700 USPS: MIPS Computer Systems, 930 E. Arques, Sunnyvale, CA 94086
apb@boole.acc.Virginia.EDU (Alan P. Batson) (10/15/90)
In article <42097@mips.mips.COM> mash@mips.COM (John Mashey) writes: >In article <38230@ut-emx.uucp> ddt@ccwf.cc.utexas.edu (David Taylor) writes: >>In article <161512.23931@timbuk.cray.com> ds@juniper09.cray.com (David Sielaff) writes: >>>In article <1990Oct9.192119.4453@murdoch.acc.Virginia.EDU> clc5q@shamash.cs.Virginia.EDU (Clark L. Coleman) writes: >> >>>>I realize that Dhrystones are not meaningful for supercomputer applications, >>>>but I thought that the scalar integer MIPS of a Cray would still exceed a >>>>PC. >Dhrystone is not geenrally meaningful these days, period. >It is well-known that compiler hacks exist (around the strcpy call), >that can boost apparent performance 30-40%, without affecting real programs >in any noticable fashion. IT almost always overstates performance relative >to a VAX, which is why it is so popular among some computer companies :-) >... While it is indeed true that a high Dhrystone value doesn't mean much at all, it is certainly the case that a system giving a LOW value has some kind of weakness. Like one of our better-known Crayettes..... Alan P. Batson Academic Computing Center apb@virginia.edu University of Virginia (804) 924-0610 Gilmer Hall Charlottesville, VA 22903
kahn@batcomputer.tn.cornell.edu (Shahin Kahn) (10/17/90)
In article <42097@mips.mips.COM> mash@mips.COM (John Mashey) writes: >SPEC as it sits, is hardly close to "the whole truth", and of course >didn't intend to be. I'd be very suspicious of anything that claimed to >be the whole truth, as opposed to a better approximation to reality. >For supercomputers, Perfect Club probably tells you more. One comment on SPEC. It is a workstation benchmark as far as I can see. For supercomputers, even Perfect is not. (Some of the codes in there may be). What is the point of running a 12 second, 16 MByte problem on a supercomputer? And Nothing, just about, scales up linearly! So it is useless to run a small version of a big problem. And, it is not always the case that you increase the dimensions and run the program again and you have a supercomputer application! At some point you need new code, new algorithm, new science. I think in some areas applications are not being developed as fast as hardware is! Can it be that too many scientists are busy making back-ups of their pc's?! Shahin. disclaimer,etc.
fouts@bozeman.bozeman.ingr.UUCP (Martin Fouts) (10/21/90)
>>>>> On 14 Oct 90 02:57:57 GMT, ddt@ccwf.cc.utexas.edu (David Taylor) said: >> Clark, when you compare PC's to Cray's, you compare VW Bug's to Ferrari's. >> Sure, Ferrari's may not get the best mileage, but there are NO substitutes. >> You can shop for 386 clones. There are no Cray clones. There use to be Cray Clones. One went bankrupt and the other is now owned by Cray. By the way, the guy who owns the Lamborgini Contach parked two spots down from by Honda might disagree with you about Ferrari substitutes. >> If you're interested in running the Dhrystone benchmark on a Cray X/MP, you're >> wasting extremely valuable clock cycles. There ought to be a law against that. >> Try comparing the Perfect Club benchmarks on a PC to a Cray. I think you'll >> find the Cray orders of magnitude faster. Pay no heed to these canabis-smoking >> "PC's-the-speed-of-Cray's" marketers. Pipe dreamers, the lot of 'em. Silly man. (;-) I can do anything I want with what I own, even if that means running integer codes on a floating point machine. A law against that is a law against property ownereship. But on a more serious front: Certain workstations achieve floating point performance within 10% of a Cray on real applications. If you have to split your Cray 10 ways, the workstation is a win. >Plus the fact that the /4 part of the machine >type means that the machine has four CPU's, which is meaningless for this >benchmark. All of the timings are for one processor. >> [... stuff ...] >And finally, Cray's are designed >for floating point, not integer arithmetic, so a lot of extra conversions >from integer format to floating point format (and back) will need to be >done. Most user codes only run on one processor, by the way. >> Amen! Would you go muddin' (Texas term meaning driving pointlessly for hours >> in a muddy field) in a Ferrari, or would you use it as a high society status >> symbol/recreational racing vehicle? >If you've stayed with me for this long, what it comes down to is this: >benchmarks, like statistics, say exactly what you want them to. They are >not lying. They're just not telling the whole truth. >> Understatement of the year. SPEC is the closest that comes to the >> "whole truth", and even it is pretty far from the mark (bad pun). >Dave Sielaff >Cray Research, Inc. >> Listen to this man. He may not be marketing (virtue or vice?), but he is >> enlightened.. ;) >> =-ddt-> There are many paths to enlightenment. -- Martin Fouts UUCP: ...!pyramid!garth!fouts (or) uunet!ingr!apd!fouts ARPA: apd!fouts@ingr.com PHONE: (415) 852-2310 FAX: (415) 856-9224 MAIL: 2400 Geng Road, Palo Alto, CA, 94303 Moving to Montana; Goin' to be a Dental Floss Tycoon. - Frank Zappa
ds@juniper09.cray.com (David Sielaff) (10/22/90)
In article <97@garth.UUCP> fouts@bozeman.bozeman.ingr.UUCP (Martin Fouts) writes: > [...stuff...] >But on a more serious front: Certain workstations achieve floating >point performance within 10% of a Cray on real applications. If you >have to split your Cray 10 ways, the workstation is a win. One of the main thrusts of my original posts was to try to expain reasons not to say "as compared to a Cray", etc. That's like saying "My laptop is faster than an IBM!" Which IBM? How many years old? Doing What? I'm not trying to deny that you have certain benchmark results, but throwing around comments like "as compared to a Cray" is just a short distance this side of meaningless. I also don't believe that a Cray is the appropriate tool for all applications. Cray machines have been designed with certain ideas mind, like fast processor speed, fast vector processing, large fast memory (both main & secondary), etc. If you need a machine like that, perhaps a Cray (Y-MP, so I don't get flamed for not specifying ;-) ) is what you should use. >>Plus the fact that the /4 part of the machine >>type means that the machine has four CPU's, which is meaningless for this >>benchmark. All of the timings are for one processor. >>> [... stuff ...] >>And finally, Cray's are designed >>for floating point, not integer arithmetic, so a lot of extra conversions >>from integer format to floating point format (and back) will need to be >>done. > >Most user codes only run on one processor, by the way. True, but my point was that if you are going to specify that your benchmark is against a four processor machine, either run it using all four processors, or specify that it was run on one processor. >>> Listen to this man. He may not be marketing (virtue or vice?), but he is >>> enlightened.. ;) > >>> =-ddt-> > >There are many paths to enlightenment. >-- >Martin Fouts > Dave Sielaff