eugene@pioneer.UUCP (02/19/87)
From: Dick Dunn <ames!ico.ISC.COM!rcd> Message-Id: <8702180627.AA05785@violet.ISC.COM> I really enjoyed this message from Dick Dunn. He raises good issues. I have edited it bit. >The Dhrystone benchmark has done something positive for us: It's a >benchmark that tells us something about the work that a lot of us do. From >what I remember of where you are and what you do (gathered from 1/86 >USENIX), you're probably among a group of card-carrying number crunchers. >For my part, I count the interval between my forays into numeric work in >days or weeks--and even that is not <heavy> numeric work, just the >existence of a %...f in a program I run! There, Dhrystone gives a measure >that doesn't involve floating point at all, which is nice. > >Another nice thing about Dhrystone is that, like Whetstone, it gives us a >measure of a complete system--CPU, bus, memory/cache, compiler, and even OS >to the extent that it might intrude (hopefully little). It's a lot better >than just raw CPU measurements. That the Dhrystone does something positive is a false sense of security. The Whetstone is very little better. No, I am not a full time card carrying number cruncher, in fact, Crays and other big machines do not spend most of their time crunching numbers, they spend it fetching from memory like every-one else (only more efficiently: chain, etc.) There is a myth which Dick propagates below when he mentions "smaller machines." Most architectures differ little from the "von Neumann." A Cray differs little architecturally from a 6502. Unique architectures include: the ICL DAP, the Goodyear MPP, the STARAN, the new Multiflow, Hypercubes (too many), the ILLIAC IV, Dennis and Arvind dataflow proposals, the Manchester and the Japanese and TI dataflow machines, the Alliant, Cedar (not the Xerox system, but U. Ill.), the IBM RP3, the Ultracomputer, C.mmp, Cm*, and numerous other projects. The issue of just what Weicker's program measures is controversial. It is certainly not separable so I think hope is in vain. Interesting you did not mention the next paragraph as part of this. Removing floating does not help the problem: you should be able to similarly separate CPU from bus, from cache, from memory, and so forth, put it all back together and you have the System, or do you? >The downfall of Dhrystone--and most other simple benchmarks--is that it >attempts to reduce the performance of a system to a single number. > . . More elaborting failure of Single figure of merit (SFM)... No disagreement here. >I think the discussion about the effects of paging (in the articles you and >I wrote) pretty well indicates the problem--do we twiddle Dhrystone so that >it <does> reflect the cost of paging as much as possible, or do we arrange >it so that it shows the paging as little as possible? The paradox (your >view of it) arises because we're trying to use one number to describe a >position in two-space (where the coordinates are raw cpu performance and >real performance under nonzero paging load). > >For my part, I'd just as soon toss in some articles to confuse the issue >whenever people try to develop a single omniscient metric for things that >are so obviously multi-valued. If they stay confused long enough, maybe >they'll start to think, just out of desperation. (I know it's a lot to >hope for!) Paging is a good example for now, but there are quite a few >other aspects of performance. I think we should measure something (not Dhrystones) in a broad, yet consistent manner. It is my thesis, and I am touring important sites around the country trying to drum up support, that we have to test performance like we do functional testing. Consider that compiler validation suites have hundreds of systematic test programs. True they are not perfect, but I think they give a better picture of how a compiler works than single programs. We must do the same for performance. The marketing types might fear some degree of consistency, but the EPA ratings did not destroy the auto market (they are two figure of merit). Everybody has their own application. With a mass of numbers, we will separate out those who are really serious about understanding just what makes a machine run. The smaller machines you will allude to (next) can certainly hock their machines based on some small set of criteria. They are not trying to compete with more expensive machines. I am only asking for a consistent and systematic form of measurement like a Metric, a Second, a Kilogram. Not a 3 accord inch. P.S. paging is only one aspect I used as an example. >I'd particularly like to be able to create grief for smaller machines which ^^ like your use of the word >have particular benchmark-oriented features. The 286 systems are the best >example--you get to choose one of perhaps six different "computational >models" depending on "how fast you want the program to run" versus "how >useful you want it to be". We need benchmarks which force such machines to >be tested on real problems--and Dhrystone is too tiny to help there. We are working on this latter. The real tough nut is: what is a real problem? How do you characterize them? What about the COBOL, FORTRAN, C, LISP, Prolog and other pieces of code out there which differ lots: how do you compare them? [Naw that's written in a language which means nothing to me...] From the Rock of Ages Home for Retired Hackers: --eugene miya NASA Ames Research Center eugene@ames-aurora.ARPA "You trust the `reply' command with all those different mailers out there?" "Send mail, avoid follow-ups. If enough, I'll summarize." {hplabs,hao,nike,ihnp4,decwrl,allegra,tektronix,menlo70}!ames!aurora!eugene
reiter@endor.UUCP (02/19/87)
In article <342@ames.UUCP> eugene@pioneer.arpa (Eugene Miya N.) writes: >From: Dick Dunn <ames!ico.ISC.COM!rcd> >>a measure of a complete system--CPU, bus, memory/cache, compiler, and even OS >>to the extent that it might intrude (hopefully little). It's a lot better >>than just raw CPU measurements. There's another aspect to performance which is not mentioned above, and that's quality of the application software. I realize this is straying pretty far from computer architecture, but from the end-user's point of view, one of the most important determiners of how fast his system runs is how well his application software is written. A user might well prefer a "slow" machine with "fast" software, and a vendor might be well advised to spend more money improving the speed of commonly used programs like editors and text processors, and less money improving the speed of his cache. To take one small example, the VAX/VMS text processor, RUNOFF, is 5-10 times faster than NROFF on the same hardware (it's true that the functionality of the programs is different, but from my point of view they're equivalent, since both are perfectly adequate for my needs), and the VAX/VMS editor EVE is similarly much faster than EMACS (and again, both do what I want). So, as a user who seems to spend most of his time doing word processing these days, I might well prefer a "1 MIPS" box which could run RUNOFF and EVE to a "3 MIPS" box which ran NROFF and EMACS. I do NOT want to get into arguments of VMS vs UNIX, RUNOFF vs NROFF, or EMACS vs EVE. The only point I'm trying to make is that the true performance of a box depends on the available application software as well as on CPU speed, cache size, compiler quality, etc. Ehud Reiter reiter@harvard (APRA,UUCP,BITNET)
tuba@ur-tut.UUCP (02/22/87)
In article <1258@husc6.UUCP> reiter@harvard.UUCP (Ehud Reiter) writes: >...as a user who seems to spend most of his time doing word processing these >days, I might well prefer a "1 MIPS" box which could run RUNOFF and EVE to >a "3 MIPS" box which ran NROFF and EMACS. Prefer to use...or prefer to pay for? Following your suggestion, performance may be measured as formatted pages of text per second. Thus price / performance will be expressed as dollars per pages per second, or dollars per page-second. If cost is a consideration, prefer the lowest dollars per page-second. jon -- Jon Krueger Department of Necessary Evil University of Rochester uucp: {seismo, allegra, decvax}!rochester!ur-tut!tuba BITNET: TUBA@UORDBV
eugene@pioneer.UUCP (02/23/87)
I discovered an error in my earlier posting of location for
various benchmark sources and their availability via NBSLIB. I mailed
out the wrong host name (it's for a host within NBS and does not use NBS
in the name). Because they are still testing nbslib, please mail me
again (sorry) for the correct host to test getting benchmark source.
--eugene miya
NASA Ames Research Center
eugene@ames-aurora.ARPA
"You trust the `reply' command with all those different mailers out there?"
"Send mail, avoid follow-ups. If enough, I'll summarize."
{hplabs,hao,nike,ihnp4,decwrl,allegra,tektronix,menlo70}!ames!aurora!eugeneram@nucsrl.UUCP (02/28/87)
Like the enquiring minds which read PEOPLE, I WAN'T TO KNOW
1. Isn't there a benchmark of benchmarks :-)
2. Benchmark names for the next generation of super-duper machines.
a. Loadstone : benchmark for machines that uses
super-conducting super-cooled (para)magnetic
materials for the CPU.
b. Sandstone: By Yr. 2000 GaAs would be the primary substrate
All Si based machines would be benchmarked with Sandstone
c. Stone: AI would have progressed so much that rocks
would be capable of thinking and computing. So plain
old stone would be the name for those class of
computing elements.
3. How does one bring a machine to its kness?
I have not seen a single machine with legs so far :-)
I was told, when I was a Kid (not that I have grown now)
that if I pull myself by my shoe laces, I would fall on my knees.
They gave it a funny name which had the word boot in it.
renu raman
...ihnp4!nucsrl!ram
Northwestern Univ. Comp. Sci. Res. Lab.
PS1: Read Somewhere (Maybe Comm. of ACM) that LLNL(Correct me again)
has
17 Crays * $5M(Average) =$80M
100 VAXens* $0.4M(Average) =$40M
150 Suns * $0.1M(Average) =$15M
1500 PCs * 3000$(Average) =$4.5M
900 Macs * 2000$(Average). =$2.0M
+ Scores of assorted junkies =$10M
Total =151.5M
What's the budget for SDI?
Are the crays LLNLs version of Personal-Super-Computer?
PS2: Miya- Thanks for the bib. Keep them coming.
Heard that somwhere in the East Coast, there is a DARPA/DoD
funded centre for the Ultra-Machine(s) of the future. Anybody
heard of such rumors?hsu@uicsrd.CSRD.UIUC.EDU (03/03/87)
ram@nucsrl.UUCP writes: > 1. Isn't there a benchmark of benchmarks :-) Actually some work has been done on evaluating "benchmarks" for interconnection networks by Steven Levitan. The conclusion was that most network metrics do not predict network performance very well, and Levitan proposes using a suite of algorithms instead. Details can be found in Levitan's paper in the '85 Proceedings of the ICPP and Levitan's PhD dissertation. Bill Hsu
gnb@bby.oz.au (Gregory N. Bond) (01/11/90)
For what it is worth:
We just got our Solbourne upgraded to a Series 5 CPU, using the new
(33MHz?) Cypres CMOS sparc chip.
Solbourne MP/OS 4.0B (which is SunOs 4.0.1 + a bit), standard
compiler. Runnung full multi user, some background stuff happening.
No other CPU bound job. dhrystone.c, vers C/1.1, 12/01/84
Check this out:
leo# cc -O -DREG=register dhrystone.c -o dryRO
leo# ./dryRO
Dhrystone(1.1) time for 500000 passes = 13
This machine benchmarks at 36275 dhrystones/second
Which is better than the fastest machine listed: the IBM3090/200.
Wowzers! I though it felt faster than my 3/80!
Greg.
--
Gregory Bond, Burdett Buckeridge & Young Ltd, Melbourne, Australia
Internet: gnb@melba.bby.oz.au non-MX: gnb%melba.bby.oz@uunet.uu.net
Uucp: {uunet,pyramid,ubc-cs,ukc,mcvax,prlb2,nttlab...}!munnari!melba.bby.oz!gnbhunter@oakhill.UUCP (Hunter Scales) (01/12/90)
gnb@bby.oz.au (Gregory N. Bond) writes: >For what it is worth: >We just got our Solbourne upgraded to a Series 5 CPU, using the new >(33MHz?) Cypres CMOS sparc chip. >Solbourne MP/OS 4.0B (which is SunOs 4.0.1 + a bit), standard >compiler. Runnung full multi user, some background stuff happening. >No other CPU bound job. dhrystone.c, vers C/1.1, 12/01/84 >Check this out: > leo# cc -O -DREG=register dhrystone.c -o dryRO > leo# ./dryRO > Dhrystone(1.1) time for 500000 passes = 13 > This machine benchmarks at 36275 dhrystones/second >Which is better than the fastest machine listed: the IBM3090/200. >Wowzers! I though it felt faster than my 3/80! >Greg. >-- >Gregory Bond, Burdett Buckeridge & Young Ltd, Melbourne, Australia >Internet: gnb@melba.bby.oz.au non-MX: gnb%melba.bby.oz@uunet.uu.net >Uucp: {uunet,pyramid,ubc-cs,ukc,mcvax,prlb2,nttlab...}!munnari!melba.bby.oz!gnb Are you sure this is right? I get *70,093* dhrystones/second on my 33 MHz 88100 (Motorola model 8612) with the Diab Data dcc88 (ver. 2.34) compiler. I get around 42k dhrystones/sec with my 20 MHz 88100 machine. And this is dhrystone 2.1. The Solbourne compiler must be broken! -- Motorola Semiconductor Inc. Hunter Scales Austin, Texas {harvard,utah-cs,gatech}!cs.utexas.edu!oakhill!hunter #include <disclaimer.h>
mash@mips.COM (John Mashey) (01/12/90)
In article <2818@cerberus.oakhill.UUCP> hunter@oakhill.UUCP (Hunter Scales) writes: > Are you sure this is right? I get *70,093* dhrystones/second > on my 33 MHz 88100 (Motorola model 8612) with the Diab > Data dcc88 (ver. 2.34) compiler. I get around 42k dhrystones/sec > with my 20 MHz 88100 machine. And this is dhrystone 2.1. > > The Solbourne compiler must be broken! Oh good: a specific person who has run this compiler. It would be very interesting to post the lines of assembler code that are generated by the strcpy function call....(since, Reinhold W. specifically says the Dhyrstone performance cannot be compared without looking at the generated code these days...) -- -john mashey DISCLAIMER: <generic disclaimer, I speak for me only, etc> UUCP: {ames,decwrl,prls,pyramid}!mips!mash OR mash@mips.com DDD: 408-991-0253 or 408-720-1700, x253 USPS: MIPS Computer Systems, 930 E. Arques, Sunnyvale, CA 94086
bzs@world.std.com (Barry Shein) (01/12/90)
I submitted the dhrystone for the 3090/200 in that list about three
years ago and I just want to say I am pleased as punch that it's
finally being punched through by most everything over $10K and
probably a few under. Took long enough!
--
-Barry Shein
Software Tool & Die, Purveyors to the Trade | bzs@world.std.com
1330 Beacon St, Brookline, MA 02146, (617) 739-0202 | {xylogics,uunet}world!bzs