rcd@ico.ISC.COM (Dick Dunn) (05/20/88)
I decided to run the Dhrystone 2 benchmark on some machines we have around
here, just to satisfy my curiosity. As I started shoveling the source code
around and setting things up, I noticed that it seemed to take a Long Time
to do some of the "make"s (all tests were on UNIX systems). So I started
keeping track of the times. I found an extreme case where, on the actual
benchmark, machine A was about 80% of the Dhrystones of machine B; however,
the time (user CPU) to do the "make" on machine B was 5 TIMES what it was
on machine A! This is pretty extreme.
It would be interesting if people made notes of compilation times. I'm not
suggesting adding this as some "formal measure" because I don't think it's
a particularly accurate measure. However, it probably adds $.02 worth of
additional information. In particular, if you're looking for a fast
machine where you're going to do a lot of development, it would be a bad
mistake to look at the Dhrystone number and think that machine B above
would be better than A, or even comparable.
--
Dick Dunn UUCP: {ncar,cbosgd,nbires}!ico!rcd (303)449-2870
...Never attribute to malice what can be adequately explained by stupidity.eugene@pioneer.arpa (Eugene N. Miya) (05/20/88)
If you are going to record compile times, please also record sizes of
executable and non-executable (*.o) files. Most Unix files are are in
bytes, I trust those with 36-bit and other size machines to round to
reasonable sizes (i.e., like 9-bit bytes).
Another gross generalization from
--eugene miya, NASA Ames Research Center, eugene@aurora.arc.nasa.gov
resident cynic at the Rock of Ages Home for Retired Hackers:
"Mailers?! HA!", "If my mail does not reach you, please accept my apology."
{uunet,hplabs,hao,ihnp4,decwrl,allegra,tektronix}!ames!aurora!eugene
"Send mail, avoid follow-ups. If enough, I'll summarize."walter@garth.UUCP (Walter Bays) (05/22/88)
In article <5213@ico.ISC.COM> rcd@ico.ISC.COM (Dick Dunn) writes: >I found an extreme case where, on the actual [Dhrystone 2.0] >benchmark, machine A was about 80% of the Dhrystones of machine B; however, >the time (user CPU) to do the "make" on machine B was 5 TIMES what it was ^^^^^^^ >on machine A! ^^^^^^^ Really machine + compiler + optimization level. Changing any of the three can make an order of magnitude or more difference. Your example points out the very different needs of development and production environments. >In particular, if you're looking for a fast >machine where you're going to do a lot of development, it would be a bad >mistake to look at the Dhrystone number and think that machine B above >would be better than A, or even comparable. True in any case. Use Dhrystones to compare one MS-DOS PC against another with identical compilers, or one Mac against another with identical compilers, or one compiler against another on the same machine. Use Dhrystones to choose workstations "in the same ballpark" to look at more closely. Don't use them to completely characterize the differences between two machines or compilers. -- ------------------------------------------------------------------------------ The U.S. is to space exploration Any similarities between my opinions as Portugal was to sea exploration. and those of the person who signs my paychecks is purely coincidental. E-Mail route: ...!pyramid!garth!walter (415) 852-2384 USPS: Intergraph APD, 2400 Geng Road, Palo Alto, California 94303 ------------------------------------------------------------------------------
rick@pcrat.UUCP (Rick Richardson) (05/29/88)
In article <5213@ico.ISC.COM> rcd@ico.ISC.COM (Dick Dunn) writes: > >I decided to run the Dhrystone 2 benchmark on some machines we have around > >It would be interesting if people made notes of compilation times. I'm not >suggesting adding this as some "formal measure" because I don't think it's >a particularly accurate measure. However, it probably adds $.02 worth of >additional information. This has been suggested before. In order for those results to have any meaning, the disk system (bus, controllers, drives) would have to be indicated in the results. That can of worms has too many sources of errors, so it is left an exercise for "the reader". E.G. we have 8 drives, 4 controllers, and 2 busses; the drives spin at 3600, have 8 heads apiece, 36 sectors/track, 15 msec track-track seek time. The controllers can overlap seek on two drives simultaneously. The controllers buffer an entire track. The busses can move 10 MB/sec. /tmp is mounted on its own filesystem. The compiler is on /, also its own filesystem, and the source/object went to a third filesystem. The three filesystems were on three different disks, but only two different controllers. UNIX was in single user mode, but with all necessary filesystems mounted. The model numbers of all the above were XXXX,YYYY,ZZZZ. UNIX had 2000 buffers for the disk cache. The filesystems hadn't been repacked for optimal ordering in 3 months; in the meantime, we'd been creating lots of little files and then deleting them in a completely random way. There's a large area of bad blocks right in the middle of /tmp, and it is quite likely the compiler temporaries were getting remapped to the spare blocks. The list goes on ... Gak! Half the time, the HZ parameter was incorrectly determined. And that's just (more or less) one independant variable. With the disk system involved there are maybe two dozen more independant variables! I suggest that these types of comparisons be made by the purchaser, on systems configured nearly identical to those being contemplated for purchase. Vendor balks? Take a walk. -- Rick Richardson, President, PC Research, Inc. (201) 542-3734 (voice, nights) OR (201) 834-1378 (voice, days) uunet!pcrat!rick (UUCP) rick%pcrat.uucp@uunet.uu.net (INTERNET)
rick@pcrat.UUCP (Rick Richardson) (05/29/88)
In article <9057@ames.arc.nasa.gov> eugene@pioneer.UUCP (Eugene N. Miya) writes: >If you are going to record compile times, please also record sizes of >executable and non-executable (*.o) files. Most Unix files are are in >bytes, I trust those with 36-bit and other size machines to round to >reasonable sizes (i.e., like 9-bit bytes). The size information of the "*.o" files are useful information. I'll include that in the next posting of 2.0 (which Reinhold Weicker has just sent me). The executable size is dependant upon libc implementation, shared or not shared, etc. I think I'll avoid that one. -- Rick Richardson, President, PC Research, Inc. (201) 542-3734 (voice, nights) OR (201) 834-1378 (voice, days) uunet!pcrat!rick (UUCP) rick%pcrat.uucp@uunet.uu.net (INTERNET)
rcd@ico.ISC.COM (Dick Dunn) (06/01/88)
In regard to Dhrystone times, I said: > >It would be interesting if people made notes of compilation times. I'm not > >suggesting adding this as some "formal measure" because I don't think it's > >a particularly accurate measure. However,... ...to which Rick Richardson replied: > This has been suggested before. In order for those results to have any > meaning, the disk system (bus, controllers, drives) would have to be > indicated in the results... > ...E.G. we have > 8 drives, 4 controllers, and 2 busses; the drives spin at 3600, have > 8 heads apiece, 36 sectors/track, 15 msec track-track seek time... My original article suggested looking at CPU times to re-make the programs. In particular, I said that I had measured, on UNIX systems, the *user CPU* time. I see no reason to think that the I/O subsystem characteristics should have any more than a minor effect on this measure--which is why I suggested it! For a uniprocessor system--or, for that matter, with a single-thread compiler--the compilation time puts a lower bound on the real time to compile. That is, I/O latency can make the real time for compilation longer than the CPU time but not shorter. Moreover, in the cases of the bad compilers I found, the total CPU time (user+sys) was more than 95% of the total elapsed time. -- Dick Dunn UUCP: {ncar,cbosgd,nbires}!ico!rcd (303)449-2870 ...If you get confused just listen to the music play...
steve@edm.UUCP (Stephen Samuel) (06/03/88)
From article <507@pcrat.UUCP>, by rick@pcrat.UUCP (Rick Richardson): > In article <5213@ico.ISC.COM> rcd@ico.ISC.COM (Dick Dunn) writes: >>I decided to run the Dhrystone 2 benchmark on some machines we have around >>It would be interesting if people made notes of compilation times. I'm not >>suggesting adding this as some "formal measure" because I don't think it's >>a particularly accurate measure. However, it probably adds $.02 worth of >>additional information. > This has been suggested before. In order for those results to have any > meaning, the disk system (bus, controllers, drives) would have to be > indicated in the results. That can of worms has too many sources (he then goes on with a 5minute description of all the weird things his (mythical?) system does. Although he's certainly right about the kind of can of worms this produces, I have to agree that compile time gives a reasonable guess as to how effecient a system is at things OTHER than a drystone If CPU time and real time are both included, then you can also make a guess as to how much room there is to improve things on the I/O side of things (even if the improvement may not be technologically possible). -- ------------- Stephen Samuel Disclaimer: You betcha! {ihnp4,ubc-vision,seismo!mnetor,vax135}!alberta!edm!steve BITNET: USERZXCV@UQV-MTS