reiter@endor.harvard.edu (Ehud Reiter) (05/12/87)
In regard to the discussion about using real programs with I/O as benchmarks,
I think one question that has to be addressed is what kind of timing should
be used. The possibilities include:
1) Elapsed Real Time. This would seem to be too dependent on the
I/O devices, and on where the files physically reside on the disk (e.g.
how long do the seeks take?). It also ignores multiprocessing.
2) Kernal Plus User Time. This would seem to be a good measure, which
has the advantage of including O.S. overhead in the timings (O.S. overhead is
part of the timing for real applications!). Problems include
-- A multitasking, multiuser kernal would have to do a lot more
bookkeeping than a single task, single user system, so the multiuser
system would seem slower. Is this "fair" - I'm not sure.
-- An I/O system which went through a lot of effort to optimize disk
accesses and file allocation would look slower than a stupid file system,
since this time measure doesn't include I/O access time. This does not
seem "fair".
3) User Time. This ignores O.S. overhead, and also penalizes a file
system which does most of its work in user mode, against a file system
which does most of its work in kernal mode.
4) I/O-less operation. We could rewrite the benchmark programs so that all
I/O is done in big blocks, outside the timing loop. Totally ignoring
I/O makes our benchmark less representative of real programs, but it
does have the advantage of making the benchmark more controlled and perhaps
less subject to fiddling by marketing types (who, for example, might try running
the program with output directed to the null device instead of a disk!).
Ehud Reiter
reiter@harvard (ARPA,BITNET,UUCP)
reiter@harvard.harvard.EDU (new ARPA)johnw@astroatc.UUCP (John F. Wardale) (05/14/87)
> [paraphrased:] should you use wall time, system+user time, or just > user time As a software developer who uses (almost excludively) "complete systems" I want benchmark timings to be real (wall, or elapsed) time. I don't care how its distributed! If/when I type "make all" (or what ever I'm working on) all that maters to me, is how long I have to sit (and read news :-) while the make is running. To average out system loading, report wall time for running it: 1) on an "empty" system 2) ten times from ten terminals 3) forking 10 parralel copies from each of 10 terminals (100 from 100 terminals would be better, but *HARD* to do in reality) John W - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Name: John F. Wardale UUCP: ... {seismo | harvard | ihnp4} !uwvax!astroatc!johnw arpa: astroatc!johnw@rsch.wisc.edu snail: 5800 Cottage Gr. Rd. ;;; Madison WI 53716 audio: 608-221-9001 eXt 110 To err is human, to really foul up world news requires the net!
collins@encore.UUCP (Jeff Collins) (05/15/87)
There has been some discussion of using a make as a performance benchmark in this news group. That seems like a good idea to me. I suggest that we use the BSD 4.3 source as the test. Ie. compile virgin 4.3 source code and report the times. I am willing to collect the times and post results. Any takers? Jeff Collins