eugene@pioneer.arpa (Eugene Miya N.) (01/12/88)
Last week, I had an idea which I'm adding to a paper (the notorious
paper) I'm writing. The idea concerns itself with the conditions
under which benchmarking is done. The grading is not directly in the
measurement scope of a benchmark itself (This, I think is best covered
by David Bailey and John Barton in the NAS Kernel benchmarks
with their 0-1000 lines of modification). The grading for measurement
conditions is as follows:
E: Measurement program run on a timeshared system with no controls
what so ever. (Assumes no logging of system load or other
measure.) This condition is perfectly fine for most
development work.
D: Measurement program with restricted limited usage (may be some
background work).
C: Conventional "stand-alone" measurement. Can be a single-user system.
This is where most benchmarks fall, and it's conveniently in the
middle as well.
B: "Stand-alone" measurement with some controls: daemons either
turned off or killed.
A: Extremely elaborate measurement and recording facilities (hardware as well as
software), high precision clock and instruction monitoring,
ability to control environment using direct control
(using commands) AND indirect control using controlled work loads.
Comments by mail please.
From the Rock of Ages Home for Retired Hackers:
--eugene miya, NASA Ames Research Center, eugene@ames-aurora.ARPA
"You trust the `reply' command with all those different mailers out there?"
"Send mail, avoid follow-ups. If enough, I'll summarize."
{uunet,hplabs,hao,ihnp4,decwrl,allegra,tektronix}!ames!aurora!eugene