aglew@urbsdc.Urbana.Gould.COM (04/22/88)
OK, how do people out their evaluate their computer system's performance? Forget that I know anything, tell me the whole story from ground 0... I know about simulations. I also know that you have to try and calibrate simulations to reality. So, how do you make the measurements necessary to calibrate a simulation to reality? Here I'm interested in micro-scale measurements, like instruction histograms, not macro-scale like I/Os. I'm also interested in applications that have a significant interactive component, ie. the classic UNIX application. These you cannot simply run at 1/10th or 1/100th speed, in trace mode of your simulator or debugger; if they're not made in real time, they're not real measurements. Seems to me that to make real-life measurements you have to have equivalent speed hardware sampling hardware events like instruction dispatch or memory access. Or, you can do time-based sampling, making the sampling frequency such that it doesn't load the system. Unfortunately, sampling is skewed by the time of the _previous_ instruction, or worse effects in high performance systems with out-of-order execution and overlap. Are there other alternatives? Are there things that hardware can do to support measurement methodolgy? Eg. taking the sampling interrupt before the commit point or putaway phase of an instruction reduces skew in time based sampling. Or how about building a circuit to cause a sampling interrupt every x000000 instructions issued, to get instruction issue based sampling instead of time based sampling (remember, I want to do sampling instead of tracing (= sampling every event) so that I've got a fighting chance of making measurements while the system is in use). Or a trap on every branch, to get basic block flow-of-control information. I get more than somewhat tired of seeing papers evaluating new machines based on the old "standard" set of benchmarks. The standard benchmarks may have some correspondence to reality, but how can I assure that they correspond to my reality. There're a lot of papers on performance evaluation using statistical techniques, simulation, etc. etc. -- but what about the real measurements that are supposed to be evaluated? aglew@gould.com