sergiop@atina.ar (Sergio Porter) (03/13/91)
We are in the process of putting together a benchmarking suite
for the Government to use in UNIX procurements.
They are aimed at evaluating in an objective way the performance
of systems offered in bids that require, most of the time, small
to medium scale systems (5-100 users range) for database and
office work.
The ultimate goal would be to compute a single number, closely
correlated to the overall performance, that can then be scaled
so that one can say "This system benchmarks at 18.5 ligth to
medium load users", or something like that.
Individual test results would be used for specifying minimun
requirements.
So far we have considered various alternatives. For each of them
I list the perceived disadvantages. Some of them may be more a
matter of taste rather than objective problems:
Benchmark Problems
------------------------- ----------------------------------------
SPEC Only CPU so far. Requires FORTRAN. Biased
towards floating point performance (yes,
I know you can use integer SPECmarks).
Dhrystones Don't know how to extend to multi-
processors. Seems to test the quality
of the compiler more than CPU performance.
MUSBUS Some tests give very unstable results
(for example, disk). Oriented to testing
soft dev load, not bussiness.
TP1 (Don't take seriously, I don't know very
much about it). Looks more a RDBMS engine
test than a system performance test. I
couldn't find source for non-RDBMS version.
bonnie, fsanalyze Don't know how to apply to multi-x systems,
where x is processor, disk or controller.
{terminal I/O tests} Nothing really good found so far.
Any suggestions wellcome. Better if source included. Actual experiences
used in large procurements, both good and bad, specially wellcome.
Royalty-free is a must, public-domain preferred. If you e-mail
replies, I will post a summary and, in the future, the final
decision taken.
Thanks in advance,
Sergio Porter
sergiop@atina.ar