tighe@convex.com (Mike Tighe) (12/07/90)
Not only is the benchmark itself worthless, but so are the times reported
by the benchmarkers. Why? Because the benchmark requires elapsed (real)
times to be reported, and this number can only be reliable when the
benchmark is run in dedicated mode. Look at these two examples:
Cray Y-MP (6ns) 5.0 elm@sprite.Berkeley.EDU
Cray YMP4/64 5.9 xxdon@monet.lerc.nasa.gov
These are basically the same machines, yet one is 15% faster. Here is
another example:
Cray XMP4/8 7.8 xxdon@monet.lerc.nasa.gov
Cray XMP (Unicos) 8.6 de5@ornl.gov
A similar difference. And for a Convex, this erroneous time was reported. I
wonder what the load was when this was run - 10?
Convex C220 9.4 xxdon@monet.lerc.nasa.gov
An advantage the RISC machines have is that they most likely to be personal
workstations, therefore when bc is run they are already in a "dedicated
mode".
--
-----------------------------------------------------------
Mike Tighe, Internet: tighe@convex.com
Voice: (214) 497-4206 Fax: (214) 497-4550
-----------------------------------------------------------elm@sprite.berkeley.edu (ethan miller) (12/07/90)
In article <109969@convex.convex.com> tighe@convex.com (Mike Tighe) writes:
%Not only is the benchmark itself worthless, but so are the times reported
%by the benchmarkers. Why? Because the benchmark requires elapsed (real)
%times to be reported, and this number can only be reliable when the
%benchmark is run in dedicated mode. Look at these two examples:
%
%Cray Y-MP (6ns) 5.0 elm@sprite.Berkeley.EDU
%Cray YMP4/64 5.9 xxdon@monet.lerc.nasa.gov
In the case of the Y-MP that I reported, I used the user+system time
charged to the program. The Y-MP has a cycle counter (and the /bin/time
command reports charged cycle counts), so I *know* the number I
reported is accurate. I didn't report elapsed wall clock time, as
it would depend a lot on how loaded the machine was.
%An advantage the RISC machines have is that they most likely to be personal
%workstations, therefore when bc is run they are already in a "dedicated
%mode".
Another advantage workstations have is that they are as fast on scalar
code. I'm sure bc was taken straight from Unix and unoptimized--after
all, how many Cray cycles are spent running bc? The startup latency
for memory requests on supercomputers is relatively high. If the
program uses non-vector accesses, each memory reference can cost
nearly 20 cycles (~120ns for a Y-MP). This is more than the
access time from cache of a load-store RISC architecture.
ethan
=================================
ethan miller--cs grad student elm@sprite.berkeley.edu
#include <std/disclaimer.h> {...}!ucbvax!sprite!elm
Witty signature line condemned due to major quake damage.meissner@osf.org (Michael Meissner) (12/08/90)
In article <109969@convex.convex.com> tighe@convex.com (Mike Tighe) writes: | Not only is the benchmark itself worthless, but so are the times reported | by the benchmarkers. Why? Because the benchmark requires elapsed (real) | times to be reported, and this number can only be reliable when the | benchmark is run in dedicated mode. ... That is certainly the conventional wisdom, but let me offer something. You should benchmark a machine in the state it's going to be used. For example, it is unrealistic to benchmark a large machine with nothing else running, if you can't afford to buy the machine for that single task, and that task only (ie, make the Cray your workstation). Thus, if fifty people are going to be sharing a machine in a timesharing fashion, you should test it with multiple jobs running. Similarly, when benchmarking single user, multitask workstations (ie, a typical UNIX box), you should have the network daemons running, X windows, a couple of xterms running, and possibly GNU emacs doing something useful. -- Michael Meissner email: meissner@osf.org phone: 617-621-8861 Open Software Foundation, 11 Cambridge Center, Cambridge, MA, 02142 Considering the flames and intolerance, shouldn't USENET be spelled ABUSENET?