[comp.unix.wizards] Between a Sun-4 and a Cray-2

tamir@CS.UCLA.EDU (07/30/87)

I would like to find out which machines are currently
available on the market whose performance is significantly
better than a Sun-4 (>> "10 MIPS") but whose cost is
significantly lower than a Cray-2 (<= $1M).

I am *NOT* interested in multiprocessors, such as Sequent,
where the performance of each processor is less than
or equal to a Sun-4.

I am particularly (but not exclusively) interested in high performance
*integer/string* operations as opposed to vectorized floating point.

Are there any such machines available on the market ?

			   Yuval Tamir

Internet: tamir@cs.ucla.edu
    UUCP: ...!{ihnp4,ucbvax,sdcrdcf,trwspp,randvax,ism780}!ucla-cs!tamir

mash@mips.UUCP (John Mashey) (07/30/87)

In article <7500@shemp.UCLA.EDU> tamir@CS.UCLA.EDU (Yuval Tamir) writes:
>I would like to find out which machines are currently
>available on the market whose performance is significantly
>better than a Sun-4 (>> "10 MIPS") but whose cost is
>significantly lower than a Cray-2 (<= $1M)....
>I am particularly (but not exclusively) interested in high performance
>*integer/string* operations as opposed to vectorized floating point.

1) As a side note, we're not sure where the 10-mips number
for the Sun-4 came from.  It looks more like 6.5-7.0 mips integer
(on the VAX 11/780 == 1 ... VAX 8700 == 6 scale).  
We got this from looking carefully at the published data,
filling in some holes in the tables, and cross-comparing with other
RISC systems. [We've got a 40-pager that analyzes the data in detail,
if anybody is interested.]

2) It would be helpful to know some more about the kinds of applications
you want to run.  Does it include scalar floating point, or no FP at all?
Performance ratios and cost/performance ratios can vary all over the
map according to the sorts of programs you want to run.  Even integer versus
string performance can vary wildly.

3) If you're looking for [for example] real 20-mips uniprocessors,
this year, the only things that do that are real mainframes, like Amdahls &
such, and the costs rapidly get up in the $1M ballpark or significantly
over. It won't be for another year that you can get real 20-mips micros.
Structurally, the computer business has tended NOT to have
ultra-high-performance integer engines that aren't also, or at least
capable of being, serious about floating point.
-- 
-john mashey	DISCLAIMER: <generic disclaimer, I speak for me only, etc>
UUCP: 	{decvax,ucbvax,ihnp4}!decwrl!mips!mash  OR  mash@mips.com
DDD:  	408-991-0253 or 408-720-1700, x253
USPS: 	MIPS Computer Systems, 930 E. Arques, Sunnyvale, CA 94086

maiden@sdcsvax.UCSD.EDU (VLSI Layout Project) (08/17/87)

I was wondering if anybody from BBN or a qualified user could
mail or post (preferable mail) information about the BBN
Butterfly performance specifications (fully loaded system).
These specifications would ideally include "best case" parallel
FP performance as well as typical case FP and integer specs.
Thank you very much.


-- 
 Edward K. Y. Jung
 The Deep Thought Group: Searching for a better way to think.
 UUCP: {seismo|decwrl}!sdcsvax!maiden
 ARPA: maiden@beowulf.ucsd.edu

whm@arizona.edu (Bill Mitchell) (08/19/87)

A couple of weeks ago in article <552@winchester.UUCP>, mash@mips.UUCP
(John Mashey) wrote:

> 1) As a side note, we're not sure where the 10-mips number
> for the Sun-4 came from.  It looks more like 6.5-7.0 mips integer
> (on the VAX 11/780 == 1 ... VAX 8700 == 6 scale).  
> We got this from looking carefully at the published data,
> filling in some holes in the tables, and cross-comparing with other
> RISC systems. [We've got a 40-pager that analyzes the data in detail,
> if anybody is interested.]

We ran our IBS (Inscrutable Benchmark Suite) on a Sun-4/260 and ...
[Before I say what we found, I must point out that Sun said that the machine
that the tests were run on was (1) a prototype (2) running an early version
of the software (3) had a 65ns rather than 60ns clock (4) had something
wrong with the floating point (5) several other things of this ilk that
amount to the suite being run under adverse conditions.] ... found that
at the cpu level the Sun-4 was about 6 (six) mips.  The same tests had been
run on an 8600 and a Sun-3/280 and we got around 4 mips for both of these
machines.

I've yet to see a non-floating point benchmark that shows 10-mip speed on
a Sun-4, using VAXs as a yardstick.

					Bill Mitchell
					whm@arizona.edu
					{allegra,cmcl2,ihnp4,noao}!arizona!whm

roy@phri.UUCP (Roy Smith) (08/21/87)

In article <552@winchester.UUCP> mash@winchester.UUCP (John Mashey) writes:
> We got this [[7 or so mips rating for Sun-4 as opposed to the 10 mips Sun
> claims -- RHS]] from looking carefully at the published data, filling in
> some holes in the tables, and cross-comparing with other RISC systems.
> [We've got a 40-pager that analyzes the data in detail, if anybody is
> interested.]

	John send me that 40-pager ("A Sun-4 Benchmark Analysis", July 23,
1987) at my request, and I've got a few comments.  First off, is I have
learned to take with a grain of salt *anything* a computer manufacturer
says about any other manufacturer's products.  With this disclaimer out of
the way, the report seems to make a lot of sense, at least on the surface.
The problem is that Sun claims 10 mips performance for the Sun-4, while
MIPS insists it's more like 7 mips.  That's a 30% difference, which in my
book just isn't that much for this type of calculation.

	Also, I don't like the performance graphs in the MIPS report.  The
graphs have "Performance" on the Y-axis and "Benchmark Type" on the X-axis,
with lines connecting the data points.  This is wrong, since it indicates
that "Benchmark Type" is a continious function, and implies that you can
interpolate between the data points to find values for intermediate types
of benchmarks.  These graphs should have been bar graphs, or scatter plots,
or some other type of plot which is more suitable to discrete data.  BTW,
the graphs look like they were done using CricketGraph on a Mac.  Very
pleasing to the eye, even if they are misleading. :-)

	One point made in the report, which I feel is vitally important, is
that floating point speed doesn't scale well with integer speed.  For us,
floating point is where it's at.  I believe the Sun-3/100, 3/200, and 4/200
series machines all use the same FPA, based on the Wietek chip set.  What
that means is that the 3/100 series has the "best" floating/integer speed
ratio, and the 4/200 series the "worst".  You get the same kind of
non-linearity with the Vax-750/780/785/8600/8[78]00 progression, I think.
-- 
Roy Smith, {allegra,cmcl2,philabs}!phri!roy
System Administrator, Public Health Research Institute
455 First Avenue, New AM serinalo
A= 

roy@phri.UUCP (Roy Smith) (08/23/87)

In article <2866@phri.UUCP> I wrote:
> the Sun-3/100, 3/200, and 4/200 series machines all use the same FPA, based
> on the Wietek chip set.  What that means is that the 3/100 series has the
> "best" floating/integer speed ratio, and the 4/200 series the "worst".

	It has since been pointed out to me that while the first sentence is
true, the second one is misleading, if not actually false.  Yes, all three
machines use the same FPA (not clear if it's actually the same board, or a
similar board based on the same chip set), but on the 4/2xx machines, the
integer cpu and the FPA are closer coupled than on the 3/2xx's, the 4/2xx has
better (presumably this means faster) memory, and better compilers.  All this
adds up to a 1.5-2X increase in floating point speed.

	Personally, I think the "better compilers" reason is a crock since
with appropriate effort, the 3/2xx could have better compilers too, but I can
understand why Sun would want to devote more programmer time to working on
the compilers for their new machines than for their old ones.  On the other
hand, since the Sun-4 series is a RISC machine, it's probably easier to write
really good compilers for it (or at least, that's the theory), so maybe I'm
being unfair (again :-)).
-- 
Roy Smith, {allegra,cmcl2,philabs}!phri!roy
System Administrator, Public Health Research Institute
455 First Avenue, New York, NY 10016

mash@mips.UUCP (John Mashey) (08/23/87)

In article <2866@phri.UUCP> roy@phri.UUCP (Roy Smith) writes:
>	John send me that 40-pager ...  a few comments.  First off, is I have
>learned to take with a grain of salt *anything* a computer manufacturer
>says about any other manufacturer's products...
ALWAYS! healthy skepticism is a Good Thing.

>...The problem is that Sun claims 10 mips performance for the Sun-4, while
>MIPS insists it's more like 7 mips.  That's a 30% difference, which in my
>book just isn't that much for this type of calculation.
Besides the usual issue that single-metric performance metrics are bad,
and that mips-numbers only get used because everybody gets forced into it,
this comment raises the interesting issue of human perception of differences.
To look at the general case, assume that a vendor claims one thing,
and reality is something else (assuming the reality is something you can
measure, i.e., NOT something like unspecified mips).
Here are some interesting questions:
	1) Do you think of the difference as ((claim - real)/claim * 100) % ?
	(i.e., the percentage of the claim that is missing)
	2) Do you think of it as ((claim - real)/real * 100) % ?
	(i.e., the percentage overstatement)
	3) For your choice of 1), or 2), at what percentage do you think it's
	worth bothering thinking that X & Y are different?
	4) Do any of you ideas change as mips ratings go up?  For example,
	maybe it's sensible to distinguish between 3.5 and 4 mips (12-14%),
	but maybe not between 9.5 and 10 (5%), and it seems irrational to
	distinguish between 59.5 and 60.
Obviously, all of these are a matter of opinion.  What I'm curious about is
how human perception works when applied to computer performance
characterization.

>	Also, I don't like the performance graphs in the MIPS report.  The
>graphs have "Performance" on the Y-axis and "Benchmark Type" on the X-axis,
>with lines connecting the data points.  This is wrong, since it indicates
>that "Benchmark Type" is a continious function, and implies that you can
>interpolate between the data points to find values for intermediate types
>of benchmarks.  These graphs should have been bar graphs, or scatter plots,
>or some other type of plot which is more suitable to discrete data.  BTW,
>the graphs look like they were done using CricketGraph on a Mac.  Very
>pleasing to the eye, even if they are misleading. :-)

Actually, they're a mixture of Cricket, Excel, and MacDraft.
There was absolutely no intention of implying interpolattion between points.

The good thing about computer graphics is that you can redraw graphs quickly.
The bad thing about it is that you can redraw them OFTEN.  We tried numerous
different ways to present the data, including various flavors of bar graphs
and scatter plots, and the ones we ended up with, somewhat to my surprise,
were much easier to understand than the others.  Here is the reasoning:
1) The goal is to show performance of machines and benchmarks, with enough
of both together to see how they compare on those benchmarks,
especially to get overall "performance envelopes".
2) If you use the bar graph that shows one benchmark versus a bunch of
machines, you get a good view of that, but you get no idea of overall
comparisons.
3) If you use the bar graph that shows several benchmarks versus a bunch of
machines, the graph gets cluttered very quickly, and it is much
harder to see a-b comparisons.  The most complex graph had 42 data points,
and it is just incomprehensible if done this way.
4) If you use scatter plots, with no lines connecting them, but little
symbols instead, you again find it gets cluttered quickly.
I've seen people end up connecting the symbols to figure out what's
happening.
5) If you use "stem-and-leaf" charts (which I like, akin to the way
I've seen some 8700-780 comparisons done), you get a real good idea
of overall performance envelopes, but it's hard to do a-b comparisons
for specific benchmarks, and it's hard to do more than about 2 machines
at once without really baroque graphics.
6) The line chart makes a few things clear, if you're lucky:
	a) machine A is always faster than machine B, for the set of benchmarks
	(the lines never cross).
	b) machine A is usually faster than B, except on a few benchmarks,
	which are made obvious by line-crossings.
	c) Missing data points are obvious [they're not as obvious on
	most bargraphs].
Point 6c) is important: one of the easiest ways to be mislead by some
numbers you get is by having partial sets of data, and getting excited
about extreme cases, without looking at overall patterns.   For example,
if all I saw was Dhrystone and character-pushing, I'd sure think
a Cray was a slow machine. [Well, not slow, but not as fast as the price.]

For this whole area, I cannot recommend the following book highly
enough.  In my opinion, it's in a league with "The Elements of Style":

Edward R. Tufte, "The Visual Display of Quantitative Information",
Graphics Press, Cheshire, Connecticut, 1983.  About $35, worth more.
-- 
-john mashey	DISCLAIMER: <generic disclaimer, I speak for me only, etc>
UUCP: 	{decvax,ucbvax,ihnp4}!decwrl!mips!mash  OR  mash@mips.com
DDD:  	408-991-0253 or 408-720-1700, x253
USPS: 	MIPS Computer Systems, 930 E. Arques, Sunnyvale, CA 94086

root@hobbes.UUCP (John Plocher) (08/26/87)

+---- John Mashey writes in <616@winchester.UUCP> ----
| 	4) Do any of you ideas change as mips ratings go up?  For example,
| 	maybe it's sensible to distinguish between 3.5 and 4 mips (12-14%),
| 	but maybe not between 9.5 and 10 (5%), and it seems irrational to
| 	distinguish between 59.5 and 60.
+----

Not talking about mips, but clock speeds:  IBM PC-AT Clones (no flames, please)
are very price competetive, and there is mucho advertizing of clock rates.
The current prices for a 10 MHz machine with 0 wait states is about $2000
while machines running at 12 MHz and 1 wait state go for about $2500.  Not too
bad a difference until you realize that the 10/0 machine runs FASTER than the
12/1 machine!  It's sobering to think that people are buying the more expensive
one just because of the numbers...  I suppose this observation does hand in
hand with the $49.99 -vs- $50 price comparisons... :-)  Most of *us* wouldn't
admit being swayed by the higher number, but isn't a 50 mips machine "better"
than a 45 mips one? :-(

-- 
John Plocher uwvax!geowhiz!uwspan!plocher  plocher%uwspan.UUCP@uwvax.CS.WISC.EDU