jareed@rodan.acs.syr.edu (Judith Ann Reed) (03/05/91)
Recently, I asked for help comparing U*ix workstations, specifically
Decstation 5000/200's, Sparc 2's, IBM RS6000/320's, and some flavor of Silicon
Graphics machine. My goals were:
* Peacefully coexist with a 14 node LAVC
* Supply a lot of compute power for "background" jobs on the U*ix server (i.e.
act as a compute server)
* Provide 3 "seats" for interactive users, plus a Macintosh running X as a
seat and X server
* Share vast amounts of disk space between VMS and U*ix via NFS
* Find a way to meaningfully evaluate the above-mentioned machines to
determine which ones most cost-effectively meet our needs
I received a number of helpful responses, and I'd like to summarize them here.
Criteria:
* Try to determine if integer or floating point performance is more important
to your application, then look at Drhystone mips vs. Mflops.
* When evaluating performance, differentiate between whether you are concerned
about aggregate performance or performance on a single job.
* Prioritize items of interest - e.g.
(1) Performance/single job is top priority
(2) Cost/seat is second top priority
* Don't forget to figure in software purchase and warranty/maintenance costs
over some period of interest, and hardware warranty/maintenance costs over
that same period of interest.
* Minimum memory configurations are ridiculously inadequate - 16 MB is a bare
minimum, 32 MB is probably ok, 64 MB is good.
* Be sure to have local page and swap space on each workstation (at least
300 MB), diskless machine problems range from poor performance to frequent
crashes.
* Consider dedicating one small workstation as an NFS server, because the
loading is a significant factor. Configure it to have /usr/... areas,
have /root/... and /swap/... locally on each machine.
* Criteria to consider in comparing small to large workstations -
Is workload dominated by relatively small jobs that can fit into
memory and disk available on a small workstation (buy several small
workstations), or by a few large batch jobs that will each use up
huge amounts of memory (buy largest workstation you can afford)?
* Check the availability of 3rd party software for the box you are considering,
how much is available and are the packages you may need available?
Also make sure, with the DEC third-party software list, that the package
runs on the RISC systems as well as vaxes.
* To do primarily data manipulation using a lot of CPU power, get a big server
with fast disks and hang X terminals off of it.
To do primarily fancy 3D graphics or software development, go for multiple
workstations.
* SCSI disks are very slow (comparatively) and can be a real bottleneck with
a fast cpu. You may want to look at IPI or SMD subsystems if your usage
is I/O intensive.
* "In evaluating the "main" machine, pick a representation job or set of
jobs and benchmark them on each machine. This is the easy part for
just single application speed. What you really need is to test the
operational environment you expect. What types of graphics jobs are
you going to be running? Are all users interested in running long
color animations simultaneously? How many tasks are likely to be
running during the day? How many tasks at night? If you have lots of
vectorizeable code, then enhancing a simple platform like a Sparcstation
with a vector board or going with a built in system like a Stardent might be
of interest."
Benchmarking:
* Don't believe manufacturer's benchmarks - get loaners for a sufficient period
to get representative code running and timed on the various platforms. (This
was repeated again and again, and I've found that vendors are very willing
to provide these loaners when they realize they are in a competitive
situation. Don't get them all at once, though - get one machine, learn
to configure it, set up your benchmarks, and then port them to other machines
one by one.)
* Check recent issues of Digital Review for CPU benchmarks, also UNIX Review
* Context switching on the RISC systems slows performance considerably. It
may be desireable to reserve one system as a compute server, others as
interactive machines.
Look at "user time" in this respect. (not sure how to do this - suggestions?)
* Be sure to compare multiple concurrent process performance vs. single job
performance.
Comments on specific platforms: (loosely grouped by type)
* If current DEC hardware is on hardware maintenance, adding more DEC machines
may simplify maintenance issues.
* VMS-ULTRIX Connections software (UCX) uses a lot of CPU to serve disks
via NFS to Unix machines.
* One person mentioned that DEC's FORTRAN compiler has some very serious bugs.
* Some people have had problems with DEC's software support, either they don't
answer questions or take a very long time doing so, or lead you astray.
* DECstations support DECnet and LAT, which may be a big consideration in a
primarily VMS shop. We already do TCPIP, so it is not so much of an issue.
* There is a known bug in the 5.4 VMS UCX software - NFS file transfers will
hang on 2 MB or greater file writes.
* Sun's implementation of NFS is the best, since they developed it.
* SUN Sparcs are the cheapest hardware
* SUN's seem to be most people's preference, for reliability, ease of
configuration.
* SunOS may be closest to a generic UNIX, and is thought to be the easiest
to port to and from.
* More people have SUN's than other machines, so support from the user
community may be easier to come by.
* "The vendors NFS packages all do byte-swapping [in a mixed-environment] as
necessary -- UNLESS you've got mixed data-type binary files (it seems Suns
need to swap word data, but not character data...)")
* "The IBM RS6000 architecture is very fast. Period.
It's also got a quite impressive performance/price ratio."
* Questions remain about the current revision of the IBM operating system.
There is a new version due out soon, however (AIX 3??) which is supposed to
solve these problems. Problems exist in shareable writeable file areas ,
multiple simultaneous processes much slower than would be expected, problems
getting knowledgeable IBM system support,
* IBM's have a data integrity feature that prevents power outages from
corrupting the file system.
* Don't forget to look at the Mips Magnum
* The Mips machines have better I/O performance than the DECstations.
* HP has a new "super" workstation coming out soon, worth a look
* DEC and IBM are comparable in performance
* Check the openness of the I/O subsystems - Sun is good, DEC seems ok,
IBM is oriented to IBM buses, Silicon Graphics had vendors with whom they
are compatible, but they are not compatible with many SCSI disk
implementations.
* "If floating point performance is your primary criteria, then the ranking
would have to be IBM followed by DEC and Silicaon Graphics, with SUN a distant
third."
* IBM has some system utilities that aid in managing the systems and networks.
Silicon Graphics has icon-based utilities that perform similar functions.
These will aid an inexperienced UNIX system administrator in setting things
up. However, after trying to get a Silicon Graphics loaner talking to the
network I've found you still have to get into the operating system to get
it right.
* Silicon Graphics windowing system is icon-oriented, and incompatible with
the rest of the world. However, Motif is due out soon for them.
Macintoshes:
* Mac X can be very slow, it needs a high-powered CX or FX with about 8 MB
memory.
* Mac X on a system running the native Macintosh OS provides "rootless"
X-windowing which does not have the full functionality of an X environment
on a U*ix or VMS system.
* Dec has Pathworks for Macintosh which provides X functionality (via Mac X)
plus everything you never knew you needed on your Mac!
In summary:
I am getting loaners of each machine and running benchmarks of our target
software. I plan on testing multiple concurrent jobs, single job performance,
multiple jobs in the background with interactive foreground performance. I/O
is not too big an issue, but I am definitely going to try to have a lot of
local disk as opposed to using NFS for everything, because serving VMS disks
via NFS on my VAXstation 3100 nearly kills it! I am leaning toward a large
CPU with Xterminals because we are primarily interested in CPU cycles.
I have yet to determine price/seat, and I intend to use final quotes to develop
alternative configurations with pricing to compare how many mips we can get for
the $$ with real configurations.
In order to develop a comparison of price/vup, I spec'd out a "minimum"
configuration for each machine from quotes. In it, I included -
16 MB memory
approx. 200 MB disk
2 years of hardware/software maintenance
19" monochrome monitor, gray scale graphics
NFS, C, FORTRAN, GKS
no additional tape/disk/cd options
I got specmarks, mips and Mflop ratings for each cpu. I got costs for the
above mentioned configuration (ask the vendor to quote items out separately so
you can mix and match options - e.g. how much is the 19" monitor, how much
memory does it come with and how much are the increments?) as best I could.
I then computed $/mip, $/Mflop, $/Specmark.
Sun, IBM and Silicon Graphics will give you a specific discount up front,
DEC gives one discount at first, then when they find they are not in the
running they up the ante.
The machines we are looking at compare as follows (in that artificial minimum
configuration).:
Sparcstation SLC: (seat/server, 16 MB total, 207 MB, 17" mono)
12.5 Dhrystone MIPS
1.2 Mflops
7.6 Specmarks (overall) (integer mean - 9.4) (floating mean - 6.6)
Sparcstation IPC: (seat/server, 24 MB total, 207 MB, 16" color, 1.44 MB)
15.8 Dhrystone MIPS
1.7 Mflops
11.8 Specmarks (overall)
Sparcstation 2: (server/seat, 16 MB total, 207 MB, 1.44 MB, 19" mono)
28.5 Dhrystone MIPS
4.2 Mflops
21 Specmarks (overall)
DECstation 5000/200: (server/seat, 16 MB total, 209 MB, 1.44 MB, 19" mono)
24 Dhrystone MIPS
3.7 Mflops
18.5 Specmarks (overall)
DECstation 5100: (server, 16 MB total, 209 MB, 1.44 MB)
19.4 Dhrystone MIPS
14.9 Specmarks (overall)
DECstation 2100: (seat/server, 16 MB total, 209 MB, 19" mono)
10.0 Dhrystone MIPS
1.2 Mflops
8.3 Specmarks (overall)
Silicon Graphics 4D/20: (seat/server, 16 MB total, 200 MB, 19" color)
10.0 Dhrystone MIPS
0.9 Mflops
Silicon Graphics 4D/35: (seat/server, 16 MB total, 200GB, 1/4" tape, 19" color)
33.0 Dhrystone MIPS
6.0 Mflops
IBM RS6000/320: (seat/server, 24 MB total, 160 MB, 19" grayscale)
27.5 Dhrystone MIPS
7.4 Mflops
24.6 Specmarks
IBM RS6000/530: (seat/server, 16 MB total, 355 MB, 19" grayscale)
34.5 Dhrystone MIPS
10.9 Mflops
32.0 Specmarks
The following ratings are based on the above configurations. I include
them in hopes they may be useful to someone. DEC is coming back with new
discounts, so their ratings may improve.
Rated by price/MIP: (don't ask what 1.000 is - use your imagination! I don't
want to quote direct pricing)
Sparcstation 2: .353
Sparcstation SLC: .389
IBM RS6000/320: .403
Silicon Graphics 4D/35: .448
Sparcstation IPC: .469
DECstation 5100: .661
IBM RS6000/530: .696
DECstation 5000/200: .737
Silicon Graphics 4D/20: 1.207
DECstation 2100: 1.146
Rated by price/Mflop: (as above)
IBM RS6000/320: 1.499
IBM RS6000/530: 2.204
Sparcstation 2: 2.395
Silicon Graphics 4D/35: 2.466
Sparcstation IPC: 4.366
DECstation 5000/200: 4.786
Sparcstation SLC: 4.872
DECstation 2100: 9.558
Silicon Graphics 4D/20: 9.914
DECstation 5100: (no figures available)
Rated by price/Specmark: (as above)
IBM RS6000/320: .450
Sparcstation 2: .479
Sparcstation IPC: .629
Sparcstation SLC: .641
IBM RS6000/530: .751
DECstation 5100: .861
DECstation 5000/200: .957
DECstation 2100: 1.381
Silicon Graphics 4D/20: (no figures available)
Silicon Graphics 4D/35: (no figures available)
*********************************************
Thanks to:
Ron Fox, NSCL, Michigan State University
Chris Stradtman, COSMIC, Athens, Ga.
Minick Rushton, Space Telescope Science Institute
John Allen
Thierry Forveille, Observatoire de Grenoble
Peter Galvin, Brown Univ. Comp. Sci.
Janet Price
Greg Pavlov, Fstrf, Amherst, N.Y.
Drew Dean, Carnegie Mellon University
Christopher J. Calabrese, AT&T Bell Laboratories, Murray Hill, NJ
Dan Packman, NCAR
Steve Alter,Transaction Technology Inc.
Richard Seymour, University of Washington, Seattle
and to anyone else I missed!
***********************************************