[comp.os.vms] Summary

jareed@rodan.acs.syr.edu (Judith Ann Reed) (03/05/91)

Recently, I asked for help comparing U*ix workstations, specifically
Decstation 5000/200's, Sparc 2's, IBM RS6000/320's, and some flavor of Silicon
Graphics machine. My goals were:
 * Peacefully coexist with a 14 node LAVC
 * Supply a lot of compute power for "background" jobs on the U*ix server (i.e.
   act as a compute server)
 * Provide 3 "seats" for interactive users, plus a Macintosh running X as a
   seat and X server
 * Share vast amounts of disk space between VMS and U*ix via NFS
 * Find a way to meaningfully evaluate the above-mentioned machines to
   determine which ones most cost-effectively meet our needs
I received a number of helpful responses, and I'd like to summarize them here.

Criteria:
* Try to determine if integer or floating point performance is more important
  to your application, then look at Drhystone mips vs. Mflops.
* When evaluating performance, differentiate between whether you are concerned
 about aggregate performance or performance on a single job.
* Prioritize items of interest - e.g.
  (1) Performance/single job is top priority
  (2) Cost/seat is second top priority
* Don't forget to figure in software purchase and warranty/maintenance costs
  over some period of interest, and hardware warranty/maintenance costs over
  that same period of interest.
* Minimum memory configurations are ridiculously inadequate - 16 MB is a bare
  minimum, 32 MB is probably ok, 64 MB is good.
* Be sure to have local page and swap space on each workstation (at least
  300 MB), diskless machine problems range from poor performance to frequent
  crashes.
* Consider dedicating one small workstation as an NFS server, because the
  loading is a significant factor. Configure it to have /usr/... areas,
  have /root/... and /swap/... locally on each machine.
* Criteria to consider in comparing small to large workstations -
	Is workload dominated by relatively small jobs that can fit into
        memory and disk available on a small workstation (buy several small
        workstations), or by a few large batch jobs that will each use up
        huge amounts of memory (buy largest workstation you can afford)?
* Check the availability of 3rd party software for the box you are considering,
  how much is available and are the packages you may need available?
  Also make sure, with the DEC third-party software list, that the package
  runs on the RISC systems as well as vaxes.
* To do primarily data manipulation using a lot of CPU power, get a big server
  with fast disks and hang X terminals off of it.
  To do primarily fancy 3D graphics or software development, go for multiple
  workstations.
* SCSI disks are very slow (comparatively) and can be a real bottleneck with
  a fast cpu. You may want to look at IPI or SMD subsystems if your usage
  is I/O intensive.
* "In evaluating the "main" machine, pick a representation job or set of
  jobs and benchmark them on each machine.  This is the easy part for
  just single application speed.  What you really need is to test the
  operational environment you expect.  What types of graphics jobs are
  you going to be running?  Are all users interested in running long
  color animations simultaneously?  How many tasks are likely to be
  running during the day?  How many tasks at night? If you have lots of
  vectorizeable code, then enhancing a simple platform like a Sparcstation
  with a vector board or going with a built in system like a Stardent might be
  of interest."

Benchmarking:
* Don't believe manufacturer's benchmarks - get loaners for a sufficient period
  to get representative code running and timed on the various platforms. (This
  was repeated again and again, and I've found that vendors are very willing
  to provide these loaners when they realize they are in a competitive
  situation. Don't get them all at once, though - get one machine, learn
  to configure it, set up your benchmarks, and then port them to other machines
  one by one.)
* Check recent issues of Digital Review for CPU benchmarks, also UNIX Review
* Context switching on the RISC systems slows performance considerably. It
  may be desireable to reserve one system as a compute server, others as
  interactive machines.
  Look at "user time" in this respect. (not sure how to do this - suggestions?)
* Be sure to compare multiple concurrent process performance vs. single job
  performance.

Comments on specific platforms: (loosely grouped by type)
* If current DEC hardware is on hardware maintenance, adding more DEC machines
  may simplify maintenance issues.
* VMS-ULTRIX Connections software (UCX) uses a lot of CPU to serve disks
  via NFS to Unix machines.
* One person mentioned that DEC's FORTRAN compiler has some very serious bugs.
* Some people have had problems with DEC's software support, either they don't
  answer questions or take a very long time doing so, or lead you astray.
* DECstations support DECnet and LAT, which may be a big consideration in a
  primarily VMS shop. We already do TCPIP, so it is not so much of an issue.
* There is a known bug in the 5.4 VMS UCX software - NFS file transfers will
  hang on 2 MB or greater file writes.

* Sun's implementation of NFS is the best, since they developed it.
* SUN Sparcs are the cheapest hardware
* SUN's seem to be most people's preference, for reliability, ease of
  configuration.
* SunOS may be closest to a generic UNIX, and is thought to be the easiest
  to port to and from.
* More people have SUN's than other machines, so support from the user
  community may be easier to come by.
* "The vendors NFS packages all do byte-swapping [in a mixed-environment] as
  necessary -- UNLESS you've got mixed data-type binary files (it seems Suns
  need to swap word data, but not character data...)")

* "The IBM RS6000 architecture is very fast.  Period.
  It's also got a quite impressive performance/price ratio."
* Questions remain about the current revision of the IBM operating system.
  There is a new version due out soon, however (AIX 3??) which is supposed to
  solve these problems. Problems exist in shareable writeable file areas ,
  multiple simultaneous processes much slower than would be expected, problems
  getting knowledgeable IBM system support,
* IBM's have a data integrity feature that prevents power outages from
  corrupting the file system.

* Don't forget to look at the Mips Magnum
* The Mips machines have better I/O performance than the DECstations.

* HP has a new "super" workstation coming out soon, worth a look

* DEC and IBM are comparable in performance
* Check the openness of the I/O subsystems - Sun is good, DEC seems ok,
  IBM is oriented to IBM buses, Silicon Graphics had vendors with whom they
  are compatible, but they are not compatible with many SCSI disk
  implementations.
* "If floating point performance is your primary criteria, then the ranking
  would have to be IBM followed by DEC and Silicaon Graphics, with SUN a distant
  third."
* IBM has some system utilities that aid in managing the systems and networks.
  Silicon Graphics has icon-based utilities that perform similar functions.
  These will aid an inexperienced UNIX system administrator in setting things
  up. However, after trying to get a Silicon Graphics loaner talking to the
  network I've found you still have to get into the operating system to get
  it right.

* Silicon Graphics windowing system is icon-oriented, and incompatible with
  the rest of the world. However, Motif is due out soon for them.

Macintoshes:
* Mac X can be very slow, it needs a high-powered CX or FX with about 8 MB
  memory.
* Mac X on a system running the native Macintosh OS provides "rootless"
  X-windowing which does not have the full functionality of an X environment
  on a U*ix or VMS system.
* Dec has Pathworks for Macintosh which provides X functionality (via Mac X)
  plus everything you never knew you needed on your Mac!

In summary:
I am getting loaners of each machine and running benchmarks of our target
software. I plan on testing multiple concurrent jobs, single job performance,
multiple jobs in the background with interactive foreground performance. I/O
is not too big an issue, but I am definitely going to try to have a lot of
local disk as opposed to using NFS for everything, because serving VMS disks
via NFS on my VAXstation 3100 nearly kills it! I am leaning toward a large
CPU with Xterminals because we are primarily interested in CPU cycles.
I have yet to determine price/seat, and I intend to use final quotes to develop
alternative configurations with pricing to compare how many mips we can get for
the $$ with real configurations.

In order to develop a comparison of price/vup, I spec'd out a "minimum"
configuration for each machine from quotes. In it, I included -
	16 MB memory
	approx. 200 MB disk
	2 years of hardware/software maintenance
	19" monochrome monitor, gray scale graphics
	NFS, C, FORTRAN, GKS
	no additional tape/disk/cd options

I got specmarks, mips and Mflop ratings for each cpu. I got costs for the
above mentioned configuration (ask the vendor to quote items out separately so
you can mix and match options - e.g. how much is the 19" monitor, how much
memory does it come with and how much are the increments?) as best I could.
I then computed $/mip, $/Mflop, $/Specmark.
Sun, IBM and Silicon Graphics will give you a specific discount up front,
DEC gives one discount at first, then when they find they are not in the
running they up the ante.
The machines we are looking at compare as follows (in that artificial minimum
configuration).:

Sparcstation SLC: (seat/server, 16 MB total, 207 MB, 17" mono)
	12.5 Dhrystone MIPS
	1.2 Mflops
	7.6 Specmarks (overall) (integer mean - 9.4) (floating mean - 6.6)
Sparcstation IPC: (seat/server, 24 MB total, 207 MB, 16" color, 1.44 MB)
	15.8 Dhrystone MIPS
	1.7 Mflops
	11.8 Specmarks (overall)
Sparcstation 2: (server/seat, 16 MB total, 207 MB, 1.44 MB, 19" mono)
	28.5 Dhrystone MIPS
	4.2 Mflops
	21 Specmarks (overall)
DECstation 5000/200: (server/seat, 16 MB total, 209 MB, 1.44 MB, 19" mono)
	24 Dhrystone MIPS
	3.7 Mflops
	18.5 Specmarks (overall)
DECstation 5100: (server, 16 MB total, 209 MB, 1.44 MB)
	19.4 Dhrystone MIPS
	14.9 Specmarks (overall)
DECstation 2100: (seat/server, 16 MB total, 209 MB, 19" mono)
	10.0 Dhrystone MIPS
	 1.2 Mflops
	 8.3 Specmarks (overall)
Silicon Graphics 4D/20: (seat/server, 16 MB total, 200 MB, 19" color)
	10.0 Dhrystone MIPS
	 0.9 Mflops
Silicon Graphics 4D/35: (seat/server, 16 MB total, 200GB, 1/4" tape, 19" color)
	33.0 Dhrystone MIPS
	 6.0 Mflops
IBM RS6000/320: (seat/server, 24 MB total, 160 MB, 19" grayscale)
	27.5 Dhrystone MIPS
	 7.4 Mflops
	24.6 Specmarks
IBM RS6000/530: (seat/server, 16 MB total, 355 MB, 19" grayscale)
	34.5 Dhrystone MIPS
	10.9 Mflops
	32.0 Specmarks

The following ratings are based on the above configurations. I include
them in hopes they may be useful to someone. DEC is coming back with new
discounts, so their ratings may improve.

Rated by price/MIP: (don't ask what 1.000 is - use your imagination! I don't
		     want to quote direct pricing)
	Sparcstation 2: 	  	.353
	Sparcstation SLC:		.389
	IBM RS6000/320:	        	.403
	Silicon Graphics 4D/35:		.448
	Sparcstation IPC: 		.469
	DECstation 5100: 	        .661
	IBM RS6000/530: 	        .696
	DECstation 5000/200:		.737
	Silicon Graphics 4D/20:        1.207
	DECstation 2100:	       1.146
Rated by price/Mflop: (as above)
	IBM RS6000/320:	               1.499
	IBM RS6000/530:		       2.204
	Sparcstation 2:		       2.395
	Silicon Graphics 4D/35:	       2.466
	Sparcstation IPC:	       4.366
	DECstation 5000/200:	       4.786
	Sparcstation SLC:	       4.872
	DECstation 2100:	       9.558
	Silicon Graphics 4D/20:	       9.914
	DECstation 5100:	       (no figures available)
Rated by price/Specmark: (as above)
	IBM RS6000/320:	                .450
	Sparcstation 2:		      	.479
	Sparcstation IPC:	       	.629
	Sparcstation SLC:	      	.641
	IBM RS6000/530:		      	.751
	DECstation 5100:	      	.861
	DECstation 5000/200:	      	.957
	DECstation 2100:	       1.381
 	Silicon Graphics 4D/20:	       (no figures available)
	Silicon Graphics 4D/35:	       (no figures available)


*********************************************
Thanks to:
Ron Fox, NSCL, Michigan State University
Chris Stradtman, COSMIC, Athens, Ga.
Minick Rushton, Space Telescope Science Institute
John Allen
Thierry Forveille, Observatoire de Grenoble
Peter Galvin, Brown Univ. Comp. Sci.
Janet Price
Greg Pavlov, Fstrf, Amherst, N.Y.
Drew Dean, Carnegie Mellon University
Christopher J. Calabrese, AT&T Bell Laboratories, Murray Hill, NJ
Dan Packman, NCAR
Steve Alter,Transaction Technology Inc.
Richard Seymour, University of Washington, Seattle
and to anyone else I missed!
***********************************************