[net.micro.mac] The art of benchmarks

bass@dmsd.UUCP (John Bass) (04/14/86)

Having taken a large amount of heat over being the slowest drive in the
bench mark list (for raw transfer speed) let me note some variables that we
took into consideration when setting our interleave value for the MacSCSI.

I did a study of disk requests to the driver last summer on my 5mb drive,
nearly all were less than a few sectors. Mathmatics REQUIRED setting the
interleave to a value of about 11:1 to optimize for the MOST COMMON type
of requests. The numbers are:

	1:	Driver setup to command out		1ms
	2:	Command transfer to controller ready	2ms
	3:	Safe PIO data transfer                  4ms
	4:      Controller delay until status ready     1ms
	5:      CPU delay until next request            3ms max 7ms
----------      -----------------------------------------------------
                Total interleave delay in ms --------  11ms    15ms 

Since at 5mbits/sec 1 sector is about 1ms the interleave was set at 11:1.
I know of NO SASI/SCSI controller that will read a sector in LESS than
4ms from select to status complete if the data is just about under the
heads. Thus I would guess the best system interleave would be about
6:1. The PIO transfer loop in the DDJ MacSCSI article is the most general
of all drivers -- it works for nearly everything with little special casing
-- it is only 4 times slower in transfer rate than IDEAL with puesdo-dma
transfers -- BUT IN OVERALL SYSTEM TIMES DESCRIBED ABOVE it only adds about
an additional 20% to the timings. Not bad for the cheapest simplest host
adapter and driver.

The DDJ driver also choose to ALWAYS present single sector requests to
the scsi interface code BECAUSE:

	1: with the HARDWARE interleave set to 11:1 it didn't matter

	2: some of the CHEAP surplus SASI controllers don't work with
	   multi-sector read/write requests -- thus for the DDJ article
	   it didn't make sense to optimize for the best possible controller.

I have driver/hardware combinations that would bench very favorably
using both the OLD MacSCSI pin board design and the right driver/controller.

Our newer clip mount MacSCSI Plus hostadapers are fully MacPlus compatable
and support much faster transfer speeds using the rom based driver.
We designed the clip mount board for Mirror Technologies (Warp 9)
and they came back to us in jan 86 in a panic to ship Pin boards when General
Computer put the squeze on AP to stop low profile logical connection shipments
(the M68000 clip used by everybody). They have had about a thousand clips on
back order for several months and have received a hand full (less than 50).
I would guess that everyone else is in the same boat -- thus Mirror shipped
between 300-500 pin board based drive subsystems while the rest of the
industry waited for clips to become available.

Since HFS has been released and larger drive/cluster sizes are in use
Mirror has contracted with a local specialist to reduce their interleave
to about 6:1 to match their drive/system throughputs today.

Even if Micah's hostadapter is fully hardware DMA I don't think that
Mr. Brechers design goals for the Micah drive are consistant with real
world requirements for several reasons:

	1: the 1:1 interleave he claims can only be had by turning off
	   interrupts -- this will certainly cause performance problems
	   with serial port traffic, Apple talk service, mouse service,
	   and background processing based on the display clock.
	   (as a side note IF interrupts ARE turned off his benchmark
	   results will be significantly in error since the system will
	   miss the clock interrupts the benchmark is based on.)

	   Display clock interrupts should occur about one per disk rev
	   and vary in length upto several milli-seconds depending on
	   who/what is linked into the service queue. This requires an
	   interleave value of several milliseconds or else a rotation will
	   be lost for every interrupt (IE minimum interleave of 2:1 or 3:1).

	   9600 baud printer traffic creates serial port interrupts
	   once per sector at 5mb/sec. This requires an interleave value
	   of between 1ms and 2ms to keep from losing interlace EVERY SECTOR
	   and having to wait upto 19ms for the disk to spin around again.
	   (IE minimum interleave of 2:1 or 3:1).

	2: Starving the serial port when used with Apple talk will make
	   using Fileserver software like MacServe jerky at best on the
	   remote workstations. The interleave pattern should include
	   several milli-seconds to account for the cpu time for a single
	   packet if apple talk is to be active. Longer delays caused by
	   seeks and error retry will cause apple talk timeouts.

	3: Missing interleave generates system throughputs that
	   vary 17:1 --- that is your transfers can take upto 1700% longer.
	   At 1:1 ANY mouse, printer, or clock interrupts WILL cause
	   the interleave to be lost resulting in about a 18.6ms rotational
	   delay.

Thus the fastest possible interleave (1:1) will probably generate in system
performance MUCH lower than expected .... where is the real benchmark that
REALLY describes the in-system performance??? ...  read/write of single
sectors/clusters from the filesystem interface?

All of this is in the noise since you can only get GOOD disk performance from
an apple filesystem (either MFS or HFS) on a NEARLY empty volume (less than
3mb) -- At 3mb used most disk systems will boot to the finder in 6-10 seconds,
at 25mb the same systems take a minute or longer -- about half the time
is needed to get the finder loaded (the trash can appears on the desk top)
and the rest of the time is rumbling in the filesystem/memory (to get up the
remaining dozen icons on the desk top). Similar delays are seen in file open,
creation and deletion requests by all applications.

I have been building/shipping MacSCSI pin board hostadapters for a full year
... Micah has shipped a few dozen units total in the last month. We offer
a host adapter with full source to allow other NON-DISK devices to be
attached to the hostadapter -- get that from Micah.

Mirror shipped our pin boards ONLY because General Computer froze up the supply
lines to their competitors like Mirror ... Mirror has talked about making
MacSCSI Plus host adapters available at a reasonable upgrade fee when the
AP clips are flowing again ... (if ever).

The advantage to the MacSCSI Plus hostadapters is that they are FULLY compatable
with ALL MacPLUS perpherials and OTHER MacPlus vendors. Mirror is 100% committed
to this host adapter because MacPlus hardware/software compatability is
IMPORTANT! They have a full line of MacPlus external perpherials in the
offering.

The Micah host adapter is not MacPlus compatable --- where will you have to go
to get a second drive or tape when you upgrade with Micah??? --- only Micah.
My Original MacSCSI pin boards aren't either ... but as soon as AP clips
are commonly available again I will take them back and upgrade them for
about $125.00 if purchased from us, or $150.00 if from one of our dealers.
($250.00ea list for new assembled units).

If there is really a warped need to go faster I can supply a drive/controller
and host adapter that is MacPlus compatable that is a full 7.5mb/sec
(that is 50% faster than Micah) -- not cheap (about $1600) ... but fast
(no other mac hostadapter is as fast). The reality is that in normal system
operation it is about the same speed at the user interface and hard to justify
the additional cost over our standard line hostadapters. Thus the bottom line
is that the our original host adapter is VERY CHEAP and the MacSCSI Plus
hostadapter costs only a little more and gives full MacPlus performance,
compatablity, and upgradeability.

To wrap this up -- Micah's cheap shot at Mirror with this shoddy benchmark
is only to try and buy market share after they have been advertising
vaporware for months -- their handouts show the VECTOR BOARD prototype
hostadapter and the full page Mag spreads show a drive mount that DOESN'T
FIT since the case CAN'T CLOSE due to blockage of the alignment plug in
the lower right of the back case.  Micah COULD NOT show this perfboard
prototype at the show -- who would stop laughing .... I have wondered
if it wasn't one of my pin boards inside since I saw one of my MacSCSI
Boot disks pop out of the demo mac during the show. They bought one
of my units very early on and drove down here and wasted a half day of
my time to help them get it going with their drive -- so goes any
profit on that sale and one more competitor in the game.

The art of benchmarking is to show your best side and the competitions
worst -- some folks seem to get carried away to the point that the
benchmark has little stake in real world usage. In this case I think
that the lack of understanding that Mr Brecher seems to have in the total system
timings may reflect other short comings in the quality of the product he is
pushing. More over IF HE REALLY DOES UNDERSTAND THESE TIMINGS AND WITHHELD
THEM then he is very guilty of presenting a falsehood on the market place.
In either case it leaves a lot of questions in my mind about both Mr. Brecher
and Micah.

My background and qualifications include nearly 15 years of systems programming
and performance tuning on IBM, DEC, UNIX and a dozen other mini/micro systems.

-- 

John Bass (DBA: Fastime, DBA:DMS Design)
DMS Design (System Design, Performance and Arch Consultants)
{dual,fortune,polyslo,hpda}!dmsd!bass     (805) 546-9141