lad@mouse.UUCP (Larry Deleski) (11/05/88)
This is a file which I downloaded from the MouseHole BBS about a year ago. At that time, Jim Reekes wrote an article on Disktimer, our beloved disk performance benchmark, and essentially exposed Disktimer for was it was (and still is), a fraud. Ephraim Vishniac would have us beleive that Reekes is a lunatic. Ephraim exhausted many hours at the keyboard disputing that article. I, for one, beleive what I read in that article. A large portion of what Reekes wrote is fact. I would like to take this opportunity to point out an error in something I said earlier. I stated that Disktimer did not use the Device Manager. It does, but not in the same way a typical SCSI driver does. What Disktimer fails to do is use the FileManager, something every driver has to do. Please read the following and decide for yourself. Reekes *does* know what he's talking about. ------------------------ DISKTIMER II vs. THE REAL WORLD DiskTimer II has been mistakenly used as the de facto standard for hard drive performance. Before I attempt to discredit any results of S. Brecher's program, I'd like to point out that I don't blame him. He is a good programmer, but I think there are some serious flaws in his benchmarkcalculations. Normally I would let this utility fade off into the sunset, but a certain hard diskvendor has taken DiskTimer and exploited it. Today, nearly every hard disk user is aware ofDiskTimer results for nearly every given drive. Brecher disclaims any similarities to realworld performance, but then what do DiskTimer results really mean? Well, they aremeaningless. Let me try to explain what disk interleaves and DiskTimer results really mean in thereal world. INTERLEAVES The ONLY reason to interleave blocks on a disk is to compensate for an interface that cannot handlethe full data transfer rate of the disk. A given disk drive has a native transfer rate that depends onthe bit density, data coding and spindle speed of the drive. If the interface of the computer is too slow for the drive's data rate, then the drive's controller chokes on reads (from disk) and starves the disk on writes (to disk). When this happens, the transfer must be suspended until thecomputer catches up. This kind of error is called a data late error. Suppose the computer was almost as fast as the disk. After the first block is read from the disk,the computer will be a bit behind in getting the data into memory. But come hell or high water,here comes the second block on the hard disk. Well, the computer just isn't ready yet ... DATA LATE ERROR ... and now we have to wait for the disk to spin completely around until the second block isagain under the drive's head. By this time the computer has been waiting for the second block. Since there are normally 26 blocks on each track, this means for every block read the computerwill have to wait for 26 more blocks to pass!!! Now THIS IS IMPORTANT!!! The block interleave is NOT a measure of performance! Just becausethe Dataframe XP use 1:1 interleave and the CMS S40 uses 6:1 does not mean the S40 is 6 timesslower. Actually, the native transfer rate of the S40 is much faster than the Dataframe XP. TheDataframe can get away with a 1:1 interleave because its transfer rate is slower than the S40. The Macintosh cannot keep up the high speed of the S40, so it is relegated to the substandardperformance of the Macintosh SCSI interface. Furthermore, 1:1 interleaving is disadvantageous due to the fact that the Macintosh only makessingle-block requests. In fact, an interleave of approximately 6:1 is optimal for handling suchrequests. Mr. Brecher himself tested this by modifying his DiskTimer (which issues multi-blockrequests) to issue single-block (or sector) requests. He ran this single-sector version ofDiskTimer, SSDiskTimer, as well as the original DiskTimer on the same drive at variousinterleaves. Below are the actual results that Mr. Brecher found. (lower times are better) INTERLEAVE CHART Interleave SSDiskTimer DiskTimer 1:1 6801 507 2:1 7206 812 3:1 7511 1116 4:1 7518 1523 5:1 6513 1827 6:1 2538 2233 7:1 2538 2537 8:1 2943 2943 As Steve says, This result clearly confirms that the about 6:1 estimate for optimal handling ofconsecutive single-block requests. It is also interesting to note that DiskTimer (withmultiple-block requests) favors a drive with a 1:1 interleave, whereas the SSDiskTimer (withsingle-block requests) demonstrates better performance at 6:1. Beside the fact the Brecher hadproven to himself that 6:1 interleave was best, he should have realized something else about hisprogram. Why do the DiskTimer results go from good to bad on the interleave scale and DiskTimer go bad to good? I'll explain this in a moment. Tom Reilly of SuperMac states; The Mac Plus is set up for a 3:1 disk interleave,... This is not accurate information. Maybe the Apple hard disk for the Macintosh Plus is set up with a 3:1interleave, but every drive manufacturer will need to determine their own interleave factor. The best interleave factor for a drive is not determine by the host computer alone. After a developerhas completely optimized his software/hardware interface, he can then test all possibleinterleaves until he achieves the desired performance. Different drive configurations will havedifferent optimal interleaves. Furthermore Tom states; The SCSI interface on the Plus requiredyou to do direct ROM calls to gain performance. Again, this is not accurate information. This isSuperMac's opinion, it is not my opinion and certainly not recommended by Apple. (The Macintosh developer guidelines clearly states never do direct ROM address calls.) I am reminded of the early Macintosh programmers that considered QuickDraw too slow. Instead ofconforming to Apple's guidelines, many decided to draw directly on the screen (ignoring windows)or jump to ROM addresses. This software blew up when Apple changed the ROMs. These programs went from high performance to no performance. We can expect Apple to continueto increase the performance and capabilities of the Macintosh ROMs, and those that continue to follow the developer's guidelines will reap the benefits without having to go back to the drawingboards to write another revision (while the customer waits). When I sit in front of a Macintosh Plus with a Dataframe XP20 or a CMS S20, I can not observeany performance differences. Both drives appear to run my applications at the same rate, eventhough DiskTimer says that the XP20 is 5.48 times faster than my CMS S20. Gary Eisenstien,Supermac's national sales manager, states the Dataframe XP is three timer faster than any otherSCSI drive available for the Macintosh. This is absolutly false! How did he ever come to believethis? He probably considers DiskTimer the final word for a drive's performance. One should not put much faith into interleave factors alone. One should understand how they affectthe user. But I would also like to point out that the FileManager of the Macintosh will only requestsingle block transfers. It is the FileManager that is responsible for all file I/O. The Macintosh will request one block, digest it, and maybe then request another block. The time the Macintoshtakes to digest a block is about the same time as it takes two blocks to pass underneath the drive's read/write heads. So, a drive with 3:1 interleave is closely in tune with the Macintosh's FileManager. Examine the chain of command from application to hard disk. Application>>FileManager>>DeviceManager>>DeviceDriver>>SCSIManager>>HDController>>HardDisk (NOTE: In some cases, such as Dataframes, the HDController is a completely separate interface board. CMS uses embedded SCSI drives, where this controller is integrated into the hard drive.) In a real world situation, an application will talk to the FileManager and request a file. TheFileManager needs to determine which blocks that the file occupies by passing a message throughthe entire chain of command to get the hard disk and read some directory informationone-block-at-a-time. Once the FileManager has found the blocks it needs, it then requests a singleblock of that file, again through all the managers, to the disk. That's one block at a time folks. DiskTimer, on the other hand, talks directly to the DeviceManager with multi-block requests. Refer to the interleave chart and look at the difference between single and multiple block requests. Clearly the results of this chart demonstrate that DiskTimer favors a drive with 1:1 interleave,and handicaps a drive that isn't. I wish the other version, DiskTimer, had been released instead. Its results are more of a real world test because of its single-block accessing method. DISKTIMER RESULTS Steve Brecher has written a benchmark program, DiskTimer, for testing the performance ofSCSI hard disks on the Macintosh. Although I respect his work, I have to critize his methods oftesting. Determining which hard disk is the better performance drive based upon DiskTimerresults can be very misleading for many reasons. For one thing, DiskTimer uses device managercalls directly. It never issues calls through the FileManager, which is the method of allapplications in the real world. DiskTimer goes directly to the device manager and requestsmultiple blocks by their block number. Applications never know what block numbers they need. This test is unfair and does not behave as a real application would. Worst of all, DiskTimer usesthe Macintosh's internal SystemTicks to calculate its resulting times. SystemTicks are in1/60th of a second and DiskTimer is attempting to measure times in 1/1000th of a second! (ie: Atypical 20MB drive's access time is 65ms and transfer rates are greater than 1MB/sec.) One very important issue that is not even being considered in DiskTimer results is whether or notthe manufacturer's SCSI drivers are performing any verification of the data that was transferred. If someone were to ignore verifying their read/writes, then they could reduce their DiskTimerresults in half! Personally, I'd rather have my data verified and ignore DiskTimer readings. Most importantly, DiskTimer should never be run on a disk containing user data. This program firstreads a block of your data, holds it in RAM, performs its test on that block, and finally writes yourdata back to the drive. It continues along with this sequence block-by-block. If you were to reset,power off, or get the bomb dialog during a DiskTimer test, then chances are you've lost data. DISKTIMER SEEK TEST The seek test does not utilize the number of data surfaces on the hard disk, the number of blocksper surface, or their size. DiskTimer can not determine if an actual seek has occurred. DiskTimer's seek test, according to the source code that I have; performs a seek by reading a block,calculates 1MB worth of (512 byte) blocks, and reading that last block. This adds two block readsto the seek time, which has nothing to do with seeks. And even more importantly, this seek testdoes not consider rotational latency. If the disk platter were required to complete an additionalrotation just to bring that second block under the disk heads, then it would add additional time tothe seek results. Moreover, the DiskTimer seek test does not consider the method of bad block allocation. Some drives store its allocated blocks on every track, others store allocation blocks at one end of the disk. If DiskTimer were to test a seek to (or across) spared blocks, it would impose unfair times for that drive. In order to perform a valid seek test, it would require a SCSI Seek command, which positions the heads without attempting to read data. At best, the read time needs to be subtracted along with the actual incurred latency. If one examines the DiskTimer results of the DataFame 20, you'll see seek times ranging from 38 to 80. How can the same hard disk display such variances? The Dataframe SCSI driver software could not possible change the true seek performance of a hard disk. The results should remain constant, because that drive has an average access time of 65ms. The only explanation for seeing such a variance is because DiskTimer's testing methods are affecting the results. FACTS ABOUT SYSTEMTICKS Consider the Macintosh's hardware limitations. In the Macintosh Plus world, the maximumtransfer rate is 320K/second. Disregarding any software limitations for the moment, theMacintosh hardware can not possibly keep up the a 1:1 interleave from the typical hard diskoperating at greater than 1MB/second. The theoretical limit is about a 3:1 interleave, as Applehas recommended. Here is a quote from Inside Macintosh (Apple's bible for the Macintosh); WARNING: Don't rely on the tick count being exact; it will usually be accurate to within one tick, but may be off by more than that. A SystemTick is 1/60th of a second. A hard disk rotates at 3600 RPM or 60 times per second. Therefore, a tick count is worth one rotation of a hard disk HYPOTHESIS What if we tested the average drive set up with the following format. 1:1 interleave 512 bytes per blocks 26 blocks per track - or - 13k per track ERROR FACTORS Tick count error is +/-1 or more. Timing a transfer of 13k, or one rotation, with SystemTickscould introduce a error of 100% or more. Because the number of ticks could be 0S, R1 or more! A tick is approximately 16.7ms while some hard disks can seek track to track in 4ms. A drive could seek across 4 or more tracks before TickCount changed. As an analogy, using ticks tomeasure a 32k read/write, would be like using a car's odometer to measure lengths of feet. CONCLUSION I understand that DiskTimer disclaims any comparisons with real world performance, but considerthe inherent errors built into the results it does present us. A 32k file or 64 blocks wouldrequire 2.46 tracks (not cylinders) of storage on the average drive. To read or write a 32k filewould require at least 2.46 rotations. SystemTicks after 2.46 rotations could be 2S or R3, andeither of these figures may be off +/- 1 or more. DiskTimer will introduce a significant errorfor each 32k read/write test. DiskTimer does 200 read/write tests and averages them to thenearest 60th of a second. Then this figure is converted to the nearest 10th of a second. Thereforeany accuracy resulting with this program, in the real or un-real DiskTimer world, would bepurely coincidental. To summarize, all DiskTimer results have been invalidated because of; 1. Multi-block requests 2. Favors a drive with 1:1 interleave 3. Ignores the FileManager calls 4. Does not perform a true seek test 5. Uses SystemTicks for calculating timing data Q.E.D. Jim Reekes (Acknowledgements to Bob Denny's material on disk interleave, Steve Brecher's source code, and other documentation that I have quoted and used as my source material.) ----------------------------
ephraim@think.COM (Ephraim Vishniac) (11/08/88)
Summary for anybody who cares: Jim Reekes presents many arguments against DiskTimer, some good, many bad, some really dumb. Larry Deleski endorses them all, but I don't. In article <698@mouse.UUCP> lad@mouse.UUCP (Larry Deleski) writes: >This is a file which I downloaded from the MouseHole BBS about a year ago. >At that time, Jim Reekes wrote an article on Disktimer, our beloved disk >performance benchmark, and essentially exposed Disktimer for was it was >(and still is), a fraud. >Ephraim Vishniac would have us beleive that Reekes is a lunatic. Ephraim >exhausted many hours at the keyboard disputing that article. I, for one, >beleive what I read in that article. A large portion of what Reekes wrote >is fact. [and the rest...] A key point here is that I think Reekes is loony not for disliking DiskTimer, but for the stupid arguments he presents against it. >I would like to take this opportunity to point out an error in something I >said earlier. I stated that Disktimer did not use the Device Manager. It >does, but not in the same way a typical SCSI driver does. What Disktimer >fails to do is use the FileManager, something every driver has to do. Still wrong. No driver uses the File Manager. The File Manager uses the drivers. Even Reekes says that! >Please read the following and decide for yourself. Reekes *does* know >what he's talking about. Let the readers decide... > DISKTIMER II vs. THE REAL WORLD >INTERLEAVES >The ONLY reason to interleave blocks on a disk is to compensate for >an interface that cannot handle the full data transfer rate of the >disk. A given disk drive has a native transfer rate that depends >onthe bit density, data coding and spindle speed of the drive. If >the interface of the computer is too slow for the drive's data rate, >then the drive's controller chokes on reads (from disk) and starves >the disk on writes (to disk). When this happens, the transfer must be >suspended until thecomputer catches up. This kind of error is called >a data late error. This is true, but becomes irrelevant to our discussion when the disk controller has a local cache which exceeds the size of a disk track. Then the crucial interface is between the disk and the controller, not between the disk subsystem (disk + controller) and the host. >Furthermore, 1:1 interleaving is disadvantageous due to the fact that the >Macintosh only makes single-block requests. Since this premise is false, all of the discussion that followed it is pointless. The Mac does make multi-block requests; not to do so would be incredibly stupid. >When I sit in front of a Macintosh Plus with a Dataframe XP20 or a >CMS S20, I can not observeany performance differences. Both drives >appear to run my applications at the same rate, eventhough DiskTimer >says that the XP20 is 5.48 times faster than my CMS S20. This is hardly surprising. As I mentioned in an earlier posting, data transfer rate is highly overrated as a performance measure. When running most applications, seek time is much more important. Data transfer rate becomes most visible in activities such as copying files. >One should not put much faith into interleave factors alone. One should >understand how they affectthe user. But I would also like to point out that >the FileManager of the Macintosh will only requestsingle block >transfers. Again, this is simply wrong, and all the discussion based on it is pointless. >DISKTIMER RESULTS >Worst of all, DiskTimer usesthe Macintosh's internal SystemTicks to >calculate its resulting times. SystemTicks are in1/60th of a second >and DiskTimer is attempting to measure times in 1/1000th of a second! >(ie: Atypical 20MB drive's access time is 65ms and transfer rates are >greater than 1MB/sec.) I very nearly agree with Jim here. These are real problems, but they can be overcome statistically. DiskTimer attempts to overcome them by repeated meaurements to gain a large sample. Brecher's error was that he starts each measurement by synchronizing with the system clock, so that (assuming a random distribution of results) he has an average underestimate of half-a-tick per measurement. For more accurate results, he should deliberately randomize his timing with respect to the clock so that over- and under-estimates are equally likely. When Brecher first came out with DiskBench (DiskTimer's predecessor) I made this same point and even wrote the code for him. He disagreed, evidently, as the change didn't survive into later versions. >One very important issue that is not even being considered in >DiskTimer results is whether or notthe manufacturer's SCSI drivers >are performing any verification of the data that was transferred. True, but how can it? If someone is silly enough to buy a disk "by the numbers" without considering quality, support, reliability, and the like, then he deserves what he gets. This point is basically irrelevant. >Most importantly, DiskTimer should never be run on a disk containing >user data. This program firstreads a block of your data, holds it in >RAM, performs its test on that block, and finally writes your data >back to the drive. It continues along with this sequence >block-by-block. If you were to reset,power off, or get the bomb >dialog during a DiskTimer test, then chances are you've lost data. Actually, DiskTimer's test consists of reading and writing the data that was already on the drive - it never overwrites the user data with anything else. So, a system failure during a write could corrupt your disk, but this is no more likely than with any other application. >DISKTIMER SEEK TEST This is prety much true. DiskTimer does an awful job with the seek test. >FACTS ABOUT SYSTEMTICKS >Disregarding any software limitations for the moment, theMacintosh >hardware can not possibly keep up the a 1:1 interleave from the >typical hard disk operating at greater than 1MB/second. The >theoretical limit is about a 3:1 interleave, as Apple has >recommended. The distinction completely lost here is between "keeping up with" and "gaining some benefit from." First, the Mac Plus can keep up with a non-cached drive (ST225N) at 2:1 interleave. Since it does this reliably, there must be some margin. Now, think about a cached drive running at 1:1. The controller can read a full track in one pass. The Mac can't read the data in that same time, but it can read it in less than two rotations. So, it gets an effective performance that's intermediate between what you see at 2:1 and what you would hope for at 1:1. Let's go back to the purpose of interleaving: matching performance. When there's no intermediate buffer (or very little), this means matching performance with the host system. When there's a sizable intermediate buffer, it means matching performance with the controller. Many sophisticated controllers don't even support variable interleaving because there's no point. The controller's internal cache effectively hides the interleaving from the host system. [The other hazard of mentioning a specific interleaving is that it implies different transfer rates for different disks. Apple's recommendation related to MFM formatted disks, which were dominant at the time. RLL formatted disks are another story.] >Here is a quote from Inside Macintosh (Apple's bible for the >Macintosh); > WARNING: Don't rely on the tick count being exact; it will usually > be accurate to within one tick, but may be off by more than that. The context, which Reekes thoughtfully omitted, states that "TickCount returns the current number of ticks since the system last started up." >A SystemTick is 1/60th of a second. A hard disk rotates at 3600 RPM >or 60 times per second. Therefore, a tick count is worth one rotation >of a hard disk >ERROR FACTORS >Tick count error is +/-1 or more. Timing a transfer of 13k, or one >rotation, with SystemTickscould introduce a error of 100% or more. >Because the number of ticks could be 0S, R1 or more! A tick is >approximately 16.7ms while some hard disks can seek track to track in >4ms. A drive could seek across 4 or more tracks before TickCount >changed. As an analogy, using ticks tomeasure a 32k read/write, >would be like using a car's odometer to measure lengths of feet. This is exactly why DiskTimer does 100 repetitions of each test. The smallest unit on my car's odometer is .1 mile (528 feet). If I drive some short distance once, I'll put little faith in what the odometer says. If I drive it a hundred times and divide the odometer reading by 100, I'll feel pretty good about the accuracy. Still, Brecher's mistake of synching to the tick count does introduce a systematic error which is easily visible on a Mac II. >CONCLUSION >To summarize, all DiskTimer results have been invalidated because of; > 1. Multi-block requests The premise was false, you'll remember. > 2. Favors a drive with 1:1 interleave If it has a significant cache. Let's not confuse different problems. > 3. Ignores the FileManager calls Irrelevant. > 4. Does not perform a true seek test True. > 5. Uses SystemTicks for calculating timing data It isn't using them that's bad, but the way they're used. So, I'd give Reekes 1.5 points out of 5, or 30%. With all its problems, DiskTimer is more accurate than that! Ephraim Vishniac ephraim@think.com Thinking Machines Corporation / 245 First Street / Cambridge, MA 02142-1214 On two occasions I have been asked, "Pray, Mr. Babbage, if you put into the machine wrong figures, will the right answers come out?"
dillon@POSTGRES.BERKELEY.EDU (Matt Dillon) (11/09/88)
Well, the way I see it (reading the two articles), DiskTimer is absolutely useless when applied to real-world file handling. However, DiskTimer isn't so bad as to preclude low-level performance comparisons between two drives... sort of the "potential" performance. It has been shown that when a filesystem makes multi-block requests to the lower level driver, performance increases in a non-trivial way. For example, BSD unix 4.1->4.2 did this (plus fixed the disk layout) and got at least a 10x improvment. C-A (Commodore-Amiga) did this and the transfer rate through the filesystem went from 60K/sec to 600K/sec. So, assuming Apple decides to apply similar fixes to their filesystem module, one ought to expect real-world numbers in the same general category as DiskTimer's results. When such a fix eventually appears one can bet the low level performance of your HD will make a big difference. -Matt
brecher@well.UUCP (Steve Brecher) (11/09/88)
In article <698@mouse.UUCP>, lad@mouse.UUCP (Larry Deleski) writes: > This is a file which I downloaded from the MouseHole BBS about a year ago. > At that time, Jim Reekes wrote an article on Disktimer, our beloved disk > performance benchmark, and essentially exposed Disktimer for was it was > (and still is), a fraud. I am the author of DiskTimer II. I am also the author of several Macintosh SCSI disk drivers, and of the firmware for several (non-SCSI) disk controllers. It is disheartening to see the republication of an article (Reekes's) which contains so much error and which I rebutted about 18(?) months ago. How a simple program like DiskTimer II, whose publication included source code, and whose user interface includes appropriate caveats about interpretation of the results could be "a fraud" is beyond me. > I would like to take this opportunity to point out an error in something I > said earlier. I stated that Disktimer did not use the Device Manager. It > does, but not in the same way a typical SCSI driver does. What Disktimer > fails to do is use the FileManager, something every driver has to do. DiskTimer II uses the disk's device driver, via _Read and _Write traps (PBRead and PBWrite) for all disk access. No device driver uses the File Manager. The File Manager uses the device driver. Applications call the File Manager which calls the Device Manager which calls the device driver which either calls the SCSI Manager or accesses the SCSI hardware directly (the latter likely to run into compatibility problems). > Please read the following [article by Reekes] and decide for yourself. > Reekes *does* know what he's talking about. Reekes has no idea of what he's talking about with respect to several issues. The following quotations indicated by ">>" are from Reekes. >> ...the Macintosh only makes single-block requests. False. The File Manager makes both single- and multiple-block requests. Reekes reiterates this error several times and bases conclusions on it. >> ...the chain of command from application to hard disk[:] >> Application>>FileManager>>DeviceManager>>DeviceDriver>> >> SCSIManager>>HDController>>HardDisk This is correct. I quote it to call it to Deleski's attention. >> DiskTimer uses the Macintosh's internal SystemTicks to calculate its >> resulting times. SystemTicks are in 1/60th of a second and DiskTimer >> is attempting to measure times in 1/1000th of a second! (ie: A typical >> 20MB drive's access time is 65ms and transfer rates are greater than >> 1MB/sec.) >> ... >> As an analogy, using ticks tomeasure a 32k read/write, would be like using a >> car's odometer to measure lengths of feet. The analogy is correct, and that's why DiskTimer II does not attempt to measure intervals such as those Reekes claims it does. DiskTimer II measures the total elapsed time to do a sequence of operations. The times measured are those reported to the user, which are usually on the orders of 10-100 deciseconds (60-600 60hz system clock interrupts). Reekes quotes Inside Macintosh to the effect that the tick count may sometimes be inaccurate by more than one tick. But the only cause of such inaccuracy is disabling of interrupts due to a continuous source of interrupts. This does not apply to DiskTimer II operations if the user follows on-screen instructions. >> DiskTimer does 200 read/write tests and averages them to the nearest >> 60th of a second. For each 100 read tests and write tests, DiskTimer II reports the *total* interval in elapsed ticks, divided by 6 (it reports deciseconds, for historical reasons). It does not attempt to measure the timing of single operations, and does no averaging. >> Most importantly, DiskTimer should never be run on a disk containing >> user data. ... If you were to reset, power off, or get the bomb dialog >> during a DiskTimer test, then chances are you've lost data. I quote this for your amusement. By this criterion, one should never run any software which writes to the disk. >> [The] seek test does not consider rotational latency. If the disk platter >> were required to complete an additional rotation just to bring that second >> block under the disk heads, then it would add additional time to the seek >> results. Rotational latency is a component of access time. As stated in the source code comments, "to a avoid constant sector skew between I/Os, which might be unfair to some drives, there is a varying delay between I/O requests. The delay is obtained via a pseudo-random number generator, using the same seed on each run." >> DiskTimer disclaims any comparisons with real world performance. DiskTimer II tells the user that "The results to not translate directly to perceived througput [sic] of typical Mac activity." Moving from response to my thoughts on this matter: DiskTimer II is a simple program that can be useful in judging the relative performance of disk subsystems on a limited set of tasks: transfers of 24KB of data to and from the disk, and a series of single-block transfers located 2048 blocks apart. The latter test is invalidated, as noted by Ephraim Vishniac, by cacheing controllers or drivers. Implementing a "real-world" benchmark of disk performance is a time-consuming task and requires a disk dedicated to the purpose, i.e., one that contains no data that needs to survive the test. Such a test requires that the exact same files be written to each disk under test and then the exact same operations performed in the same order. Only a laboratory has the resources to set up such tests. DiskTimer II was provided as a simple, non-destructive test that any user can run. It should not be relied upon as a complete measurement of disk subsystem performance. Its measurements, where not invalidated by special considerations, provide a few data points that may sometimes be useful to some people. I would have no trouble believing that a superior tool was available (I have not looked at SCSI Evaluator). But there's a lot of ignorance about the technicalities of disk performance testing, so be careful about accepting anyone's results unless you fully understand the test techniques.
alexis@ccnysci.UUCP (Alexis Rosen) (11/12/88)
I would really like to see some hard information come out of this little flame war. Right now, we have very little. The Reeks article was a fair start, and now if Ephraim cares to answer it we'll really be getting somewhere. I would like to point out one thing, though- While I'm not expert on this subject, there are some things I do know: In article <698@mouse.UUCP> lad@mouse.UUCP (Larry Deleski) writes: >I would like to take this opportunity to point out an error in something I >said earlier. I stated that Disktimer did not use the Device Manager. It >does, but not in the same way a typical SCSI driver does. What Disktimer >fails to do is use the FileManager, something every driver has to do. > >Please read the following and decide for yourself. Reekes *does* know >what he's talking about. But Larry apparently does not. To quote from the Reeks article which he appended to this same posting: >[Disk requests go through this chain:] >Application>>FileManager>>DeviceManager>>DeviceDriver>>SCSIManager>> >HDController>>HardDisk So drivers NEVER use the File Manager; it uses them. In fact the entire paragraph seems garbled. Larry might be trying to say that DiskTimer II never uses the File Manager, whereas normal programs always do. If that's true, it sounds like a valid criticism. Regardless of the merits of this argument, Larry is not demonstarting the kind of expertese that makes me want to trust his judgement on this matter. One other point- while the Reeks article sounds fairly expert, there are a few major oddities that stand out. At one point he says the seek time of the drive he's discusing is 65 ms, and thus should always yield the same seek numbers. The 'seek time' is really an average time, which bears only vauge resemblance to any individual seek. (It is is true that the same seek should take the same amount of time, not counting rotational latency.) Whether this invalidates his point depends on whether DiskTimer always seeks between the same exact tracks each time it is run on any one disk. Now, would the real expert please stand up? p.s. Please don't construe any of this as a personal attack of either Larry or Jim Reeks. I'll leave that to others :-) Flames to /dev/null. ---- Alexis Rosen alexis@dasys1.UUCP or alexis@ccnysci.UUCP Writing from {allegra,philabs,cmcl2}!phri\ The Big Electric Cat uunet!dasys1!alexis Public UNIX {portal,well,sun}!hoptoad/
alexis@ccnysci.UUCP (Alexis Rosen) (11/15/88)
In article <30643@think.UUCP> ephraim@vidar.think.com.UUCP (Ephraim Vishniac) writes (in an article about J. Reeks vs. DiskTimer II): > ... As I mentioned in an earlier posting, data >transfer rate is highly overrated as a performance measure. When >running most applications, seek time is much more important. Data >transfer rate becomes most visible in activities such as copying >files. This is what I used to think. I always looked for drives with low seek times. So I bought a Maxtor 140 MB drive in early '87. This drive performed as I expected and I was very happy with it. Later I bought a Quantum 80 MB unit with a seek time 50% slower than the Maxtor (28 ms as opposed to 18 ms) which nevertheless performed much faster in many operations. The speed came from use of the then-new "fake DMA" mode which used hardware-assisted "blind" reads and writes on the Mac II. So transfer speed had a major impact on perceived speed. I still buy drives with the lowest seek times, though (currently CDC/Imprimis Wrens). This is because I tend to use them for Multi-user file service and database accesss. Typically, that kind of application is more likely to cause lots of slow disk seeks. To summarize, what Ephraim says is true for me. I don't think it's true for the majority of single-user systems (with the exception of some database jobs). ---- Alexis Rosen alexis@dasys1.UUCP or alexis@ccnysci.UUCP Writing from {allegra,philabs,cmcl2}!phri\ The Big Electric Cat uunet!dasys1!alexis Public UNIX {portal,well,sun}!hoptoad/
ephraim@think.COM (Ephraim Vishniac) (11/16/88)
In article <989@ccnysci.UUCP> alexis@ccnysci.UUCP (Alexis Rosen) writes:
+In article <30643@think.UUCP> ephraim@vidar.think.com.UUCP (Ephraim Vishniac)
+writes (in an article about J. Reeks vs. DiskTimer II):
+> ... As I mentioned in an earlier posting, data
+>transfer rate is highly overrated as a performance measure. When
+>running most applications, seek time is much more important.
+This is what I used to think. ... Later I bought a Quantum 80 MB
+unit with a seek time 50% slower than the Maxtor (28 ms as opposed to
+18 ms) which nevertheless performed much faster in many operations.
+The speed came from use of the then-new "fake DMA" mode which used
+hardware-assisted "blind" reads and writes on the Mac II. So transfer
+speed had a major impact on perceived speed.
But the Q280 also has a large (60KB? I don't remember exactly.)
on-board cache. So the seeks may be slower, but they are also
somewhat less frequent. This is a prime example of why benchmarking
or buying disks is so damned complicated. We need an actual usage
test, but I doubt we'll ever agree on one.
Ephraim Vishniac ephraim@think.com
Thinking Machines Corporation / 245 First Street / Cambridge, MA 02142-1214
On two occasions I have been asked, "Pray, Mr. Babbage, if you put
into the machine wrong figures, will the right answers come out?"alexis@ccnysci.UUCP (Alexis Rosen) (11/17/88)
In article <31573@think.UUCP> ephraim@vidar.think.com.UUCP (Ephraim Vishniac) writes: "In article <989@ccnysci.UUCP> alexis@ccnysci.UUCP (Alexis Rosen) writes: "+In article <30643@think.UUCP> ephraim@vidar.think.com.UUCP (Ephraim Vishniac) "+writes (in an article about J. Reeks vs. DiskTimer II): "+> ... As I mentioned in an earlier posting, data "+>transfer rate is highly overrated as a performance measure. When "+>running most applications, seek time is much more important. " "+This is what I used to think. ... Later I bought a Quantum 80 MB "+unit with a seek time 50% slower than the Maxtor (28 ms as opposed to "+18 ms) which nevertheless performed much faster in many operations. "+The speed came from use of the then-new "fake DMA" mode which used "+hardware-assisted "blind" reads and writes on the Mac II. So transfer "+speed had a major impact on perceived speed. " "But the Q280 also has a large (60KB? I don't remember exactly.) "on-board cache. So the seeks may be slower, but they are also "somewhat less frequent. This is a prime example of why benchmarking "or buying disks is so damned complicated. We need an actual usage "test, but I doubt we'll ever agree on one. How large is large? My understanding was that it was a one-track cache. This is, I believe, exactly the same as the Maxtor. In any event, a one-track cache can't have any effect on seeks, can it? I thought the primary use of this kind of cache is to eliminate rotational latency delays on requests for different sectors on the same track. There is another factor- since the Maxtor had more platters (and thus more R/W heads) it actually had to do less track-to-track seeking (more info under the heads at any one moment). Of course, there could conceivably be a disk so fragmented that this would be irrelevant. But in fact, my case was a pretty controlled test: One large 40-odd MB datafile, pre-allocated and guaranteed contiguous. ---- Alexis Rosen alexis@dasys1.UUCP or alexis@ccnysci.UUCP Writing from {allegra,philabs,cmcl2}!phri\ The Big Electric Cat uunet!dasys1!alexis Public UNIX {portal,well,sun}!hoptoad/