becker@iris.ucdavis.edu (Jeffrey Becker) (04/11/90)
Hi. We have just completed a new release of the IOSTONE file system performance benchmark program. It is a set of 'C' routines designed to provide a comprehensive measure, including components of disk performance, CPU performance on file system tasks, and buffer cache performance. If you are interested in receiving the code, please send me e-mail. We are anxious to hear about any feedback you might have, as well as IOSTONE performance results from your systems. Thanks. -jeff
uppal@hpindda.HP.COM (Sanjay Uppal) (04/13/90)
Could you give us some context for this "new release" please ? Is this one of the benchmarks SPEC is considering for IO performance ? Does it do remote as well as local file/disk performance ? Is it limited to UNIX systems ? How does it compare to existing IO benchmarks ? Sanjay Uppal <uppal> (NN9T, VU2SMT) phone: (408) 447-3864 Hewlett-Packard (IND) uucp: ...!hplabs!hpda!uppal NS arpa: uppal%hpda@hplabs.hp.com
becker@iris.ucdavis.edu (Jeffrey Becker) (10/17/90)
In article <143502@@sun.Eng.Sun.COM>, lm@@slovax.Sun.COM (Larry McVoy) writes: > In article <2387@@van-bc.wimsey.bc.ca> jtc@@van-bc.wimsey.bc.ca (J.T. Conklin) writes: > >In article <14900016@@hpdmd48.boi.hp.com> sritacco@@hpdmd48.boi.hp.com (Steve Ritacco) writes: > > IOstone is misnamed and misleading. It ought to be called "cachestone" or > "buffer cachestone". It does *not* measure I/O performance. It measures > cache performance. We are sorry that you were mislead. We would like to clarify a few points: 1. IOStone was designed to measure IO system performance on REAL SYSTEM WORKLOADS. Consequently, our synthetic workloads are derived from REAL PROGRAM TRACES. You point out our program performs well with a large disk cache. We completely agree with you. However, note that IO system workloads also benefit greatly from large disk caches. 2. IOStone was designed to measure TOTAL IO SYSTEM PERFORMANCE. We do not desire to measure the performance of individual components of the IO system performance such as the disk, controller or the I/O channel alone. Rather, we want to know how these perform in conjunction with the other parts of the I/O SYSTEM, such as the cpu and the operating system. 3. We designed IOStone to be as simple to use and portable as possible. We do not claim it is the MOST accurate measure of IO performance that can be constructed, but we do feel that it achieves a good compromise between portability, simplicity and accuracy. We feel that IOStone is the best widely used indicator of IO system performance that is currently available. We do not mind criticism, but we prefer it to be CONSTRUCTIVE! Other postings in this debate have mentioned the great difficulties inherent in producing a comprehensive IO system metric. Please consider these issues carefully before launching into another tirade. We welcome any CONSTRUCTIVE SUGGESTIONS you might have in designing an IO system metric. If you would like a free copy of the IOStone code or a copy of our Computer Architecture News (June 90) article, send email to becker@iris.ucdavis.edu Jeff Becker Arvin Park
staff@cadlab.sublink.ORG (Alex Martelli) (10/19/90)
becker@iris.ucdavis.edu (Jeffrey Becker) writes: ... >We welcome any CONSTRUCTIVE SUGGESTIONS you might have in designing an >IO system metric. I have one, and as a beta-site for IOstone made it (but never got any feedback about it, nor about all I reported - maybe my mail was flakier than usual): do NOT place open() and close() in a tight inner loop!!! Do the system-traces show that io-intensive programs open() and close() a file each time they need to read()/write() a few hundreds or thousands bytes from/to it...? If so, they look suspiciously like NFS-based I/O to me, rather than 'real' disk I/O!-) Then again, maybe the final IOstone does NOT have the inner-loop open() and close(); although I didn't hear anything from you about this point, surely I wasn't the only one to make it - it's a rather clear one. I do not have the hard data to back this opinion up, but I believe that open() and close() are quite a bit less frequent than read() and write(), so it woule be quite reasonable to accept some performance hit in the former calls if it bought you a somewhat smaller increase in the latter ones... but such a design would lead to poor IOstone numbers (in the 'beta' release, which is all I was able to see). Alex Martelli - CAD.LAB s.p.a., v. Stalingrado 45, Bologna, Italia Email: (work:) staff@cadlab.sublink.org, (home:) alex@am.sublink.org Phone: (work:) ++39 (51) 371099, (home:) ++39 (51) 250434; Fax: ++39 (51) 366964 (work only; any time of day or night). -- Alex Martelli - CAD.LAB s.p.a., v. Stalingrado 45, Bologna, Italia Email: (work:) staff@cadlab.sublink.org, (home:) alex@am.sublink.org Phone: (work:) ++39 (51) 371099, (home:) ++39 (51) 250434; Fax: ++39 (51) 366964 (work only; any time of day or night).
lm@slovax.Sun.COM (Larry McVoy) (10/23/90)
In article <7834@ucdavis.ucdavis.edu> becker@iris.ucdavis.edu (Jeffrey Becker) writes: >In article <143502@@sun.Eng.Sun.COM>, lm@@slovax.Sun.COM (Larry McVoy) writes: >> In article <2387@@van-bc.wimsey.bc.ca> jtc@@van-bc.wimsey.bc.ca (J.T. Conklin) writes: >> >In article <14900016@@hpdmd48.boi.hp.com> sritacco@@hpdmd48.boi.hp.com (Steve Ritacco) writes: >> >> IOstone is misnamed and misleading. It ought to be called "cachestone" or >> "buffer cachestone". It does *not* measure I/O performance. It measures >> cache performance. > >We feel that IOStone is the best widely used indicator of IO system >performance that is currently available. Widely used? By whom? > Jeff Becker > Arvin Park Perhaps you would care to comment on why this benchmark was rejected as an addition to the standard SPEC benchmarks? Especially in light of the fact that SPEC would *love* to add an IO benchmark to their suite that does what you claim yours does. Come on guys, it's obvious that you blew it with this benchmark. Instead of arguing endlessly about how great it is, why don't you go off and try again? --- Larry McVoy, Sun Microsystems (415) 336-7627 ...!sun!lm or lm@sun.com
davidsen@crdos1.crd.ge.COM (Wm E Davidsen Jr) (10/23/90)
In article <144018@sun.Eng.Sun.COM> lm@sun.UUCP (Larry McVoy) writes: | Perhaps you would care to comment on why this benchmark was rejected as | an addition to the standard SPEC benchmarks? Especially in light of the | fact that SPEC would *love* to add an IO benchmark to their suite that | does what you claim yours does. There are many possible reasons, most of which are related to the fit between IOstone and what SPEC wants to include as an IO benchmark. My impression is that they want something to measure the performance of the disk drives rather than the effective performance as seen by a typical program. This is undoubtedly only part of the reason. | Come on guys, it's obvious that you blew it with this benchmark. Instead of | arguing endlessly about how great it is, why don't you go off and try again? The nice thing about this is that you have such a detached attitude and have presented a great list of technical flaws ;-) IOstone returns results which match *in ratio* measurements of production IO bound programs. Therefore many of us find the results useful for comparing machine to one another as platforms for running that load. -- bill davidsen (davidsen@crdos1.crd.GE.COM -or- uunet!crdgw1!crdos1!davidsen) VMS is a text-only adventure game. If you win you can use unix.
lm@slovax.Sun.COM (Larry McVoy) (10/24/90)
In article <2783@crdos1.crd.ge.COM> davidsen@crdos1.crd.ge.com (bill davidsen) writes: >In article <144018@sun.Eng.Sun.COM> lm@sun.UUCP (Larry McVoy) writes: >| Come on guys, it's obvious that you blew it with this benchmark. Instead of >| arguing endlessly about how great it is, why don't you go off and try again? > > The nice thing about this is that you have such a detached attitude >and have presented a great list of technical flaws ;-) IOstone returns >results which match *in ratio* measurements of production IO bound >programs. Therefore many of us find the results useful for comparing >machine to one another as platforms for running that load. Your statment that IOstone matches real programs is unsubstantiated. I have not been able to find any such program. You want to understand, apparently, why I don't like this benchmark. OK, here's why. IOstone claims to measure IO performance, not cache performance. It explicitly tries to clear the cache but fails to do so in the face of large caches such as those provided by SunOS, Mach, and Multics. This gives us results that can not be considered apples to apples comparisons. It would be OK if the designers understood the differences between various caching systems and were trying to measure those differences. If they were trying to do that they could have done so much more effectively by #!/bin/sh SIZE=1024 while true do BS=1024 COUNT=`expr $SIZE / $BS` dd if=/dev/zero of=/tmp/XXX bs=$BS count=$COUNT > /dev/null if [ $? -ne 0 ] then echo can\'t write the file, probably too big. exit 0 fi time dd if=/tmp/XXX of=/dev/null > /dev/null SIZE=`expr $SIZE * 2` done And printed a graph of machine, filesize, time. This would clearly show "who's got a big one" in terms of caches, that is :-) They showed up at Sun one day and wanted to talk to the file system people about our "buffer cache" and how it managed to work so fast. They didn't realize that anything but buffer cache based file systems existed. They *designed in* knowledge of the buffer cache. They claim their system is based in file system traces. I'd like to see the traces. I'd like to see the user program that goes out and tries to clear the cache before running. I've taken lots of file system traces and the traces that I've seen don't like anything like the IO that results from running IOstone. If you really trust this benchmark, go buy Suns. Suns look fantastic under this benchmark. Don't blame me when you try out your new Sun and find that, surprise, it is not 100X faster after all. I'm fighting this benchmark because it gives you some information and tells you how that information is to be interpreted. In particular, it tells you about data movement and calls that data movement IO rates. The data that it is moving on some machines may be IO, but on Sun machines it is almost always data from the cache. I don't call data in the cache IO; maybe you do. If you want an example of a much more reasonable IO benchmark, and dd just doesn't cut it for you, then take a look at Tim Bray's benchmark, bonnie. That benchmark measures IO rates. If you want a benchmark that measures cache hits versus cache misses, you can do better than IOstone. You could run MusBus or any number of light weight multi user benchmarks. They do a fairly good job of measuring your caching / paging / swapping system. --- Larry McVoy, Sun Microsystems (415) 336-7627 ...!sun!lm or lm@sun.com
gillies@m.cs.uiuc.edu (10/24/90)
In all this IOStone talk, I only hear about disk performance and disk
cacheing. That is not enough.
Paging is important
Network throughput is important
Serial I/O (like a 56Kbaud download from another machine) is
important.
High Bandwidth Display is *very* important.
In fact, it's my impression that manufacturers are MORE likely to
screw up these other areas, rather than disk I/O. There seems to be
more disk I/O standardization (SCSI, Unix, stdio.h) than ethernet,
video display, or serial line driver interface standardization.
Are these other IO performances tested by IOStone?
Don W. Gillies, Dept. of Computer Science, University of Illinois
1304 W. Springfield, Urbana, Ill 61801
ARPA: gillies@cs.uiuc.edu UUCP: {uunet,harvard}!uiucdcs!gillieslm@slovax.Sun.COM (Larry McVoy) (10/25/90)
In article <3300200@m.cs.uiuc.edu> gillies@m.cs.uiuc.edu writes: > >In all this IOStone talk, I only hear about disk performance and disk >cacheing. That is not enough. > >Paging is important >Network throughput is important >Serial I/O (like a 56Kbaud download from another machine) is > important. >High Bandwidth Display is *very* important. > >In fact, it's my impression that manufacturers are MORE likely to >screw up these other areas, rather than disk I/O. There seems to be >more disk I/O standardization (SCSI, Unix, stdio.h) than ethernet, >video display, or serial line driver interface standardization. > >Are these other IO performances tested by IOStone? Nope. IOstone does not even pretend to deal with these areas. It pretends to deal with only disk I/O and doesn't do that right. translate("IOstone") { return ("CacheStone"); } Now I don't like IOstone, for reasons that have been widely discussed and confirmed by others in private mail (thanks guys), so you may want to hear it from the authors. But they'll tell you it is just a disk benchmark. --- Larry McVoy, Sun Microsystems (415) 336-7627 ...!sun!lm or lm@sun.com