martin@mwtech.UUCP (Martin Weitzel) (06/08/90)
Some time ago Conor P. Cahil posted the following request: >I am trying to collect data on the performance of the different tape >backup systems. [....] >If you do run the test please send me the following info: > One file system on my machine produced the following results: CPU: 20 MHz 80386 (ACER 32/20 w 8 MB RAM) Tape drive: Archive 60MB 1/4" Cartridge (FT 60 & SC 422R) Disk Drive & controller: CDC WREN V (383H) & WD 1007V-SE2 OS: 386/ix R 2.0.2 Command: /bin time sh -c "find /usr -print | cpio -ocO /dev/tape" Time: Real: 9:08.5 User: 21.6 System: 2.38.3 Size of file system: 68338 blocks (reported by du -s) 80066 blocks written by cpio The times rate slightly above those that Conor P. Cahill found for his (very similar) system - but I would like to comment a little on mine. As I was never quite satisfied with the speed of my streamer I played around in the last few days and have written some test programs to see where the bottlenecks are. You are invited to read on, though be warned: It's not a very short article. Furthermore I must admit that I have not much background with respect to the internals of streamers (and the QIC-02 standard?). I would appreciate corrections from everybody with better technical background. 1) Problems with meassuring the time: ----------------- Concerning the method (proposed by the original poster) for meassuring the time (/bin/time sh -c "find .... | cpio ...."), someone else (Walter Mecky) has allready pointed out that only the "Real Time" may be meaningful, as parent-child relationships in the shell usually will not deliver the times of the *left* side of a pipeline. (If you don't believe, try "factor 40000000003" - "4", nine "0", "3" - vs. "factor 40000000003 | cat". In the latter case you will only see the system/user times for cat, which are negligible.) If 60 MB cartridge is constantly streaming, the write speed is ~83 KB/sec or 5 MB/min. So if you take "start-up" and "turn-direction" time into the calculation, the above times could not be much better. Further if the tape is constantly streaming and stops only to turn direction, one should be careful not to mis-interpret some timings: If it happens that the last (partially) written track was the 2nd, 4th, 6th or 8th, timing results are not likely to change within about 7 MB of data written to the tape, if you use the driver that rewinds the tape finally. This can have some impact on the avarage result depending on the number of blocks totally written (if it are at least 30 MB as in the original posting suggested, the transfer rate in MB/min may vary by 20% in the worst case as a quick calculation shows). Note also that if the goal is to evaluate tape performance the timing results should rather be based on the number of blocks written to the tape, instead on the size of the file system (otherwise you include the quality of the backup program, ie. by how much the data is expanded by the backup program). Further taking the number of blocks written to the tape as base of the evaluation makes it a little easier to compare the results against the optimum of a constantly streaming tape. 2) Influence of buffer size. ------------------ After having made the above observations, I tried to find the maximum speed with which cpio could work, if it had *not* to write to the tape. In case of my "/usr"-file system, it was more than 250 KB/sec (or 15 MB/min) so that it should be not much of a problem to keep the tape streaming. But a little thought quickly showed that a large buffer could rather tend to make the tape stop than a small buffer. Why? Well, a test program(%1) has shown that writing to the tape is handled slightly asynchroneous and it would be an advantage if the buffers get filled rapidly. If eg. you specify buffer size of around 250 KB, it would keep the tape streaming for three seconds, then need one second to fill the buffer again. This will stop and restart the tape. During most of the first three seconds cpio will not continue to read the disk and when cpio reads the streamer will wait. It may even happen that you reduce cpio-s effective throughput below that of the streamer, though throughput would have been more than sufficient with a small buffer! Practical experience emphasized this theory and of course it seems clear that frequent stops of the tape are performance killers. After this I tried with various buffer sizes and observed that the default - specifying non at all, neither "B" nor "Cnnn" - would keep the tape better streaming than everything else. What I'm not quite sure of is the influence of "cpio ... >/dev/tape" vs. "cpio ... -O /dev/tape". IMHO the latter performs a bit better, though I'm not quite sure. As it has other advantages over output redirection, the second is clearly preferable anyway. 3) Backing up the spool directory with my news ------------------ Trying to do this (*lots* of small files) dramatically changed figures. Here the throughput from "cpio ... >/dev/null" falls below the streaming rate of the tape and goes as low as 45 KB/sec (2.7 MB/min) in some cases(%2). As on this partition there are not only the news but also some larger files, average throughput is approx. 75 KB/sec - still not fast enough to keep up with the tape. Could a large buffer at least help now by "smoothening" the throughput a little, so that the tape does not stop so often? Can we get at least somewhere near 75 KB/sec (currently it seems that caused by the numereous stops of the tape throughput is even below 45 KB/sec)? As backup typically is done when the load of the system is near to zero, it would seem advisable to give huge amounts of memory to the buffer - PROVIDED IT IS SIMULTANEOUSLY FILLED AND EMPTIED(%3). The effects of buffering and especially the recommendable buffer size depend much on the characteristics of the file system: Optimally numereous small files would be alternatingly cpio-ed with some large files, but in any case some really large files should come at the beginning. It may be helpful to optimize that by meassuring the "peaks" and the "over time distribution" of the transfer rate(%2), but even then it may not be possible to achieve satisfactory results in each and every case; ideally the buffering algorithms should be somewhat "adaptive"(%3). 4) Conclusions ------------------ If your tape doesn't stream, you should first evaluate the transfer rate which is requiered to keep the tape streaming. Then take a measurement of the rate with which the data can be read from disk. If both are very close, it might eventually be helpful if you try to avoid the "find"-side of the pipe to cpio by running find seperatly and storing the results in a file which you give later as standard input to cpio. Note also that recent finds have a "-cpio" option, which also may be helpful in reducing the overhead. (Cpio-ing directly from find supports much less options than if you execute cpio directly, but it may suffice for regular backups.) If your meassurements show that disk reads are two to three times faster than tape writes and that the latter are somewhat asynchroneous, you are generally better off if you choose a small buffer. Especially keep in mind that a large buffer alone seems never sufficient. Good results would require simultaneous operation on different buffers. I've heard that some cpio-s claim they do so, but at least for 386/ix I doubt and it couldn't be confirmed from the manuals, though this could be considered a detail of the implementation and would not necessarily have to be mentioned. (Well, even important things would not necessarily have to be mentioned as the current manuals show, but that's quite a different story ... 1/2:-)) So you may either look for some PD backup program that does multi- buffering or you may wait until I come up with a solution that can be tacked to the end of a pipeline(%3). If there is sufficient interest from the net and I find the time to write some documentation, I'll make my programs available. %1: A program that allows you to check the best streaming rate of your tape (and decide whether streaming is possible at all) and that lets you estimate the degree of "asynchronity" of the write requests to your tape driver. %2: A program which shows you peak and average data rates with which the files can be read from the disk. %3: A program that smoothens throughput on the way from the disk to the tape. (I allready have written one, but it's not quite what I finally want. Especially the buffering algorithms will have to be changed a little and made "intelligent".) This program will be usable in a pipe *after* cpio and do its best to get the streaming rate of the tape close to the rate with which the data is read from disk. -- Martin Weitzel, email: martin@mwtech.UUCP, voice: 49-(0)6151-6 56 83
pfrennin@altos86.Altos.COM (Peter Frenning) (06/08/90)
I can substantiate Martins findings. Last night I did some experimenting of my own (I mailed results directly to Conner instead of posting), I also tried to up the buffer size from 10240 to 102400, and saw to my surprise that instead of decreasing the elapsed time went up from 9:50 to about 12:30 secs. (~~50 Mb used of 285 Mb). Given the hints in Martins article, I'll experiment some more over the week-end. If anything interesting comes up I'll post. +-------------------------------------------------+---------------------------+ |Peter Frenning, Altos Computer Systems, San Jose | ***** TANSTAAFL ***** | |2641 Orchard Parkway, San Jose, CA 95134 | There Ain't No Such Thing | |pfrennin@Altos.COM (..!uunet|sun!altos!pfrennin) | As A Free Lunch (Heinlein)| +-------------------------------------------------+---------------------------+
dag@fciva.FRANKLIN.COM (Daniel A. Graifer) (06/12/90)
Just to add my two cents. We've tried all kinds of tricks to get the QIC-24 cartridges on our Prime EXLs (386/25 MultiBusII, SCSI, Maxtor Drives, unknown tape maker, AT&T SysV 3.1) to stream. Cpio -C, afio, etc. Recently, we re-MKFSed our user partitions with gap=1 as recommended by Prime for their new fast file system (called Blocked-Bundled I/O). Voila! Tape streams continuously using cpio -B. Fill a 60MB tape in about 15 minutes (vs. 35 before). Draw what lessons you will from this. (I've learned to at least TRY what the vendor recommends! Actually, I've wanted to do it for ages, I just didn't have the full day to devote to backing up and restoring our two 663MB disks!). Good luck, Dan -- Daniel A. Graifer Franklin Mortgage Capital Corporation uunet!dag@fmccva.franklin.com 7900 Westpark Drive, Suite A130 (703)448-3300 McLean, VA 22102
mark@intek01.UUCP (Mark McWiggins) (06/14/90)
martin@mwtech.UUCP (Martin Weitzel) writes: >So you may either look for some PD backup program that does multi- >buffering ... Get afio from UUNET or elsewhere. I use: afio -o -c 22 -f -s 59m /dev/rmt/c2s0 and it streams like crazy. -- Mark McWiggins Integration Technologies, Inc. (Intek) +1 206 455 9935 DISCLAIMER: I could be wrong ... 1400 112th Ave SE #202 Bellevue WA 98004 uunet!intek01!mark Ask me about C++!