acp@ms.uky.edu (ACPNET consultant) (03/09/90)
This is on an AT&T 6386 WGS running Sysv/386 3.2.1.
The hard disk in the machine is ~145 megabytes, and the tape drive only
holds 60meg, so I'm trying to fit a full disk backup on one tape with
something like
find . -print | cpio -oc | compress | dd [blocking options] > /dev/rmt/c0s0
Now, including dd in the pipe is *loads* faster than not including
it, but I haven't been able to find any dd options which let the
tape stream. It keeps stopping and backing up. When using something
like ibs=32k obs=512 (the tape writes 512-byte records), dd reports
no full input blocks, all partial blocks--I assume this corresponds to
bursts produced by compress. Something like bs=512 produces all full
blocks but doesn't affect the tape speed.
I assume something as slow as a tape drive could keep up with compress
if the buffering was done properly; it seems that when dd has a large
input buffer, it should hold off writing until the buffer was full;
this would at least allow the tape to stream for several records while
dd fills up its input buffer again.
Should I be using something other than dd? Any other suggestions?
Kenneth Herron
--
acp@ms.uky.edu University of Kentucky ACP Network Consultant
ukma!acp Dept. of Mathematics, room 715 POT (606) 257-2975
Lexington, KY 40506plocher@sally.Sun.COM (John Plocher) (03/09/90)
+-- In <14496@s.ms.uky.edu> acp@ms.uky.edu (ACPNET consultant) writes | This is on an AT&T 6386 WGS running Sysv/386 3.2.1. | | find . -print | cpio -oc | compress | dd [blocking options] > /dev/rmt/c0s0 | it, but I haven't been able to find any dd options which let the | tape stream. It keeps stopping and backing up. When using something | like ibs=32k obs=512 (the tape writes 512-byte records), dd reports +-- Try something like find . -print | cpio -oc | compress | dd of=/dev/rmt/c0s0 -obs 1024k this writes 1Mb at a time to the tape - and the drive *will* stream. Don't worry about tape blocking - the QIC tapes are set up to do all blocking internally, unlike the 9 track reel tape drives you may be familiar with... You may also want to explore the different cpio options avaliable: -I rawdevice which selects the Input device, -O rawdevice which selects the Output device, and -C size which specifies a blocking size -I and -O will detect end of tape and prompt you to change tapes and press return when ready to continue ... find . -print | cpio -oc -O /dev/rst/c0s0 -C 102400 Lastly, there are several programs on the net that are replacements for either cpio (afio), or dd (ddd) which allow double buffering, multi-process buffering, ... which may be another solution to your problems... -John
hcj@lzsc.ATT.COM (HC Johnson) (03/10/90)
In article <14496@s.ms.uky.edu>, acp@ms.uky.edu (ACPNET consultant) writes: > This is on an AT&T 6386 WGS running Sysv/386 3.2.1. > > find . -print | cpio -oc | compress | dd [blocking options] > /dev/rmt/c0s0 > > Now, including dd in the pipe is *loads* faster than not including > it, but I haven't been able to find any dd options which let the > tape stream. It keeps stopping and backing up. When using something If you want your tape to screem ( in bursts) try: dd of=/dev/rmt/c0s0 of=512b this will output 512 BLOCKS of 512 at a time. The tape will run continuously for the big write, then rest while the buffer is refilled by the input to dd. Howard C. Johnson ATT Bell Labs att!lzsc!hcj hcj@lzsc.att.com
nvk@ddsw1.MCS.COM (Norman Kohn) (03/10/90)
In article <14496@s.ms.uky.edu> acp@ms.uky.edu (ACPNET consultant) writes: >find . -print | cpio -oc | compress | dd [blocking options] > /dev/rmt/c0s0 > >Now, including dd in the pipe is *loads* faster than not including >it, but I haven't been able to find any dd options which let the >tape stream. It keeps stopping and backing up. When using something >like ibs=32k obs=512 (the tape writes 512-byte records), dd reports >no full input blocks, all partial blocks--I assume this corresponds to >bursts produced by compress. Something like bs=512 produces all full >blocks but doesn't affect the tape speed. > Try dd ibs=5120 obs=512k (as I recall, this is what ISC uses when loading the system from tape). With ISC it does indeed produce improvement, perhaps even better than bs=512k which I have also used. While I haven't looked, I don't think that the blocks written on tape are any different: cpio and the tape driver still prepare the blocks they want, and all you've done is modify the amount of data that arrives at one time down the pipe. Reading, of course, you reverse the respective ibs/obs sizes. -- Norman Kohn | ...ddsw1!nvk Chicago, Il. | days/ans svc: (312) 650-6840 | eves: (312) 373-0564
cpcahil@virtech.uucp (Conor P. Cahill) (03/12/90)
In article <1990Mar10.041015.23468@ddsw1.MCS.COM> nvk@ddsw1.MCS.COM (Norman Kohn) writes: >In article <14496@s.ms.uky.edu> acp@ms.uky.edu (ACPNET consultant) writes: >>find . -print | cpio -oc | compress | dd [blocking options] > /dev/rmt/c0s0 >> >> [discusion of how to get dd to stream deleted] >> >Try dd ibs=5120 obs=512k While all the suggestions that I have seen for this may actually get the tape to stream on some systems, there is no answer for every system. In addition, if there is some other high priority stuff going on, there is no command that will get the tape to stream always. For example, the following dd will stream an entire tape to a file without stopping at all: dd if=/dev/rmt0 of=junkfile bs=50k However, if I am working on the console and cause just a few lines to be output on the console, the streaming will stop. This even occurs if I use bs=2048k. Whenever I type ls -l on the console the tape drive will stop and restart. So the answer is: If you are working on the console, you may never get the tape to stream for very large chunks. If you are working elsewhere, you might still have problems because of what you are running on the system. If you system is totally quiet (except for the backup stuff itself) you "should" be able to get the data to stream. PS - Another part of this is how fast the compress can feed the data to dd. If your tape drive can get ahead of the compress output, you can't stream. That is why some people were saying to use ...|dd obs=big_value that way dd stored up the data and then fed it to the tape in one big chunk. I guess I am just rambling on now so I will stop. Remember there is no one solution, and sometimes none at all, to getting a tape drive to stream. -- Conor P. Cahill (703)430-9247 Virtual Technologies, Inc., uunet!virtech!cpcahil 46030 Manekin Plaza, Suite 160 Sterling, VA 22170
friedl@mtndew.UUCP (Steve Friedl) (03/14/90)
It was suggested that this be used to send compressed data to a tape drive: > find . -print | cpio -oc | compress | dd [blocking options] > /dev/rmt/c0s0 And HC Johnson suggests: > > If you want your tape to screem ( in bursts) try: > dd of=/dev/rmt/c0s0 of=512b > this will output 512 BLOCKS of 512 at a time. The tape will run continuously > for the big write, then rest while the buffer is refilled by the input to dd. The adventurous might want to investigate another approach that might help the tape stream even more. The problem with the huge blocksizes is that it stops while the buffer refills, so why not use two processes sharing (at least) two buffers. The parent process reads into some huge buffer, then forks a child to write the buffer to the output place (the tape or whatever). While the child is writing the buffer, the parent is busy filling a second buffer. Once the second one is full, the parent waits for the child to finish, then forks again to let it write. This does involve a process switch each block, but if the blocks are large enough it might not be too bad. Of course, you've got to have copy-on-write VM or this become ridiculous really quickly. I have not actually done this, but I bet a donut that AT&T has. Their "ctccpio" command feeding the stupid 23MB tape unit on the 3B2 has lots of process switches, and I think this is how they manage to keep the tape streaming. It trashes the machine for the duration, but it is >much< faster than a dd with even a large blocksize (it could also be because ctccpio turns on stream mode with some ioctl). You could probably do an even better job on this with a reader and a writer with a common set of shared-memory buffers. Then you get the speed of double buffering without the process switches. While writing this article I started diddling with some actual code to handle the process-switch part. After ten minutes it was just about finished, and I might post it in a few days if I ever get it working. I suppose one could really get a big win by building the shared-memory version right into compress, so there was no extra pipeio going on. Thoughts, anybody? Steve -- Stephen J. Friedl, KA8CMY / Software Consultant / Tustin, CA / 3B2-kind-of-guy +1 714 544 6561 voice / friedl@vsi.com / {uunet,attmail}!mtndew!friedl "How could anybody look at Miss April and *not* believe in a God?" - me
les@chinet.chi.il.us (Leslie Mikesell) (03/15/90)
In article <371@mtndew.UUCP> friedl@mtndew.UUCP (Steve Friedl) writes: >The adventurous might want to investigate another approach that >might help the tape stream even more. The problem with the huge >blocksizes is that it stops while the buffer refills, so why not >use two processes sharing (at least) two buffers. >The parent process reads into some huge buffer, then forks a >child to write the buffer to the output place (the tape or >whatever). While the child is writing the buffer, the parent is >busy filling a second buffer. Once the second one is full, the >parent waits for the child to finish, then forks again to let it >write. "afio" does this if you use the -f flag and it works pretty well. >I have not actually done this, but I bet a donut that AT&T has. >Their "ctccpio" command feeding the stupid 23MB tape unit on the >3B2 has lots of process switches, and I think this is how they >manage to keep the tape streaming. It trashes the machine for >the duration, but it is >much< faster than a dd with even a large >blocksize (it could also be because ctccpio turns on stream mode >with some ioctl). The second guess is correct - afio also knows how to stream the 3B2 tape if you compile the option in and use the -C flag. Turning on streaming mode is not something an ordinary program like dd should attempt because (a) it is wierd - see the code in afio, and (b) writing the wrong sized buffer will crash the machine. The 2nd process doesn't really gain that much because the tape controller eats most of the CPU anyway (23M tape). >You could probably do an even better job on this with a reader >and a writer with a common set of shared-memory buffers. Then >you get the speed of double buffering without the process >switches. I started on this a long time ago, but after crashing the 3B2 I moved the test code to a 3B1 and found that writing to a device from shared memory would crash the machine and that you couldn't malloc() anything after attaching a shared memory segment (3B1 only). Again, the 3B1 floppy controller eats the CPU so there isn't a big gain anyway. I recently tried the same code on a 386 and found that it would run about 50-70K/sec from a cpio of the (ESDI) disk to the (125M) tape. The program uses an arbitrary number of arbitrary sized blocks in shared memory and can copy over to local memory on the device side if needed. The tricky part is that it tries to guess when the tape has stopped and lets most of the shared memory fill up before re-starting. It can usually do slightly better than afio even though there is an extra pipe involved (from cpio). >Thoughts, anybody? I'll mail you a copy - if you have time you can clean it up and post it or let me know what you think of the concept. Les Mikesell les@chinet.chi.il.us