shenkin@cunixf.cc.columbia.edu (Peter S. Shenkin) (12/19/90)
I can't figure out how to make bru do the following things; can someone
enlighten me?
1. Backup all files in a directory EXCEPT the ones I list; for example,
all of /usr except /usr/src and /usr/people. I know I can make a list
of the files I want in a file, then use 'bru -c - < filename', but then
I have to create this file again everytime I do a backup, since I do
not expect the contents of /usr to be static. The FM does not cover
this, so it's possible that it's not doable, yet this is hard to believe.
2. Recognize either a specified or the default tape drive using the -e
option. The FM says that with -e, "If the media size is unknown or
unspecified it is assumed to be infinite." In fact, it always seems
to assume it is infinite, even if specified. The FM also says that
/dev/tape is used as the default, so my reading says that -e should
assume this default; however, even if I say 'bru -e -f /dev/tape', bru
still assumes the media size is infinite. Without any -f option, 'bru -x'
does in fact seem to know how long a tape is, so why doesn't 'bru -e',
and why can't it figure this out even with -f?
It is also a real pain that -e does not work with -Z; the FM does not
mention this limitation; however, an attempt to do this elicits an error
message. I submit that just about the only time you need -e is when you use
-Z; yet that's when you can't get it.
Does anyone out there have or know of a set of scripts for setting up
backups, apportioning directories to tapes, keeping records of what was
backed up on which tape, etc.? Is there anything is any of the archives
(eg, comp.unix.sources) that anyone has found useful in practice?
I also find the system manager backup utility worse than useless, since it
not only doesn't extend bru's capabilities; rather, it limits them to
an unusably small subset.
I am using the QIC-150 drive on a PI (4d25tg) running 3.2.1.
-P.
************************f*u*cn*rd*ths*u*cn*gt*a*gd*jb**************************
Peter S. Shenkin, Department of Chemistry, Barnard College, New York, NY 10027
(212)854-1418 shenkin@cunixf.cc.columbia.edu(Internet) shenkin@cunixf(Bitnet)
***"In scenic New York... where the third world is only a subway ride away."***Dan Karron@UCBVAX.BERKELEY.EDU (12/19/90)
I have scripts I use to do what you want (I think?@!). I use find, egrep -v, fgrep -v, and other home brew filters to make a list of files to archive, not to archive, and so on. Then, as a last step I use bru or tar. I also would like to add that bru does not even know what machine it is on ! There are no brutab entries for the sgi blest tape drivers, or any tapes ! I posted a complaint about no bru entries, or enough info in the man pages to write bru entries for various tapes and drives. There should be an entry for each model drive and each size tape media. If you want to see how I do this, let me know, and I will clean up the code and put in on my anon ftp. Cheers! dan. +-----------------------------------------------------------------------------+ | karron@nyu.edu (E-mail alias that will always find me) | | Fax: 212 340 7190 * Dan Karron, Research Associate | | . . . . . . . . . . . . . . * New York University Medical Center | | 560 First Avenue \*\ Pager <1> (212) 397 9330 | | New York, New York 10016 \**\ <2> 10896 <3> <your-number-here> | | (212) 340 5210 \***\_________________________________________ | | Main machine: karron.med.nyu.edu (128.122.135.3) IRIS 85GT | +-----------------------------------------------------------------------------+
olson@anchor.esd.sgi.com (Dave Olson) (12/22/90)
In <1990Dec18.211216.24299@cunixf.cc.columbia.edu> shenkin@cunixf.cc.columbia.edu (Peter S. Shenkin) writes: | 1. Backup all files in a directory EXCEPT the ones I list; for example, | all of /usr except /usr/src and /usr/people. I know I can make a list | of the files I want in a file, then use 'bru -c - < filename', but then | I have to create this file again everytime I do a backup, since I do | not expect the contents of /usr to be static. It is relatively rare to want to back up a whole directory tree except for a few files, so none of the standard backup programs provided with IRIX have this capablity. Generating the filelist as above is the only way. | 2. Recognize either a specified or the default tape drive using the -e | option. The FM says that with -e, "If the media size is unknown or | unspecified it is assumed to be infinite." In fact, it always seems | to assume it is infinite, even if specified. The FM also says that | /dev/tape is used as the default, so my reading says that -e should | assume this default; however, even if I say 'bru -e -f /dev/tape', bru | still assumes the media size is infinite. Without any -f option, 'bru -x' | does in fact seem to know how long a tape is, so why doesn't 'bru -e', | and why can't it figure this out even with -f? I'm not clear on what you mean by 'recognize' here. By default, bru uses the 'normal' method of doing multivolume tapes on IRIX (for the SCSI and and xm tape drives), which is to write until EOT is detected, then switch to a new volume. To do this, the tape sizes are set to 0K in /etc/brutab. There is absolutely no way to determine tape lengths ahead of time, since e.g. a QIC 150 drive could have either QIC150 or QIC120 cartridges, in any of several lengths. This is true of most types of tape drives. If you are certain of the capacity, and are sure you will use exactly the same tape type and length for each volume, you could use the -s option to specify the length, or edit /etc/brutab. | It is also a real pain that -e does not work with -Z; the FM does not | mention this limitation; however, an attempt to do this elicits an error | message. I submit that just about the only time you need -e is when you use | -Z; yet that's when you can't get it. The reason it doesn't do this is that it would actually have to read and compress every single file in order to determine how much space the compressed files would require. This is incredibly slow. If you really wanted to know, you could always do something gross like: bru -cvfZ /dev/null .... | tail I agree that the above issues should be clarified in the man page. It will be fixed in the next release. -- Dave Olson Life would be so much easier if we could just look at the source code.
shenkin@cunixf.cc.columbia.edu (Peter S. Shenkin) (12/24/90)
> = olson@anchor.esd.sgi.com (Dave Olson) The overall question is one of backup strategy. I think that one would like to be able put together a strategy that backs up the entire file system, or a specified part of it, on close to the minimal number of tapes, such that no archive takes up more than a single tape. Why this last proviso? -- after all, one of bru's strengths is that its archives CAN span several tapes. The reason is that it becomes extremely laborious to restore or backup a given file or group of files from or to a multi-volume archive. It cannot be done unattended. If I know that what I need to restore is on a given tape, or that what I need to backup will fit on a single tape, I can stick it in, type my command and go home. It seems reasonable that bru should provide, if not complete means of doing this, at least sufficient flexibility to allow me to do it by means of additional programs such as scripts. In article <1990Dec21.174239.11753@odin.corp.sgi.com> you write: [[ re: backing up a directory "except for" certain files or subdirectories ]] > >It is relatively rare to want to back up a whole directory tree except >for a few files..... I must say, I am surprised to hear this. I would think that backing up root except for /usr, /tmp and /debug, and backing up /usr except for /usr/people, would be the most common operations, aside from backing up /usr/people. What alternative do most people use, when doing maintenance backups? In retrospect, the ability to specify "backup everything in some filesystem except something else," where the something-else might be taken from a file, for instance, would be a frill, rather than a necessity. One can certainly work around this by using either a pipe or else a file containing the file-names to be backed up. Though then the information you have to keep around to tell you what the hell is on the tape is much more verbose. -------------- [[ re: bru knowing about the length of tapes ]] >I'm not clear on what you mean by 'recognize' here... >...There is absolutely no way to determine tape lengths >ahead of time, since e.g. a QIC 150 drive could have either QIC150 or >QIC120 cartridges, in any of several lengths.... Thanks -- I understand now. I was assuming bru used the size field in /etc/brutab. I noted that some of these were non-zero, but I see now that these are not for the QIC-150 drive. It seems that there would be no problem in making size correspond to the size of the recommended tape -- if the actual tape were shorter, then presumably the end-of-tape signal would over-ride it, no? Now, a naive question. If I do want to edit /etc/brutab, how do I figure out how to set the size? For instance, I have a DC 6150 which is 620 Ft. in length, and that also says 12,500 FTPI on it. What is the size? Section 7 of the IRIX System Administrator's Guide is called "Disk and Cartridge Devices", and on the first page it tells me that one of the topics it will discuss is "tape device sizes", but as far as I can see there is no reference whatsoever to this subject in the chapter, and the tps and mtio man pages are equally unenlightening. --------------- [[ bru -eZ ]] > >The reason it doesn't do this is that it would actually have to read and >compress every single file in order to determine how much space the >compressed files would require. This is incredibly slow.... But if I could do this, I could do something like bru -evZ / >& tmpfile at the beginning of my backup day. It would take a few hours to run, but then I could feed tmpfile into an awk script or other simple program which would divvy the files up into single-volume-sized groups, and then I could bru the groups one by one. It would seem simpler to allow than to disallow this, so why prevent this use? >Life would be so much easier if we could just look at the source code. ... and maybe change it, too! -P. ************************f*u*cn*rd*ths*u*cn*gt*a*gd*jb************************** Peter S. Shenkin, Department of Chemistry, Barnard College, New York, NY 10027 (212)854-1418 shenkin@cunixf.cc.columbia.edu(Internet) shenkin@cunixf(Bitnet) ***"In scenic New York... where the third world is only a subway ride away."***
olson@anchor.esd.sgi.com (Dave Olson) (12/27/90)
In <1990Dec24.155048.29640@cunixf.cc.columbia.edu> shenkin@cunixf.cc.columbia.edu (Peter S. Shenkin) writes: | > = olson@anchor.esd.sgi.com (Dave Olson) | | In article <1990Dec21.174239.11753@odin.corp.sgi.com> you write: | [[ re: backing up a directory "except for" certain files or subdirectories ]] | > | >It is relatively rare to want to back up a whole directory tree except | >for a few files..... | | I must say, I am surprised to hear this. I would think that backing | up root except for /usr, /tmp and /debug, and backing up /usr except for | /usr/people, would be the most common operations, aside from | backing up /usr/people. What alternative do most people use, when | doing maintenance backups? There is an option (-m) that says don't cross mount points. This makes it trivial to back up only the root partition, or only the /usr partition, ignoring any submounts (efs, dbg, or nfs). This would also ignore /debug when backing up root. Of course, this doesn't help for /tmp, but see below. As for backing up all of /usr except /usr/people, how about find /usr/* -prune -print | egrep -v /usr/people | bru -vcm - My main point here is that no backup program will provide all the options that every user/system administrator wants. It should make it POSSIBLE to do most of them, and the ability to accept a filename or directory list from standard input satisfies a great number of the requests. On the other hand, adding options for every exclusion/selection mechanism would make bru even slower, larger, and harder to maintain than it is today. Since most people who do backups regularly (and don't all of us :)) set up shell scripts of some form anyway, the amount of extra typing doesn't seem to be much of an issue. | [[ re: bru knowing about the length of tapes ]] | >...There is absolutely no way to determine tape lengths | >ahead of time, since e.g. a QIC 150 drive could have either QIC150 or | >QIC120 cartridges, in any of several lengths.... | | Thanks -- I understand now. I was assuming bru used the size field in | /etc/brutab. I noted that some of these were non-zero, but I see now | that these are not for the QIC-150 drive. | | It seems that there would be no problem in making size correspond to | the size of the recommended tape -- if the actual tape were shorter, | then presumably the end-of-tape signal would over-ride it, no? When you use the -s option, or a size in brutab, bru does NOT handle the EOT from the driver correctly. This was a design decision on the author's part (not from SGI). A lot of the 'design decisions' in bru seem to have been made (or justified) on the basis that bru was supposed to be portable across a wide range of machines and operating systems, and a number of them don't give reliable end-of-media indications. | Now, a naive question. If I do want to edit /etc/brutab, how do I | figure out how to set the size? For instance, I have a DC 6150 which is | 620 Ft. in length, and that also says 12,500 FTPI on it. What is the size? QIC 150 tapes hold VERY close to 150 Mb, hence the name. They usually hold 1-2 Mbytes more, but the exact amount depends on a number of factors. If you always use fairly new tapes, and get them from the same manufacturer, your best bet is to use dd to size a couple of them and use the lowest value, if you want to use -s, or the size field in brutab. | Section 7 of the IRIX System Administrator's Guide is called "Disk and | Cartridge Devices", and on the first page it tells me that one of the | topics it will discuss is "tape device sizes", but as far as I can see | there is no reference whatsoever to this subject in the chapter, and the | tps and mtio man pages are equally unenlightening. I'll pass this on to the documentation people. | [[ bru -eZ ]] | But if I could do this, I could do something like | bru -evZ / >& tmpfile | at the beginning of my backup day. It would take a few hours to run, | but then I could feed tmpfile into an awk script or other simple program which | would divvy the files up into single-volume-sized groups, and then I could | bru the groups one by one. I've been convinced by you and other people's responses that -eZ should work. I'll change that for the next major release, probably with a warning message at the start that it will take a long time. Note that making such a partitioning at any time prior to the backup always introduces the 'risk' that the size of that directory tree may increase due to new files, growing files, core files, etc., so it may not fit no matter what you do... -- Dave Olson Life would be so much easier if we could just look at the source code.
shenkin@cunixf.cc.columbia.edu (Peter S. Shenkin) (12/29/90)
[Consisting of further comments on Dave's reply, plus a new thread at the end.] In article <1990Dec26.200730.7738@odin.corp.sgi.com> olson@anchor.esd.sgi.com (Dave Olson) writes: >In <1990Dec24.155048.29640@cunixf.cc.columbia.edu> shenkin@cunixf.cc.columbia.edu (Peter S. Shenkin) writes: >| > = olson@anchor.esd.sgi.com (Dave Olson) > >| [[ bru -eZ ]] >| But if I could do this, I could do something like >| bru -evZ / >& tmpfile >| at the beginning of my backup day. It would take a few hours to run, >| but then I could feed tmpfile into an awk script or other simple program which >| would divvy the files up into single-volume-sized groups, and then I could >| bru the groups one by one. > >I've been convinced by you and other people's responses that -eZ >should work. I'll change that for the next major release, probably >with a warning message at the start that it will take a long time. Thanks, Dave. Just a comment: 'bru -eZ' won't take any longer than 'bru -cZ', just as 'bru -e' won't take any longer than 'bru -c', and warning messages get to be very annoying. I suggest you save the warning message for the manual -- and put it under Z, not e! In fact, my 4d25 can't stream the tape drive using 'bru -cZ', even with virtually nothing else going on. >Note that making such a partitioning at any time prior to the backup >always introduces the 'risk' that the size of that directory tree may >increase due to new files, growing files, core files, etc., so it may >not fit no matter what you do... This is possible, but hardly likely if you are either (1) leaving a margin of error appropriate to your system, or (2) backing up a single-user workstation by popping a tape in as you leave for the day. Yes, of course you might have jobs running in background that could be growing files, but I still think we deserve to be able to make the best guess we can, and of course bear the consequences when we guess wrong. After all, if you couldn't shoot yourself in the foot with it, it wouldn't be UNIX. NEW THREAD: I've been thinking about something that I first noticed several years ago when restoring a multi-user VAX from a 0-level (ie, full) dump tape plus several incrementals, following a disk crash. When you do such a restore, you get all the files that were there as of the time of the last incremental, but you also get files -- a whole lot of files, in my experience -- that users had deleted since the 0-level dump was made. That is, you don't really restore the file system; you get a lot of chaff in there along with all the wheat. I personally found that weeding the extraneous stuff out was a real chore. And where disk space is tight, this process could actually overflow available storage. Is this enough of a problem for people to consider a backup strategy that eliminates the problem? One way to do this would be: each time you do an incremental backup, make a complete list of all files present, as well as copies of those files that have changed "recently". When restoring from an incremental tape, have an option that deletes from disk any file that is not on the list -- or, alternatively, instead of deleting it, putting it in a special place, such as a duplicate file-tree built under, say '/delete'. It would be possible for users to emulate this functionality as follows. Each time an backup is done, first write a "table of contents" of the file system to some place on the disk, and make sure this table is included in the backup. If it becomes necessary to do a complete restore, a new table of contents could be made following the final incremental restore. A program could read this new table, and look check for the presence of each entry in the old table, then make a list of the entries in 'new' that are not in 'old'. In fact, if the same program makes both tables, then 'diff' will suffice for this. Then another program could go through the 'diff' list and delete or move the files in it. 'find / -print' could be used to make the tables. Well, now that I've said that, it seems so straightforward that I have no suggestions to Dave. But it seems such a departure from the backup strategies I am aware of that I'd like to get peoples' opinions of it. I think this would also lengthen the practical time interval necessary between full backups. It may be that some people do this already, but if so I'm unaware of it. -P. ************************f*u*cn*rd*ths*u*cn*gt*a*gd*jb************************** Peter S. Shenkin, Department of Chemistry, Barnard College, New York, NY 10027 (212)854-1418 shenkin@cunixf.cc.columbia.edu(Internet) shenkin@cunixf(Bitnet) ***"In scenic New York... where the third world is only a subway ride away."***
dixons%phvax.dnet@SMITHKLINE.COM (12/29/90)
Peter- With regard to your desire for an incremental backup strategy that takes care of the problem of deleted files returning when an series of incremental backups are retrieved: GNU tar does exactly what you suggest. That is, in incremental save mode, it saves a copy of each directory as it goes. Then upon restores, it can put the directories back by deleting files which should no longer be there. Someone posted some mods to get GNU tar working on Irises a while back. I have been using it for a year or so without problems because I wanted the incremental capability. Scott Dixon (dixons@smithkline.com)
system@LNS61.TN.CORNELL.EDU (SYSTEM MANAGER) (01/04/91)
Peter, A minor correction to your posting about VMS Backup: In a recent posting (Message-Id: <1990Dec28.170722.18489@cunixf.cc.columbia.edu> Date: 28 Dec 90 17:07:22 GMT, Subj: Re: Questions bru-ing in my mind) you stated >When you do such a restore, you get all the files that were there as of >the time of the last incremental, but you also get files -- a whole lot of >files, in my experience -- that users had deleted since the 0-level dump >was made. This implies that either you neglected to specify /INCREMENTAL when you restored the incremental tapes, or that the incremental Backup was done incorrectly. VMS BACKUP normally records all of the directory information (but not all of the files), and an incremental restore will automatically remove the files which were deleted from disk between the two Backups. Just a quibble: I don't like seeing misinformation being spread. Selden E. Ball, Jr. (Wilson Lab's network and system manager) Cornell University Voice: +1-607-255-0688 Laboratory of Nuclear Studies FAX: +1-607-255-8062 Wilson Synchrotron Lab BITNET: SYSTEM@CRNLNS Judd Falls & Dryden Road Internet: SYSTEM@LNS61.TN.CORNELL.EDU Ithaca, NY, USA 14853-8001 HEPnet/SPAN: LNS61::SYSTEM = 44283::SYSTEM P.S. another quibble: CUNIXF is not a registered Bitnet node. s.
shenkin@cunixf.cc.columbia.edu (Peter S. Shenkin) (01/04/91)
In article <9101031844.aa00963@VGR.BRL.MIL> system@LNS61.TN.CORNELL.EDU (SYSTEM MANAGER) writes: >Peter, > >A minor correction to your posting about VMS Backup: > .... > >This implies that either you neglected to specify /INCREMENTAL when you >restored the incremental tapes, or that the incremental Backup was done >incorrectly. VMS BACKUP normally records all of the directory >information (but not all of the files), and an incremental restore >will automatically remove the files which were deleted from disk >between the two Backups. I never said VMS. I said VAX. This was under either 4.1bsd or Ultrix; I forget which. In any case, if VMS backup does this, I'm happy to hear that great minds and DEC engineers think alike :-). -P. ************************f*u*cn*rd*ths*u*cn*gt*a*gd*jb************************** Peter S. Shenkin, Department of Chemistry, Barnard College, New York, NY 10027 (212)854-1418 shenkin@cunixf.cc.columbia.edu(Internet) shenkin@cunixf(Bitnet) ***"In scenic New York... where the third world is only a subway ride away."***
fnf@riscokid.UUCP (Fred Fish) (01/10/91)
In article <9012190504.AA27505@karron.med.nyu.edu> karron@cmcl2.nyu.edu writes: >I also would like to add that bru does not even know what machine it is on ! >There are no brutab entries for the sgi blest tape drivers, or any tapes ! > >I posted a complaint about no bru entries, or enough info in the man >pages to write bru entries for various tapes and drives. > >There should be an entry for each model drive and each size tape media. BRU is licensed from Enhanced Software Technologies (1-602-820-0042). I'm sure if you give them a call and explain your situation, that you have insufficient documentation from SGI, they would be happy to sell you a complete manual for the product at a reasonable cost for the manual plus shipping and handling. The SGI version is also quite old, the current version is 12.7. -Fred
fnf@riscokid.UUCP (Fred Fish) (01/10/91)
In article <1990Dec24.155048.29640@cunixf.cc.columbia.edu> shenkin@cunixf.cc.columbia.edu (Peter S. Shenkin) writes: >It seems that there would be no problem in making size correspond to >the size of the recommended tape -- if the actual tape were shorter, >then presumably the end-of-tape signal would over-ride it, no? The problem here is that many drivers provide no way to distinguish between bumping into the end of the tape and hitting a hard error that one cannot advance past. Bru assumes that any unrecoverable read or write error indicates end of tape unless the "advance" flag is set. size= advance Action 0 not set Assume an infinite length tape, any hard error means end of tape. >0 not set Assume that the tape is the specified size. Any hard error before reaching that size may be either eot or a real error. Issue warning and switch to next media. 0 set Assume an infinite length tape, any hard error is either eot or real error. Do NOT issue warning, switch to next media. >0 set Assume that the tape is the specified size. Any hard error before reaching that size is a tape error that can be advanced past by continuing to read or write. Do this until we get to the given size. Note that the last case is the killer if your drive does NOT advance and you have advance set. Bru will continue trying to read or write for however long it takes to get to where it thinks the end of media should be. Most 1/4" tapes go belly up at the first read/write error and refuse to have anything further to do with that tape. >Now, a naive question. If I do want to edit /etc/brutab, how do I >figure out how to set the size? For instance, I have a DC 6150 which is >620 Ft. in length, and that also says 12,500 FTPI on it. What is the size? The simpliest way is to leave the brutab size parameter at 0K, and write enough data with the -v option to fill up a tape. Notice the running size total and subtract off about 5-10% for a "fudge factor". Use this size in your brutab entry. Another way is "dd if=<some very large file> of=/dev/rmt/tp ibs=1k obs=1k" and see how many blocks it takes for dd to bump into the end of the media. >But if I could do this, I could do something like > bru -evZ / >& tmpfile >at the beginning of my backup day. It would take a few hours to run, >but then I could feed tmpfile into an awk script or other simple program which >would divvy the files up into single-volume-sized groups, and then I could >bru the groups one by one. It would seem simpler to allow than to disallow >this, so why prevent this use? You can do this with "bru -cvZf /dev/null / >tmpfile". Internally, the -e option is almost identical to the -c option, except that no I/O buffers are built or written.
fnf@riscokid.UUCP (Fred Fish) (01/10/91)
In article <1990Dec28.170722.18489@cunixf.cc.columbia.edu> shenkin@cunixf.cc.columbia.edu (Peter S. Shenkin) writes: >I've been thinking about something that I first noticed several years ago >when restoring a multi-user VAX from a 0-level (ie, full) dump tape plus >several incrementals, following a disk crash. > >When you do such a restore, you get all the files that were there as of >the time of the last incremental, but you also get files -- a whole lot of >files, in my experience -- that users had deleted since the 0-level dump >was made. That is, you don't really restore the file system; you get >a lot of chaff in there along with all the wheat. I personally found that >weeding the extraneous stuff out was a real chore. And where disk space is >tight, this process could actually overflow available storage. The brumenus product from EST includes a facility called "snapshots", in which a log is kept of all the files in a given tree. Each backup has a snapshot associated with it. Given the snapshots, you can do diffs between them to find out interesting things such as what files were added between any two backups, what files were changed, and what files were deleted. So, you can use the "what files were deleted" info to remove files that disappeared between two backups (say A and B), by restoring A, deleting the appropriate files, and then restoring B. It's relatively straight forward for a source customer to extract this snapshot and "snapdiff" facility from the menus code and use them in their own shell scripts. Also in there is a "namefilter" which does the inclusion/exclusion list sort of thing that everyone seems to want. -Fred