bernsten@phoenix.Princeton.EDU (Dan Bernstein) (03/03/89)
The lack of protection against completely unlinking UNIX files, methods of setting up a ``trash can,'' etc., are frequent subjects of discussion here. Consider the following idea: If every file has an extra link in a special directory meant only for such links, then one doesn't even need to change how rm works in order to preserve files; removing a file or moving another on top of it will not destroy it. To be more specific, in your home directory have a .Copies directory, inaccessible except at select times. In .Copies are files COPY000000, COPY000001, etc., and a file, linklist, that maps each copy number to one or more pathnames. Every COPY file is originally a hard link to a file outside .Copies; if a COPY file has only a single link, the original must have been deleted. The ``emptytrash'' command thus finds and unlink any COPY file with a single link. ``setuptrash'' recursively finds all files within your home directory (except in .Copies) and links each to a COPY file, checking inode numbers to determine whether it's repeating itself. ``unrm'' will probably be the most difficult command to write, because even if each of the above makes a valiant effort to maintain the linklist, /bin/mv and the rename() call don't. Intelligently finding the correct file could be quite a chore when someone just says ``unrm core.'' But simply assuming that ``core'' means the current working directory slash core would be good enough for most cases. It is hoped that the extra links themselves within .Copies do not take up enough file space that they must be removed often, for setuptrash is naturally continuous from the previous setuptrash, doing no extra ln's. (It would be dangerous to have two links within .Copies to the same file, for then that file will never be removed automatically; it is perhaps worth an extra program to check for this.) setuptrash doesn't really need to be done in the foreground, so even with large directories the user need not perceive the delay. emptytrash needs only search a single (though humungous) directory, so it could probably be done in every .logout. Dealing with files moving around and directories moving around makes automatic unrm operation more difficult, unless mv and mvdir are rewritten to maintain linklist. If setuptrash were running continuously this would not be a problem, but more realistically, it is for unrm to figure out which file you mean and give you a choice if necessary. A more serious problem is that a new file created after the latest setuptrash (e.g., during a session if setuptrash is run by .login) will not be preserved. It seems to me that this is just something to warn the user about---that new files are considered in some sense temporary until you run setuptrash. Note that setting this up would change the perceived behavior of rm slightly: at least on this system, rm asks for confirmation if the file has any strange bits set, but only if the file has exactly one link left. With all the .Copies COPY files running around, rm would never ask for this confirmation. (Which makes sense as it's so easy to unrm.) Mail me any corrections, questions, comments, or implementations. I will post a summary in a week. ---Dan Bernstein, bernsten@phoenix.princeton.edu
bernsten@phoenix.Princeton.EDU (Dan Bernstein) (03/24/89)
I did promise a few weeks ago to summarize this in a week... For those who missed the original posting, I was proposing that rather than somehow convert every rm into an mv (unlink() into rename()), you could prepare beforehand for losing files by making an extra link somewhere safe with ln (link()). (That's what I meant to say, anyway.) After all, link() followed by unlink() is almost the same as to rename(). I received several responses, each of which I summarize (in detail) below. Skip to GENERAL THOUGHTS at the end if you don't want to read eighty lines of discussion... Stephen C. North (hector!north, ulysses!north, hector.homer.nh.att.com) prefers the ``old alias "rm" trick'' for its simplicity. I'd say that there is less to think about for the user, but often-noted problems include: 1. If the file preservation is too invisible, you'll be too careless under a system or shell without it. 2. How do you make sure that all unlink()s use the alias? 3. How do you make sure that shell scripts that really should delete a file actually use the real rm? Nevertheless, the ``old alias "rm" trick'' does in practice prove quite useful. James R. Drinkwater (jd@csd4.milw.wisc.edu) sees the problem that every file in the file system would take up an extra inode. He said that when he wanted a file back, it was because of accidental deletion (e.g., rm a* without remembering attn.important) rather than later deciding he needed a file. He made a general proposal that the trash directory contain not only the deleted files but also soft links to their original positions; this is an excellent idea that applies to all trash directory methods. He also proposed that deleted files could be dumped to tape in the end rather than really erased; I think this would require superuser support and also that everybody use the same trash method---jd proposed a global trash directory. He pointed out that files should remain at least a fixed (though user-defined) time; I'd say that if the trash is emptied automatically, this had better be true, whil if the trash is emptied manually, it shouldn't. Christopher J. Calabrese (ulysses!cjc@research, cjc@ulysses.att.com) also pointed out that ``you really want to emptytrash only files over a certain age.'' He criticized my proposal, saying it would require too much overhead and too many ``huge and unnecessary directories'' to maintain, as well as time; this is basically correct. He brought up the problem of distinguishing between deleting when you want a copy preserved and deleting when you don't. He said that people most often delete files that they just created, and that for this reason changing rm's behavior is better than my proposal. Paul English (ileaf!io!speed!pme@eddie.mit.edu, pme@speed.io.uucp) also prefers the idea of changing rm's behavior. He proposed that rather than doing mv, safe rm should make a hard link to the file and then remove the original file. This is, like my method, more restricted than mv, which (on newer systems) can transfer files across filesystems; forcing a physical transfer of a potentially gigantic file is dubious, so I agree that an rm alias should understand the necessity of staying within a filesystem. Eli ? (echarne@orion.cf.uci.edu) mentioned that on his system, file names beginning with a comma are automatically removed after a few days, and that thus a safe way of removing files is to rename them to ,-files. I've observed this elsewhere (# files are also commonly removed); renaming files that way seems to me a very good solution. Barry Shein (bzs@xenna.encore.com) also observed that you usually delete what you're currently working on. He pointed out again the fundamental problem of convincing all programs to unlink() safely---except those shell scripts that should really erase the file (aargh)... He proposed that if UNIX supported real event signals (wake me up when a process does X, and pause that process in the meantime) one could easily trap all unlink()s, and noted that one can effectively do this by using NFS. He mentioned that some editors and other utilities unlink and then recreate the file, which deserves some discussion: The more common action (shell >, vi, most other programs) is to simply write over the file. This means that trapping unlink() won't stop most changes, and brings to light the fact that version numbering in UNIX is a very very tricky subject. What do you do if a process keeps a file open? Do you say the version number increases on each write() (very inefficient) or on each close()? How do you distinguish between files that should not be version numbered and files that should, and what about disk space? I am tempted to say that because of the unified UNIX philosophy for dealing with everything as just some type of file, version numbering is impossible---but I remember hearing someone mention it is possible, and if I do make my claim, Murphy will insure that I am publicly proven wrong. Carl Witty (cwitty@csli.stanford.edu) wondered what mvdir is (it's a general term covering whatever you have to do to move a directory---on BSD, mv can do mvdir, within filesystems...). He reminds us that ``the only cost for an extra hard link is the space in the directory file, which is certainly manageable.'' Of course, this is the opposite view to jd, who worries about all the extra inodes needed. I agree with cwitty; I've never seen more than half the inodes used, on any filesystem. Jerry Peek (jdpeek@rodan.acs.syr.edu) supports my idea and has been looking forward to this summary. Well, now you have it. Kevin Braunsdorf (ksb@j.cc.purdue.edu) said that at Purdue there are three entombing schemes, of which the best one, maintained by Matt Bradburn (mjb@staff.cc.purdue.edu), is a library redefining unlink(), link(), and rename() to safer versions. ``It works.'' Larry Wall (lwall@devvax.jpl.nasa.gov, lwall@jpl-devvax.jpl.nasa.gov) criticized my scheme since it doesn't work across filesystems, and thus doesn't work over his account. He would rather see a trashcan in each subdirectory; this is an interesting idea. GENERAL THOUGHTS The first person who reads this far wins a ... :-) If UNIX were the type of system where version numbering were possible (oops, I mean common, really I do) then the problem of file deletion would be trivial. But version numbering is not possible (oops, common) in UNIX. Changing the low-down behavior of at least unlink() and possibly link() and rename(), by (slow) NFS trickery or by a safe-rm library, would completely solve the problem of files being accidentally deleted. Perhaps the kernel should support this. However, this leaves the problem of files that you really want deleted, or the fact that this is not (yet?) the standard and thus programs will be written for the old standard, or shell scripts that only want a temporary file, or ... . So it's not a simple problem. As for the idea of a more long-term link() to make unlink() more safe, the responses have convinced me that without kernel support this is not an appropriate use of resources for all files. However, it would be useful as a ``preserve'' program that you explicitly invoke upon files that you do not want deleted at any cost. preserve would not stop any changes, and it would have to list all those programs that unlink() and recreate files as ``preserve will not work with these, sorry,'' but it would prevent accidental deletion of the named files. So you would just preserve your most important files, as a last resort. There could be advantages to writing preserve as simply a process that keeps the file open. This is a shorter-term solution, giving Murphy a great excuse to crash the machine; but it would not require an extra filesystem entry, and it would be trivial to include automatic warnings every so often if the file is accidentally removed. ``Mail from username... Subject: preserving "foo". To recover "foo", type "unrm foo"...'' Or I suppose the file could be re-instated in a trash directory by that process... ---Dan Bernstein, bernsten@phoenix.princeton.edu
ekrell@hector.UUCP (Eduardo Krell) (03/25/89)
In article <7360@phoenix.Princeton.EDU> bernsten@phoenix.Princeton.EDU (Dan Bernstein) writes: >If UNIX were the type of system where version numbering were possible >(oops, I mean common, really I do) then the problem of file deletion >would be trivial. But version numbering is not possible (oops, common) >in UNIX. It's not common, but it has been done. Dave Korn and/or I are going to present a paper called "The 3-D File System" at the next summer Usenix conference in Baltimore. If you listen to the presentation or read the paper in the proceedings, I think you'll realize it adds version numbering and more to the Unix file system. Eduardo Krell AT&T Bell Laboratories, Murray Hill, NJ UUCP: {att,decvax,ucbvax}!ulysses!ekrell Internet: ekrell@ulysses.att.com
jik@athena.mit.edu (Jonathan I. Kamens) (03/26/89)
In article <7360@phoenix.Princeton.EDU> bernsten@phoenix.Princeton.EDU (Dan Bernstein) writes: >For those who missed the original posting, I was proposing that rather >than somehow convert every rm into an mv (unlink() into rename()), you >could prepare beforehand for losing files by making an extra link >somewhere safe with ln (link()). (That's what I meant to say, anyway.) >After all, link() followed by unlink() is almost the same as to rename(). (I had missed the original posting.... Oh, well :-) There are two problems with using link() or with keeping a file open in order to prevent it from being lost. Both of these mechanisms are useless under NFS (and possibly other protocols, but I only know for sure about NFS) -- soft links are meaningless because an NFS filesystem can be mounted anywhere and paths are therefore meaningless, and hard links won't work very well either because the trash directory has to be on the same filesystem as the deleted file. If I mount "/foo/bar/baz" from another machine onto "/mnt" on my machine, and I use an rm that uses a link() call to delete a file, where is it going to put the link? Keeping a file open also does nothing under NFS -- there's no state, so the NFS server doesn't realize that the file is open and it will go away. I hear that the next version of NFS supplies the capability to do file locking, but I don't know if this includes keeping track of busy files. This discussion came up some time last year in comp.unix.wizards. At that point, I was working on a suite of file-deletion (and undeletion) utilities for Project Athena. That set of programs has been finished and is going in the next Project Athena software release. The particular attractiveness of the "delete" program is that it can exactly emulate the behavior of rm and rmdir if that is desired, with the result that you can do a couple of aliases and then never realize that you're using delete (unless you need to undelete something)! If people are interested in getting the sources to the programs (delete, undelete, expunge, purge, and lsdel), I can probably send people sources (after I check with my boss and get a copy of the copyright notice that has to appear at the top of each file :-). It's only been tested on BSD, but I assume that it will work with minimal modification on other Unix platforms as well. Jonathan Kamens USnail: MIT Project Athena 410 Memorial Drive, No. 223F jik@Athena.MIT.EDU Cambridge, MA 02139-4318 Office: 617-253-4261 Home: 617-225-8218
john@anasaz.UUCP (John Moore) (03/26/89)
In article <10113@bloom-beacon.MIT.EDU> jik@athena.mit.edu (Jonathan I. Kamens) writes:
]If people are interested in getting the sources to the programs
](delete, undelete, expunge, purge, and lsdel), I can probably send
]people sources (after I check with my boss and get a copy of the
Better yet, why not post them to unix.sources?
--
John Moore (NJ7E) mcdphx!anasaz!john asuvax!anasaz!john
(602) 861-7607 (day or eve)
The opinions expressed here are obviously not mine, so they must be
someone else's. :-)guy@auspex.UUCP (Guy Harris) (03/27/89)
>Keeping a file open also does nothing under NFS -- there's no state, >so the NFS server doesn't realize that the file is open and it will go >away. I hear that the next version of NFS supplies the capability to >do file locking, but I don't know if this includes keeping track of >busy files. I'm not sure what you mean by "the next version of NFS". The *current* version of the ONC/NFS source as distributed by Sun includes a network service to perform byte-span locking (which subsumes file locking) over the network; said service is different from the file access service that NFS refers to in the strict sense. I think the previous version of the ONC/NFS source had it as well. The next version of the NFS *protocol* doesn't include support for byte-span locking; that's still left up to a separate service and protocol. Since the locking is a separate service, and since it's accessed on typical UNIX implementations only through the "fcntl" system call (POSIX/SVID-style), neither Version 2 (the current version of the protocol) nor Version 3 (the next version of the protocol) of the NFS service keeps track of files that a client has open, so no, this doesn't include keeping track of busy files.
hedrick@geneva.rutgers.edu (Charles Hedrick) (03/27/89)
> soft links are > meaningless because an NFS filesystem can be mounted anywhere and > paths are therefore meaningless There are two ways to make soft links work across NFS. One is to use a mounting convention that gives the same path names on all systems. We often use /machinename as the top level directory, even on the machine itself. So a link to /machine/usr/bin/foo works anywhere. The other is for all of your symbolic links to be relative. E.g. if you want /usr/bin/foo to be a link to /usr/ucb/bar, define the link as ../ucb/bar. If you want it to be a link to /bin/bar, define the link as ../../bin/bar. This presupposes that you mount all file systems from a given system at the same point in the local file system, of course. > I hear that the next version of NFS supplies the capability to > do file locking, but I don't know if this includes keeping track of > busy files. File locking (actually System V record locking) has been present in NFS on Suns since at least release 3.2. It appears that the implementation in 4.0.2 actually works. (4.0.1 is the current released version, but you can get 4.0.2 of the lock daemon from the Sun software support people.) NFS does manage to keep files that are still open around. Try the following: tail -f foo & rm foo ls .nfs* You'll see that foo has been renamed to .nfsXXX. Now if you kill the tail, .nfsXXX will go away. I'm not sure quite how that interacts with statelessness. It's possible that if you open a file, remove it, and then crash before closing it, that the .nfsXXX file will stay on the the file server. I haven't looked at the code that carefully. Despite all the comments about how NFS violates "Unix semantics", we've not run into anything that failed across NFS, aside from bugs in implementations.
jik@athena.mit.edu (Jonathan I. Kamens) (03/27/89)
In article <Mar.26.19.51.41.1989.8372@geneva.rutgers.edu> hedrick@geneva.rutgers.edu (Charles Hedrick) writes: > >There are two ways to make soft links work across NFS. One is to use >a mounting convention that gives the same path names on all systems. >We often use /machinename as the top level directory, even on the >machine itself. So a link to /machine/usr/bin/foo works anywhere. >The other is for all of your symbolic links to be relative. E.g. if >you want /usr/bin/foo to be a link to /usr/ucb/bar, define the link as >../ucb/bar. If you want it to be a link to /bin/bar, define the link >as ../../bin/bar. This presupposes that you mount all file systems >from a given system at the same point in the local file system, of >course. Both of these will work fine when you're in a relatively small system and NFS is only exported to a small number of hosts. However, when you have more than 1000 workstations and more machines of other types, all of which can NFS mount nearly any directory off of any fileserver on campus on any mount point, it stops being feasible to force directories always to get mounted in the same place and always at the same level from the root. In fact, the capability to mount in different locations is used a lot here at Athena. Both of the points you made will help make links *more* reliable, but none of them will make the links completely reliable, and (in my opinion) that's what you need when you are going to be using the links for file removal. When we first started going through design reviews for delete et al, a system of ".deleted" directories storing symbolic links to deleted files was proposed. However, as we talked more and more about it, it degenerated into something like this: "When a file is deleted, it is renamed from foo to .#foo so that it becomes invisible to the user and will be cleaned up in the nightly filesystem sweeps after a few days. Then, the delete program starts walking up the directory tree until it encounters either (a) a directory named .deleted or (b) the top of the filesystem (NFS or whatever); while doing this it keeps track of how many levels up it's gone and all of the directories it's transversed up in order to be able to do a link back to the deleted file. If it gets to the top of the filesystem and doesn't find a .deleted directory, it tries to create one. If it fails, it starts walking back *down* the directory chain until it finds a directory in which it can create a .deleted directory, coming, at worst, back to the original directory from which it started. By this point, it should have been able to create a .deleted directory, and it makes a link in that directory to the deleted file." If you think about it, you'll realize that this is the only way to guarantee that a link will be created when working with NFS, unless you have all filesystems mounted on all machines at all times, something which we most definitely do not. So, as you can see there are certain things that are *not* feasible with NFS in certain environments. Jonathan Kamens USnail: MIT Project Athena 410 Memorial Drive, No. 223F jik@Athena.MIT.EDU Cambridge, MA 02139-4318 Office: 617-253-4261 Home: 617-225-8218 P.S. Oh, so that's what all those .nfsXXX files are from :-)
jpn@genrad.uucp (John P. Nelson) (03/28/89)
>NFS does manage to keep files that are still open around. Try the following: > tail -f foo & > rm foo > ls .nfs* >You'll see that foo has been renamed to .nfsXXX. Now if you kill the >tail, .nfsXXX will go away. This is only true if the 'rm' and the 'tail' occur on the same machine. Try it with two different machines: No .nfsXXX file is created. You get assorted bizzare behaviors when this happens. Actually, I discovered this because we had an application where this behavior was very annoying. >I'm not sure quite how that interacts >with statelessness. It's possible that if you open a file, remove it, >and then crash before closing it, that the .nfsXXX file will stay on >the the file server. Yes, it does. >Despite all the comments about how NFS violates "Unix semantics", >we've not run into anything that failed across NFS, aside from bugs in >implementations. We had a ascii "database" with both readers and writers: The readers simply open the file, but the writers use locking and always create a new file, then rename the new file as the old file (expecting old readers to keep their descriptors to the old file). Unfortunately, this doesn't work under NFS (at least not in sun 3.x: I don't know about sun 4.x). john nelson UUCP: {decvax,mit-eddie}!genrad!jpn smail: jpn@lightning.genrad.com
dgk@ulysses.homer.nj.att.com (David Korn[drew]) (03/29/89)
I implemented a version of unlink(name) that did a rename() to a file with the same name in a sub-directory called .trashcan, if this directory exists. I modified rm to use this unlink, and I modified rmdir to consider a directory empty if the only thing it it was a .trashcan directory. This scheme has the advantage of having the link() always on the same file system. Also, only files that have a .trashcan subdirectory are saved. Thus, you can have a single rm command that works on directories that you want files saved, and to remove trash from the .trashcan directory. Of course, I added a line to cron to clean out .trashcan directories every day.