jfh@rpp386.cactus.org (John F. Haugh II) (10/09/90)
In article <1990Oct7.155203.13283@hq.demos.su> avg@hq.demos.su (Vadim G. Antonov) writes: > Hi! I think the problem of login trojan horses has a quite simple > solution: > > 1) There should be UNMASKABLE method of killing all programs > reading this terminal. It may be a predefined sequence > of characters or a special hardware signal like BREAK or > CARRIER DROP. This feature should be hard-coded in TTY > driver. This is often refered to as "the secure attention key", or SAK for short. The idea is that some special key sequence is associated with the user wanting to validate the path to the system. By pressing the SAK sequence the user is certain they are speaking directly to the system, and not to a trojan horse. The BREAK key is a typical choice since it is baud-rate insensitive. Any key sequence which is dependent on the correct baud rate being set at any moment is likely to be circumvented by a clever hacker switching baud rates on you. > 2) All processes, associated with a TTY should be killed > (as SIGHUP does) andprotected processes should be > RE-ASSOCIATED with an unique TTY-id (which actually > does not exist). Killed is a tad strong. The only real requirement is that unauthorized access to the device be revoked. You can do this simply by marking a file table entry which references the device as "stale" and returning an error on any attempt (other than close, I suppose ...) to use that descriptor. In practice it seems that being overly agressive with your desires to kill processes makes users unhappy. Suitably privileged processes would be able to re-open the device, allowing daemons such as biff and month to avoid losing touch with reality ... You might want to get a copy of the DoD Orange Book, if you don't have one already. -- John F. Haugh II UUCP: ...!cs.utexas.edu!rpp386!jfh Ma Bell: (512) 832-8832 Domain: jfh@rpp386.cactus.org "SCCS, the source motel! Programs check in and never check out!" -- Ken Thompson
cowan@marob.masa.com (John Cowan) (10/09/90)
In article <18578@rpp386.cactus.org>, jfh@rpp386.cactus.org (John F. Haugh II) writes: >In article <1990Oct7.155203.13283@hq.demos.su>, > avg@hq.demos.su (Vadim G. Antonov) writes: >> [stuff about secure access key deleted] >You might want to get a copy of the DoD Orange Book, if you don't >have one already. Er, I know it's 1990 and all, but can people in the .SU domain get copies of DoD information? -- cowan@marob.masa.com (aka ...!hombre!marob!cowan) e'osai ko sarji la lojban
avg@hq.demos.su (Vadim G. Antonov) (10/11/90)
In article <18578@rpp386.cactus.org> jfh@rpp386.cactus.org (John F. Haugh II) writes: >You might want to get a copy of the DoD Orange Book, if you don't >have one already. Thank you, John, I'll ask USA DoD to send me a copy of it together with a map of current locations of SAM radars. :-) :-) :-) Vadim Antonov, DEMOS, Moscow, USSR (It is a joke!)
jfh@rpp386.cactus.org (John F. Haugh II) (10/11/90)
In article <27120124.5C36@marob.masa.com> cowan@marob.masa.com (John Cowan) writes: >Er, I know it's 1990 and all, but can people in the .SU domain get copies >of DoD information? my copy does not include any restrictions on which countries the information may be exported to. presumably the orange book can be sent anywheres i want. the foreword explains how to get copies if you are part of the public. i think even people in ussr are part of "the public". i would expect that any citizen of any country can send a draft payable in u.s. funds and get a copy. there is even an electronic version floating around, but i understant that moscow is being fed at 1200 baud or so from finland, and i don't think news should get clogged up just so one person can have a copy of the orange book. -- John F. Haugh II UUCP: ...!cs.utexas.edu!rpp386!jfh Ma Bell: (512) 832-8832 Domain: jfh@rpp386.cactus.org "SCCS, the source motel! Programs check in and never check out!" -- Ken Thompson
steve@nuchat.UUCP (Steve Nuchia) (10/11/90)
In article <1990Oct10.175043.10305@hq.demos.su> avg@hq.demos.su (Vadim G. Antonov) writes: >Thank you, John, I'll ask USA DoD to send me a copy of it together >with a map of current locations of SAM radars. :-) :-) :-) The Orange book is completely non-classified -- it bears not even a Confidential or NoForeign. The first item in the list of purposes in my 1985 edition says: To provide a standard to manufacturers as to what security features to build into their new and planned commercial products ... I bought my copy by walking into a GPO (Government Printing Office) storefront and handing them money. Anybody could do the same. In the SU you could probably just ask your cab driver :-) There is an address for "the public" to get copies: Office of Standards and Products, National Computer Security Center Fort Meade, MD 20755-6000 Attention: Chief, Computer Security Standards They probably want hard currency, and almost certainly won't tell you anything about radars, but it's worth a try, no? -- Steve Nuchia South Coast Computing Services (713) 964-2462 "To learn which questions are unanswerable, and _not_to_answer_them; this skill is most needful in times of stress and darkness." Ursula LeGuin, _The_Left_Hand_of_Darkness_
emv@math.lsa.umich.edu (Edward Vielmetti) (10/11/90)
In article <18583@rpp386.cactus.org> jfh@rpp386.cactus.org (John F. Haugh II) writes:
my copy does not include any restrictions on which countries the
information may be exported to. presumably the orange book can be
sent anywheres i want.
it was posted to comp.doc, and it's ftp'able or uucp'able from uunet
and from midgard.ucsc.edu.
--Ed
Edward Vielmetti, U of Michigan math dept <emv@math.lsa.umich.edu>
moderator, comp.archivespaul@prcrs.UUCP (Paul Hite) (10/12/90)
In article <18583@rpp386.cactus.org>, jfh@rpp386.cactus.org (John F. Haugh II) writes: > In article <27120124.5C36@marob.masa.com> cowan@marob.masa.com (John Cowan) writes: > >Er, I know it's 1990 and all, but can people in the .SU domain get copies > >of DoD information? > > there is > even an electronic version floating around, but i understant that > moscow is being fed at 1200 baud or so from finland, and i don't > think news should get clogged up just so one person can have a copy > of the orange book. Actually the Orange Book was published in comp.doc a few months ago. That's where I got my copy. (But 1200 baud? Sheesh.) Paul Hite PRC Realty Systems McLean,Va uunet!prcrs!paul (703) 556-2243 You can't tell which way the train went by studying its tracks.
sarima@tdatirv.UUCP (Stanley Friesen) (10/13/90)
In article <18578@rpp386.cactus.org> jfh@rpp386.cactus.org (John F. Haugh II) writes: >This is often refered to as "the secure attention key", or SAK for >short. The idea is that some special key sequence is associated with >the user wanting to validate the path to the system. ... >Killed is a tad strong. The only real requirement is that unauthorized >access to the device be revoked. You can do this ... >In practice it seems that being overly agressive with your >desires to kill processes makes users unhappy. Suitably privileged >processes would be able to re-open the device, allowing daemons such as >biff and month to avoid losing touch with reality ... A tad strong indeed. Excessively strong I would say. Under UNIX I would suggest that the SAK suspends all jobs currently attached to the terminal, and mark the file descriptor for suspension on future I/O attampts. Perhaps the suspended processes could be aged and killed if not restarted in a reasonable amount of time, but certainly not immediately. Why suspension? As protection against accidental or malicious use of the SAK in the middle of an important activity (like editing a large source file). This would allow the user to reconnect the session again later. (Of course the login process should match sessions with users, so someone else cannot attach to my session, but this too is possible). Without a restart capability the SAK feature is just too much. -- --------------- uunet!tdatirv!sarima (Stanley Friesen)
greywolf@unisoft.UUCP (The Grey Wolf) (10/18/90)
In article <18578@rpp386.cactus.org> jfh@rpp386.cactus.org (John F. Haugh II) writes: # In article <1990Oct7.155203.13283@hq.demos.su> avg@hq.demos.su (Vadim G. Antonov) writes: # # > 2) All processes, associated with a TTY should be killed # > (as SIGHUP does) andprotected processes should be # > RE-ASSOCIATED with an unique TTY-id (which actually # > does not exist). # # Killed is a tad strong. The only real requirement is that unauthorized # access to the device be revoked. You can do this simply by marking a # file table entry which references the device as "stale" and returning # an error on any attempt (other than close, I suppose ...) to use that # descriptor. I believe this is what the vhangup() call does. Or used to. I don't know what has taken its place, but I have heard rumours of the demise of this call... # John F. Haugh II UUCP: ...!cs.utexas.edu!rpp386!jfh # Ma Bell: (512) 832-8832 Domain: jfh@rpp386.cactus.org # "SCCS, the source motel! Programs check in and never check out!" # -- Ken Thompson I wonder, incidentally, why does close() return something? Is it just that it's a system call? What checks for close()'s return value? I could see it being used for security checks and such, I suppose (verifying that "fd %d had to be closed -- CHECK THIS"). -- "This is *not* going to work!" "Well, why didn't you say so before?" "I *did* say so before!" ...!{ucbvax,acad,uunet,amdahl,pyramid}!unisoft!greywolf
bzs@world.std.com (Barry Shein) (10/18/90)
>I wonder, incidentally, why does close() return something? Is it just that >it's a system call? What checks for close()'s return value? I could see it >being used for security checks and such, I suppose (verifying that "fd %d >had to be closed -- CHECK THIS"). Trying to close a closed file descriptor will cause -1 to be returned and errno set to EBADF. Close can also be interrupted (EINTR), I would guess that's most likely to happen with socketio. These are conditions a program might want to check for, even if (in the first case) just to check for logic bugs in the program. -- -Barry Shein Software Tool & Die | {xylogics,uunet}!world!bzs | bzs@world.std.com Purveyors to the Trade | Voice: 617-739-0202 | Login: 617-739-WRLD
jik@athena.mit.edu (Jonathan I. Kamens) (10/18/90)
In article <BZS.90Oct17215707@world.std.com>, bzs@world.std.com (Barry Shein) writes: |> >I wonder, incidentally, why does close() return something? Is it just that |> >it's a system call? What checks for close()'s return value? I could see it |> >being used for security checks and such, I suppose (verifying that "fd %d |> >had to be closed -- CHECK THIS"). Programmers failing to check the return value of close() in their programs is one of the never-ending problems we have had here at Project Athena. If you are working on a remote filesystem (such as NFS or AFS), your file is often not completely flushed to the remote filesystem until you close it. Therefore, if you are (for example) close to your quota, and you do close(), it is quite possible that the close() will put you over your quota and therefore fail. However, programmers who don't check close() never find out that the file didn't get written because the user went over his quota, and this can have very bad effects, for example, GNU emacs 18.54 (I don't know about 18.55, we haven't switched to it yet) didn't check close() when saving files, so a user might spend several hours working on a paper, then save it (without getting any errors from emacs) and log out. The next time he logs in, the paper is empty, and all his work has been lost. A worse example is something like ispell, a spelling program that saves a copy of the original file before spell-checking it. A previous version (actually, we may not have gotten around to fixing this bug yet, I don't remember) would save both the back-up and spell-checked file without checking close()'s return value -- the result is that both files were truncated and the user left with nothing if they could not be written for some reason. Unix programmers of the world -- PLEASE do not assume that all the world's UFS and that all of the world's filesystems have the same semantics as local disks. They DON'T. Close() returns a value because it is possible for it to fail. -- Jonathan Kamens USnail: MIT Project Athena 11 Ashford Terrace jik@Athena.MIT.EDU Allston, MA 02134 Office: 617-253-8085 Home: 617-782-0710
steve@nuchat.UUCP (Steve Nuchia) (10/18/90)
In article <BZS.90Oct17215707@world.std.com> bzs@world.std.com (Barry Shein) writes: >Trying to close a closed file descriptor will cause -1 to be returned >and errno set to EBADF. Close can also be interrupted (EINTR), I would You get EBADF if you pass it something that isn't an open file descriptor, whether it used to be one or not. If you have a 4.2 BSD or earlier "more" (maybe still, I dunno) ask it for help N times, where N is approximately equal to the maximum number of file descriptors your kernel allows. This demonstration "worked" (ie, broke) last time I tried it on a Sun. I'm told that the code is passing a FILE * to close(2), and not checking the return value. Naughty Naughty. On the other hand, you should never check for an error that you don't know what to do about. :-) -- Steve Nuchia South Coast Computing Services (713) 964-2462 "Could we find tools that would teach their own use, we should have discovered something truly beyond price." Socrates, in Plato's Republic
brnstnd@kramden.acf.nyu.edu (Dan Bernstein) (10/19/90)
In article <1990Oct18.121818.9956@athena.mit.edu> jik@athena.mit.edu (Jonathan I. Kamens) writes: > If you are working on a remote filesystem (such as NFS or AFS), your file is > often not completely flushed to the remote filesystem until you close it. > Therefore, if you are (for example) close to your quota, and you do close(), > it is quite possible that the close() will put you over your quota and > therefore fail. This is a failure of NFS, not a reason to check close(). (Well, close() should be restarted on EINTR, but that's a side issue.) It's not right to give the programmer the burden of working around NFS's bugs. Same for AFS. You allude to a real problem, though: an application may have to deal with disk-full errors on any write(). An fwaste(fd,n) syscall, to allocate n bytes for a file, would solve this. (Note that this is different from the proposed fextend(fd,n), which changes the length of the file but zero-fills rather than allocate space on disk. ftruncate(fd,n) is the opposite of each.) ---Dan
jik@athena.mit.edu (Jonathan I. Kamens) (10/19/90)
In article <19547:Oct1818:25:2690@kramden.acf.nyu.edu>, brnstnd@kramden.acf.nyu.edu (Dan Bernstein) writes: |> This is a failure of NFS, not a reason to check close(). (Well, close() |> should be restarted on EINTR, but that's a side issue.) It's not right |> to give the programmer the burden of working around NFS's bugs. Same for |> AFS. Hogwash. If close() is specified as returning an error value, then it is reasonable for it to sometimes return an error value, and it is also reasonable for the programmer to be expected to check and deal with its return value. The error is that when kernels first started getting code put into them that would set errno to something other than EBADF in response to a close(), the man page for close() (at least in BSD4.3, which is what I'm looking at) was not updated to reflect the new possible errno values. Of course, there are other man pages besides the one for close() that don't list all possible errno values, so that's nothing unusual. Just as it is reasonable for write() to return an error when the disk is full, it is reasonable for close() to do so, and the man page should be updated to reflect that. You seem to be saying that filesystems should never be allowed to postpone writing to disk until close(), which is a completely reasonable thing for a filesystem to want to do (*especially* remote filesystems), just because close() should be easier for the programmer to deal with. Feh. The whole world isn't UFS anymore, and to try to pretend that it is and not update our system calls to reflect the fact that it isn't is to unnecessarily limit progress. -- Jonathan Kamens USnail: MIT Project Athena 11 Ashford Terrace jik@Athena.MIT.EDU Allston, MA 02134 Office: 617-253-8085 Home: 617-782-0710
ske@pkmab.se (Kristoffer Eriksson) (10/19/90)
In article <19547:Oct1818:25:2690@kramden.acf.nyu.edu> brnstnd@kramden.acf.nyu.edu (Dan Bernstein) writes: >This is a failure of NFS, not a reason to check close(). It's not right >to give the programmer the burden of working around NFS's bugs. Same for AFS. If I remember correctly, physical I/O-errors are sometimes reported not on the write() that wrote the data that provoked the error, but on the next I/O-call. This is because the data is buffered in the filesystem cache before it is physically written out, and the error may occur after the write() has already returned. The error is reported back to the application at the first opportunity after it occured, and this may sometimes be the close() call. Also, close() may flush buffered data from the cache, wich can also result in new I/O-errors. Therefore, close() can fail on any filesystem, even local ones, and should always be checked (provided that you care about your data). -- Kristoffer Eriksson, Peridot Konsult AB, Hagagatan 6, S-703 40 Oerebro, Sweden Phone: +46 19-13 03 60 ! e-mail: ske@pkmab.se Fax: +46 19-11 51 03 ! or ...!{uunet,mcsun}!sunic.sunet.se!kullmar!pkmab!ske
richard@aiai.ed.ac.uk (Richard Tobin) (10/19/90)
In article <BZS.90Oct17215707@world.std.com> bzs@world.std.com (Barry Shein) writes: >>I wonder, incidentally, why does close() return something? >Trying to close a closed file descriptor will cause -1 to be returned >and errno set to EBADF. Close can also be interrupted (EINTR), And with at least one network file system, errors such as "disk full" may not be reported until the file is closed, due to client buffering. Failure to check for this leads to editors truncating files and other such unpleasantness. -- Richard -- Richard Tobin, JANET: R.Tobin@uk.ac.ed AI Applications Institute, ARPA: R.Tobin%uk.ac.ed@nsfnet-relay.ac.uk Edinburgh University. UUCP: ...!ukc!ed.ac.uk!R.Tobin
bzs@world.std.com (Barry Shein) (10/20/90)
>From: jik@athena.mit.edu (Jonathan I. Kamens) >In article <BZS.90Oct17215707@world.std.com>, bzs@world.std.com (Barry Shein) writes: No, Barry Shein does not "writes" what you quoted, try to keep your attributions accurate or leave them out altogether. Thank you. -- -Barry Shein Software Tool & Die | {xylogics,uunet}!world!bzs | bzs@world.std.com Purveyors to the Trade | Voice: 617-739-0202 | Login: 617-739-WRLD
ts@cup.portal.com (Tim W Smith) (10/21/90)
< Programmers failing to check the return value of close() in their programs < is one of the never-ending problems we have had here at Project Athena. < < If you are working on a remote filesystem (such as NFS or AFS), your file i s < often not completely flushed to the remote filesystem until you close it. < Therefore, if you are (for example) close to your quota, and you do close(), < it is quite possible that the close() will put you over your quota and < therefore fail. This sounds like very bad file system design. If you are over some quota, the write() that caused you to go over the quota should return an error. The local system should determine your quota and remaining space and check this on each write. Furthermore, when the close() fails, you now have a program that knows that some amount of previously written data is not valid. How can it determine how much? Furthermore, can the program recover the data from the system? Or does this mean that a program should keep a copy in memory of all data that is hard to reproduce until it closes the file? In summary, this behaviour of a file system is not acceptable. Tim Smith
boyd@necisa.ho.necisa.oz (Boyd Roberts) (10/23/90)
In article <19547:Oct1818:25:2690@kramden.acf.nyu.edu> brnstnd@kramden.acf.nyu.edu (Dan Bernstein) writes: > >This is a failure of NFS, not a reason to check close(). (Well, close() >should be restarted on EINTR, but that's a side issue.) It's not right >to give the programmer the burden of working around NFS's bugs. Same for >AFS. > Always, always, always check the return values of system calls. It costs nothing and can save you from potential embarassment at some laster time. It something has an error return you _must_ check for it. It's not a matter of `working around ... bugs'. It's all about good programming style. It's incredibly useful to have good diagnostics for even the worst end case. Take NFS for example. When it breaks I'd like to know which system call failed and why. That way you _know_ what's going on, or at least you have a handle on it. There's nothing worse than silence when something goes weird on you. One horror story springs to mind with setuid(2) and setgid(2). This (large) piece of code started off running as root and then went setgid() followed by setuid() to become right `user'. Did it check what setuid() and setgid() returned? How many of you check what those two return? Anyone? Anyway, the setuid() and setgid() both FAILED which caused the software to fail in strange ways. On a `real' UNIX machine this wouldn't be a problem. But, welcome to SCO and their C2 security nightmare. Because this stuff was invoked by init it had no `login user id'. The documentation for set[ug]id(2) states that both calls would fail if the `login user id' was not set. This thing can be set once and _only once_. It's done by login(1) and su(1). Goodbye UNIX semantics -- hello Mom*. Anyway, if the programmer of this piece of code (not me) had checked what set[gu]id() returned it would have been far easier to track down. At least, I would have had a pointer to the real problem rather than having to trek through a large software package with only a vague idea of what the problem was. Boyd Roberts boyd@necisa.ho.necisa.oz.au ``When the going gets wierd, the weird turn pro...'' * With apologies to Scatterbrain.
jfh@rpp386.cactus.org (John F. Haugh II) (10/23/90)
In article <35111@cup.portal.com> ts@cup.portal.com (Tim W Smith) writes: >This sounds like very bad file system design. If you are over some quota, >the write() that caused you to go over the quota should return an error. >The local system should determine your quota and remaining space and >check this on each write. you have just required that all write() operations be performed synchronously. there are open flags defined to support synchronous i/o - use them, or tell your vendor to include them in your version of the o/s, but don't hobble applications that aren't as serious about their output as yours. -- John F. Haugh II UUCP: ...!cs.utexas.edu!rpp386!jfh Ma Bell: (512) 832-8832 Domain: jfh@rpp386.cactus.org "SCCS, the source motel! Programs check in and never check out!" -- Ken Thompson
brnstnd@kramden.acf.nyu.edu (Dan Bernstein) (10/24/90)
In article <1885@necisa.ho.necisa.oz> boyd@necisa.ho.necisa.oz (Boyd Roberts) writes: > Always, always, always check the return values of system calls. Of course. I check for every error I know how to handle. I put /*XXX*/ when there may be errors that I can't handle. > It something has an error return you > _must_ check for it. And if something is not documented as returning error X, and there's no logical reason to expect it to, and there's no good way to handle the error if it does come up? > Did it check what setuid() and setgid() returned? How many of > you check what those two return? Anyone? I do. See the pty source. You were saying? ---Dan
boyd@necisa.ho.necisa.oz (Boyd Roberts) (10/25/90)
In article <5238:Oct2322:14:3690@kramden.acf.nyu.edu> brnstnd@kramden.acf.nyu.edu (Dan Bernstein) writes: > >And if something is not documented as returning error X, and there's no >logical reason to expect it to, and there's no good way to handle the >error if it does come up? > Dan, you twist and turn -- like a twisty turny thing. A decent interface returns ok or error. Now there may be multiple error classes, but all you want to distinguish is success or failure. Failure is usually disaster, but in some cases you can act on the error type and try to recover. What are you trying to say? A new type of error has occured and because you aren't prepared for it you say ``there's no good way to handle [it]''. Are you saying that you treat it as success or do you just ignore it? Surely you mean that if you don't know how to handle a specific recovery (for some unspecified error type) it's still treated as failure? Boyd Roberts boyd@necisa.ho.necisa.oz.au ``When the going gets wierd, the weird turn pro...''
brnstnd@kramden.acf.nyu.edu (Dan Bernstein) (10/26/90)
In article <1893@necisa.ho.necisa.oz> boyd@necisa.ho.necisa.oz (Boyd Roberts) writes: > In article <5238:Oct2322:14:3690@kramden.acf.nyu.edu> brnstnd@kramden.acf.nyu.edu (Dan Bernstein) writes: > >And if something is not documented as returning error X, and there's no > >logical reason to expect it to, and there's no good way to handle the > >error if it does come up? > Dan, you twist and turn -- like a twisty turny thing. You're the one trying to change the problem by generalizing it too far. See below. > A decent interface returns ok or error. Now there may be multiple error > classes, but all you want to distinguish is success or failure. Failure > is usually disaster, but in some cases you can act on the error type and > try to recover. This generalization isn't true. See below. > What are you trying to say? A new type of error has occured and because > you aren't prepared for it you say ``there's no good way to handle [it]''. > Are you saying that you treat it as success or do you just ignore it? In the *particular* case of close(), I just ignore errors that I don't know how to handle. After all, I use close() merely to indicate to the system that I'm done with a descriptor. (That is what it's meant for--- right?) If close() gives EINTR, it might not have completed. This is just like any other system call. I know a good way to handle this: simply retry the close(). The system is telling me ``whoops, I know you were giving me an extra assertion to work with, but I was distracted---say it again?'' If close() gives EBADF, my program (which I assume keeps proper track of descriptors) has made an incorrect assumption about its external environment. It might have been passed a closed stdin, for instance. In most cases the right way to handle this error is to do nothing and return success. Some programs might want to report the problem back to the user, but there's no behavior that becomes undefined if they don't. There is no reason that I have to handle either error. My goal is only to give the system as much information as possible, so that it can pass timely EOFs down pipes, free up the resources for the descriptor so that I'll have more chances to open() later, etc. Now let's take an error matching my quoted description. close() is not documented as returning EDQUOT. There's no logical reason to expect it to return EDQUOT---even if write()'s buffering weren't hidden below its interface, it could easily return the error accurately. There's no good way to handle EDQUOT if it comes up---even if I had some reason to suspect that close() could return the error, I simply *cannot* replay data that might have been buffered in the same file by a previous process. Tell me: Why should I handle EDQUOT? Why should I interpret it as some sort of error? Who benefits if I thrash about upon this error? > Surely you mean that if you don't know how to handle a specific recovery > (for some unspecified error type) it's still treated as failure? Not at all. This might be true for calls that give me information, but close() is not such a call. Do you check the return value of assert()? ---Dan
jik@athena.mit.edu (Jonathan I. Kamens) (10/26/90)
In article <8645:Oct2521:49:5790@kramden.acf.nyu.edu>, brnstnd@kramden.acf.nyu.edu (Dan Bernstein) writes: |> Now let's take an error matching my quoted description. close() is not |> documented as returning EDQUOT. "The close(2) man page (at least on my system) has not been updated to reflect the fact that close() can return EDQUOT. Just like it doesn't say (at least on my system) that close can return EINTR. I have to check for EINTR, even though it isn't documented, but I don't have to check for EDQUOT." Very logical. |> There's no logical reason to expect it |> to return EDQUOT "In my opinion, there is no logical reason to expect it to return EDQUOT." That's wonderful, but several filesystem implementors and kernel wprogrammers have disagreed with you. Some of the code written by those implementors and programmers are in wide use all over the Unix world. So, your program can either accept the decision they've made, and be more robust, or ignore it, and work less robustly and less reliably. |> ---even if write()'s buffering weren't hidden below its |> interface, it could easily return the error accurately. "In my opinion, it should be possible to implement any filesystem, local or remote, so that the kernel can always return quota errors to a user process immediately upon write(), and not delay them until close()." You suggested one possible way to do this in another posting, Dan, and I asked you to expand on it, since I didn't understand what you were saying. I have seen no response to my request. Why not? In any case, I will repeat what I've said already -- Dan, in *my* opinion, it is *not* necessarily possible to implement any filesystem reasonably and guarantee immediate quota reports. As far as I can tell, there are several possible ways to do this: 1. Each time a remote filesystem/device is accessed for the first time, the kernel has to get and store the user's quota on that device. Each time *any* filesystem operation is performed on that device from that point forward, the kernel has to adjust the quota value appropriately. This means write()s, seek()s, unlink()s, mkdir()s, link()s, symlink()s, mknod(), and anything else I forgot. 2. All filesystem calls must be completely synchronous, and must be completed by the kernel before control returns to the user process. This way, the kernel can get quota errors from the fileserver immediately. 3. The kernel has to "reserve" space on the remote filesystem before performing any file operations, and "ask" for more space whenever it uses up the space it's asked for. Now, let's analyze why I think that none of these approaches is reasonable: 1. First of all, there are many calls that affect the filesystem for which the kernel *does not need to know* the effect on the filesystem in terms of space. By requiring the kernel to keep track of quotas, you are requiring that it understand the lowest level details of the remote filesystem, so that (for example) it knows that adding a file to a directory will take away xx bytes from the user's quota. All levels of abstraction and protocol layers vanish when the local kernel has to be all-knowing about the remote filesystem. Of course, you could require that the remote filesystem *tell* the client kernel how much space was allocated/released by each filesystem operation. But that's just a degeneration into case (2), which I will deal with below. The other problem is that remote filesystems can have more than one kernel operating on them at a time. Kernel A can't know if Kernel B is also operating on the filesystem, with the net result being complete quota chaos if, for example, I'm compiling a program for two different platforms on two different machines in the same remote filesystem. You could solve this problem by requiring the remote filesystem to notify all current clients of the space allocation/freeing whenever any client performs any filesystem operation. Gee, now there's a model of efficiency for you. Now the fileserver has to keep a current list of all clients who are using each filesystem on it (if that's even possible), and has to communicate with them whenever anybody does a filesystem operation. 2. One of the greatest features of AFS, for example, is that files don't actually go out over the network until you're done modifying them (unless you do a seek(), I believe). Requiring completely synchronous operation would eliminate much of the speed advantage of using AFS. I like AFS. In my opinion, it's a great filesystem. I'm willing to pay the cost of AFS's great performance by checking for EDQUOT after close() in my programs. 3. Have you ever rented a car? You know how they put a really big almost-charge (I forget what the word they use for it is) on they credit card in case you do something nasty? Did you know that if that puts you over your limit, you won't be able to use the card for anything else until you return the car, and possibly not for several days after that, since the car rental company may not bother to clear the charge until it expires automatically. Would you like the same thing to be true with filesystems? I think I may need a meg in the filesystem, so I ask for it. But in reality, the user process for which I asked for that meg is only going to write a 1 meg file. That 1 meg puts me close to my quota. Now, no other kernel anywhere can write to my filesystem until the user process finishes. If I crash, how does the remote fileserver find out that the pre-allocation is no longer needed? |> There's no good |> way to handle EDQUOT if it comes up "In my opinion, there's no good way to handle EDQUOT if it comes up." As I have already pointed out, Dan, almost all of our filesystem access at Athena is on filesystems that have the "problem" we are discussing. And yet we have very little, if any, problem with programs that can't "handle EDQUOT" in a "good way". You have asserted, as fact, that there is no good way, and yet I can give you any number of examples of programs that have found a good way. Therefore, your "fact" must not be a "fact". Now, it is true that *for some programs*, there will not be any good way to deal with EDQUOT from close. Just as, for example, /bin/login can't deal with the setuid() failing, so it gives up. We have that problem because Ultrix doesn't deal with user ID's as high as BSD 4.3 does, so some of our users can't log in on our Ultrix workstations. The worst thing a program can do is say, "Woops, I just noticed an error," and abort. |> ---even if I had some reason to |> suspect that close() could return the error, I guess that the fact that two of the most popular remote filesystems in use today allow close() to return the error is not "some reason to suspect" that it will happen? You're living in a fantasy world, Dan. You don't want close() to return EDQUOT. Well, it does. Either your programs deal with that fact, or they fail inexplicably. |> I simply *cannot* replay |> data that might have been buffered in the same file by a previous |> process. This is a straw man, Dan. No one is claiming that every program can recover from a failing close(). It's a fact of life that programs sometimes encounter error conditions from which they cannot recover. If you cannot recover, you give up, after telling the user that you cannot recover. If you *can* recover, on the other hand, you do. It's that simple. |> Tell me: Why should I handle EDQUOT? Why should I interpret it |> as some sort of error? Who benefits if I thrash about upon this error? You should handle it because your program can be more robust if you handle it. You should handle it because if you don't, your programs will fail and the user will never find out about it. You should handle it because it's there, and because until it's *not* there, your customers are going to complain about your failure to handle it. If you want to go on a crusade against close() failing with EDQUOT, feel free to do so. But I submit that until your crusade is successful, you are nothing but stupid to say, "Since it shouldn't fail with EDQUOT, I'm not going to check to see if it fails with EDQUOT." And I'd say that emacs users benefit incredibly from the fact that emacs "thrash[es] about upon this error." Because they don't lose files that they would otherwise lose without ever being told that they were lost. |> Not at all. This might be true for calls that give me information, but |> close() is not such a call. As I have already illustrated at length in other messages, this assertion is no longer true. In modern, 1990 Unix, close() does give the programmer information. As I've already said, if you choose to ignore that information because you believe religiously that it shouldn't exist, then your code is less robust than it can be. |> Do you check the return value of assert()? This is just stupid, and it's just like your argument about setuid() returning ENOENT. Are you claiming that there is just ONE version of Unix, ANYWHERE, where it is even remotely possible that assert() will return a useful value? I don't know of any. I doubt there are any. Assert() is not documented as *ever* returning an error indication. On the other hand, close() *is* documented as returning an error indication, and furthermore, there are already many versions of Unix for which close() can fail with EDQUOT. They exist. They are being used. And you have to cope with them. -- Jonathan Kamens USnail: MIT Project Athena 11 Ashford Terrace jik@Athena.MIT.EDU Allston, MA 02134 Office: 617-253-8085 Home: 617-782-0710
brnstnd@kramden.acf.nyu.edu (Dan Bernstein) (10/26/90)
In article <18619@rpp386.cactus.org> jfh@rpp386.cactus.org (John F. Haugh II) writes: > In article <35111@cup.portal.com> ts@cup.portal.com (Tim W Smith) writes: > >This sounds like very bad file system design. If you are over some quota, > >the write() that caused you to go over the quota should return an error. > >The local system should determine your quota and remaining space and > >check this on each write. > you have just required that all write() operations be performed > synchronously. Not at all. It doesn't even require that the server hear about the operation, because clients can be given ``windows'' upon the disk space. ``Here, you can assume you have at least 5M until I tell you otherwise.'' > there are open flags defined to support synchronous > i/o - use them, or tell your vendor to include them in your version > of the o/s, but don't hobble applications that aren't as serious > about their output as yours. What several participants in this discussion seem to be missing is that EDQUOT is *not* as disastrous as EIO and should *not* require such drastic measures as fsync()/O_SYNC. If the CPU and disks are perfectly reliable, then why should such synchronization be required? To put it differently, why can't an application be reliable under quota errors without going to all the real work necessary for general reliability? EDQUOT is a relatively ``normal'' condition compared to EIO. ---Dan
buck@siswat.UUCP (A. Lester Buck) (10/26/90)
In article <8645:Oct2521:49:5790@kramden.acf.nyu.edu>, brnstnd@kramden.acf.nyu.edu (Dan Bernstein) writes: > In the *particular* case of close(), I just ignore errors that I don't > know how to handle. After all, I use close() merely to indicate to the > system that I'm done with a descriptor. (That is what it's meant for--- > right?) No, it is also meant for the sytem to tell you that your file entered a stable state, which is a much more important use now that processes can have hundreds of files open. > There's no good > way to handle EDQUOT if it comes up---even if I had some reason to > suspect that close() could return the error, I simply *cannot* replay > data that might have been buffered in the same file by a previous > process. Tell me: Why should I handle EDQUOT? Why should I interpret it > as some sort of error? Who benefits if I thrash about upon this error? > ---Dan Your data is probably quite safe in the filesystem, but the quota machinery just won't let it be closed. I can think of several ways to handle an EDQUOT error on close, depending on how much work has just gone into computing the contents of the file. For an interactive user, we back off and fire up a shell prompt with the message "Over quota: delete other files, abort, retry? :-)" with appropriate machinery to allow the user to make an intelligent decision. For a non-interactive user who just spent $1000 of compute time on this file, I might suggest running a .overquotarc file in their root directory, or deleting the largest, oldest files under the assumption that they can be recovered from system backups, or send mail to the user, wait for a human to come by and mount a tape, then lseek to the beginning of the file and write it to tape. The idea that there is "no good way to handle EDQUOT" is bogus. I am not proposing that every application offer this functionality, only that some important applications might need, and should certainly be given the opportunity, to recover from an error on close. -- A. Lester Buck buck@siswat.lonestar.org ...!uhnix1!lobster!siswat!buck
brnstnd@kramden.acf.nyu.edu (Dan Bernstein) (10/26/90)
In article <1990Oct26.005843.12463@athena.mit.edu> jik@athena.mit.edu (Jonathan I. Kamens) writes: > In article <8645:Oct2521:49:5790@kramden.acf.nyu.edu>, brnstnd@kramden.acf.nyu.edu (Dan Bernstein) writes: > |> Now let's take an error matching my quoted description. close() is not > |> documented as returning EDQUOT. > "The close(2) man page (at least on my system) has not been updated to reflect > the fact that close() can return EDQUOT. Just like it doesn't say (at least > on my system) that close can return EINTR. I have to check for EINTR, even > though it isn't documented, but I don't have to check for EDQUOT." Are you trying to restate what I said? Okay, here's your mistakes. Sentence 2: My man page does say that close() can return EINTR. Sentence 3: I have *never* said that I have to check for EINTR. It's reasonable to expect a programmer to check for EINTR, because there's a good handling method: to restart the system call. But this is not something I have to do, particularly in programs where close() cannot be interrupted by signal handlers. In other words, I do *not* have to check for *any* error. This applies to EINTR and EDQUOT. The difference between EINTR and EDQUOT is that there's a sensible way to handle the first. As I explain in a previous article, there is no correct way to handle the second, short of requiring every single program to fsync() as if EDQUOT were EIO. > |> There's no logical reason to expect it > |> to return EDQUOT > "In my opinion, there is no logical reason to expect it to return EDQUOT." What is this with you and your wimpy presentation style? Of course it's my opinion; do you think I'd say something if I had a different opinion? > So, your program can > either accept the decision they've made, and be more robust, Sorry, bud. You're wrong. > |> ---even if write()'s buffering weren't hidden below its > |> interface, it could easily return the error accurately. > "In my opinion, it should be possible to implement any filesystem, local or > remote, so that the kernel can always return quota errors to a user > process immediately upon write(), and not delay them until close()." This is true for every currently operating filesystem I know about. > You suggested one possible way to do this in another posting, Dan, and I asked > you to expand on it, since I didn't understand what you were saying. I have > seen no response to my request. Why not? Sorry, I've been saving up a few articles to respond to in a batch. I thought the analogy to TCP windowing was reasonably clear. > 3. The kernel has to "reserve" space on the remote filesystem before > performing any file operations, and "ask" for more space whenever it uses up > the space it's asked for. Yep. > 3. Have you ever rented a car? You know how they put a really big > almost-charge (I forget what the word they use for it is) on they credit card [ continues the interesting analogy ] Are you assuming that network filesystem programmers aren't intelligent enough to fix these failures, perhaps along the lines I explain in a previoius article? > You have asserted, as fact, that there is no good way, and > yet I can give you any number of examples of programs that have found a good > way. I have asserted, as opinion, that there is no good way. In a previous article I challenge you to demonstrate that good way in an important class of cases where I'm pretty sure it's impossible. I agree that it may be possible to handle EDQUOT on close() properly in the special case that you've opened the file you're accessing, and you know that no other process is writing to the same system buffer. > You're living in a fantasy world, Dan. You don't want close() to return > EDQUOT. Well, it does. Either your programs deal with that fact, or they > fail inexplicably. Naw. Take pty, for example. It will not ``fail inexplicably'' because a close() fails with EDQUOT. It'll just have one fewer descriptor to play with. As it isn't descriptor-happy like some programs (e.g., more), this simply *cannot* cause a failure---pty only closes a few (I think at most four) descriptors it gets from the outside, and them only upon exiting. Where's the problem? > |> Do you check the return value of assert()? > This is just stupid, and it's just like your argument about setuid() returning > ENOENT. As both this and the ENOENT argument were satires of your continual failure to present rational justifications for your statements, I agree entirely. ---Dan
ske@pkmab.se (Kristoffer Eriksson) (10/26/90)
In article <8645:Oct2521:49:5790@kramden.acf.nyu.edu> brnstnd@kramden.acf.nyu.edu (Dan Bernstein) writes: > Tell me: Why should I handle EDQUOT? Why should I interpret it >as some sort of error? Who benefits if I thrash about upon this error? As the user of a program I expect it to report ALL unexpected errors, no matter whether the program knows how to recover from them or not. *I* might be able to correct the error situation, and most importantly, I absolutely demand to be informed if my data possibly did not make it to the disk. That the program doesn't know how to recover from the error is no excuse for not telling me about it. After all, there's no problem detecting that there was some kind of error, the only problem occurs if you absolutely must have the program automatically recover from the error without involving the user. -- Kristoffer Eriksson, Peridot Konsult AB, Hagagatan 6, S-703 40 Oerebro, Sweden Phone: +46 19-13 03 60 ! e-mail: ske@pkmab.se Fax: +46 19-11 51 03 ! or ...!{uunet,mcsun}!sunic.sunet.se!kullmar!pkmab!ske
jfh@rpp386.cactus.org (John F. Haugh II) (10/26/90)
In article <12402:Oct2605:19:2790@kramden.acf.nyu.edu> brnstnd@kramden.acf.nyu.edu (Dan Bernstein) writes: >> you have just required that all write() operations be performed >> synchronously. > >Not at all. It doesn't even require that the server hear about the >operation, because clients can be given ``windows'' upon the disk space. >``Here, you can assume you have at least 5M until I tell you >otherwise.'' bzzt. wrong answer. process A writes some stuff and still has 5MB. process B writes some stuff and still has 5MB. process A goes crazy, writes a ton of data and uses up everything it has, plus the 5MB. now, do you extend the quota of process B by another 5MB? how does the server distinguish between process A and process B when both exist over in the client end of the world? you have also just added state - process A can't write and process B can. now generalize it to a person who wants to grossly exceed their quota, creates fifty process and starts writing. process number 1 .. 50 respectively reach their quotas and get to write 5MB for free. increase "50" until you see with my point. >What several participants in this discussion seem to be missing is that >EDQUOT is *not* as disastrous as EIO and should *not* require such >drastic measures as fsync()/O_SYNC. If the CPU and disks are perfectly >reliable, then why should such synchronization be required? To put it >differently, why can't an application be reliable under quota errors >without going to all the real work necessary for general reliability? >EDQUOT is a relatively ``normal'' condition compared to EIO. the administration of the machine needs to answer that question. given that most programs do not check for errors on write system calls (try this: "$ (ulimit 0; passwd joe)" on a System V machine. actually, don't - it trashes your password file.), why would someone want to intentionally increase the number of write system call failures??? the answer from a programmers point of view should be that if the administration is stupid enough to put their trust in quotas, they deserve the grief their users give them. this may not be a nice answer, but i like it ;-) this is an administrative problem that can't be solved by programmers. -- John F. Haugh II UUCP: ...!cs.utexas.edu!rpp386!jfh Ma Bell: (512) 832-8832 Domain: jfh@rpp386.cactus.org "SCCS, the source motel! Programs check in and never check out!" -- Ken Thompson
lwall@jpl-devvax.JPL.NASA.GOV (Larry Wall) (10/28/90)
In article <1246@prcrs.UUCP> paul@prcrs.UUCP (Paul Hite) writes:
: Paul Hite PRC Realty Systems McLean,Va uunet!prcrs!paul (703) 556-2243
: You can't tell which way the train went by studying its tracks.
Can so. You just check to see which way the bugs are squished.
(I guess I shouldn't have stayed up to watch the time change. At least
it belongs in this newsgroup, being about bugs and internals...)
Larryboyd@necisa.ho.necisa.oz (Boyd Roberts) (11/05/90)
In article <8645:Oct2521:49:5790@kramden.acf.nyu.edu> brnstnd@kramden.acf.nyu.edu (Dan Bernstein) writes: >... Tell me: Why should I handle EDQUOT? Why should I interpret it >as some sort of error? Who benefits if I thrash about upon this error? > It is an error because close() returned -1. I don't care what errno is set to. The close() still failed. What do you believe? errno, or the system call returning -1? I believe the system call. If I don't understand errno I use a perror(3) equivalent to complain about it. You seem to say that you know better than the system call. How can you? Maybe the system call is broken? Maybe your code is broken? Maybe the universe is broken? But you _have_ to believe the system call return. You have no other logical choice. > >Not at all. This might be true for calls that give me information, but >close() is not such a call. Do you check the return value of assert()? > I _know_ the return value of the assert. All is ok when my assert returns. My code, or the condition in the assert is broken. And, no I haven't been stupid enough to defeat assert and have it return on failure. Sure, the assert implementation could be broken (hell, some bunch of fools even broke cat(1)!). Initially I'll believe that I'm wrong and that assert is right. Once I'm convinced otherwise it's time to fill out the bug report, _AND NOT ONE SECOND BEFORE_!! Dan, I reckon you just post for the sake of it. You like playing Devil's Bozo. Boyd Roberts boyd@necisa.ho.necisa.oz.au ``When the going gets wierd, the weird turn pro...''
brnstnd@kramden.acf.nyu.edu (Dan Bernstein) (11/06/90)
In article <1914@necisa.ho.necisa.oz> boyd@necisa.ho.necisa.oz (Boyd Roberts) writes: > It is an error because close() returned -1. Are you paying attention? It is *not* a failure for my program to ignore a close() error; it performs properly, complies with all the standards, etc. My program does not receive any information from close() that matters for the correctness of its operation. Your repeated assertions like > You have no other logical choice. are absolutely ridiculous. Suppose I admit that I'm supposed to report an error. In this case, the program does not have any connection with the user. It would be inappropriate to use syslog because the program cannot afford to block and syslog is unreliable. As I asked before, what do I gain by thrashing about if close() returns EDQUOT? The situation would be different if I opened and closed lots of files. Then the operation of my program might be jeapordized by a failing close(). But that isn't the case here. The situation might be different if I had opened and written to the file I'm closing. If I know nobody else has written to the file, I can reasonably recover from an EDQUOT error upon close(). But that isn't the case here. You say that you *must* inform the user of an error. Have you ever seen the quota system in action? Did you notice that the kernel blares some obnoxious messages at the user to indicate quota-related write errors? I think you just post for the hell of it, Boyd. You like playing Devil's Fool. ---Dan