chased@rbbb.Eng.Sun.COM (David Chase) (01/26/91)
brnstnd@kramden.acf.nyu.edu (Dan Bernstein) writes: >Now you can talk all you want about reallocating memory (btw, there's no >safe way to use realloc(), but you knew that) to read in as many users >as possible. I'll skip the comments about a quadratic time >requirement... I'll bite -- are you saying that reallocating memory to obtain a larger array and copying old into new always takes ("requires") quadratic time? (Quadratic time in what? I assume number of array elements.) I always double the size when I do this, by the way. 1 + 2 + 4 + 8 + ... + 2^N = 2 * (2^N) - 1, which is linear in 2^N, which is the number of array elements. But I'm sure you knew that, so you must have been talking about something else. What were you talking about? Inquiring minds want to know. David Chase Sun Microsystems
brnstnd@kramden.acf.nyu.edu (Dan Bernstein) (01/29/91)
In article <6662@exodus.Eng.Sun.COM> chased@rbbb.Eng.Sun.COM (David Chase) writes: > I always double the size when I do this, by the way. Then you are using poor programming techniques. No program should take up half the memory of your machine when it could take up a fourth with hardly any effort on the programmer's part. Any single-pass ``users'' has to either use quadratic time or be a rude abuser of resources. Why not use two passes and be done with it? ---Dan
chased@rbbb.Eng.Sun.COM (David Chase) (01/30/91)
brnstnd@kramden.acf.nyu.edu (Dan Bernstein) writes: >Any single-pass ``users'' has to either use quadratic time or be a rude >abuser of resources. Why not use two passes and be done with it? No reason not to do that, and that's not what I was arguing (hard to imagine rudely abusing resources on a machine large enough to support enough users that the users command would rudely abuse resources, but so it goes). Maybe this is a sensible tradeoff on the machines that you use. However, you said that quadratic time was "required". No "or abuse resources" to qualify that statement. I'm interested in knowing how you arrived at this conclusion, since, in fact, it is false. I figure you'll find some way to redefine your way out of this blunder; just wanted to be sure that you did in it in public so that all could see that you aren't the infallible "mathematician" that you claim you are. NB -- doubling yields peak waste of 66% (N bytes in old, 2N in new, copy not yet made, only N useful bytes), not 50%. See? You didn't even get that right. Be more careful when you post, ok? Takes people's time to read these things. Yours, David Chase Sun Microsystems (and member of the Dan Bernstein USENET fan-the-flames club)
jfh@rpp386.cactus.org (John F Haugh II) (01/30/91)
In article <15325:Jan2903:19:4991@kramden.acf.nyu.edu> brnstnd@kramden.acf.nyu.edu (Dan Bernstein) writes: >Any single-pass ``users'' has to either use quadratic time or be a rude >abuser of resources. Why not use two passes and be done with it? No, it doesn't. It can get it exactly right the first time and be done with it. fd = open ("/etc/utmp", O_RDONLY); fstat (fd, &statb); users = (struct utmp *) malloc (statb.st_size); read (fd, (char *) users, statb.st_size); This has certain problems, all of which Dan will point out, but any non-atomic operation has other "problems". I would argue that this has "less" problems because it is "more" atomic and far less complex. Besides, this is my posting, and quite clearly I am right about everything I post [ ;-)'s for the humor impaired only ] As for doubling realloc, given the large number of demand paged systems these days, doubling realloc often is a big win because the wasted pages are never touched, so they neither take physical memory nor backing store. Doubling realloc avoids the silly practice of gradually incrementing your memory up to the appropriate size. That is a sure performance sucker if ever there was one. -- John F. Haugh II UUCP: ...!cs.utexas.edu!rpp386!jfh Ma Bell: (512) 832-8832 Domain: jfh@rpp386.cactus.org "13 of 17 valedictorians in Boston High Schools last spring were immigrants or children of immigrants" -- US News and World Report, May 15, 1990
rang@cs.wisc.edu (Anton Rang) (01/30/91)
In article <15325:Jan2903:19:4991@kramden.acf.nyu.edu> brnstnd@kramden.acf.nyu.edu (Dan Bernstein) writes: >Any single-pass ``users'' has to either use quadratic time or be a rude >abuser of resources. Why not use two passes and be done with it? Umm, are linked lists in disfavor in this group, or am I missing something obvious here? Anton +---------------------------+------------------+-------------+ | Anton Rang (grad student) | rang@cs.wisc.edu | UW--Madison | +---------------------------+------------------+-------------+
jef@well.sf.ca.us (Jef Poskanzer) (01/30/91)
In the referenced message, brnstnd@kramden.acf.nyu.edu (Dan Bernstein) wrote: }Any single-pass ``users'' has to either use quadratic time or be a rude }abuser of resources. Why not use two passes and be done with it? Well, no. There's at least one way of figuring out how much space to malloc without having to read utmp an extra time. --- Jef Jef Poskanzer jef@well.sf.ca.us {apple, ucbvax, hplabs}!well!jef "If you don't use Saber to develop your next C program, you're a dork." -- Brian Reid
oz@yunexus.yorku.ca (Ozan Yigit) (01/31/91)
In article <15325:Jan2903:19:4991@kramden.acf.nyu.edu> brnstnd@kramden.acf.nyu.edu (Dan Bernstein) writes: >Any single-pass ``users'' has to either use quadratic time or be a rude >abuser of resources. I think *unnecessary* multiple passes on disk files could best be described as "a rude abuse of resource". >Why not use two passes and be done with it? To rephrase: ... you are using poor programming techniques. No program should make multiple passes over a disk file when it could accomplish its task with a single pass with hardly any effort on the programmer's part. oz
brnstnd@kramden.acf.nyu.edu (Dan Bernstein) (02/01/91)
In article <21548@yunexus.YorkU.CA> oz@yunexus.yorku.ca (Ozan Yigit) writes: > To rephrase: ... you are using poor programming techniques. No program > should make multiple passes over a disk file when it could accomplish its > task with a single pass with hardly any effort on the programmer's part. If you spent a minute putting your priorities into order you'd realize that this is an entirely ridiculous generalization. If two programs are just as fast, just as small, and take just as little programmer (and user) effort, then it simply doesn't matter whether one of them uses more passes than the other. To rephrase: You don't seem to know what good programming techniques are. Tradeoffs between speed, space, and human effort are important. Tradeoffs between ``multiple passes'' and ``single pass'' are entirely irrelevant when they aren't reflected in speed, space, or human effort. ---Dan
dik@cwi.nl (Dik T. Winter) (02/02/91)
In article <5883:Feb102:05:4991@kramden.acf.nyu.edu> brnstnd@kramden.acf.nyu.edu (Dan Bernstein) writes: > In article <21548@yunexus.YorkU.CA> oz@yunexus.yorku.ca (Ozan Yigit) writes: > > To rephrase: ... you are using poor programming techniques. No program > > should make multiple passes over a disk file when it could accomplish its > > task with a single pass with hardly any effort on the programmer's part. > > If you spent a minute putting your priorities into order you'd realize > that this is an entirely ridiculous generalization. If two programs are > just as fast, just as small, and take just as little programmer (and > user) effort, then it simply doesn't matter whether one of them uses > more passes than the other. > Ah, but that is the problem. Just as fast comes in flavors. You have CPU time, real time, IO time and some more. If one of the programs uses more passes it is not equally fast IO time wise (unless you are using SSD on a Cray of course). (Moreover, a program doing IO might well block another program that wants to do IO.) -- dik t. winter, cwi, amsterdam, nederland dik@cwi.nl
geoff@zoo.toronto.edu (Geoffrey Collyer) (02/02/91)
Dan Bernstein, in response to Ozan Yigit: > To rephrase: You don't seem to know what good programming techniques are. From pty/master.c: (void) lseek(fdsess,(long) 0,0); (void) read(fdsess,(char *) &newuid,sizeof(int)); (void) sprintf(newsuid,"%d",newuid); (void) chdir(".."); if (chdir(newsuid) == -1) { (void) mkdir(newsuid,0700); (void) chdir(newsuid); } (void) sprintf(foo,"../%d/%s",uid,fnsess); (void) rename(foo,fnsess); (void) sprintf(foo,"../%d/%s",uid,fnre); (void) rename(foo,fnre); /* in case we're already disconnected */ -- Geoff Collyer utzoo!geoff, zoo.toronto.edu!geoff
brnstnd@kramden.acf.nyu.edu (Dan Bernstein) (02/02/91)
In article <1991Feb2.045119.22199@zoo.toronto.edu> geoff@zoo.toronto.edu (Geoffrey Collyer) writes: > Dan Bernstein, in response to Ozan Yigit: > > To rephrase: You don't seem to know what good programming techniques are. > From pty/master.c: Yes, what's your point? I have several hundred notes on pieces of pty that could be improved. In the list I find ``check for phys I/O errors on reading fdsess''; that's the only problem with the section of code you quoted. Are you worried about all the ignored return codes? I'll go through them one by one. Sure, I could've programmed more defensively, but the only possible response to a fatal error would be cutting off the user program from its input and output. > (void) lseek(fdsess,(long) 0,0); This is guaranteed to work. The third 0 could be changed to some symbolic constant (like L_SET), but as you know 0 is more portable. > (void) read(fdsess,(char *) &newuid,sizeof(int)); This is guaranteed to work, because fdsess is a regular file that was created with at least the int. The only possible problem is a hard I/O error or other fatal system error. > (void) sprintf(newsuid,"%d",newuid); This is guaranteed to work; newsuid has enough space. > (void) chdir(".."); This is guaranteed to work. > if (chdir(newsuid) == -1) > { > (void) mkdir(newsuid,0700); This cannot fail unless some renegade sysadmin changes the mode of the session directory while pty is running. 0700 could be made a symbolic constant too, but see above. > (void) chdir(newsuid); > } Cannot fail. > (void) sprintf(foo,"../%d/%s",uid,fnsess); > (void) rename(foo,fnsess); Guaranteed to work, as above. > (void) sprintf(foo,"../%d/%s",uid,fnre); > (void) rename(foo,fnre); /* in case we're already disconnected */ Ditto. ---Dan
brnstnd@kramden.acf.nyu.edu (Dan Bernstein) (02/02/91)
In article <2886@charon.cwi.nl> dik@cwi.nl (Dik T. Winter) writes: > Ah, but that is the problem. Just as fast comes in flavors. You have CPU > time, real time, IO time and some more. If one of the programs uses more > passes it is not equally fast IO time wise (unless you are using SSD on > a Cray of course). (Moreover, a program doing IO might well block another > program that wants to do IO.) See, you're proving my point. You don't really care about the number of passes; you care about speed (including any effect on the speed of other programs). It's simply not true that any program using more passes must use more time; that's only a rule of thumb. Suppose you're given two programs, one of which uses fewer passes but is twice as slow. Which do you really care about: the number of passes, or the speed? ---Dan
jfh@rpp386.cactus.org (John F Haugh II) (02/02/91)
In article <1991Feb2.045119.22199@zoo.toronto.edu> geoff@zoo.toronto.edu (Geoffrey Collyer) writes: >Dan Bernstein, in response to Ozan Yigit: >> To rephrase: You don't seem to know what good programming techniques are. > >From pty/master.c: I'm sure it was very hard to write. Therefore, it should be hard to read, and hard to maintain, thus promoting job security. What exactly does it do? -- John F. Haugh II UUCP: ...!cs.utexas.edu!rpp386!jfh Ma Bell: (512) 832-8832 Domain: jfh@rpp386.cactus.org "I've never written a device driver, but I have written a device driver manual" -- Robert Hartman, IDE Corp.
dhesi%cirrusl@oliveb.ATC.olivetti.com (Rahul Dhesi) (02/03/91)
In <5883:Feb102:05:4991@kramden.acf.nyu.edu> brnstnd@kramden.acf.nyu.edu (Dan Bernstein) writes: >Tradeoffs between ``multiple passes'' and ``single pass'' are entirely >irrelevant when they aren't reflected in speed, space, or human effort. It's often easier to adapt a single-pass program to handle previously-unforeseen needs than to similarly adapt a multipass program. A good example is what happens if you decide later that you would like to handle standard input. Now if you have a multipass implementation, you either have to completely redo it, or use a possibly huge temp file, or do buffering in blocks which may involve a heavy penalty in speed and memory usage. -- Rahul Dhesi <dhesi%cirrusl@oliveb.ATC.olivetti.com> UUCP: oliveb!cirrusl!dhesi
das@eplunix.UUCP (David Steffens) (02/03/91)
In article <15151:Feb207:16:0691@kramden.acf.nyu.edu>, brnstnd@kramden.acf.nyu.edu (Dan Bernstein) says: > ... It's simply not true that any program using more passes must > use more time; that's only a rule of thumb... Eh? Your statement is only applicable when comparing a one-pass program that uses algorithm XYZ with a two-pass program that uses algorithm ABC, where the speed of ABC compensates for the second pass over the file. A program that reads a file ONCE is ALWAYS faster than a program that uses virtually the SAME algorithm AND reads the file TWICE. And that was precisely the point of several earlier articles, namely: <19010@rpp386.cactus.org> jfh@rpp386.cactus.org (John F Haugh II) <22937@well.sf.ca.us> jef@well.sf.ca.us (Jef Poskanzer) <21548@yunexus.YorkU.CA> oz@yunexus.yorku.ca (Ozan Yigit) -- David Allan Steffens | I believe in learning from past mistakes... Eaton-Peabody Laboratory | ...but does a good education require so many? Mass. Eye & Ear Infirmary, 243 Charles Street, Boston, MA 02114 {harvard,mit-eddie,think}!eplunix!das (617) 573-3748 (1400-1900h EST)
ok@goanna.cs.rmit.oz.au (Richard A. O'Keefe) (02/04/91)
In article <2886@charon.cwi.nl>, dik@cwi.nl (Dik T. Winter) writes: > Ah, but that is the problem. Just as fast comes in flavors. You have CPU > time, real time, IO time and some more. If one of the programs uses more > passes it is not equally fast IO time wise ... I am firmly in the "read it once" camp, but it's worth pointing out that operating system features can invalidate some obvious conclusions. One of the things that I have found tricky is benchmarking programs that read files under UNIX; the disc block cache means that the second time a program reads a file much of it is still there in memory. In the case of a program reading /etc/utmp (often rather small) twice in rapid succession, the second read stands a good chance of finding the file still in memory. On another operating system, or with different loads, things might be different. Provided a program is *documented* as relying for its efficiency on such operating system features, I think that's fair enough. -- The Marxists have merely _interpreted_ Marxism in various ways; the point, however, is to _change_ it. -- R. Hochhuth.
ok@goanna.cs.rmit.oz.au (Richard A. O'Keefe) (02/04/91)
In article <14994:Feb207:10:4791@kramden.acf.nyu.edu>, brnstnd@kramden.acf.nyu.edu (Dan Bernstein) writes: > > if (chdir(newsuid) == -1) { > > (void) mkdir(newsuid,0700); > This cannot fail unless some renegade sysadmin changes the mode of the > session directory while pty is running. Er, what about the disc running out of I-nodes? Or going over quota? Or a file server dropping dead? > > (void) chdir(newsuid); > > } > Cannot fail. Can fail if the preceding mkdir() failed -- see above. There is *always* another way for things to go wrong. -- The Marxists have merely _interpreted_ Marxism in various ways; the point, however, is to _change_ it. -- R. Hochhuth.
oz@yunexus.yorku.ca (Ozan Yigit) (02/04/91)
In article <5883:Feb102:05:4991@kramden.acf.nyu.edu> (Dan Bernstein) burps: >In article <21548@yunexus.YorkU.CA> oz@yunexus.yorku.ca (Ozan Yigit) writes: >> To rephrase: ... you are using poor programming techniques. No program >> should make multiple passes over a disk file when it could accomplish its >> task with a single pass with hardly any effort on the programmer's part. > >If you spent a minute putting your priorities into order you'd realize >that this is an entirely ridiculous generalization. Well, I am sure you would know about ridiculous generalizations. Those were your own words plus a trivial meaning-preserving transformation. keep up the cyclically self-disputing silliness. oz
igb@fulcrum.bt.co.uk (Ian G Batten) (02/04/91)
In article <14994:Feb207:10:4791@kramden.acf.nyu.edu> brnstnd@kramden.acf.nyu.edu (Dan Bernstein) writes: > > > if (chdir(newsuid) == -1) > > { > > (void) mkdir(newsuid,0700); > > This cannot fail unless some renegade sysadmin changes the mode of the Ah! I must go out and buy the patented ``Dan Bernstein never-fill'' file system, which has infinite data space and none of those nasty limits of the number of inodes. You can never, ever, in user space ensure that you have space to do IO as there will always be a timing window between your statfs (or whatever) and the actual IO. > > (void) chdir(".."); > > This is guaranteed to work. Really. Tell me about it. It was some utility --- I forget what --- screaming ``can't find ..'' which alerted me on the occasion that a renegade disk controller sprayed a data block accross a few directories on my news machine. Seeing that and rapidly shutting the machine down allowed me to rescue it.
chip@tct.uucp (Chip Salzenberg) (02/04/91)
According to brnstnd@kramden.acf.nyu.edu (Dan Bernstein): >> (void) read(fdsess,(char *) &newuid,sizeof(int)); > >This is guaranteed to work, because fdsess is a regular file that was >created with at least the int. The only possible problem is a hard I/O >error or other fatal system error. That particular "fatal" error -- disk failure -- will not kill the program. Instead, it will result in a pty program continuing to run using wrong data that could have been detected and ignored. This is "good programming practice?" >> if (chdir(newsuid) == -1) >> { >> (void) mkdir(newsuid,0700); > >This cannot fail unless some renegade sysadmin changes the mode of the >session directory while pty is running. Not to protect against that possibility is a strong temptation, but it certainly is not "good programming practice." >> (void) chdir(newsuid); >> } > >Cannot fail. Unless the mkdir() failed. -- Chip Salzenberg at Teltronics/TCT <chip@tct.uucp>, <uunet!pdn!tct!chip> "I want to mention that my opinions whether real or not are MY opinions." -- the inevitable William "Billy" Steinmetz
tchrist@convex.COM (Tom Christiansen) (02/05/91)
From the keyboard of oz@yunexus.yorku.ca (Ozan Yigit):
:In article <5883:Feb102:05:4991@kramden.acf.nyu.edu> (Dan Bernstein) burps:
:>In article <21548@yunexus.YorkU.CA> oz@yunexus.yorku.ca (Ozan Yigit) writes:
:>> To rephrase: ... you are using poor programming techniques. No program
:>> should make multiple passes over a disk file when it could accomplish its
:>> task with a single pass with hardly any effort on the programmer's part.
:>
:>If you spent a minute putting your priorities into order you'd realize
:>that this is an entirely ridiculous generalization.
:
:Well, I am sure you would know about ridiculous generalizations. Those
:were your own words plus a trivial meaning-preserving transformation.
:
:keep up the cyclically self-disputing silliness.
Well, that's what you get when you trade postings with Dan. It's
like playing patty cake with a schizoid King Kong.
--tom
--
"Still waiting to read alt.fan.dan-bernstein using DBWM, Dan's own AI
window manager, which argues with you 10 weeks before resizing your window."
### And now for the question of the month: How do you spell relief? Answer:
U=brnstnd@kramden.acf.nyu.edu; echo "/From: $U/h:j" >>~/News/KILL; expire -f $Utchrist@convex.COM (Tom Christiansen) (02/05/91)
From the keyboard of dhesi%cirrusl@oliveb.ATC.olivetti.com (Rahul Dhesi):
:In <5883:Feb102:05:4991@kramden.acf.nyu.edu>
:brnstnd@kramden.acf.nyu.edu (Dan Bernstein) writes:
:
:>Tradeoffs between ``multiple passes'' and ``single pass'' are entirely
:>irrelevant when they aren't reflected in speed, space, or human effort.
:
:It's often easier to adapt a single-pass program to handle
:previously-unforeseen needs than to similarly adapt a multipass
:program. A good example is what happens if you decide later that you
:would like to handle standard input. Now if you have a multipass
:implementation, you either have to completely redo it, or use a
:possibly huge temp file, or do buffering in blocks which may involve a
:heavy penalty in speed and memory usage.
Very true. Consider Henry Spencer's fiendishly clever "nroff -man in awk"
program (awf), to my mind the pinnacle of awk programming. He runs it in
three stages for various reasons, but this makes communication and feedback
between passes difficult at best. See his fascinating paper on ``Awk as a
Major System's Programming Language'' in the proceedings from last month's
Dallas USENIX in which he discusses this and a bunch of other issues.
It's a similar problem, by the way, to the one that occurs in streams like:
a | b | c | d | e | f | g | h | i > omega
It's a bummer when you realize that portions c and g need to talk
to each other.
It occurs to me that this discussion is about general programming, not
just C programming, so I've directed followups to comp.unix.programmer.
Of course, if you don't like this, edit your headers; for example, the
overly motivated might reasonably select alt.religion.computers. :-)
--tom
--
"Still waiting to read alt.fan.dan-bernstein using DBWM, Dan's own AI
window manager, which argues with you 10 weeks before resizing your window."
### And now for the question of the month: How do you spell relief? Answer:
U=brnstnd@kramden.acf.nyu.edu; echo "/From: $U/h:j" >>~/News/KILL; expire -f $Ubrnstnd@kramden.acf.nyu.edu (Dan Bernstein) (02/05/91)
In article <4702@goanna.cs.rmit.oz.au> ok@goanna.cs.rmit.oz.au (Richard A. O'Keefe) writes: > Er, what about the disc running out of I-nodes? Or going over quota? > Or a file server dropping dead? Well, yes, but those fall under the blanket of ``fatal errors'' which I mentioned at the top of the article. (Actually, going over quota shouldn't happen unless the system is strangely configured, but this is beside the point.) There's a push-pull here between ``don't catch errors you can't handle sensibly'' and ``don't dig your hole too deep.'' ---Dan
brnstnd@kramden.acf.nyu.edu (Dan Bernstein) (02/05/91)
Ya know, shouldn't there be a comp.programming group for programming discussions that aren't related to any particular language? We have IBM and UNIX and games programming groups, but no general comp.programming to be the practical side of comp.software-eng. I wonder. In article <21594@yunexus.YorkU.CA> oz@yunexus.yorku.ca (Ozan Yigit) writes: > >In article <21548@yunexus.YorkU.CA> oz@yunexus.yorku.ca (Ozan Yigit) writes: > >> No program > >> should make multiple passes over a disk file when it could accomplish its > >> task with a single pass with hardly any effort on the programmer's part. > Those > were your own words plus a trivial meaning-preserving transformation. Be serious. Time, space, and human effort are by far the most important three factors in programming. When you can get a noticeable improvement in one without noticeably hurting another, you should do so. That's what I've been saying all along. Now you bring in the fourth god of ``number of passes.'' Wtf are you talking about? The number of passes matters *only* in how it affects time, space, and human effort. Lots of examples bear this out. I still say that your generalization about the number of passes is ridiculous. It doesn't match anything I've ever said by a ``trivial meaning-preserving transformation,'' and I wish you would stop putting words into my mouth. ---Dan
brnstnd@kramden.acf.nyu.edu (Dan Bernstein) (02/05/91)
In article <W~%&'N@uzi-9mm.fulcrum.bt.co.uk> igb@fulcrum.bt.co.uk (Ian G Batten) writes: > > > (void) chdir(".."); > > This is guaranteed to work. > Really. Tell me about it. I started the article with a blanket disclaimer saying that the code still had to handle hard I/O errors. Do I need to say ``other than the fatal errors mentioned above'' in every sentence so that you don't quote me out of context? Sheesh. ---Dan Stupidity, n.: An overwhelming desire to rewrite one-line shell scripts as 36-line Perl scripts so they run 6% faster. See Christiansen, Tommy.
ambar@ora.com (Jean Marie Diaz) (02/05/91)
From: brnstnd@kramden.acf.nyu.edu (Dan Bernstein) Date: 2 Feb 91 07:10:47 GMT > (void) read(fdsess,(char *) &newuid,sizeof(int)); This is guaranteed to work [...] > (void) chdir(".."); This is guaranteed to work. > (void) chdir(newsuid); > } Cannot fail. [and so forth] I commend to you a paper titled "Can't Happen, or /*NOTREACHED*/, or Real Programs Dump Core", by Ian Darwin & Geoff Collyer. It can be found in the proceedings of the Winter Usenix Conference, Dallas 1985, pages 136-151. Or FTP to cs.toronto.edu will get you the Postscript version under doc/programming/canthappen.PS. Since "anything that can go wrong will go wrong", it is a programmer's responsibility to deal with Murphy as gracefully as possible. Dumping core is never graceful. AMBAR
gwyn@smoke.brl.mil (Doug Gwyn) (02/05/91)
In article <29778:Feb419:26:3791@kramden.acf.nyu.edu> brnstnd@kramden.acf.nyu.edu (Dan Bernstein) writes: >Now you bring in the fourth god of ``number of passes.'' Wtf are you >talking about? The number of passes matters *only* in how it affects >time, space, and human effort. Lots of examples bear this out. An /etc/utmp inspector could easily be foiled is it relies on the contents of /etc/utmp being unchanged between its first and second passes. It's bad enough not being able to lock the file for one pass, but that's probably easier to cope with than a phase error.
oz@yunexus.yorku.ca (Ozan Yigit) (02/05/91)
In article <29778:Feb419:26:3791@kramden.acf.nyu.edu> Dan Bernstein writes: >Ya know, shouldn't there be a comp.programming group for programming >discussions that aren't related to any particular language? Why? So you can endlessly and vituperatively argue the good programming techniques within a 40-line C-boredom that core dumps when its required file is missing? >... and I wish you would stop putting words into my mouth. I couldn't possibly do that. Your foot is in there. oz
mike@BRL.MIL (Michael John Muuss <mike>) (02/05/91)
In article <14994:Feb207:10:4791@kramden.acf.nyu.edu>, brnstnd@kramden.acf.nyu.edu (Dan Bernstein) writes: |> Are you worried about all the ignored return codes? I'll go through them |> one by one. Sure, I could've programmed more defensively, but the only |> possible response to a fatal error would be cutting off the user program |> from its input and output. |> |> Cannot fail. |> Guaranteed to work, as above. |> Ditto. |> |> ---Dan Dear Dan - I have seen every one of the system-calls you claim as "guaranteed to work" FAIL, in similar usage. (I've seen sprintf() fail too, but checking it's return code is problematic, due to the differences between BSD and SYSV implementations, so portability usually wins here). Filesystems filling up, memory failures, failing disk drives, corrupted program memory images, NFS outages and server malfunctions, etc, etc. ".." can unexpectedly vanish, or take you to unexpected places. ".." may not even be of type=directory! In my 17 years of using UNIX, I've seen it all. Not often, but not "never". "The problem with making programs foolproof is that the fools are so damned clever!" That goes double for hardware. Having programs not checking the error returns can make the job of troubleshooting and repairing malfunctions much more difficult. Yes, error checking can demand 3-4 times as much code as the actual task at hand. For essential systems programs, and criticial user applications, the error checking is well worth it. I don't think that I would consider the application under discussion "critical", so your (void)ing the error returns in this application was entirely appropriate. But, I didn't want you (or other readers of this list) to think that the sys-calls which you considered "guaranteed" to work actually would be, under all circumstances. They are not. Best, -Mike Muuss Advanced Computer Systems Ballistic Research Laboratory APG, MD 21005-5066
ksb@nostromo.cc.purdue.edu (Kevin Braunsdorf) (02/05/91)
In article <29778:Feb419:26:3791@kramden.acf.nyu.edu> brnstnd@kramden.acf.nyu.edu (Dan Bernstein) writes: >Be serious. Time, space, and human effort are by far the most important >three factors in programming. Yeah, the time and space I have to spend tracking down sloppy code that: (a) has no error checks (b) doesn't identify itself (c) outputs the *wrong* error string... The *small* time it takes to type in if (0 != chdir(acTo)) { fprintf(stderr, "%s: chdir: %s: %s\n", progname, acTo, strerror(errno)); exit(9); } (which is a macro in vi, like: map! ^F fprintf(stderr, "%s: ;: %s: %s\n", progname, ;, strerror(errno));^[F; or: map! ^P fprintf(stderr, "%s: ;: %s\n", progname, strerror(errno));^[F; ) is better than either (void)chdir(acTo); or if (-1 == chdir(acTo)) { perror("chdir"); exit(1); } the second one is the one that really pisses me off. With a little more care the program could lead off with a *name* so long pipes wouldn't be so hard to debug under the UNIX shell. Yeah, Dan, the time I care about is not the first coder, it is the maintainer (most of the time is spent in maintaining a system), and the user writting his first pipeline (who thinks: chdir: Not a typewriter is not a great error message, and thinks all of UNIX is cryptic). -- "We'll crash and burn, it's hard to look away!!" kayessbee, Kevin Braunsdorf, ksb@cc.purdue.edu, pur-ee!ksb, purdu!ksb
brnstnd@kramden.acf.nyu.edu (Dan Bernstein) (02/05/91)
Summary: Chip complains that a read() from disk into a ``newuid'' integer may fail and hence have pty switch to the wrong uid. But I explicitly initialized newuid to uid just to handle this possibility. Can people stop taking my code out of context? Fgs, right above the section that Geoff quoted is the comment /* XXX: We should have some error recovery here! */ I freely admit that the code isn't perfect. But I already have enough safeguards in place that I'm not worried about any huge disasters, and I don't need a dozen people telling me that my programming technique has problems when they haven't even bothered to read my code. In article <27AD7A46.343B@tct.uucp> chip@tct.uucp (Chip Salzenberg) writes: > >> (void) read(fdsess,(char *) &newuid,sizeof(int)); > That particular "fatal" error -- disk failure -- will not kill the > program. Instead, it will result in a pty program continuing to run > using wrong data that could have been detected and ignored. This is > "good programming practice?" So how would you like to handle it, Chip? Die with a fatal error, possibly killing a truly critical system program running under pty? You should already have known that the value read in---newuid---is initialized to uid. Now I know there are some wacko UNIX systems out there, but I've never seen an I/O error on the first block of a disk file that resulted in read() returning less than 4 bytes of that block. Either the entire block gets read in, or none of it does. (Sure, it's still worth testing for the error---if you can figure out a sane way to handle it.) So, Chip, all that will happen upon an I/O error in that section is that pty will continue with the same uid as it had before. If communication (or the disk) breaks down entirely then the signaller process will lose track of the master process, but it is practically impossible that the section of code you quoted will result in a wrong uid, even in the already disastrous situation that there are physical disk errors. You say ``it will result in a pty program continuing to run using wrong data''; I say that you don't know what you're talking about. I put that newuid = uid initialization in for a reason, and I am getting rather sick of people drawing incorrect conclusions about my code because Geoff took it out of context. Nobody's programming is perfect; the best I hope I can say is that I've gotten better over the years. I do think, however, that I have some degree of common sense about what can fail and what can't, and when I've already put a comment into my program saying ``We should test for foo'' it gets rather annoying to see a dozen people saying ``Dan, you don't use good programming technique, because you don't test for foo. Don't you realize what a programming criminal you are?'' Sheesh. Don't you read code before you criticize it? > >> if (chdir(newsuid) == -1) > >> (void) mkdir(newsuid,0700); > >This cannot fail unless some renegade sysadmin changes the mode of the > >session directory while pty is running. > Not to protect against that possibility is a strong temptation, but it > certainly is not "good programming practice." Well, golly gee, I suppose every program should encrypt all its internal data and throw away the key, just in case some renegade sysadmin is trying to corrupt its operation with ptrace()! Yeah! Chip, be reasonable. You can't demand of system programs that they check for external system consistency at every step. Defensive programming is fine, but screwing around with correct code just because ``a sysadmin might go out of his way to damage the internal state of the system'' can be pretty stupid. ---Dan
flint@gistdev.gist.com (Flint Pellett) (02/06/91)
brnstnd@kramden.acf.nyu.edu (Dan Bernstein) writes: >Summary: Chip complains that a read() from disk into a ``newuid'' >integer may fail and hence have pty switch to the wrong uid. But I >explicitly initialized newuid to uid just to handle this possibility. >Can people stop taking my code out of context? ... >So, Chip, all that will happen upon an I/O error in that section is that >pty will continue with the same uid as it had before. If communication >(or the disk) breaks down entirely then the signaller process will lose >track of the master process, but it is practically impossible that the >section of code you quoted will result in a wrong uid, even in the >already disastrous situation that there are physical disk errors. You >say ``it will result in a pty program continuing to run using wrong >data''; I say that you don't know what you're talking about. >I put that newuid = uid initialization in for a reason... I haven't followed this entire thread, but I think I've seen enough of it to make a couple general observations that may or may not apply directly here, which other people might be able to learn from. 1. If you post any code on usenet, expect it to be criticized. If you can't handle lots of criticism, don't post the code. 2. It would be nice, if people criticizing code did so as if were a professional code review. In the first place, remember that in a code review it is the code that is being criticized, NOT the programmer. There is a difference between someone "being wrong on this point" and "not knowing what they are talking about". If you don't have any respect for the person whose code you are criticizing, my suggestion is to not bother doing it: you aren't going to help them or yourself. 3. Regarding the code: I didn't see the entire example code, but from the ensuing discussion, I conclude that there was a problem in the comments in the code. It was stated that the initialization was put in "for a reason". If the comments were complete, that would have been obvious, and in fact that whole paragraph that appears above explaining what the reasons were, and the reasons why the error conditions are being ignored, should have been there as a comment, even if the result was 15 lines of comment for 1 line of code. When code is commented like that, then a maintainer who comes along trying to get/keep the code working in a slightly different environment can quickly determine if the "reason" for the check is still valid, instead of having to spend who knows how much time trying to figure out what the reason for it was. 4. A good maintainer (or criticizer) always assumes the original author did things for reasons. If you assume that the original author was just stupid and didn't know what they were talking about, it may save you the time of figuring out what the reason for having a check was, making it ok to remove it or alter it. Then you "test" it, it works ok, and you can go on your way, leaving a time-bomb sitting around waiting to blow up when the reason for the test you screwed up becomes apparent.) -- Flint Pellett, Global Information Systems Technology, Inc. 1800 Woodfield Drive, Savoy, IL 61874 (217) 352-1165 uunet!gistdev!flint or flint@gistdev.gist.com
terryl@sail.LABS.TEK.COM (02/06/91)
In article <14994:Feb207:10:4791@kramden.acf.nyu.edu> brnstnd@kramden.acf.nyu.edu (Dan Bernstein) writes: >In article <1991Feb2.045119.22199@zoo.toronto.edu> geoff@zoo.toronto.edu (Geoffrey Collyer) writes: >> Dan Bernstein, in response to Ozan Yigit: >> > To rephrase: You don't seem to know what good programming techniques are. >> From pty/master.c: > >Yes, what's your point? I have several hundred notes on pieces of pty >that could be improved. In the list I find ``check for phys I/O errors >on reading fdsess''; that's the only problem with the section of code >you quoted. (Delete MANY lines of him defending his code) Well, I have no idea what his point was, but I have one: If I were your manager, I'd ding you REAL hard for writing semi-obscure code with such a dearth of comments. Now, if he deleted them, then fine, but since I didn't see you post any in your defense, I'll have to assume that they don't exist. __________________________________________________________ Terry Laskodi "There's a permanent crease of in your right and wrong." Tektronix Sly and the Family Stone, "Stand!" __________________________________________________________
brnstnd@kramden.acf.nyu.edu (Dan Bernstein) (02/06/91)
In article <1123@gistdev.gist.com> flint@gistdev.gist.com (Flint Pellett) writes: > 1. If you post any code on usenet, expect it to be criticized. You're right, I overreacted. Sorry. I was hoping for some constructive criticism, though, and it doesn't help to be told something that I had already noted in the code. > 2. It would be nice, if people criticizing code did so as if were a > professional code review. Yes. Among other things, please check the original source, and when it has a comment like /* XXX: We should have some error recovery here! */ then assume that the author does realize that the code can be improved. > 3. Regarding the code: I didn't see the entire example code, but from > the ensuing discussion, I conclude that there was a problem in the > comments in the code. It was stated that the initialization was put > in "for a reason". If the comments were complete, that would have > been obvious, Agreed, and I think that the comment makes it clear enough in context. Geoff, why did you feel compelled to take the code out of context? > 4. A good maintainer (or criticizer) always assumes the original > author did things for reasons. Although in practice most things are done for no reason at all. :-) ---Dan
greywolf@unisoft.UUCP (The Grey Wolf) (02/06/91)
In article <5883:Feb102:05:4991@kramden.acf.nyu.edu> brnstnd@kramden.acf.nyu.edu (Dan Bernstein) writes: > >To rephrase: You don't seem to know what good programming techniques >are. Tradeoffs between speed, space, and human effort are important. >Tradeoffs between ``multiple passes'' and ``single pass'' are entirely >irrelevant when they aren't reflected in speed, space, or human effort. As I'm sure someone will/has pointed out already (haven't seen it yet, but then news has strange timing problems), multiple-pass contemplation of a file is stable IFF the file is guaranteed not to be physically rearranged between passes. (Of course the same can be said of a single pass thru a file, but the window is (usually) *much* smaller). Examples where multiple passes work is on (normally) stable files, such as running them thru a compiler or running a disk thru fsck. (Of course, even there, anomalies happen...) Side issue: -- has anyone managed a single-pass fsck? That would be tricky at best, possibly disastrous. I thought for a moment that instead of doing five passes, one could use five file descriptors to handle each phase, seeing as the size of a disk partition is effectively fixed (last I checked, anyway...). But I guess the passes need to happen in a particular order... > >---Dan -- thought: I ain't so damb dumn! | Your brand new kernel just dump core on you war: Invalid argument | And fsck can't find root inode 2 | Don't worry -- be happy... ...!{ucbvax,acad,uunet,amdahl,pyramid}!unisoft!greywolf
boyd@necisa.ho.necisa.oz.au (Boyd Roberts) (02/06/91)
In article <29421:Feb419:13:5891@kramden.acf.nyu.edu> brnstnd@kramden.acf.nyu.edu (Dan Bernstein) writes: >Well, yes, but those fall under the blanket of ``fatal errors'' which I >mentioned at the top of the article. (Actually, going over quota >shouldn't happen unless the system is strangely configured, but this is >beside the point.) But you'd check for exceeding your quota on close(2) wouldn't you Dan? Boyd Roberts boyd@necisa.ho.necisa.oz.au ``When the going gets wierd, the weird turn pro...''
geoff@zoo.toronto.edu (Geoffrey Collyer) (02/06/91)
Dan, I didn't exclude the XXX comment from the pty code excerpt to hide anything, it merely seemed redundant given the code that followed (surely all those (void) casts are a reminder that this code is incomplete and needs to be finished; to intend otherwise would be unprofessional). From pty.1: .SH BUGS None known, but they're probably hiding somewhere. Fascinating; this would seem to be at odds with the XXX comment. I also don't consider that code excerpt to be at all atypical of pty; I have read the entire pty source, and the excerpted code is fairly typical, in fact it was picked at random. Given your cavalier disregard for errors and apparent incomprehension of possible sources of error, I don't think you are in any position to criticise another's programming technique, let alone Ozan's. I shall be polite here and refrain from any further comment on the programming technique so stunningly displayed by pty. And I'm not going to argue this for ten weeks. Don't you have a job or thesis to keep you busy? -- Geoff Collyer utzoo!geoff, zoo.toronto.edu!geoff
terryl@sail.LABS.TEK.COM (02/07/91)
In article <29421:Feb419:13:5891@kramden.acf.nyu.edu> brnstnd@kramden.acf.nyu.edu (Dan Bernstein) writes: >In article <4702@goanna.cs.rmit.oz.au> ok@goanna.cs.rmit.oz.au (Richard A. O'Keefe) writes: >> Er, what about the disc running out of I-nodes? Or going over quota? >> Or a file server dropping dead? > >Well, yes, but those fall under the blanket of ``fatal errors'' which I >mentioned at the top of the article. (Actually, going over quota >shouldn't happen unless the system is strangely configured, but this is >beside the point.) There's a push-pull here between ``don't catch errors >you can't handle sensibly'' and ``don't dig your hole too deep.'' Now, I'm reading between the lines here, but this is what I see: Many people have pointed out ways things can go wrong, but you just hand- wave them away as "fatal errors" (errors that you can't possibly fix). Now, what I see (and from examining the code fragment from pty/master.c) is that quite a few of the things you claim "cannot happen", could very well happen, but your program will blindly plow on, possibly dumping core. <ENTER VERY IMHO MODE> There's nothing I hate more than programs that silently fail, because I can't tell WHY they failed. Even just a simple "I can't do this because of that" is preferable (like perror(filename)). At least I can say "Aha, over quota; better delete some files." Also, don't tell me "Cannot open file", even if file is a real live filename. Tell me WHY you can't open it. <EXIT VERY IMHO MODE> __________________________________________________________ Terry Laskodi "There's a permanent crease of in your right and wrong." Tektronix Sly and the Family Stone, "Stand!" __________________________________________________________
tchrist@convex.COM (Tom Christiansen) (02/07/91)
From the keyboard of terryl@sail.LABS.TEK.COM:
: There's nothing I hate more than programs that silently fail, because I
:can't tell WHY they failed. Even just a simple "I can't do this because of that"
:is preferable (like perror(filename)). At least I can say "Aha, over quota;
:better delete some files." Also, don't tell me "Cannot open file", even if file
:is a real live filename. Tell me WHY you can't open it.
Well, perror(filename) is far better than nothing, but there two major
problems:
1) It's not enough information.
2) Sometimes you have no stderr.
Regarding problem 1, I like to see error messages in this form:
program: operations on object: reason
as in:
my_prog: cannot creat /foo/bar: permission denied
The my_prog is especially important in programs in pipelines. I once
grepped through all the system source (ok, it was BSD, but that doesn't
let the rest of the world off the hook) for things that said effectively:
printf(stderr, "cannot open: %s\n", filename);
and was horrified and angered. They all deserve fixing.
On problem #2, I think that if you have no stderr, (and even if you do
for system programs) you should also syslog the problem. It's not a
perfect solution, but it's surely better than nothing. I'm aware of
Dan's arguments about syslog not being perfect, such as: you can spoof
it, because you can get denial of service problems, because it's not
(indefinitely) extensible. While these may be all true, it sure beats
a boot to the head. I just hate system programs that don't syslog.
And you don't want to hear what I have to say about systems without
syslog, and a modern one at that. I simply don't use them.
As was mentioned earlier, *every* system call should *always*
be checked, even if you "know" it can't fail. Look in the classic
paper referenced earlier. One of its examples was from the init code;
something like:
open("/", 0);
dup2(0,1);
dup2(0,2);
These can all fail. I'd like the program to know it. I'd also
like it if it found some way to tell me, but at least it shouldn't
ignore them just because it can't think of something better to do.
I know people say not to check for errors that you don't know how to
handle. I think they're wrong. That's what asserts are all about.
I've spent more in the last 10 years tracking down program that don't
detect error conditions, or if they do, don't give me reasonable
messages, than I like to think about, all because some programmer
was too lazy or short-sighted to deal with the problem in an
adequate and informative manner.
Yes, I've attempted to move the thread again. I don't think it
belongs in BSD bugs anymore, although maybe Bostic et alii will
notice the BSD reference above and amend any remaining deficiencies.
My check was many, many releases ago -- I don't how bad recent
Berkeley code is at this.
--tom
--
"Still waiting to read alt.fan.dan-bernstein using DBWM, Dan's own AI window
manager, which argues with you for 10 weeks before resizing your window."
### And now for the question of the month: How do you spell relief? Answer:
U=brnstnd@kramden.acf.nyu.edu; echo "/From: $U/h:j" >>~/News/KILL; expire -f $Uboyd@necisa.ho.necisa.oz.au (Boyd Roberts) (02/07/91)
Come on Dan, I've told you before. Check those error conditions. Weird things happen. Just chmod 0 /dev/tty and watch your UNIX machine go absolutely ga-ga on you, and _none_ of tools that break will say anything of any worth about the problem. Presumably because they were written with the philosophy that states `X will ever happen'. And I thought it was warm here. There's nothing quite like a Dan flame-fest. Boyd Roberts boyd@necisa.ho.necisa.oz.au ``When the going gets wierd, the weird turn pro...''
brnstnd@kramden.acf.nyu.edu (Dan Bernstein) (02/07/91)
In article <2013@necisa.ho.necisa.oz.au> boyd@necisa.ho.necisa.oz.au (Boyd Roberts) writes: > Come on Dan, I've told you before. Check those error conditions. Boyd, I *am* checking those error conditions. There are a few isolated sections of code in pty (one being the section Geoff quoted) where errors that might be handled are ignored. I realize that this 4% of the code needs to be redone to handle errors better. How do you think it comes across, Boyd, when a programmer puts a comment into some code saying ``this needs foo''; someone quotes the code without the comment; and then you pop in and say ``Come on, I've told you before. You need foo.'' I think it casts some doubt on your academic standards: you haven't bothered to read code before you criticize it, even in public. > Weird things happen. Just chmod 0 /dev/tty and watch your UNIX > machine go absolutely ga-ga on you, and _none_ of tools that break > will say anything of any worth about the problem. pty: fatal: cannot reopen pty for /dev/tty: Permission denied. Boyd, fuck off. Or learn to read the code you criticize. Your choice. ---Dan
stripes@eng.umd.edu (Joshua Osborne) (02/08/91)
In article <1991Feb4.195805.16710@ora.com>, ambar@ora.com (Jean Marie Diaz) writes: [...] > Since "anything that can go wrong will go wrong", it is a programmer's > responsibility to deal with Murphy as gracefully as possible. Dumping > core is never graceful. It isn't normally graceful, but in bizzare failure cases I *want* a core dump so I can see what went on to get me to the "can't happen" default in a switch (I want to know more then just the improper switch control variable). That's why my code has a ton of asserts, or for more end-user style code a assert-like call "make_core(int expr, char *reason)" which prints some standard message, the reason string, dumps some stuff in a file, dumps a core the tells the user to save the core & the error file, and who to mail or phone... (the core file isn't kept with that name, too many autodeleters nail core files...) Of corse more normal error conditions get more normal messages! I have redirected follow-ups to me because I don't know what newsgroup generic programing flames/tips should go to. Hopefully I did this right, I couldn't find it in the manpages, so I'll have to rely on my faulty memmory... -- stripes@eng.umd.edu "Security for Unix is like Josh_Osborne@Real_World,The Multitasking for MS-DOS" "The dyslexic porgramer" - Kevin Lockwood "CNN is the only nuclear capable news network..." - lbruck@eng.umd.edu (Lewis Bruck)
dbc@cimage.com (David Caswell) (02/09/91)
.From pty/master.c: . . (void) lseek(fdsess,(long) 0,0); . (void) read(fdsess,(char *) &newuid,sizeof(int)); . (void) sprintf(newsuid,"%d",newuid); . should be (void) lseek(fdsess,0L,0); saves some keystokes, after all what else in the world is important?
mcdaniel@adi.com (Tim McDaniel) (02/13/91)
mike@BRL.MIL (Michael John Muuss <mike>) writes:
Having programs not checking the error returns can make the job of
troubleshooting and repairing malfunctions much more difficult.
Yes, error checking can demand 3-4 times as much code as the actual
task at hand. For essential systems programs, and criticial user
applications, the error checking is well worth it.
A number of places, including Applied Dynamics (plug plug), use an
exception-handling library. If a function fails, it raises an
exception, causing the invocation of any code-specific exception
handlers (and, in some implementations, outputs a message if not
handled). Many implementations use preprocessor trickery at some
point, to make the code look nice.
There's an incredible feeling of freedom in simply writing
buf = MEMmalloc(STlen(s) + 1);
and knowing that the function won't return if it fails, and to write
the one line
EMMfatal("MEMmalloc", MEMoutOfMemory, "");
to do everything needed to wrap up the program with an informative
message. We have much more robust code as a result of such a library.
Also, error handling is centralized, so all the messages have the same
appearance, and it's easy to redirect them to a new destination.
This topic has been discussed in comp.lang.c before, and it's not
relevant to C per se anyway, so I won't go into more details.
--
"I'd hang you from my nipples ... but it would shock the children."
Tim McDaniel Applied Dynamics Int'l.; Ann Arbor, Michigan, USA
Internet: mcdaniel@adi.com UUCP: {uunet,sharkey}!amara!mcdanielwolfe@dds4.uucp (Peter Wolfe) (02/14/91)
In article <MCDANIEL.91Feb12173940@dolphin.adi.com> mcdaniel@adi.com (Tim McDaniel) writes: >A number of places, including Applied Dynamics (plug plug), use an >exception-handling library. If a function fails, it raises an >exception, causing the invocation of any code-specific exception >handlers (and, in some implementations, outputs a message if not >handled). Many implementations use preprocessor trickery at some >point, to make the code look nice. Any possibility (hint, hint) of such a library being available to the world at large? -- Peter Wolfe (604-270-1171) uunet!van-bc!dds4!wolfe Digital Dispatch Systems, Richmond BC Canada