lee@ssc-vax.UUCP (Lee Carver) (08/13/88)
Why should file descriptor closing neccesarily close the file pointer? Especially when there are more then one file descriptors associated with the file pointer. The following is ~180 lines of discussion. --- The plan I was trying to build a nice, prompting, validating input reader for adding to shell scripts. The idea was to run this program (we'll call it readtkn), and have it read and validate the user's input, then write the result to stdout. Managed to build something more useful then not. Typical usage might be: # this file is 'demo' set src=`readtkn 'Enter source file > ' opt opt opt` set dst=`readtkn 'Enter destination > ' opt opt opt` cp $src $dst Obviously, expand this to your heart's content. Presumably, the options in the above example constrain the user's input to valid file names, etc. --- The problem Unfortunately, we ran into serious problems during testing, and any other time that a script using readtkn is sent a file of responses instead of reading the user's terminal. We test our package by running the scripts with known answers, and verifying the expected behavior, along the lines of: # this file is 'test' demo << EOF model /tmp/model EOF diff model /tmp/model The second execution of readtkn finds an end of file. It seems that when the first execution of readtkn terminates, it closes the file descriptor (exit semantics). Thus, when the second readtkn runs, it is handed the file descriptor of a closed file, and read EOF. My understanding of the UNIX process and file structure tells me that each file descriptor is associated with a file pointer. When readtkn is run (by fork), it creates a new file descriptor to the original file pointer. Thus, if either the child (readtkn) or parent (shell/demo) change the file pointer (seek, read, etc.), the other one is affected. The problem is the close, which is called automatically for every open file descriptor on exit. The close eliminates the file pointer, even though there are two file descriptors associated with it. It seems to me that this should not be so. The file pointer that the first readtkn closes is shared by a file descriptor in the shell. Now it must close its descriptor/connection, but why should it cause the file pointer to be closed as well? In fact, one might be inclined to argue that the file pointer is "owned" by the parent, not the child. So what is the child doing closing the file pointer that it does not own? Why does this work at all if stdin is /dev/tty? Apparently, the shell reopens stdin if it is closed by a child process, but I'm not sure. --- The proposals Clearly, there are programs that rely on these semantics. So we cannot change the semantic of the close, at least in the normal case. That eliminates the proposal that only the "owner" of the file pointer can close the file pointer. The next alternative is new fcntl option to mark a file descriptor as "don't close on exit". This is somewhat similar to the F_SETFD option to "close on exec". Personally, I don't like this, since I'm not sure I could manage it. My feeling is that a "new" close call should be provided (disconnect?). The semantics of disconnect would be to close the file pointer only when the last file descriptor is disconnected. An equivalent variation would be a fcntl option to activate these semantics on close could be added. A problem with these proposals is the interaction of stdio. If the input actually read by readtkn is less then the amount pre-fetched by stdio, we need to clean up. The un-read but pre-fetched bytes need to be restored to the file. Perhaps an lseek to the last delivered byte is all that is needed. Hopefully I'm missing something obvious. If not, or you have a better idea, send me mail. --- The workaround On our system at least (see disclosure), we were able to get things to work by wrapping a shell script around readtkn in the following style: # this file is 'readtkn', a wrapper for the program readtkn.exe if test $AUTOMATED then read scrap echo $scrap | readtkn.exe $* else readtkn.exe $* endif This does work, but seems quite inelegant. Also, it limits you to full lines in readtkn (no single character reads). Also, you have to have explict knowledge of nesting because of the AUTOMATED variable. Without that, interactive users don't get their prompts until after they supply the correct answer (sigh). --- The disclosure This was actually done in full flower on an Apollo system with their proprietary Aegis operating system, and proprietary "/com/sh" shell. After complaining to them about this weirdness, I discover that it also happens on un-adulterated UNIX (well BSD 4.3). So, if the problem statement isn't exactly UNIX-ese, I'm sorry. These sample programs have been run, with the indicated results, on ssc-bee, a BSD 4.3 VAX-11/785. My plan is to tell Apollo how I'd like to see it fixed. Since it seems broken on UNIX too, maybe we can all benefit. --- The details So, now we conclude with the actual file sources. The file names should be clear. The body of each file is indented three spaces, and each file is terminated with the line ' *** EOF ***'. test driver script: sample << EOF AB EOF *** EOF *** sample, the script called from above: readtkn readtkn *** EOF *** results of running the test driver: A --- END OF FILE --- *** EOF *** desired results, if stdin stayed open: A B *** EOF *** readtkn.c, the source of readtkn: #include <stdio.h> main ( argc, argv ) int argc; char **argv; { int ch; ch = getchar (); if ( ch == EOF ) puts ( "--- END OF FILE ---" ); else { putchar ( ch ); putchar ( '\n' ); } exit (0); } *** EOF *** --- The signature Please mail me your comments and suggestions. I'll summarize what comes in. Thanks. Lee Carver Boeing Aerospace csnet: lcarver@boeing.com uucp: {...}!uw-beaver!ssc-vax!ssc-bee!lee
chris@mimsy.UUCP (Chris Torek) (08/13/88)
In article <1122@ssc-bee.ssc-vax.UUCP> lee@ssc-vax.UUCP (Lee Carver) writes: >Why should file descriptor closing neccesarily close the file >pointer? Especially when there are more then one file descriptors >associated with the file pointer. The following is ~180 lines of >discussion. It does not; and the discussion is pointless, since the entire mechanism is different. (And boy does Lee feel silly :-) ...) >The second execution of readtkn [where the first was from a file, not >a terminal, and both are from the same script] finds an end of file. The second finds EOF because the first read all the data and left the seek pointer of the underlying file descriptor---shared between each invocation of readtkn and the shell that starts them, since the shell provides it and creates it only once---pointing at the end of the file. For instance, if a.out is compiled from the program #include <stdio.h> main() { char buf[100]; (void) fgets(buf, sizeof buf, stdin); (void) fputs(buf, stdout); exit(0); } then running the command (a.out > /dev/null; a.out) < /etc/termcap will *not* print the second line of /etc/termcap! Instead, it will print a (probably partial) n'th line, by reading something from 512, 1K, 2K, 4K, 8K, or 16K bytes after the first character (the number of bytes skipped depends on your Unix variant and the block size of the file system in which /etc/termcap resides). What happened? Simple: stdio read the first n kbytes of /etc/termcap, returned the first line, the first a.out exited, and the second a.out read the second n kbytes. How can you work around it? (1) Stop using stdio. The resulting code will be considerably less efficient. (2) Use stdio, but use a separate file, or reset the seek pointer, for each invocation of the program: (1) a.out < /etc/termcap > /dev/null a.out < /etc/termcap # prints first line (2) /* seektozero: */ main() { (void)lseek(0,0L,0); exit(0); } (a.out >/dev/null; seektozero; a.out) </etc/termcap # also prints first line (2, improved) (a.out.new >/dev/null; a.out) </etc/termcap # prints second line The modified a.out.new has one new line before exit(0): /* long ftell(); /* should be declared in stdio.h */ (void) lseek(stdin, ftell(stdin), 0); -- In-Real-Life: Chris Torek, Univ of MD Comp Sci Dept (+1 301 454 7163) Domain: chris@mimsy.umd.edu Path: uunet!mimsy!chris
ok@quintus.uucp (Richard A. O'Keefe) (08/14/88)
In article <1122@ssc-bee.ssc-vax.UUCP> lee@ssc-vax.UUCP (Lee Carver) writes: >Why should file descriptor closing neccesarily close the file >pointer? Especially when there is more than one file descriptor >associated with the file pointer.>Typical usage might be: > # this file is 'demo' > set src=`readtkn 'Enter source file > ' opt opt opt` > set dst=`readtkn 'Enter destination > ' opt opt opt` > cp $src $dst > > # this file is 'test' > demo << EOF > model > /tmp/model > EOF > diff model /tmp/model >The second execution of readtkn finds an end of file. The problem isn't what you think it is. It's actually quite simple. Your program probably uses stdio. By default, stdio is LINE buffered when reading from terminals and BLOCK buffered when reading from other devices. Your program presumably calls fgets() or getc() to read the response, but stdio will actually grab as much as it can get. So when you run demo with a "here" file, the very first 'readtkn' is asking for BUFSIZ characters (might be 512, or 1024, might even be more) and is thus getting ALL of the "here" file. The second execution of 'readtkn' is getting an end of file indication because the first one really did eat all the characters. The remedy is simple. Only your program needs to change. Before reading from stdin, do setbuf(stdin, (char*)NULL); /* all flavours */ or setlinebuf(stdin); /* BSD */ or use setvbuf in System V.