[comp.unix.wizards] file descriptor vs. file pointer closing

lee@ssc-vax.UUCP (Lee Carver) (08/13/88)

Why should file descriptor closing neccesarily close the file
pointer?  Especially when there are more then one file descriptors
associated with the file pointer.  The following is ~180 lines of
discussion.

--- The plan

I was trying to build a nice, prompting, validating input reader for
adding to shell scripts.  The idea was to run this program (we'll call
it readtkn), and have it read and validate the user's input, then
write the result to stdout.  Managed to build something more useful
then not.

Typical usage might be:
   
   # this file is 'demo'
   set src=`readtkn 'Enter source file > ' opt opt opt`
   set dst=`readtkn 'Enter destination > ' opt opt opt`
   cp $src $dst

Obviously, expand this to your heart's content.  Presumably, the
options in the above example constrain the user's input to valid file
names, etc.

--- The problem

Unfortunately, we ran into serious problems during testing, and any
other time that a script using readtkn is sent a file of responses
instead of reading the user's terminal.  We test our package by
running the scripts with known answers, and verifying the expected
behavior, along the lines of:

   # this file is 'test'
   demo << EOF
   model
   /tmp/model
   EOF
   diff model /tmp/model

The second execution of readtkn finds an end of file.

It seems that when the first execution of readtkn terminates, it
closes the file descriptor (exit semantics).  Thus, when the second
readtkn runs, it is handed the file descriptor of a closed file, and
read EOF.

My understanding of the UNIX process and file structure tells me that
each file descriptor is associated with a file pointer.  When readtkn
is run (by fork), it creates a new file descriptor to the original
file pointer.  Thus, if either the child (readtkn) or parent
(shell/demo) change the file pointer (seek, read, etc.), the other one
is affected.

The problem is the close, which is called automatically for every open
file descriptor on exit.  The close eliminates the file pointer, even
though there are two file descriptors associated with it.

It seems to me that this should not be so.  The file pointer that the
first readtkn closes is shared by a file descriptor in the shell.  Now
it must close its descriptor/connection, but why should it cause the
file pointer to be closed as well?

In fact, one might be inclined to argue that the file pointer is
"owned" by the parent, not the child.  So what is the child doing
closing the file pointer that it does not own?

Why does this work at all if stdin is /dev/tty?  Apparently, the shell
reopens stdin if it is closed by a child process, but I'm not sure.

--- The proposals

Clearly, there are programs that rely on these semantics.  So we
cannot change the semantic of the close, at least in the normal case.
That eliminates the proposal that only the "owner" of the file pointer
can close the file pointer.

The next alternative is new fcntl option to mark a file descriptor as
"don't close on exit".  This is somewhat similar to the F_SETFD option
to "close on exec".  Personally, I don't like this, since I'm not sure
I could manage it.

My feeling is that a "new" close call should be provided
(disconnect?).  The semantics of disconnect would be to close the file
pointer only when the last file descriptor is disconnected.  An
equivalent variation would be a fcntl option to activate these
semantics on close could be added.

A problem with these proposals is the interaction of stdio.  If the
input actually read by readtkn is less then the amount pre-fetched by
stdio, we need to clean up.  The un-read but pre-fetched bytes need to
be restored to the file.  Perhaps an lseek to the last delivered byte
is all that is needed.

Hopefully I'm missing something obvious.  If not, or you have a better
idea, send me mail.

--- The workaround

On our system at least (see disclosure), we were able to get things to
work by wrapping a shell script around readtkn in the following style:

     # this file is 'readtkn', a wrapper for the program readtkn.exe
     if test $AUTOMATED then
	read scrap
	echo $scrap | readtkn.exe $*
     else
     	readtkn.exe $*
	endif

This does work, but seems quite inelegant.  Also, it limits you to
full lines in readtkn (no single character reads).  Also, you have to
have explict knowledge of nesting because of the AUTOMATED variable.
Without that, interactive users don't get their prompts until after
they supply the correct answer (sigh).

--- The disclosure

This was actually done in full flower on an Apollo system with their
proprietary Aegis operating system, and proprietary "/com/sh" shell.
After complaining to them about this weirdness, I discover that it
also happens on un-adulterated UNIX (well BSD 4.3).  So, if the
problem statement isn't exactly UNIX-ese, I'm sorry.

These sample programs have been run, with the indicated results, on
ssc-bee, a BSD 4.3 VAX-11/785.

My plan is to tell Apollo how I'd like to see it fixed.  Since it
seems broken on UNIX too, maybe we can all benefit.

--- The details

So, now we conclude with the actual file sources.  The file names
should be clear.  The body of each file is indented three spaces, and
each file is terminated with the line ' *** EOF ***'.  

test driver script:
   sample << EOF
   AB
   EOF
   *** EOF ***
 
sample, the script called from above:
   readtkn
   readtkn
   *** EOF ***

results of running the test driver:
   A
   --- END OF FILE ---
   *** EOF ***
 
desired results, if stdin stayed open:
   A
   B
   *** EOF ***
 
readtkn.c, the source of readtkn:
   #include <stdio.h>
   
   main ( argc, argv )
   int argc;
   char **argv;
   {  int ch;
   
      ch = getchar ();
   
      if ( ch == EOF )
         puts ( "--- END OF FILE ---" );
      else {
         putchar ( ch );
         putchar ( '\n' );
         }
   
      exit (0);
      }
   *** EOF ***

--- The signature

Please mail me your comments and suggestions.  I'll summarize what
comes in.  Thanks.

Lee Carver
Boeing Aerospace

csnet:  lcarver@boeing.com
uucp:   {...}!uw-beaver!ssc-vax!ssc-bee!lee

chris@mimsy.UUCP (Chris Torek) (08/13/88)

In article <1122@ssc-bee.ssc-vax.UUCP> lee@ssc-vax.UUCP (Lee Carver) writes:
>Why should file descriptor closing neccesarily close the file
>pointer?  Especially when there are more then one file descriptors
>associated with the file pointer.  The following is ~180 lines of
>discussion.

It does not; and the discussion is pointless, since the entire mechanism
is different.  (And boy does Lee feel silly :-) ...)

>The second execution of readtkn [where the first was from a file, not
>a terminal, and both are from the same script] finds an end of file.

The second finds EOF because the first read all the data and left the
seek pointer of the underlying file descriptor---shared between each
invocation of readtkn and the shell that starts them, since the shell
provides it and creates it only once---pointing at the end of the file.
For instance, if a.out is compiled from the program

	#include <stdio.h>
	main() {
		char buf[100];
		(void) fgets(buf, sizeof buf, stdin);
		(void) fputs(buf, stdout);
		exit(0);
	}

then running the command

	(a.out > /dev/null; a.out) < /etc/termcap

will *not* print the second line of /etc/termcap!  Instead, it will
print a (probably partial) n'th line, by reading something from 512,
1K, 2K, 4K, 8K, or 16K bytes after the first character (the number of
bytes skipped depends on your Unix variant and the block size of the
file system in which /etc/termcap resides).

What happened?  Simple: stdio read the first n kbytes of /etc/termcap,
returned the first line, the first a.out exited, and the second a.out
read the second n kbytes.

How can you work around it?  (1) Stop using stdio.  The resulting code
will be considerably less efficient.  (2) Use stdio, but use a separate
file, or reset the seek pointer, for each invocation of the program:

(1)
	a.out < /etc/termcap > /dev/null
	a.out < /etc/termcap		# prints first line

(2)
	/* seektozero: */ main() { (void)lseek(0,0L,0); exit(0); }

	(a.out >/dev/null; seektozero; a.out) </etc/termcap
					# also prints first line

(2, improved)
	(a.out.new >/dev/null; a.out) </etc/termcap
					# prints second line

The modified a.out.new has one new line before exit(0):

	/*	long ftell();	/* should be declared in stdio.h */

		(void) lseek(stdin, ftell(stdin), 0);
-- 
In-Real-Life: Chris Torek, Univ of MD Comp Sci Dept (+1 301 454 7163)
Domain:	chris@mimsy.umd.edu	Path:	uunet!mimsy!chris

ok@quintus.uucp (Richard A. O'Keefe) (08/14/88)

In article <1122@ssc-bee.ssc-vax.UUCP> lee@ssc-vax.UUCP (Lee Carver) writes:
>Why should file descriptor closing neccesarily close the file
>pointer?  Especially when there is more than one file descriptor
>associated with the file pointer.>Typical usage might be:
>   # this file is 'demo'
>   set src=`readtkn 'Enter source file > ' opt opt opt`
>   set dst=`readtkn 'Enter destination > ' opt opt opt`
>   cp $src $dst
>
>   # this file is 'test'
>   demo << EOF
>   model
>   /tmp/model
>   EOF
>   diff model /tmp/model
>The second execution of readtkn finds an end of file.

The problem isn't what you think it is.  It's actually quite simple.

Your program probably uses stdio.  By default, stdio is LINE buffered
when reading from terminals and BLOCK buffered when reading from
other devices.  Your program presumably calls fgets() or getc() to
read the response, but stdio will actually grab as much as it can get.
So when you run demo with a "here" file, the very first 'readtkn' is
asking for BUFSIZ characters (might be 512, or 1024, might even be more)
and is thus getting ALL of the "here" file.  The second execution of
'readtkn' is getting an end of file indication because the first one
really did eat all the characters.

The remedy is simple.  Only your program needs to change.
Before reading from stdin, do
	setbuf(stdin, (char*)NULL);	/* all flavours */
or	setlinebuf(stdin);		/* BSD */
or use setvbuf in System V.