[comp.lang.c] Does NEWLINE always flush stdio buffer?

knudsen@ihlpl.ATT.COM (Knudsen) (06/30/89)

I'd always understood that printf'ing any string ending in '\n'
(newline) would flush the I/O buffer.
However, I've found that this will not always flush if stdout has been
redirected to a pipe.

Possibly the failure is related to whether or not the byte preceding
the newline is a printable ASCII character or not.
I know about fflush(), which always works, but it can be a pain
to call it in some cases.

Does anyone know the "official" rules of the traditional stdio
library?  (Not the ANSII standard; I'm dealing with an older
system).  Thanks, mike k
-- 
Mike Knudsen  Bell Labs(AT&T)   att!ihlpl!knudsen  knudsen@ihlpl.att.com
  Round and round the while() loop goes;
  "Whether it stops," Turing says, "no one knows!"
Shotguns -- just say PULL!

chris@mimsy.UUCP (Chris Torek) (07/02/89)

In article <11012@ihlpl.ATT.COM> knudsen@ihlpl.ATT.COM (Knudsen) writes:
>I'd always understood that printf'ing any string ending in '\n'
>(newline) would flush the I/O buffer.

Nope.

>However, I've found that this will not always flush if stdout has been
>redirected to a pipe.

Yep.  Nor (on 4BSD Unix) a socket (there, a pipe *is* a socket), nor
(on any Unix, and any pANS system) any ordinary file.

>Does anyone know the "official" rules of the traditional stdio
>library?

The `official' rule is that `interactive' output (like that going to
a terminal or a window) is supposed to be line buffered by default,
and that any other output is supposed to be fully buffered by default.
An application can decide for itself by using setbuf or setvbuf.

I feel the best approach is for applications to set all output files
to `fully buffered', and then to use fflush whenever the output is
supposed to appear.  This never produces surprises, and it also tends
to run faster.
-- 
In-Real-Life: Chris Torek, Univ of MD Comp Sci Dept (+1 301 454 7163)
Domain:	chris@mimsy.umd.edu	Path:	uunet!mimsy!chris

scs@adam.pika.mit.edu (Steve Summit) (07/02/89)

In article <11012@ihlpl.ATT.COM> knudsen@ihlpl.ATT.COM (Knudsen) writes:
>I'd always understood that printf'ing any string ending in '\n'
>(newline) would flush the I/O buffer.
>However, I've found that this will not always flush if stdout has been
>redirected to a pipe.
>
>I know about fflush(), which always works, but it can be a pain
>to call it in some cases.

Some will disagree, but I believe it is best to call fflush
explicitly whenever the output should be seen right now, such as
after prompts, and after intermediate output that precedes long
compute-intensive activity.

Others have also found fflush inconvenient, which is why various
special cases have been added in attempt to obviate its use, such
as that a newline can flush output, as can a pending read
(actually, getchar or the equivalent) on stdin.  (One or both of
these behaviors is often referred to as "line buffering.")  The
trouble is, as you have already discovered, that line buffering
is not sufficient, among other things because it applies, for
efficiency reasons, only if isatty(fileno(stdout)), and not if
stdout is a pipe.

(Normally I deprecate counter-functional design decisions based
solely on efficiency concerns, but in this case the decision is
correct: if line buffering is to be turned on implicitly at all,
it must be done so only for interactive invocations, and isatty
is the closest convenient thing we have for determining
interactivity.  Stdio buffering exists in the first place at
least partially for efficiency reasons: doing low-level,
system-call I/O on a character-at-a-time basis is notoriously,
and truly, inefficient, i.e. it does make a difference in
practice.  Line buffering partially removes the advantages of
full buffering; it must not be turned on implicitly when
batch-style jobs are generating large quantities of output
redirected to a file.)

I have rarely found it inconvenient to call fflush explicitly --
if a program asks the user more than a couple of questions, I
usually have a centralized prompt-and-get-answer routine, and I
can place a single call to fflush there.  For cases other than
prompts, there is usually one place within a main processing loop
somewhere where an fflush can be added to make sure any pending
output has been displayed.

It is possible to turn off buffering entirely, if line buffering
is inadequate and explicit fflush'es are somehow unworkable,
although to do so is worse from a performance standpoint than not
using stdio at all.  That is, an unbuffered printf("Hello, world!\n")
is even worse than write(1, "Hello, world!\n", 14) because the
former will typically generate fourteen calls to write(), each
passing one character.  (This is why programs like tar with the v
option perform poorly on heavily-loaded machines or over network
connections; tar's verbose output is printed to stderr which is
normally unbuffered.)

It is rather straightforward to put special cases into the stdio
code so that printf and the like on unbuffered streams call
write() with as many characters as are available, without doing
any buffering.  Unfortunately, I have not seen a distributed
version of stdio which does so.

                                            Steve Summit
                                            scs@adam.pika.mit.edu

P.S.  Note that it is unclear to talk about "the stdio buffer;"
there is typically one buffer per open stream: stdin, stdout, and
any fopen'ed streams.

gwyn@smoke.BRL.MIL (Doug Gwyn) (07/02/89)

In article <11012@ihlpl.ATT.COM> knudsen@ihlpl.ATT.COM (Knudsen) writes:
>I'd always understood that printf'ing any string ending in '\n'
>(newline) would flush the I/O buffer.
>...
>Does anyone know the "official" rules of the traditional stdio library?

Yes, they're described in TFM.  By default (i.e. unless setbuf() or
setvbuf() is used to change it), terminal output is line-buffered,
stderr is unbuffered, and all other I/O streams are fully (block)
buffered.

henry@utzoo.uucp (Henry Spencer) (07/02/89)

In article <12375@bloom-beacon.MIT.EDU> scs@adam.pika.mit.edu (Steve Summit) writes:
>... Line buffering partially removes the advantages of
>full buffering; it must not be turned on implicitly when
>batch-style jobs are generating large quantities of output
>redirected to a file.)

Unfortunately there is a further complication.  Terminal output definitely
should be line buffered or something similar.  (One can argue that doing
e.g. a single printf as one write, plus some care with fflush, ought to
be sufficient and actual line buffering causes more trouble than it is
worth.)  Disk output pretty definitely should be fully buffered.  But
what about pipes?  One really wants the final destination of output to
propagate back up a pipeline, so earlier stages use buffering appropriate
for the final destination.  In particular, a pipeline whose output is to
a terminal should *not* be doing full buffering at any stage.  This isn't
easy to do.  It might be reasonable to default to line buffering for pipes,
since modern Unixes very seldom implement them as anonymous disk files
(for which block alignment is a significant performance issue) and line
buffering gets you most of the performance win.
-- 
$10 million equals 18 PM       |     Henry Spencer at U of Toronto Zoology
(Pentagon-Minutes). -Tom Neff  | uunet!attcan!utzoo!henry henry@zoo.toronto.edu

gwyn@smoke.BRL.MIL (Doug Gwyn) (07/02/89)

In article <1989Jul2.040259.3872@utzoo.uucp> henry@utzoo.uucp (Henry Spencer) writes:
>It might be reasonable to default to line buffering for pipes, ...

Line buffering has an additional flaw -- it requires that almost every
character being output be tested against '\n'.

As I recall, the original STDIO implementations did not treat "terminal
devices" (i.e. those responding to tty-oriented ioctls) specially.  Line
buffering appears to have been added later for two main reasons:
	(1) to guarantee that prompts etc. were seen before requesting
	    input from a terminal
	(2) unbuffered output was painfully slow

As has already been noted in this discussion, applications can solve both
problems by using full buffering and issuing an fflush() before an input
request.  That's actually what I prefer, and is essentially how my portable
"Dx" IPC package (part of a large project currently under development)
expects to be used.  My belief is that line buffering is an attempt to
automate a solution to a certain class of application design problems,
without taking into account the possibility you brought up or other,
similar difficulties with automating the determination of when to flush
and when not to flush output buffers.  As seems to always happen, the
attempt to make system software "smart" fails because it takes genuine
intelligence to really solve the problem right, and automated algorithms
following fixed rules cannot properly deal with the entire spectrum of
possibilities.

The proposed ANSI Standard for C suggests (sort of a soft requirement)
what are essentially the current UNIX System V STDIO rules for line-buffered
streams: not only should the output buffer be flushed when a new-line is
encountered or the buffer fills up, but also ALL line-buffered output streams
should be flushed whenever input is requested from an unbuffered stream or
when line-buffered input has to refill its buffer.  The cost of implementing
this can be substantial!  Since it is only a "soft" requirement, I wouldn't
mind at all if implementations claimed that "interactive devices" never
exist, thereby obtaining full buffering by default except for stderr.
Portable applications need to issue explicit fflush()es at the appropriate
times anyway, so we might as well be using full buffering all the time.

peter@ficc.uu.net (Peter da Silva) (07/03/89)

In article <10487@smoke.BRL.MIL>, gwyn@smoke.BRL.MIL (Doug Gwyn) writes:
> The proposed ANSI Standard for C suggests (sort of a soft requirement)
> what are essentially the current UNIX System V STDIO rules for line-buffered
> streams: not only should the output buffer be flushed when a new-line is
> encountered or the buffer fills up, but also ALL line-buffered output streams
> should be flushed whenever input is requested from an unbuffered stream or
> when line-buffered input has to refill its buffer.  The cost of implementing
> this can be substantial!

No, it is not. In fact, if (popping down to the implementations I'm familiar
with) _filbuf(stdin) calls _flsbuf(stdout) you get all the advantages of
line-buffered output, obviate the need for most explicit fflush() calls, and
have almost no overhead. This behaviour was first implemented (to my
knowledge) in the Berkeley stdio library, and is (I believe) what the dpANS
is referring to.

In fact, if you implement smart flushing, you can abandon the overhead of
line buffering completely, without breaking any working programs.
-- 
Peter da Silva, Xenix Support, Ferranti International Controls Corporation.

Business: uunet.uu.net!ficc!peter, peter@ficc.uu.net, +1 713 274 5180.
Personal: ...!texbell!sugar!peter, peter@sugar.hackercorp.com.

dhesi@bsu-cs.bsu.edu (Rahul Dhesi) (07/04/89)

In article <18351@mimsy.UUCP> chris@mimsy.UUCP (Chris Torek) writes:
   I feel the best approach is for applications to set all output files
   to `fully buffered', and then to use fflush whenever the output is
   supposed to appear.  This never produces surprises...

...except under VAX/VMS which, when you run a batch job, interprets

     fflush(stdout);
	  
to mean:

     printf("\n");
-- 
Rahul Dhesi <dhesi@bsu-cs.bsu.edu>
UUCP:    ...!{iuvax,pur-ee}!bsu-cs!dhesi

foessmei@lan.informatik.tu-muenchen.dbp.de (Reinhard Foessmeier) (07/05/89)

In article <11012@ihlpl.ATT.COM> knudsen@ihlpl.ATT.COM (Knudsen) writes:
>I'd always understood that printf'ing any string ending in '\n'
>(newline) would flush the I/O buffer.
 ...
>Does anyone know the "official" rules of the traditional stdio
>library?  (Not the ANSII standard; I'm dealing with an older
>system).  Thanks, mike k

The System V manual says (setbuf (3) that "by default, output to a terminal
line is buffered and all other input/output is fully buffered".  So, the
behavior observed by you seems to be correct standard.

Reinhard F\"ossmeier, Technische Univ. M\"unchen |  "Lasciate ogni speranza,
foessmeier@infovax.informatik.tu-muenchen.dbp.de |      voi che entrate!"
   [ { relay.cs.net | unido.uucp } ]             |      (Dante, Inferno)

djones@megatest.UUCP (Dave Jones) (07/06/89)

From article <8045@bsu-cs.bsu.edu), by dhesi@bsu-cs.bsu.edu (Rahul Dhesi):
) In article <18351@mimsy.UUCP) chris@mimsy.UUCP (Chris Torek) writes:
)    I feel the best approach is for applications to set all output files
)    to `fully buffered', and then to use fflush whenever the output is
)    supposed to appear.  This never produces surprises...
) 
) ...except under VAX/VMS which, when you run a batch job, interprets
) 
)      fflush(stdout);
) 	  
) to mean:
) 
)      printf("\n");
) -- 
) Rahul Dhesi <dhesi@bsu-cs.bsu.edu)
) UUCP:    ...!{iuvax,pur-ee}!bsu-cs!dhesi


This came up a year or two ago, and somebody said that it was
going to be "fixed in the next release".  Do you have the latest
and greatest?

diamond@diamond.csl.sony.junet (Norman Diamond) (07/06/89)

In article <11012@ihlpl.ATT.COM> knudsen@ihlpl.ATT.COM (Knudsen) writes:

>I'd always understood that printf'ing any string ending in '\n'
>(newline) would flush the I/O buffer.

On the BSD 4.3 system I use, fprintf to --> stderr <--, with stderr
defaulted to a --> terminal <--, and the string ending in --> '\n' <--,
STILL doesn't flush automatically.  I learned fflush very quickly.

--
Norman Diamond, Sony Computer Science Lab (diamond%csl.sony.jp@relay.cs.net)
 The above opinions are claimed by your machine's init process (pid 1), after
 being disowned and orphaned.  However, if you see this at Waterloo, Stanford,
 or Anterior, then their administrators must have approved of these opinions.

gwyn@smoke.BRL.MIL (Doug Gwyn) (07/06/89)

In article <4861@ficc.uu.net> peter@ficc.uu.net (Peter da Silva) writes:
>>The cost of implementing this can be substantial!
>No, it is not. In fact, if ... _filbuf(stdin) calls _flsbuf(stdout) ...

It is not just stdout, but ALL terminal-associated streams that are
supposed to be flushed.  Also, during output it is necessary to check
each character against new-line to implement line buffering.

I stand by my assessment that line buffering is a mistake.

peter@ficc.uu.net (Peter da Silva) (07/09/89)

In article <10490@smoke.BRL.MIL>, gwyn@smoke.BRL.MIL (Doug Gwyn) writes:
> In article <4861@ficc.uu.net> peter@ficc.uu.net (Peter da Silva) writes:
> >>The cost of implementing this can be substantial!
> >No, it is not. In fact, if ... _filbuf(stdin) calls _flsbuf(stdout) ...

> It is not just stdout, but ALL terminal-associated streams that are
> supposed to be flushed.

A little odd, but defensible.

I don't know if this is a mistake or not, but I'd like to ask how many
terminal-associated streams the majority of applications open...

> Also, during output it is necessary to check
> each character against new-line to implement line buffering.

Only if you want to implement line-buffering. Smart buffering (where you
flush output buffers when you go to fill input buffers) almost completely
removes any need for line buffering, and radically reduces the need for
explicit fflush()ing.

> I stand by my assessment that line buffering is a mistake.

I agree. Line buffering is a mistake. But if you have a look at what I wrote
you'll see I didn't advocate line buffering.
-- 
Peter da Silva, Xenix Support, Ferranti International Controls Corporation.
Business: peter@ficc.uu.net, +1 713 274 5180. | "WHAT HAPPENED TO ALL
Personal: peter@sugar.hackercorp.com.   `-_-' |  THE WOMEN IN TEXAS?"
Quote: Have you hugged your wolf today?  'U`  |  -- ACS1W@jane.uh.edu (meesh)

meissner@tiktok.dg.com (Michael Meissner) (07/16/89)

In article <4861@ficc.uu.net> peter@ficc.uu.net (Peter da Silva) writes:
| No, it is not. In fact, if (popping down to the implementations I'm familiar
| with) _filbuf(stdin) calls _flsbuf(stdout) you get all the advantages of
| line-buffered output, obviate the need for most explicit fflush() calls, and
| have almost no overhead. This behaviour was first implemented (to my
| knowledge) in the Berkeley stdio library, and is (I believe) what the dpANS
| is referring to.
| 
| In fact, if you implement smart flushing, you can abandon the overhead of
| line buffering completely, without breaking any working programs.

This is not completely true.  I've run into programs, which use stdio
for output, and raw read's for input.  In this case, the output is NOT
flushed.
--
Michael Meissner, Data General.
Uucp:		...!mcnc!rti!xyzzy!meissner		If compiles were much
Internet:	meissner@dg-rtp.DG.COM			faster, when would we
Old Internet:	meissner%dg-rtp.DG.COM@relay.cs.net	have time for netnews?