knudsen@ihlpl.ATT.COM (Knudsen) (06/30/89)
I'd always understood that printf'ing any string ending in '\n' (newline) would flush the I/O buffer. However, I've found that this will not always flush if stdout has been redirected to a pipe. Possibly the failure is related to whether or not the byte preceding the newline is a printable ASCII character or not. I know about fflush(), which always works, but it can be a pain to call it in some cases. Does anyone know the "official" rules of the traditional stdio library? (Not the ANSII standard; I'm dealing with an older system). Thanks, mike k -- Mike Knudsen Bell Labs(AT&T) att!ihlpl!knudsen knudsen@ihlpl.att.com Round and round the while() loop goes; "Whether it stops," Turing says, "no one knows!" Shotguns -- just say PULL!
chris@mimsy.UUCP (Chris Torek) (07/02/89)
In article <11012@ihlpl.ATT.COM> knudsen@ihlpl.ATT.COM (Knudsen) writes: >I'd always understood that printf'ing any string ending in '\n' >(newline) would flush the I/O buffer. Nope. >However, I've found that this will not always flush if stdout has been >redirected to a pipe. Yep. Nor (on 4BSD Unix) a socket (there, a pipe *is* a socket), nor (on any Unix, and any pANS system) any ordinary file. >Does anyone know the "official" rules of the traditional stdio >library? The `official' rule is that `interactive' output (like that going to a terminal or a window) is supposed to be line buffered by default, and that any other output is supposed to be fully buffered by default. An application can decide for itself by using setbuf or setvbuf. I feel the best approach is for applications to set all output files to `fully buffered', and then to use fflush whenever the output is supposed to appear. This never produces surprises, and it also tends to run faster. -- In-Real-Life: Chris Torek, Univ of MD Comp Sci Dept (+1 301 454 7163) Domain: chris@mimsy.umd.edu Path: uunet!mimsy!chris
scs@adam.pika.mit.edu (Steve Summit) (07/02/89)
In article <11012@ihlpl.ATT.COM> knudsen@ihlpl.ATT.COM (Knudsen) writes: >I'd always understood that printf'ing any string ending in '\n' >(newline) would flush the I/O buffer. >However, I've found that this will not always flush if stdout has been >redirected to a pipe. > >I know about fflush(), which always works, but it can be a pain >to call it in some cases. Some will disagree, but I believe it is best to call fflush explicitly whenever the output should be seen right now, such as after prompts, and after intermediate output that precedes long compute-intensive activity. Others have also found fflush inconvenient, which is why various special cases have been added in attempt to obviate its use, such as that a newline can flush output, as can a pending read (actually, getchar or the equivalent) on stdin. (One or both of these behaviors is often referred to as "line buffering.") The trouble is, as you have already discovered, that line buffering is not sufficient, among other things because it applies, for efficiency reasons, only if isatty(fileno(stdout)), and not if stdout is a pipe. (Normally I deprecate counter-functional design decisions based solely on efficiency concerns, but in this case the decision is correct: if line buffering is to be turned on implicitly at all, it must be done so only for interactive invocations, and isatty is the closest convenient thing we have for determining interactivity. Stdio buffering exists in the first place at least partially for efficiency reasons: doing low-level, system-call I/O on a character-at-a-time basis is notoriously, and truly, inefficient, i.e. it does make a difference in practice. Line buffering partially removes the advantages of full buffering; it must not be turned on implicitly when batch-style jobs are generating large quantities of output redirected to a file.) I have rarely found it inconvenient to call fflush explicitly -- if a program asks the user more than a couple of questions, I usually have a centralized prompt-and-get-answer routine, and I can place a single call to fflush there. For cases other than prompts, there is usually one place within a main processing loop somewhere where an fflush can be added to make sure any pending output has been displayed. It is possible to turn off buffering entirely, if line buffering is inadequate and explicit fflush'es are somehow unworkable, although to do so is worse from a performance standpoint than not using stdio at all. That is, an unbuffered printf("Hello, world!\n") is even worse than write(1, "Hello, world!\n", 14) because the former will typically generate fourteen calls to write(), each passing one character. (This is why programs like tar with the v option perform poorly on heavily-loaded machines or over network connections; tar's verbose output is printed to stderr which is normally unbuffered.) It is rather straightforward to put special cases into the stdio code so that printf and the like on unbuffered streams call write() with as many characters as are available, without doing any buffering. Unfortunately, I have not seen a distributed version of stdio which does so. Steve Summit scs@adam.pika.mit.edu P.S. Note that it is unclear to talk about "the stdio buffer;" there is typically one buffer per open stream: stdin, stdout, and any fopen'ed streams.
gwyn@smoke.BRL.MIL (Doug Gwyn) (07/02/89)
In article <11012@ihlpl.ATT.COM> knudsen@ihlpl.ATT.COM (Knudsen) writes: >I'd always understood that printf'ing any string ending in '\n' >(newline) would flush the I/O buffer. >... >Does anyone know the "official" rules of the traditional stdio library? Yes, they're described in TFM. By default (i.e. unless setbuf() or setvbuf() is used to change it), terminal output is line-buffered, stderr is unbuffered, and all other I/O streams are fully (block) buffered.
henry@utzoo.uucp (Henry Spencer) (07/02/89)
In article <12375@bloom-beacon.MIT.EDU> scs@adam.pika.mit.edu (Steve Summit) writes: >... Line buffering partially removes the advantages of >full buffering; it must not be turned on implicitly when >batch-style jobs are generating large quantities of output >redirected to a file.) Unfortunately there is a further complication. Terminal output definitely should be line buffered or something similar. (One can argue that doing e.g. a single printf as one write, plus some care with fflush, ought to be sufficient and actual line buffering causes more trouble than it is worth.) Disk output pretty definitely should be fully buffered. But what about pipes? One really wants the final destination of output to propagate back up a pipeline, so earlier stages use buffering appropriate for the final destination. In particular, a pipeline whose output is to a terminal should *not* be doing full buffering at any stage. This isn't easy to do. It might be reasonable to default to line buffering for pipes, since modern Unixes very seldom implement them as anonymous disk files (for which block alignment is a significant performance issue) and line buffering gets you most of the performance win. -- $10 million equals 18 PM | Henry Spencer at U of Toronto Zoology (Pentagon-Minutes). -Tom Neff | uunet!attcan!utzoo!henry henry@zoo.toronto.edu
gwyn@smoke.BRL.MIL (Doug Gwyn) (07/02/89)
In article <1989Jul2.040259.3872@utzoo.uucp> henry@utzoo.uucp (Henry Spencer) writes: >It might be reasonable to default to line buffering for pipes, ... Line buffering has an additional flaw -- it requires that almost every character being output be tested against '\n'. As I recall, the original STDIO implementations did not treat "terminal devices" (i.e. those responding to tty-oriented ioctls) specially. Line buffering appears to have been added later for two main reasons: (1) to guarantee that prompts etc. were seen before requesting input from a terminal (2) unbuffered output was painfully slow As has already been noted in this discussion, applications can solve both problems by using full buffering and issuing an fflush() before an input request. That's actually what I prefer, and is essentially how my portable "Dx" IPC package (part of a large project currently under development) expects to be used. My belief is that line buffering is an attempt to automate a solution to a certain class of application design problems, without taking into account the possibility you brought up or other, similar difficulties with automating the determination of when to flush and when not to flush output buffers. As seems to always happen, the attempt to make system software "smart" fails because it takes genuine intelligence to really solve the problem right, and automated algorithms following fixed rules cannot properly deal with the entire spectrum of possibilities. The proposed ANSI Standard for C suggests (sort of a soft requirement) what are essentially the current UNIX System V STDIO rules for line-buffered streams: not only should the output buffer be flushed when a new-line is encountered or the buffer fills up, but also ALL line-buffered output streams should be flushed whenever input is requested from an unbuffered stream or when line-buffered input has to refill its buffer. The cost of implementing this can be substantial! Since it is only a "soft" requirement, I wouldn't mind at all if implementations claimed that "interactive devices" never exist, thereby obtaining full buffering by default except for stderr. Portable applications need to issue explicit fflush()es at the appropriate times anyway, so we might as well be using full buffering all the time.
peter@ficc.uu.net (Peter da Silva) (07/03/89)
In article <10487@smoke.BRL.MIL>, gwyn@smoke.BRL.MIL (Doug Gwyn) writes: > The proposed ANSI Standard for C suggests (sort of a soft requirement) > what are essentially the current UNIX System V STDIO rules for line-buffered > streams: not only should the output buffer be flushed when a new-line is > encountered or the buffer fills up, but also ALL line-buffered output streams > should be flushed whenever input is requested from an unbuffered stream or > when line-buffered input has to refill its buffer. The cost of implementing > this can be substantial! No, it is not. In fact, if (popping down to the implementations I'm familiar with) _filbuf(stdin) calls _flsbuf(stdout) you get all the advantages of line-buffered output, obviate the need for most explicit fflush() calls, and have almost no overhead. This behaviour was first implemented (to my knowledge) in the Berkeley stdio library, and is (I believe) what the dpANS is referring to. In fact, if you implement smart flushing, you can abandon the overhead of line buffering completely, without breaking any working programs. -- Peter da Silva, Xenix Support, Ferranti International Controls Corporation. Business: uunet.uu.net!ficc!peter, peter@ficc.uu.net, +1 713 274 5180. Personal: ...!texbell!sugar!peter, peter@sugar.hackercorp.com.
dhesi@bsu-cs.bsu.edu (Rahul Dhesi) (07/04/89)
In article <18351@mimsy.UUCP> chris@mimsy.UUCP (Chris Torek) writes:
I feel the best approach is for applications to set all output files
to `fully buffered', and then to use fflush whenever the output is
supposed to appear. This never produces surprises...
...except under VAX/VMS which, when you run a batch job, interprets
fflush(stdout);
to mean:
printf("\n");
--
Rahul Dhesi <dhesi@bsu-cs.bsu.edu>
UUCP: ...!{iuvax,pur-ee}!bsu-cs!dhesi
foessmei@lan.informatik.tu-muenchen.dbp.de (Reinhard Foessmeier) (07/05/89)
In article <11012@ihlpl.ATT.COM> knudsen@ihlpl.ATT.COM (Knudsen) writes: >I'd always understood that printf'ing any string ending in '\n' >(newline) would flush the I/O buffer. ... >Does anyone know the "official" rules of the traditional stdio >library? (Not the ANSII standard; I'm dealing with an older >system). Thanks, mike k The System V manual says (setbuf (3) that "by default, output to a terminal line is buffered and all other input/output is fully buffered". So, the behavior observed by you seems to be correct standard. Reinhard F\"ossmeier, Technische Univ. M\"unchen | "Lasciate ogni speranza, foessmeier@infovax.informatik.tu-muenchen.dbp.de | voi che entrate!" [ { relay.cs.net | unido.uucp } ] | (Dante, Inferno)
djones@megatest.UUCP (Dave Jones) (07/06/89)
From article <8045@bsu-cs.bsu.edu), by dhesi@bsu-cs.bsu.edu (Rahul Dhesi):
) In article <18351@mimsy.UUCP) chris@mimsy.UUCP (Chris Torek) writes:
) I feel the best approach is for applications to set all output files
) to `fully buffered', and then to use fflush whenever the output is
) supposed to appear. This never produces surprises...
)
) ...except under VAX/VMS which, when you run a batch job, interprets
)
) fflush(stdout);
)
) to mean:
)
) printf("\n");
) --
) Rahul Dhesi <dhesi@bsu-cs.bsu.edu)
) UUCP: ...!{iuvax,pur-ee}!bsu-cs!dhesi
This came up a year or two ago, and somebody said that it was
going to be "fixed in the next release". Do you have the latest
and greatest?
diamond@diamond.csl.sony.junet (Norman Diamond) (07/06/89)
In article <11012@ihlpl.ATT.COM> knudsen@ihlpl.ATT.COM (Knudsen) writes: >I'd always understood that printf'ing any string ending in '\n' >(newline) would flush the I/O buffer. On the BSD 4.3 system I use, fprintf to --> stderr <--, with stderr defaulted to a --> terminal <--, and the string ending in --> '\n' <--, STILL doesn't flush automatically. I learned fflush very quickly. -- Norman Diamond, Sony Computer Science Lab (diamond%csl.sony.jp@relay.cs.net) The above opinions are claimed by your machine's init process (pid 1), after being disowned and orphaned. However, if you see this at Waterloo, Stanford, or Anterior, then their administrators must have approved of these opinions.
gwyn@smoke.BRL.MIL (Doug Gwyn) (07/06/89)
In article <4861@ficc.uu.net> peter@ficc.uu.net (Peter da Silva) writes: >>The cost of implementing this can be substantial! >No, it is not. In fact, if ... _filbuf(stdin) calls _flsbuf(stdout) ... It is not just stdout, but ALL terminal-associated streams that are supposed to be flushed. Also, during output it is necessary to check each character against new-line to implement line buffering. I stand by my assessment that line buffering is a mistake.
peter@ficc.uu.net (Peter da Silva) (07/09/89)
In article <10490@smoke.BRL.MIL>, gwyn@smoke.BRL.MIL (Doug Gwyn) writes: > In article <4861@ficc.uu.net> peter@ficc.uu.net (Peter da Silva) writes: > >>The cost of implementing this can be substantial! > >No, it is not. In fact, if ... _filbuf(stdin) calls _flsbuf(stdout) ... > It is not just stdout, but ALL terminal-associated streams that are > supposed to be flushed. A little odd, but defensible. I don't know if this is a mistake or not, but I'd like to ask how many terminal-associated streams the majority of applications open... > Also, during output it is necessary to check > each character against new-line to implement line buffering. Only if you want to implement line-buffering. Smart buffering (where you flush output buffers when you go to fill input buffers) almost completely removes any need for line buffering, and radically reduces the need for explicit fflush()ing. > I stand by my assessment that line buffering is a mistake. I agree. Line buffering is a mistake. But if you have a look at what I wrote you'll see I didn't advocate line buffering. -- Peter da Silva, Xenix Support, Ferranti International Controls Corporation. Business: peter@ficc.uu.net, +1 713 274 5180. | "WHAT HAPPENED TO ALL Personal: peter@sugar.hackercorp.com. `-_-' | THE WOMEN IN TEXAS?" Quote: Have you hugged your wolf today? 'U` | -- ACS1W@jane.uh.edu (meesh)
meissner@tiktok.dg.com (Michael Meissner) (07/16/89)
In article <4861@ficc.uu.net> peter@ficc.uu.net (Peter da Silva) writes: | No, it is not. In fact, if (popping down to the implementations I'm familiar | with) _filbuf(stdin) calls _flsbuf(stdout) you get all the advantages of | line-buffered output, obviate the need for most explicit fflush() calls, and | have almost no overhead. This behaviour was first implemented (to my | knowledge) in the Berkeley stdio library, and is (I believe) what the dpANS | is referring to. | | In fact, if you implement smart flushing, you can abandon the overhead of | line buffering completely, without breaking any working programs. This is not completely true. I've run into programs, which use stdio for output, and raw read's for input. In this case, the output is NOT flushed. -- Michael Meissner, Data General. Uucp: ...!mcnc!rti!xyzzy!meissner If compiles were much Internet: meissner@dg-rtp.DG.COM faster, when would we Old Internet: meissner%dg-rtp.DG.COM@relay.cs.net have time for netnews?