wayne@ames.arc.nasa.gov (Wayne Hathaway) (06/10/88)
I hate to bother Wizards with what is probably a very simple question,
but it stumps me. I have this very simple program that periodically
does an fprintf(stderr,...) to indicate progress. The messages show
up as written, as expected. Now I want to pipe this output through
awk. This works fine, EXCEPT that all the lines are buffered in the
originating process until it exits, then they all blast out at once.
I realize I can put fflush()s in the original program, but isn't that
against the "filters" philosophy? I mean, if it doesn't buffer when
writing to the terminal, why does it buffer when writing to a pipe?
I have done nothing special in the program about buffering, by the
way, and the environment is SunOS 3.3.
Thanx for any enlightenment!
Wayne Hathaway ultra!wayne@Ames.ARPA
Ultra Network Technologies
2140 Bering drive with a domain server:
San Jose, CA 95131 wayne@Ultra.COM
408-922-0100gwyn@brl-smoke.ARPA (Doug Gwyn ) (06/10/88)
In article <16124@brl-adm.ARPA> ultra!wayne@ames.arc.nasa.gov (Wayne Hathaway) writes: >I mean, if it doesn't buffer when writing to the terminal, why does >it buffer when writing to a pipe? People keep dorking around with stdio buffering in an attempt to be "helpful". As usual, it just messes things up in some cases. stderr is supposed to start out unbuffered, ALWAYS. It doesn't, though, on SunOS. A way to fix this is to setbuf(stderr,(char*)0); at the beginning of your program.
lm@arizona.edu (Larry McVoy) (06/10/88)
In article <8063@brl-smoke.ARPA> gwyn@brl.arpa (Doug Gwyn (VLD/VMB) <gwyn>) writes: >stderr is supposed to start out unbuffered, ALWAYS. It doesn't, >though, on SunOS. WHAT?!?!?!?! Why the hell not? Whose bright idea was that? And why don't they unfix it? -- Larry McVoy lm@arizona.edu or ...!{uwvax,sun}!arizona.edu!lm
gp@picuxa.UUCP (Greg Pasquariello X1190) (06/10/88)
In article <16124@brl-adm.ARPA> ultra!wayne@ames.arc.nasa.gov (Wayne Hathaway) writes: >I hate to bother Wizards with what is probably a very simple question, >but it stumps me. I have this very simple program that periodically >does an fprintf(stderr,...) to indicate progress. > > Wayne Hathaway ultra!wayne@Ames.ARPA You can turn buffering off for stderr (or any stream) with setbuf(). The syntax for stderr is setbuf(stderr, (char *) NULL). (where NULL is 0). -- ========================================================================= Greg Pasquariello AT&T Product Integration Center ihnp4!picuxa!gp 299 Jefferson Rd, Parsippany, NJ 07054 =========================================================================
david@linc.cis.upenn.edu (David Feldman) (06/10/88)
A self proclaimed novice asked why stderr gets buffered when piped in Sun 0S 3.3. Well, I can give an answer based on Ultrix experience. When piping, stderr gets buffered so that it may be separated from stdout. Stdout goes through the pipe, and when it is closed, the stderr buffer gets flushed through the pipe. That is assuming you have redirected stderr through the pipe also. This is a documented feature, and I believe it is a csh thing. I can't remember off hand. I don't think fflush() will help, especially if it is done in csh. One way csh could implement this feature is to attach stderr to a file and then throw the file down the pipe when stdout closes. Any csh hackers know the details on this thing? I am guessin'. Dave Feldman david@linc.cis.upenn.edu
chris@mimsy.UUCP (Chris Torek) (06/11/88)
In article <4999@super.upenn.edu> david@linc.cis.upenn.edu (David Feldman) writes: >A self proclaimed novice asked why stderr gets buffered when piped in >Sun OS 3.3. > >Well, I can give an answer based on Ultrix experience. When piping, >stderr gets buffered so that it may be separated from stdout. No. Buffering is a user-level concept; piping and descriptor merging is a kernel level concept. The two are not supposed to to be mixed together. Stderr gets buffered because it improves performance and raises benchmark numbers. It also makes correct programs fail, and should not be done casually. >... That is assuming you have redirected stderr through the pipe also. >This is a documented feature, and I believe it is a csh thing. It has nothing to do with the shell being used. >I don't think fflush() will help ... It will. >... One way csh could implement this feature is to attach stderr to >a file and then throw the file down the pipe when stdout closes. This would generally destroy performance, as the shell would have to remain in the path of a pipeline. Currently `a |& b' simply runs a and b `together' (under csh, a is started first) with a's stdout and stderr (fds 1 and 2) both going to a pipe that is b's stdin (fd 0). The pipe is created with kernel concepts (file descriptors) and the programs are run without giving them any knowledge as to where those descriptors connect. -- In-Real-Life: Chris Torek, Univ of MD Comp Sci Dept (+1 301 454 7163) Domain: chris@mimsy.umd.edu Path: uunet!mimsy!chris
jfh@rpp386.UUCP (John F. Haugh II) (06/11/88)
the reason printf buffers when writing to a non-terminal is because it is cheaper (cpu-wise) to wait until an entire buffer is full. if you really want to see the output immediately regardless of the output destination, you must use fflush. this is a very common situation when using fprintf or printf for debugging a program which core dumps. - john.
gwyn@brl-smoke.ARPA (Doug Gwyn ) (06/11/88)
In article <4999@super.upenn.edu> david@linc.cis.upenn.edu.UUCP (David Feldman) writes: >Well, I can give an answer based on Ultrix experience. When piping, >stderr gets buffered so that it may be separated from stdout. Stdout >goes through the pipe, and when it is closed, the stderr buffer gets flushed >through the pipe. That is assuming you have redirected stderr through the >pipe also. This is a documented feature, and I believe it is a csh thing. What the hell are you talking about? The whole reason for the invention of the standard error output as distinct from standard output is to AVOID error output getting mixed with legitimate output in pipelines etc. I don't like csh, but I won't accuse it of directing standard error output into pipes. Indeed I think you'd have to work fairly hard to get it to do so. My guess for the reason that stderr is (line-)buffered on some BSD-derived systems is that Bill Shannon once thought it would be a good idea and just did it. (Apologies if I'm maligning him.) It is also possible that someone thought it would improve network performance for error output to be sent in large chunks rather than a character at a time.
roy@phri.UUCP (Roy Smith) (06/12/88)
gwyn@brl.arpa (Doug Gwyn (VLD/VMB) <gwyn>) writes: > I won't accuse [csh] of directing standard error output into pipes. > Indeed I think you'd have to work fairly hard to get it to do so. Depends on what you call "fairly hard"; "foo |& bar" does the trick. > someone thought it would improve network performance for error output to > be sent in large chunks rather than a character at a time. Any system which produces so much error output that you have to worry about how efficiently the error messages are delivered is getting far too many errors. Better you spend you time figuring out how to fix the system than deliver the error messages faster. -- Roy Smith, System Administrator Public Health Research Institute 455 First Avenue, New York, NY 10016 {allegra,philabs,cmcl2,rutgers}!phri!roy -or- phri!roy@uunet.uu.net
jas@rain.rtech.UUCP (Jim Shankland) (06/14/88)
In article <4999@super.upenn.edu> david@linc.cis.upenn.edu.UUCP (David Feldman) writes: >When piping, stderr gets buffered so that it may be separated from stdout. >Stdout goes through the pipe, and when it is closed, the stderr buffer gets >flushed through the pipe. That is assuming you have redirected stderr through >the pipe also. This is a documented feature, and I believe it is a csh thing. >I can't remember off hand. I don't think fflush() will help, especially >if it is done in csh. One way csh could implement this feature is to >attach stderr to a file and then throw the file down the pipe when stdout >closes. Any csh hackers know the details on this thing? I am guessin'. I'll say. Jim Shankland ..!ihnp4!cpsc6a!\ sun!rtech!jas ..!ucbvax!mtxinu!/ "The road to hell ... is where the heart is"
rbj@ICST-CMR.ARPA (Root Boy Jim) (06/16/88)
? From: Roy Smith <roy@phri.uucp>
? gwyn@brl.arpa (Doug Gwyn (VLD/VMB) <gwyn>) writes:
?
? > I won't accuse [csh] of directing standard error output into pipes.
? > Indeed I think you'd have to work fairly hard to get it to do so.
?
? Depends on what you call "fairly hard"; "foo |& bar" does the trick.
Gee, Doug, RTFM! The 4.2 manual gives the example:
(command > stdout) >& stderr; for some reason it was dropped in the 4.3
manual. The same should work for pipes as well: (cmd1 > stdout) |& cmd2.
Maybe you should *use* something before you put it down.
? Roy Smith, System Administrator
? Public Health Research Institute
? 455 First Avenue, New York, NY 10016
? {allegra,philabs,cmcl2,rutgers}!phri!roy -or- phri!roy@uunet.uu.net
(Root Boy) Jim Cottrell <rbj@icst-cmr.arpa>
National Bureau of Standards
Flamer's Hotline: (301) 975-5688
The opinions expressed are solely my own
and do not reflect NBS policy or agreement
My name is in /usr/dict/words. Is yours?gwyn@brl-smoke.ARPA (Doug Gwyn ) (06/16/88)
In article <16176@brl-adm.ARPA> rbj@ICST-CMR.ARPA (Root Boy Jim) writes: >Gee, Doug, RTFM! ... for some reason it was dropped in the 4.3 manual. Which is what I have. R'ing TFM doesn't do much good in such cases. >Maybe you should *use* something before you put it down. >? > I won't accuse [csh] of directing standard error output into pipes. Far from "putting it down", here I was defending csh against what appeared (and still appears) to be an erroneous claim about it. >? > Indeed I think you'd have to work fairly hard to get it to do so. Several people pointed out that |& provides a convenient way to do this, so I was mistaken about how hard it was. It's nice, but not as useful as one might think, since the standard output and standard error output are in general unsynchronized. The data flowing down the pipeline can consequently look pretty messy.