bert@let.rug.nl (Bert Bos) (02/27/91)
I tried to fork a process and redirect both input and output to pipes. Both processes could work in parallel and as soon as one of the two had some text ready, it would be written out on the pipe for the other process to read. One of the processes could even be a program such as Awk (started with exec()). Problem is, the pipes appear to buffer such large amounts of text, that one process only gets input after the other has already finished. How do I force the pipes to pass on text one line at a time? I'm using calls such as pipe(), dup2(), select(), read(), etc, but I couldn't find anything in the manuals. -- "Always remember, however, that there's Bert Bos (bert@let.rug.nl) usually a simpler and better way to do Alfa-informatica something than the first way that pops RijksUniversiteit Groningen into your head." (D.E. Knuth, TeXbook) Groningen, The Netherlands
tchrist@convex.COM (Tom Christiansen) (02/27/91)
From the keyboard of bert@let.rug.nl (Bert Bos): :I tried to fork a process and redirect both input and output to pipes. :Both processes could work in parallel and as soon as one of the two :had some text ready, it would be written out on the pipe for the other :process to read. One of the processes could even be a program such as :Awk (started with exec()). : :Problem is, the pipes appear to buffer such large amounts of text, :that one process only gets input after the other has already finished. :How do I force the pipes to pass on text one line at a time? You must have control of the program that's doing the output. If you don't have that program's source code, you're out of luck. Many programs use stdio, and for these you must force a flush as soon as want the output to show up. From C, you can use fflush(3S) now and then, or set buffering to be line- or un-buffered. See the setbuf(3S) and related functions. This is how it's usually done. Now, in the case of awk, you're doubly out of luck: even with the source code, you aren't going to be able to make it flush its buffers when you want, because there's no way for awk to do that. Ok, I take that back: you run it through the awk-to-perl translator, because in perl it's easy to flush your stdio buffers. In fact, one person posted a short sh/awk script a few weeks ago with just your problem. The perl solution solved his problem. Ok, I'll take it back again. If you run your program under a debugger, you can probably hammer the stdio flags. Depending on your implementation you might even be able to patch the binary. But these are really very last ditch efforts, and I wouldn't want to admit to having done them. :-) --tom -- "UNIX was not designed to stop you from doing stupid things, because that would also stop you from doing clever things." -- Doug Gwyn Tom Christiansen tchrist@convex.com convex!tchrist
brnstnd@kramden.acf.nyu.edu (Dan Bernstein) (02/27/91)
In article <1991Feb27.014643.7915@convex.com> tchrist@convex.COM (Tom Christiansen) writes: > : How do I force the pipes to pass on text one line at a time? > You must have control of the program that's doing the output. If you > don't have that program's source code, you're out of luck. Uh, no. On BSD machines you just run ``pty program'' instead of ``program'', and if ``program'' uses stdio it will buffer as if it were talking to a tty. This solves the problem at hand. If you don't have the program's source code, and you're using System V, you're out of luck. ---Dan
gwyn@smoke.brl.mil (Doug Gwyn) (02/28/91)
In article <1585@gufalet.let.rug.nl> bert@let.rug.nl (Bert Bos) writes: >Problem is, the pipes appear to buffer such large amounts of text, >that one process only gets input after the other has already finished. >How do I force the pipes to pass on text one line at a time? >I'm using calls such as pipe(), dup2(), select(), read(), etc, but I >couldn't find anything in the manuals. Since (unless your pipe implementation is horribly broken) each write() produces an independent chunk in the pipeline, I would guess that what you are actually having trouble with is buffering in the applications, not in the kernel. If stdio is being used, be sure to fflush() the output to ensure that it is write()n to the pipe, and use setbuf() to disable input buffering on the reader. Obviously, if either or both of the communicating processes are provided by somebody else, you may not be able to change how they operate.
rbj@uunet.UU.NET (Root Boy Jim) (03/05/91)
In article <1585@gufalet.let.rug.nl> bert@let.rug.nl (Bert Bos) writes: >Problem is, the pipes appear to buffer such large amounts of text, >that one process only gets input after the other has already finished. >How do I force the pipes to pass on text one line at a time? I don't think that pipes are any different than plain files as far as buffering is concerned. If there is data available to read, it should be made available, byte by byte. >I'm using calls such as pipe(), dup2(), select(), read(), etc, but I >couldn't find anything in the manuals. Ah, but what are you using to write? Printf uses stdio, and will most likely be buffered. Use sprintf followed by write, or fflush to force data thru the pipe. -- [rbj@uunet 1] stty sane unknown mode: sane