zdenko@katzo.rice.edu (zdenko tomasic) (06/11/91)
Does anybody know how to change the pipe buffer PIPE_BUF? I suspect that it has a value of 512 bytes (POSIX conformance?), but according to /usr/include/sys/limits.h, it can be increased to 32KB at run-time. I was not able to find out from info how to achieve that. Default pipe behavior in AIX is rather inefficient with too many context switches. Thanks for your attention. -- ___________________________________________________________________ Zdenko Tomasic, Rice U., Chem. Dept., P.O. Box 1892, Houston, Tx 77251 INTERNET: zdenko@katzo.rice.edu ___________________________________________________________________
john@crcaus.cactus.org (John R. Miller) (06/12/91)
In article <1991Jun10.185820.8120@rice.edu> zdenko@katzo.rice.edu (zdenko tomasic) writes: >Does anybody know how to change the pipe buffer PIPE_BUF? It can't be changed. >... I suspect >that it has a value of 512 bytes (POSIX conformance?), nope >... but according >to /usr/include/sys/limits.h, it can be increased to 32KB at >run-time. I was not able to find out from info how to achieve that. It's always 32k. You can demonstrate this by doing a blocking write of 32k to a pipe that has one byte in it already. The write will block because AIX guarantees it will deliver writes of less than 32k atomically, but it can't because there isn't enough room in the pipe. If the pipe is empty the write will succeed, because there is enough room to write it all at once. If the write is larger than 32k the write will succeed but it won't be delivered to the pipe (and thus readers) atomically. Atomic writes aren't guaranteed for writes larger than 32k. >Default pipe behavior in AIX is rather inefficient with too many >context switches. If you are seeing too many context switches, I don't think it's caused by pipe buffers filling up. disclaimer: I've been wrong before. -- John R. Miller 13102 Briar Hollow Dr. Austin, Texas 78729 hm: 512/331-0155 john@crcaus.cactus.org or ..cs.utexas.edu!bigtex!crcaus!john wk: 512/823-3867 john@glasnost.austin.ibm.com
zdenko@katzo.rice.edu (zdenko tomasic) (06/15/91)
In article <320@crcaus.cactus.org> john@crcaus.UUCP (John R. Miller) writes: > > .... good points on PIPE_BUF ... > ... >If you are seeing too many context switches, I don't think it's caused ^^^^^^^^^^^^^^^^ >by pipe buffers filling up. Actually, I should mention that the whole thing started when I just let the shell set up the pipe for 2 user filters. The context switches were excessive (in millions) and the cpu time was way up just for the input part of the receiving end of the pipe (I did not measure the sending filter timing). After playing around with pipes set up explicitly through a C program and changing the pipe buffer sized with setbuffer (Chuck Karsh was kind to point that in his e-mail to me), I could push the number of context switches up by reducing the pipe buffer size. Since the pipe size buffer of less than 80 bytes roughly gives the number of context switches in the right ballpark, I conclude that the (csh,ksh) shells use line buffering for pipes, i.e. pipe buffers are then rather small --> too many context switches. I guess somebody assumed that stdin and stdout will allways involve terminal io and left everything line buffered at all times. The work-around is to make your own pipes and avoid shell pipes. I initially assumed (mislead by comments in the /usr/include/sys/limits.h) that 512 was the default PIPE_BUF and I am convinced now that it is not so (and even if it were it would not produce the observed number of context switches). I hope somebody could verify the above and change shell pipe buffering as other systems (e.g. MIPS) do not have such problems. There is no reason to make piping through shell as expensive as it is now, is there? > >disclaimer: I've been wrong before. me, too! > >-- >John R. Miller 13102 Briar Hollow Dr. Austin, Texas 78729 >hm: 512/331-0155 john@crcaus.cactus.org or ..cs.utexas.edu!bigtex!crcaus!john >wk: 512/823-3867 john@glasnost.austin.ibm.com -- ___________________________________________________________________ Zdenko Tomasic, Rice U., Chem. Dept., P.O. Box 1892, Houston, Tx 77251 INTERNET: zdenko@katzo.rice.edu ___________________________________________________________________