mwm@eris.BERKELEY.EDU (Mike (My watch has windows) Meyer) (06/06/87)
In article <279@louie.udel.EDU> rminnich@udel.EDU (Ron Minnich) writes:
<6. Do MCC pipes work as named pipes (a la sys V), or just the
< equivalent of the unix '|'?
Uh, MCC pipes (and also Matt's pipe device, which is PD) are more than
the "|". They put pipes in the file system name space, where you can
use the for magic. To wit:
run edit from ram:input to pipe:edit-tmp with s:edit-script
run edit from ram:control to pipe:control-tmp with s:edit-control
edit from pipe:edit-tmp to ram:output with pipe:edit-control
Pipe:file is a pipe. Such things run an order of magniude faster than
going through ram:.
I *think* that's what SysV pipes do, but the closest I can come to a
SysV box is Unicos, which was based on SysV before there was a relase
number, and has been severely hacked upon. Someone wanna tell me what
named pipes really do?
Thanx,
<mike
--
How many times do you have to fall Mike Meyer
While people stand there gawking? mwm@berkeley.edu
How many times do you have to fall ucbvax!mwm
Before you end up walking? mwm@ucbjade.BITNETpeter@sugar.UUCP (06/10/87)
You create a named pipe with a mknod call. As a special case anyone can do this. The pipe is therefore just another special file. You can open it and write to it just like a file. You can open it and read from it similarly. Unlike PIPE: pipes, a reader doesn't hang on opening... it hangs on a read. Similarly a writer doesn't hang: it can just write its bytes and get out of there without waiting for a reader to show up. You also don't hang on close waiting for your coconspiritor to finish. Otherwise it's pretty much the same. It would be nice if the MCC PIPE: would handle more packets. For example, Examine and ExNext.
dillon@CORY.BERKELEY.EDU (Matt Dillon) (06/12/87)
>You create a named pipe with a mknod call. As a special case anyone can do >this. The pipe is therefore just another special file. You can open it >and write to it just like a file. You can open it and read from it similarly. >Unlike PIPE: pipes, a reader doesn't hang on opening... it hangs on a read. >Similarly a writer doesn't hang: it can just write its bytes and get out of >there without waiting for a reader to show up. You also don't hang on close >waiting for your coconspiritor to finish. Otherwise it's pretty much the same. > >It would be nice if the MCC PIPE: would handle more packets. For example, >Examine and ExNext. You are, fortunetly, wrong. I wrote PIPE:, and it neither blocks on Open() nor blocks on Write(). In fact, I certainly hope MCC pipes have a provision for non-infinite buffer sizes, or they are useless to me. Perhaps you are refering to a very early version of my PIPE:? Specifically, you can specify the buffer size for PIPE: pipes easily enough: 'pipe:name/b#'. If you want an infinite buffer size (writer NEVER blocks), then 'pipe:name/n' suffices. Without any options, the pipe uses a 4K buffer. For non infinite buffer sizes, my PIPE: allows the writer to write until the buffer is full, then waits for it to drain down to half before allowing the writer to write again. If the reader can keep up with the writer, then the writer never blocks. The reason for this is obvious: reduce the number of task switches while still attaining 100% CPU utilization. Now you may ask, 'why not just make it use infinite buffer sizes as a default'?? That's an easy question to answer. By example I will describe how I construct my VD0: disk from floppy power up: given A 600K file on the floppy containing the COMPRESSED, ARCHIVED VD0: (As in backup, not arc), I setup two pipes between three programs which will run concurrently. (A) A Copy from the 600K file to the first PIPE: specifying a relatively large, but not infinite, buffer size (I use a 128K buffer) (B) A DECOMPRESS from the first PIPE: to a second PIPE: using an infinite buffer size. (C) A de-archive (restore) from the second PIPE:. restore takes the decompressed archive file and reconstructs the entire VD0:. Now, I don't want to use an infinite buffer size for the first PIPE: because the Copy is *much* faster than decompress, and would basically get most of the 600K file in RAM: before decompress got through the first 50K. 600K + memory that decompress uses + a partially restored VD0: is quite a bit more than the measily 2.5Meg I've got. I can use an infinite buffer size for the second PIPE: because Restore can read data much faster than the decompressor can decompress it. The net result is 100% CPU utilization. What you see is 128K bytes read from the floppy, about a 10 second wait while uncompress reduces the pipe buffer to around 64K, then a 64K byte read from the floppy, another 5 seconds, another 64K byte read, etc... until all 600K has gone through. Meanwhile, uncompress has enjoyed *never* having to block when reading the compressed file from the first PIPE: In fact, that first PIPE: rarely gets smaller than 64K until Copy is finished reading the disk file. -Matt
hatcher@INGRES.BERKELEY.EDU.UUCP (06/12/87)
Both Peter & Matt's comments about pipes were quite interesting. Unfortunately Matt neglected to answer Peter's comment about wanting pipes to support Examine and Exnext. This is a very important feature. The ideal, which tends to be discovered and rediscovered endlessly, is to have a consistent name space where everything is a first class citizen. The reason is simple...when things are very consistent, then software tends to work automatically with objects that the author didn't have in mind a priori. The Amiga currently has *almost* consistent universal name space, as illustrated by the fact that Matt can write a pipe: device in the first place, and any software can write to it as if it were a regular file. But unlike other devices which contain file names, the pipe: device cannot be listed (Examine/Exnext etc). If it supported those packet types then it would behave a lot more like our other devices. A practical example: I have sometimes used more than one pipe at once (good old multitasking Amiga!), and after scrolling several CLI's and swapping windows around, wanted to check what was going on with the pipes. The ideal way would be to do "dir pipe:". A second problem was that I had gotten a writer going to a pipe, then it took a while to set up the reader, but I couldn't remember the name I'd used for the pipe (and the command had scrolled off the top of the CLI window). Again, being able to do "dir pipe:" would fix that. These are just a few examples; consistency and universality of interfaces are well established principles by now. There are many discussions of these principles in histories of Unix and Smalltalk, to pick two easy examples. One ignores history at ones own peril (and to the inconvenience of clients of your software). Oh, a final note: it's nice to have a root in which to look at a name space. That's why devices on Unix live in the file system in /dev -- it's not a peculiarity, it's a very basic feature of a uniform name space. It would be very nice if we could do "dir ::" or something and discover a list of all devices. Doug Merritt ucbvax!unisoft!certes!doug
dillon@CORY.BERKELEY.EDU (Matt Dillon) (06/12/87)
>Both Peter & Matt's comments about pipes were quite interesting. >Unfortunately Matt neglected to answer Peter's comment about wanting >pipes to support Examine and Exnext. I'm convinced already.... shouldn't be too hard to implement. -Matt
peter@sugar.UUCP (Peter DaSilva) (06/16/87)
In article <8706120539.AA16159@cory.Berkeley.EDU>, dillon@CORY.BERKELEY.EDU (Matt Dillon) writes: > >You create a named pipe with a mknod call. As a special case anyone can do > >this. The pipe is therefore just another special file. You can open it > >and write to it just like a file. You can open it and read from it similarly. > >Unlike PIPE: pipes, a reader doesn't hang on opening... it hangs on a read. > >Similarly a writer doesn't hang: it can just write its bytes and get out of > >there without waiting for a reader to show up. You also don't hang on close > >waiting for your coconspiritor to finish. Otherwise it's pretty much the same. > > > >It would be nice if the MCC PIPE: would handle more packets. For example, > >Examine and ExNext. > > You are, fortunetly, wrong. I wrote PIPE:, and it neither blocks > on Open() nor blocks on Write(). In fact, I certainly hope MCC pipes have > a provision for non-infinite buffer sizes, or they are useless to me. Perhaps > you are refering to a very early version of my PIPE:? I don't know what version I'm talking about. Did you write the PIPE: device in the MCC toolkit, as well as the one whose source is floating around (that is, are they the same program)? It may well be a very early version that I'm using here. Let me check: The posted date is 6th Feb 1987. The buffer size is stated to be 4K. It does direct write-through when writing to a pending read (nice touch). The source contains the notation that if you open-write<4K bytes-close, the close will wait until a reader opens the pipe. That's what I mean by "waiting for your co-conspiritor to finish". > Specifically, you can specify the buffer size for PIPE: pipes easily > enough: 'pipe:name/b#'. If you want an infinite buffer size (writer NEVER > blocks), then 'pipe:name/n' suffices. Without any options, the pipe uses a > 4K buffer. There doesn't seem to be any provision for this. > Now you may ask, 'why not just make it use infinite buffer sizes > as a default'?? That's an easy question to answer. By example I will Nope, I didn't ask this one. It's pretty obvious that yanking all of memory in a multitasking system isn't always appropriate. I'm not sure I would use the infinite buffer size feature at all. > VD0: is quite a bit more than the measily 2.5Meg I've got. I can use > an infinite buffer size for the second PIPE: because Restore can > read data much faster than the decompressor can decompress it. But because of this, the pipe will always be almost empty, so you might as well give it a 4K buffer size as well. It might be noted here that the UNIX pipes have a 4K buffer size. Now, a question and a request: Does your new PIPE: support the packets Examine and ExNext use? If so, then you could just "dir pipe:" to get a list of active pipes, right? The request: I want to use your PIPE: device as the skeleton for an improved console device. May I? How would I use it with Aztec (compatibility mode for sure, right)? Can I call Intuition from a device (opening windows, etc) or do I have to handcraft windows from the lower level primitives? I assume the former, but I've visited the Guru enough to know how often my assumptions are unwarranted.
peter@sugar.UUCP (Peter DaSilva) (06/16/87)
> space. That's why devices on Unix live in the file system in /dev > -- it's not a peculiarity, it's a very basic feature of a uniform > name space. It would be very nice if we could do "dir ::" or something > and discover a list of all devices. You can do an assign with no args and get the list. From a program, however, it's a bit more complex. Maybe when I finish my console window device I can add a device device. In the meantime, use the code in my STDFILE package (to be posted soon). I have come to prefer the Amiga device naming convention to the UNIX one... if only it used the UNIX path naming conventions with them.
hatcher@INGRES.BERKELEY.EDU (Doug Merritt) (06/24/87)
In article <187@sugar.UUCP> peter@sugar.UUCP (Peter DaSilva) writes: Me: >> space. That's why devices on Unix live in the file system in /dev >> -- it's not a peculiarity, it's a very basic feature of a uniform >> name space. It would be very nice if we could do "dir ::" or something >> and discover a list of all devices. Peter: :Summary: List devices? Just say Assign! :You can do an assign with no args and get the list. From a program, however, :it's a bit more complex. Naturally we all know that Assign gives you a list of devices, but that is not the point (BTW I have a C function that searches the list for a device of a given name, email to "ucbvax!unisoft!certes!doug" if you want it). The point was about a single "uniform name space", not whether the two separate name spaces could be accessed. This is an important design principle, and not one that I am inventing off the top of my head. Before you start disagreeing with me too strongly on the issue, it wouldn't hurt to look up some of the papers published on the subject that explain the issues far better than I can. There is a clear, concise paper on the subject by the venerable Rob Pike and P.J. Weinberger called "The Hideous Name" on pg 563 of the Summer 1985 Usenix Proceedings (although their paper is oriented towards network names, the ideas are universal enough to be applicable). There've been lots of others published too, but I don't have references handy. But I'll plod ahead in my own fashion for those of you who may be interested. :Maybe when I finish my console window device I can :add a device device. In the meantime, use the code in my STDFILE package :(to be posted soon). I have come to prefer the Amiga device naming convention :to the UNIX one... if only it used the UNIX path naming conventions with :them. All of this sounds good. I don't have any argument with keeping the Amiga device naming convention. I didn't mean that AmigaDos should have a filesystem just exactly like Unix, I just meant that at least with Unix you can list out devices in directories just like any other file. And your "device device" could support that. Change the syntax of my original suggestion to "dir devices:" if you like that better than "dir ::". Another aspect of a universal name space is the /procs directory in Unix System 3, that contains a list of active processes (that again can be listed with the "ls" command). I think this is a good idea. Why have many interfaces for programs to deal with, when you could give just one uniform interface? Well, unfortunately, sometimes you need custom interfaces. But there's no conflict on the Amiga. You just support all the basic things like Lock, Open, Read, Examine, ExNext for any and all objects (files, directories, devices, tasks, processes, etc), and then add other message types as necessary to provide for the special functionality of that particular object. Just make sure that it always responds to the minimal message set. Regards, Doug Merritt ucbvax!ingres!hatcher (thru Jun 28 only!) or ucbvax!unisoft!certes!doug
dillon@CORY.BERKELEY.EDU (Matt Dillon) (06/24/87)
>I don't know what version I'm talking about. Did you write the PIPE: device >in the MCC toolkit, as well as the one whose source is floating around (that >is, are they the same program)? It may well be a very early version that I'm >using here. Let me check: Slight misunderstanding... you were talking about the MCC PIPE:, and I was talking about my PIPE: (I did not write the MCC pipe... I'm not connected with them in any way). >> VD0: is quite a bit more than the measily 2.5Meg I've got. I can use >> an infinite buffer size for the second PIPE: because Restore can >> read data much faster than the decompressor can decompress it. > >But because of this, the pipe will always be almost empty, so you might as well >give it a 4K buffer size as well. It might be noted here that the UNIX pipes >have a 4K buffer size. If the program writing to the pipe has a large buffer size (8K is not uncommon), then it will block every time it does a write until the reader is ready. Thus, to completely ensure the program does not block (assuming the reader can read it faster), the pipe would have to be a bit larger... say 16K. >Now, a question and a request: > >Does your new PIPE: support the packets Examine and ExNext use? If so, then you >could just "dir pipe:" to get a list of active pipes, right? I've been trying... and trying... and trying... I can't figure out why it is crashing.... Can somebody post a *working* device driver which uses Examine/ExNext???? >The request: I want to use your PIPE: device as the skeleton for an improved >console device. May I? How would I use it with Aztec (compatibility mode for >sure, right)? Can I call Intuition from a device (opening windows, etc) or do >I have to handcraft windows from the lower level primitives? I assume the >former, but I've visited the Guru enough to know how often my assumptions are >unwarranted. Anybody who wishes is certainly welcome to strip my source and use it as a skeleton. Since it is copyright, I have to give my permission: You have my permission (reasonable approximation of my signature: Matthew Dillon). Poof, done. Anybody who needs the source can also mail me directly and I'll send it to him/her. Devices are processes. Thus, you can call any library you wish. However, you have to be careful not to make a library call that will result in DOS trying to talk to your process. You can even open-up a CON: window and write to it... though that is *very* precarious. -Matt
qix@ihlpa.ATT.COM (Puckett) (06/25/87)
In article <8706240726.AA06112@cory.Berkeley.EDU>, dillon@CORY.BERKELEY.EDU (Matt Dillon) writes: > >Does your new PIPE: support the packets Examine and ExNext use? If so, then you > >could just "dir pipe:" to get a list of active pipes, right? > > I've been trying... and trying... and trying... I can't figure > out why it is crashing.... Can somebody post a *working* device driver > which uses Examine/ExNext???? P: supports these. Here is a fragment of the code. The complete sources have been posted to comp.sources.amiga and sent to Fred Fish. Hope this helps. -Ed Puckett. /*--------------------------------------------------------------------------- ** PipeExamine() responds to Examine requests. For locks on the handler, the ** address first item of the pipelist is stored in the DiskKey field for ** PipeExNext()'s reference. */ void PipeExamine (pkt) struct DosPacket *pkt; { struct FileInfoBlock *fib; struct FileLock *lock; PIPEDATA *pipe; void FillFIB(); pkt->dp_Res1= 1; /* no error, for now */ pkt->dp_Res2= 0; fib= (struct FileInfoBlock *) BPTRtoCptr (pkt->dp_Arg2); if ((lock= (struct FileLock *) BPTRtoCptr (pkt->dp_Arg1)) == NULL) { pkt->dp_Res1= 0; pkt->dp_Res2= ERROR_OBJECT_NOT_FOUND; } else { if ((pipe= (PIPEDATA *) lock->fl_Key) == NULL) /* then this is a lock on the handler */ { FillFIB ( fib, FirstItem (&pipelist), HandlerName, (FIBF_EXECUTE | FIBF_DELETE), 1, 0, 0, &PipeDate ); } else { FillFIB ( fib, NULL, pipe->name, (FIBF_EXECUTE | FIBF_DELETE), -1, pipe->buf->len, 1, &pipe->accessdate ); } } ReplyPkt (pkt); } /*--------------------------------------------------------------------------- ** PipeExNext() responds to ExNext requests. The DiskKey field of the ** FileInfoBlock is assumed to be a pointer to the next pipe in the pipelist ** which is to be listed in the directory. We then scan pipelist for this ** pointer, and upon finding it, store its information in the FileInfoBlock ** and store the address of the next pipe in pipelist in the DiskKey field. ** If the pipe is not found in the list, or if DiskKey is NULL, then ** ERROR_NO_MORE_ENTRIES is returned. ** By rescanning the list each time, deletion of a pipe cannot hurt us ** by causing a dangling pointer in DiskKey -- we just end the directory ** listing there. This can cause incomplete directory information for the ** cleint, however, if the last listed pipe is deleted before the client's ** next ExNext() call. */ void PipeExNext (pkt) struct DosPacket *pkt; { struct FileLock *lock; struct FileInfoBlock *fib; PIPEDATA *listitem, *pipe; void FillFIB(); pkt->dp_Res1= 0; /* error, for now */ if ((lock= (struct FileLock *) BPTRtoCptr (pkt->dp_Arg1)) == NULL) { pkt->dp_Res2= ERROR_INVALID_LOCK; goto EXNEXTREPLY; } if (lock->fl_Key != NULL) /* then an individual pipe */ { pkt->dp_Res2= ERROR_OBJECT_WRONG_TYPE; goto EXNEXTREPLY; } pkt->dp_Res2= ERROR_NO_MORE_ENTRIES; /* until found otherwise */ fib= (struct FileInfoBlock *) BPTRtoCptr (pkt->dp_Arg2); if ((listitem= (PIPEDATA *) fib->fib_DiskKey) == NULL) goto EXNEXTREPLY; for (pipe= (PIPEDATA *) FirstItem (&pipelist); pipe != NULL; pipe= (PIPEDATA *) NextItem (pipe)) if (listitem == pipe) break; if (listitem == pipe) /* then found next entry */ { FillFIB ( fib, NextItem (listitem), listitem->name, (FIBF_EXECUTE | FIBF_DELETE), -1, listitem->buf->len, 1, &listitem->accessdate ); pkt->dp_Res1= 1; pkt->dp_Res2= 0; } EXNEXTREPLY: ReplyPkt (pkt); } /*--------------------------------------------------------------------------- ** PipeParentDir() responds to ParentDir requests. */ void PipeParentDir (pkt) struct DosPacket *pkt; { struct FileLock *lock; void InitPipedirLock(); InitPipedirLock (); pkt->dp_Res2= 0; if ((lock= (struct FileLock *) BPTRtoCptr (pkt->dp_Arg1)) == NULL) { pkt->dp_Res1= 0; pkt->dp_Res2= ERROR_INVALID_LOCK; } else { if (lock->fl_Key == NULL) /* then lock is on handler */ pkt->dp_Res1= 0; /* root of current filing system */ else pkt->dp_Res1= CptrtoBPTR (PipedirLock); } ReplyPkt (pkt); } * * * * * /*--------------------------------------------------------------------------- ** FillFIB() fills a FileInfoBlock with the specified information. Note ** that handlers must store BSTR's in the FileInfoBlock. */ static void FillFIB (fib, DiskKey, FileName, Protection, Type, Size, NumBlocks, Datep) struct FileInfoBlock *fib; LONG DiskKey; char *FileName; /* null-terminated */ LONG Protection; LONG Type; LONG Size; LONG NumBlocks; struct DateStamp *Datep; { fib->fib_DiskKey= DiskKey; fib->fib_DirEntryType= Type; CstrtoBSTR (FileName, fib->fib_FileName, sizeof (fib->fib_FileName)); fib->fib_Protection= Protection; fib->fib_EntryType= Type; /* ??? */ fib->fib_Size= Size; fib->fib_NumBlocks= NumBlocks; CopyMem (Datep, &fib->fib_Date, sizeof (struct DateStamp)); fib->fib_Comment[0]= '\0'; /* empty BSTR */ }