pete@violet.berkeley.edu ( Pete Goodeve ) (03/06/88)
In an earlier message [Subject: Re: the good old "tools vs. integrated
systems" debate] I suggested that we needed some thought on the business
of communicating between programs using ports and messages. So here's the
way I've been thinking, anyway... Apologies for it getting a mite long.
By the way, I forgot to mention AREXX last time, and everybody promptly set
me straight on that. I'm not sure that AREXX is quite what I'm looking
for, though. I'm not even sure how relevant it is to the points that I'm
concerned with at the moment. It's hard to tell, because I haven't had a
chance to look at it yet. All I know is what I can remember from Bill
Hawes' talk at BADGE, and from reading a REXXText. It's obviously a very
comprehensive command language, for situations where you need that sort of
thing [but NOT a pretty language... my impressions on its structure from
the manual were definitely un-positive].
I don't think it's necessarily particularly incompatible with my concerns
here either. The thing I feel is most important is that when one program
gets a message from another -- via AREXX or wherever -- there should be a
standard way for it to know what to expect as the contents of that message
and how to handle it. Remember that we're not just talking command lines
here: there are all sorts of things we might want to pass between programs.
I was saying before that there are several levels that we have to look at,
from the basic use of ports up to maybe very complex information protocols.
Let's start at the bottom and work up, as even the basic matter of messages
and ports seems to need some standardization.
First of all, people still get it wrong. VideoScape-3D is an example: it
supposedly can be synchronized to an external task if it finds a couple of
public ports declared; the trouble is, both task- and reply-ports have to
be created (and therefore owned) by the external task, and so there's no
way that the Waiting V-3D can be signalled when the task replies!
Obviously if even a top notch guy like Allen can slip up (he admits he was
in a hurry...(:-)) there's need for an expansion on the good ol' RKM. Even
the KickStart Guide skimps a bit here, I think.
Things aren't helped by some limitations in the design of the message/port
system. It's fine for a group of tasks that are well behaved towards each
other, but there are loopholes when two independent programs want to
communicate with each other. Consider one program that has a desire to
send a message to another program's named port, IF THAT EXISTS; one
sequence you might think of is:
....
his_port = FindPort("his_port");
....
while (appropriate) {
.....
if (his_port != NULL)
PutMsg(his_port,mymsg);
....
}
....
You can see that this has problems, because "his_port" could go away at any
time and the program wouldn't know about it. Even if it did a FindPort
each time, this still isn't clobber-proof because it COULD happen that the
port went away just at that moment. The only perfect method is to always
use the sequence:
Forbid();
his_port = FindPort("his_port");
if (his_port != NULL)
PutMsg(his_port,mymsg);
Permit();
This is cumbersome, and has quite a bit of overhead each time, because
FindPort has to search. Obviously we want something a little better for a
general protocol. I can suggest two possible paths to take. The first is
straightforward; the second needs more work, but I can see advantages.
The simple approach is to decide on a standard protocol in which, to open
communication, the sending program does something like:
Forbid();
his_port = FindPort("his_port");
if (his_port != NULL)
PutMsg(his_port,LOCKMSG);
Permit();
The program owning "his_port" notes (and replies to, naturally) the
LOCKMSG, and will promise not to go away until it receives a corresponding
RELEASEMSG from the sender. Then the sender can use the his_port handle
with impunity until it wants to end the conversation. It would be a good
idea to also reserve a special cut-off message for the "his_port" owner to
send back if it wanted to end the conversation, to keep things symmetrical.
By implication we have two ports, "his_port" and the reply port of the
first program. This should be all we need for two-way communication, as
long as the reply port is prepared to handle messages other than replies.
I don't see any difficulties here, and we don't want more ports around than
we need.
That's the first approach, and I can't really see any reasons not to adopt
it for situations where two programs want to talk directly to each other,
but I keep visualizing scenarios which beg for a little more flexibility.
I think I'm going to leave expanding on these scenarios for another time
(I'd like to avoid too many topics in one heading -- there's probably
already too much here, but nevah mind...). I'll just say for now that I
think they need some kind of central "message broker" process. This could
either be a process with a single recognized message port or a resident
library.
There's one other thing to look at first, and that's the format and content
of the messages being passed around. As I said in that earlier note, I
don't think we need anything very elaborate for a basic structure --
something like:
struct IPCMsg {
struct Message ipc_Msg;
ULONG ipc_ID;
/* Data follows... */
};
ipc_ID would indicate the type of data concerned; ipc_Msg.mn_Length would
be (4 + length_of_data). I suggest that the ID should normally be a four
character string pretty much like an IFF ID word -- in fact we should
reserve "FORM" etc. for messages that ARE IFF objects. To cover all
situations, one should be allowed less than four characters, but always
left-justified (padding could be spaces or nulls -- I see no preference);
on the other hand if the first byte was ZERO, the ID would be a "private"
one agreed between the programs involved; maybe we should also reserve
codes 00000000-000000FF for universally agreed shorthands; two of these
could well be the LOCKMSG and RELEASEMSG above. A couple of basic IDs that
immediately come to mind are "TEXT", for simple ASCII, and "CLI ", for DOS
command lines (to a shell). Decisions on particular IDs would be handled in
the same way as IFFs are now -- through a clearing house.
The rest of the structure should be filled in properly, too, and we should
be sure that the standard rules are followed. (I presume -- but haven't
checked -- that PutMsg sets the ln_Type to NT_MESSAGE, and ReplyMsg sets it
to NT_REPLYMSG. Anyone know for sure?) The receiving program should be
able to assume that if mn_Reply is NULL it should dispose of the message's
memory when done; if it is non-zero, of course, the message MUST be
replied (possibly with data and/or ID changed).
Yeah.. well that's quite enough for now. Nothing very exotic, but maybe
it'll be a start on a foundation we can build a flexible IPC protocol from.
I'll churn out my rationale for a message broker as soon as I can.
Meanwhile, I'll sit back and wait for the flak on this lot.
-- Pete --pete@violet.berkeley.edu ( Pete Goodeve ) (03/06/88)
[...Yes I did post this and the previous article to comp.sys.amiga.tech,
-- then read Bryce's note about it's non-existence, so I'm back here with
a repeat...]
In my previous article [IPC (1): Ports & Messages] I was arguing that we
need a properly defined protocol for passing messages between programs,
and I suggested that we might need something more general than
individually created links between specific programs. In this article I
want to look at that idea in more detail.
Ron Minnich's "tools-based hypercard" concept, which started this whole
discussion off, is just a nebulous glimmer at this stage, but if you
envisage it as I do it will be as a suite of small programs, any one of
which can call on any other to perform the task it is designed for, using
information provided by -- or generating information for -- the invoking
program. As the whole point is to keep the system modular and adaptable,
it would be a horrendous task to keep track of a network of ports belonging
to the individual programs.
It will be much cleaner if a module that wants something done can "say to
the air", so to speak, "I want information on this; does anyone have it?",
or "I have this new information: does anyone want it?". In other words, a
Broadcast message. Once a broadcast request has been answered, the two
programs can always set up private communication ports of course, but this
might not always be the way to go either. You can broadcast specific
information as well as requests, and the advantage of this is that you
don't have to be concerned how many others are listening: any program that
wants the information can pick it up and use it; if no one wants it, it
will just drift off into the ozone.
Another example: we have a "tools-oriented" publisher package (I wish!),
together with our favorite editor and paint package, which also have such
facilities [Stop dreaming, Peter...]. The publisher has no edit or paint
facilities of its own: it can read existing files, but all manipulation is
done by the other programs. Suppose we start up the publisher and ask it
to format a particular text file; it reads the file as it is at that
moment, but at the same time sets up to receive any new information that
might be generated about that file. We look at the results previewed on
the screen, and decide that we want to re-edit a paragraph, so we click on
the editor icon (on the workbench -- the publisher doesn't know about it
specifically). The editor first off sends out a message "What text file?",
which the publisher --knowing which file we are working on -- will respond
to. Note that in the general case SEVERAL programs might respond, and the
editor would then pop up a requester asking the user to select one. The
editor then reads the original file (it won't try to work on the
publisher's data image -- no telling what format it is in...), and you can
start editing. Every significant change you make to the text (with some
appropriate atomicity) is put into a message which the editor tosses out.
The publisher is looking for these, so each time it updates its own preview
image suitably.
That's a sketch which might give you some idea of my dreams -- I've left
out most of the details, and ALL the snags. This sort of scenario could
also be handled by straight message passing between the two programs, but I
like the general idea of the editor, for instance, not having to know or
care what use is going to be made of the information it tosses out. The
program receiving it might be an incremental compiler rather than a
publishing package, or it could be just a communication link, sending it
(via DNET, naturally...) to some remote installation.
In this situation, there probably wouldn't be more than one other program
interested in a particular message, but how about a scientific data
acquisition module, continuously collecting data? You would want a
permanent record of it all on disk, perhaps compressed by a second module,
and you might also want to do some analysis (or a couple of kinds of
analysis) on it in real time as it arrived. Here there are several
modules, all working on the same stream of messages from the first module.
Closer to the needs of most people, there's been some desire expressed on
the net recently for VT100 to have "plug-in modules" for specific purposes.
The "Broadcast message" mechanism would be a natural for this. In fact I'd
go one further and just have a fairly dumb Communications Module (without
screen display or keyboard input) that simply sends text block messages it
receives out the serial port, and broadcasts incoming characters to whoever
wants them. If all messages ended up being one character long, we could
hit severe speed restrictions, but I think there may be ways around this.
I'll leave that for further thought, though.
Alright, let's get to some specifics. What I'm proposing is a "message
broker", which is built from a single public port with a process attached
[or a process with a port attached; whatever]. It could also be
constructed as a resident library, but for now let's go with it as a
process. Other programs would pass it messages regarding their interest in
a particular "topic" (using the standard "IPC" protocol defined in the
previous article). These messages would initially be one of: "I want to
know about this topic", or "I want to give information on this topic".
The topic is a text string -- whatever is suitable --, but this only has to
be used for the initial contact. In the ID word at the head of the message
data area the broker will return a handle value, to be used when actual
information is passed on that topic. Topics are kept (with their handles)
in a lookup list, which is added to as necessary. The requesting program
supplies a reply port as usual, and this is filed away by the broker under
that topic.
When the broker receives a data message on a topic, it looks up -- using
the handle -- which reply ports should be informed, makes duplicates of the
message as necessary, and dispatches these. Normally the original message
will only be replied when all the spawned copies have been replied; the
sender could probably also specify that it wanted an immediate reply if the
message didn't contain any pointers to other memory. You won't be able to
pass information back in replied messages, though, because they're copies.
If we keep the handle less than 32 bits it can be one of the "private" IDs
discussed in the last article, and can go in the ipc_ID slot. The message
itself also needs an type ID, so that the receiving program will know how
to handle it, and this would presumably go into the next word. The data
area proper would follow this, so the message structure would be -- aside
from the extra word -- just the same as a standard IPC message.
struct BrokerMsg {
struct Message brkr_Msg;
ULONG brkr_Topic;
ULONG brkr_ID;
/* Data follows...*/
};
With a scheme like this, a receiving program gets only the messages it is
interested in without having to know who or where the originator is. To me
this looks like a good foundation for a flexible network of programs. The
disadvantages as usual are in extra overhead, both in message structure and
the time taken in duplication and so on. So now tell me all the other
objections...
-- Pete --peter@nuchat.UUCP (Peter da Silva) (03/07/88)
THe idea of a 4 character ID and a length byte sounds good. I'd like to suggest that a single byte mightn't be enough. IFF might be a good starting point. Each message would be an IFF chunk containing an ID and the usual IFF data for that chunk type. I think FindPort should be good enough for simple stuff. More complex situations could be handled by virtual device drivers. Matt Dillon's sample drivers seem like a good starting point. -- -- a clone of Peter (have you hugged your wolf today) da Silva `-_-' -- normally ...!hoptoad!academ!uhnix1!sugar!peter U -- Disclaimer: These aren't mere opinions... these are *values*.
pete@violet.berkeley.edu (03/07/88)
In article <738@nuchat.UUCP> peter@nuchat.UUCP (Peter da Silva) writes: >THe idea of a 4 character ID and a length byte sounds good. I'd like to >suggest that a single byte mightn't be enough. > ... Sorry... somebody didn't make it clear (me?). Stuart's structure had a long size value (which I think is too much); I'm suggesting using the standard mn_Length slot which is a UWORD. It probably would be a mistake to encourage messages longer than 64K! (If absolutely necessary, the data section of some types of messages could be pointers to public memory.) >IFF might be a good starting point. Each message would be an IFF chunk >containing an ID and the usual IFF data for that chunk type. > ... I think that's overkill. See my posting yesterday... >I think FindPort should be good enough for simple stuff. More complex >situations could be handled by virtual device drivers. Matt Dillon's Yes, with proper safeguards -- again see posting "IPC (1):...". -- Pete --
dillon@CORY.BERKELEY.EDU (Matt Dillon) (03/08/88)
Here is a more substantiated description of my ideas for IPC.
Note that it is still INCOMPLETE! For instance, the actual format of
the macro command string is not specified as yet. THIS IS A BRAINSTORM.
Essentially, the whole thing is hidden by a run time library,
with only the tip showing (a port structure) so you can use Wait(), etc...
While all of you are talking about low level formats and assume nobody
will ever want to upgrade the system in an upward compatible manner,
I've been doing some serious thinking on what kind of capability I would
want THAT DOESN'T CURRENTLY EXIST.
IFF is definately the way to go. Naturally, we don't want to
encumber the application programmer with figuring out all the offsets
and stacking, etc...
Ending comment: You asked for it!
eport = SarcInit(applname, ops, cmdvect)
An application logs its name with SARC. The application name need not
be unique, and will automatically be qualified with a two digit number
00-99 to make it unique (you must provide enough space in the string
for this). applname must also exist throughout the session.
ops is a string containing the following options:
(none at the moment)
cmdvect is a vector to your command interpreter which is called when
doing synchronous SARC commands. SarcDo()/SarcSend() is fully
reentrant. For instance, if you are in the middle of a SarcRead()
which has blocked, and some other application sends you a command,
cmdvect will be used to execute the command without interrupting
the SarcRead() (effectively, SarcRead() detects the reception, calls
cmdvect, then returns to it's wait).
(void) SarcUnInit(eport)
Closes the SARC port and unlogs the application. Any pending commands
will be returned (and removed if the SARC port was the returnee).
REMOTE CONTROL AND COMMAND EXECUTION
SARC remote control and command execution occurs through the extended
PORT structure returned by SarcInit(). Remote commands are sent to
the port via the SARC library from other processes. Macro commands
may be sent to SARC which causes SARC to interpret them and relay
them to one or more other processes (including the possibility of
putting some of these commands back in your lap).
error = SarcDoCmd(eport, command, outstream, instream, errstream)
Execute a synchronous macro command. This call sends the specified
command and streams to SARC, which interprets and executes it.
This call does NOT return until the command is complete, but may make
calls to CMDVECT specified in SarcInit() to accomplish the command.
Any stream which is NULL will become a NUL stream.
error = SarcSendCmd(eport, command, outstream, instream, errstream)
Execute an asynchronous macro command. Exactly like SarcDoCmd(),
but returns immediately. You can tell when the command is complete
when SarcRemoteCmd() returns a non-zero asycmdcomplete. You can
stack multiple SarcSendCmd()'s which will be run in order.
You MUST use SarcRemoteCmd() when using SarcSendCmd() or
completion messages may build up on eport.
NOTE: SarcDoCmd() waits for any pending SarcSendCmd()'s to complete
before running. This also clears the asynccmdcomplete messages
automatically (asynccmdcomplete set to 0).
error = SarcRemoteCmd(eport, &asyccmdcomplete, cmdvec)
When you have determined something has been sent to eport, make a
call to SarcRemoteCmd() to interpret the message. If you give a
non-NULL &asycmdcomplete, it will be filled with the number of
SarcSendCmd() requests which have completed since the last time you
asked.
If you give a non-null cmdvec, that cmdvec becomes the global
cmdvec (replacing the one given in SarcInit()), and if any remote
commands are ready, will be called in the following format:
(*cmdvec)(command, outstream, instream, errstream)
Otherwise, if cmdvec is NULL, pending remote commands stay pending.
STREAMS
SARC STREAMS are used to transfer data between applications. Streams
can operate in one of two modes: IFF or RAW. Additionaly, mode
mixing is allowed. (Process A writes in RAW mode while process B
reads in IFF mode).
Various calls are available to do operations synchronously,
asynchronously, with, and without buffer copying.
Streams are strictly common-ground datapaths. To get two-way
communication you must open two independant streams, write to one and
read from the other (while the other task reads from one and writes to
the other).
A SARC stream is front-ended by a PORT structure. Thus, one can
'WaitPort' or Wait() on its signal to wait for something to happen.
This does not neccesarily mean that data is available for reading,
but may also occur when, for example, an asynchronous write has
completed. The Use of Wait() is usually coupled with the "n"
A special stream "NUL" exists which always returns an EOF and is a
sink for write's.
stream = SarcOpen(name, modes, &error)
Open a SARC stream. If the name buffer supplied holds a zero-length
string, a unique name of 9 digits is generated and loaded into the
name buffer. Otherwise, the supplied name is used as a common reference
allowing processes to rendezvous with each other. If "c" is not
specified, the stream name must already exist (either currently openned
by another process or holding data).
Note: "Exclusive access" means that only one SarcOpen() is allowed on a
specific descriptor. You can still SarcDup() it with different modes
and pass it to a remote process as stdin, stdout, or stderr.
modes: "r" read
"w" write
"s" seek (random access), else assumes sequential access
"c" create it if it doesn't exist, error if it does
"i" IFF, else RAW
"F" use an actual FILE to hold data (always in RAW mode).
Otherwise, a memory buffer is used.
"R" don't block on reads when requested data is not
available.
"X" Exclusive access
error is set to 0 on success, or an error code on failure:
-1 : not found
-2 : already open for exclusive access, etc..
error = SarcSeek(stream, position, how)
Seek within a file. Provides random access capability. You must
have specified "s" in the SarcOpen().
IFF streams: SarcSeek() works ONLY within the lowest level data
sections.
newstream=SarcDup(stream, modes)
Get a duplicate descriptor with different modes. For instance, you
might have openned an exclusive access stream for read, and now want
another descriptor that can be written to. You must specify all
applicable modes except for "c", which doesn't make sense in this
case.
Usually used to make one way link-pairs and then using one as the stdin,
stdout, or stderr streams.
newstream = SarcDup(oldstream, "ws");
error = SarcClose(stream)
Close a SARC stream. An error might be returned as follows:
-1 Nobody ever read the data
-2 Somebody didn't read all the data
-3 Illegal or Incomplete IFF usage
-4 IFF too complex
-5 Ran out of memory
-8 NOT IFF
-9 IFF ERROR
error = SarcDelete(name)
Delete the specified SARC file. The file is removed from the
namespace. However, if the file is currently open those descriptors
are still valid and the memory taken up by the file is not freed
until those descriptors are closed. This works even if the file
is a real file (but only if all entities go through SARC to access
it).
(void) CheckSarcStream(stream, &error, &readytoread, &asywritecomplete)
All variables are pointers to longs. Any argument may be passed as
NULL if you aren't interested in the information.
error: Current status 1 active
0 EOF
-? some error
readytoread: N # bytes available to be read (RAW)
N # blocks (read calls) that can be done (IFF)
asywritecomplete: N # of asynchronous write requests which have
completed since the last time you requested
this status (passed a non-null variable
as &asywritecomplete)
bytes = SarcRead(stream, buf, bytes)
Read from a stream into a buffer. This call blocks until the specified
number of bytes has been read or an EOF or error occurs.
If the stream has been specified as IFF, decoded IFF data is read as
shown in the example. Essentially, control information is qualified
by a +,-, or =. The block of data after an '=' is always data and
not qualified. Note that the bytes returned can be less than the
bytes requested (block oriented). You MUST specify at least a 16
byte buffer when reading control information. CHUNK data can be
read in multiples of any size. The number of bytes in the CHUNK data
is given in the preceding "=????nnnnnnnnnn\0" buffer.
11= SarcRead(s, buf, 32); "+FORM\08SVX\0"
16= SarcRead(s, buf, 32); "=VHDR0000000020\0" (20 byte chunk)
20= SarcRead(s, buf, 1000); a Vhdr structure
16= SarcRead(s, buf, 32); "=NAME0000000008\0" (8 byte chunk)
8= SarcRead(s, buf, 53); a Name structure (just the name)
16= SarcRead(s, buf, 32); "=(c) 0000000018\0" (18 byte cpy right)
5= SarcRead(s, buf, 5); /* don't have to read it all at once */
13= SarcRead(s, buf, 40);
16= SarcRead(s, buf, 32); "=BODY0000001022\0"
1022= SarcRead(s, buf, 1022); The body data...
6= SarcRead(s, buf, 32); "-FORM\0"
0= SarcRead(s, buf, 32); EOF
This works with SarcARead() as well.
bytes = SarcARead(stream, &buf, bytes)
Same as SarcRead(), but uses SARC's internal buffer (maybe even the
writer's buffer) to hold the data. The 'buf' pointer is modified to
point to the buffer. The buffer is good until the next SarcRead() or
SarcARead() call from that stream.
This call may cause SARC to reorganize its internal buffers if the
requested # bytes do not occur contiguously internally.
IFF: Same as SarcRead().
NOTE: If -1 specified for bytes, all the available data is made
contiguous and returned.
If -2 specified for bytes, no reorganization occurs and you
get data fragments (THIS IS THE PREFERED METHOD OF USING
SARCAREAD).
bytes = SarcWrite(stream, buf, bytes)
Write data from a buffer to a SARC stream. The buffer is copied into
SARC's internal buffers. This call never blocks.
When writing IFF, the chunk names are written separately from the
chunks. Stacking is accomplished by specifying the chunk-name-type
as +,-, or =:
+ Level down
- Level up
= Low level chunk
This allows you to organize your IFF easily. SARC automatically
puts it in the correct format (adds padding and length specs, etc...)
and checks it for validity.
SarcWrite(s, "+FORM8SVX", 9);
SarcWrite(s, "=VHDR", 5);
SarcWrite(s, &Vhdr, sizeof(Vhdr));
SarcWrite(s, "=NAME", 5);
SarcWrite(s, Name, strlen(name));
SarcWrite(s, "=(c) ", 5);
SarcWrite(s, Cpw, strlen(Cpw));
SarcWrite(s, "=BODY", 5);
SarcWrite(s, Body, 1024);
SarcWrite(s, "-FORM", 5);
bytes = SarcDoWrite(stream, buf, bytes)
Write data from a buffer to a SARC stream. This call blocks until
the recipient has read and processed the data. The caller's buffer is
used (usually means no data copying).
bytes = SarcSendWrite(stream, buf, bytes)
Same as SarcDoWrite(), but this call does not block. The caller's
buffer is used and thus should not be modified until the operation
is complete. Any number of Send's may be queued on a stream.
Calling SarcDoWrite() after a Send is equivalent to waiting for
all Sends to complete AND the Do to complete.
>
>A proposal
>
>Yeah, let's do it! It's high time that the Amiga had a standard method of
>inter-process communication. With products like ARexx and Browser coming
>out, we may get stuck with a "de-facto" standard which may not be well
>designed, or several "standards" that are mutually incompatable. ("It's
>supports ARexx, but does it support Browser?")
I agree, but let's do it right. One reason the clipboard device
never caught on (to non-commercial programs) is that it is difficult to
use.
ARexx is interesting, but also somewhat convoluted and takes too
much overhead.
Whatever *we* decide on, it must be made relatively easy to use.
But we don't simply want a stream oriented IPC. People don't take enough
time to think of easy ways to provide maximum power and capability.
This is what I envision:
-Low per-task memory overhead
-Symbolic Asynchronous Remote-Control capability (domain based)
e.g. one application can send symbolic commands to another...
A data stream, as part of a command, would be a minor item in
this system.
(DME.file1(unblock block block) DME.file2(bcopy))
Also, have a standard format for SARC menu item simulation.
e.g. DME(File-Quit)
-Macro Expansion capability (which is a superset of what ARexx
would give us) e.g. program X gives the driver a symbolic macro
which the driver executes, possibly causing remote-control
commands be sent back to program X and/or to other applications.
-Fully reentrant and stackable remote commands, with infinite
loop detection, etc...
e.g. program X sends a Macro to the driver, which sends resolved
commands back to program X which just happen to be macros in
program X which is expanded again and sent as a Macro to the
driver, which sends resolved commands to program's X and Y,
program X executing the command, etc....
Also allowing program X to handle such things synchronously or
asynchronously (not wait for macro completion before continuing,
but not getting confused either).
-Exit recovery. Allow program X to exit at any time without
screwing up any macro's in progress (have them gracefully fall
to their deaths). i.e. calling SarcClose() clears anything
pending.
-Standard command format. Ability to specify logical or physical
streams (i.e. supply your own functions or use DOS functions,
etc...).
-Low level IFF decoder for arbitrary IFF streams. Since IFF is
structured, the decoder would not have to know specific formats,
-Ditto Low level IFF encoder for arbitrary IFF chunks.. Structures
the chunks for you.
-Run time Library to handle interaction.
I could work on something like this after I get DNET out.
-Mattpeter@nuchat.UUCP (Peter da Silva) (03/09/88)
Good call on the task going away. For the case of the Browser messages I
suggested the other day I don't think that it's a problem.
(1) You can lock before sending any messages to the browser: there are
only going to be two messages from you to it in any two-way conversation,
anyway: 'ADDT' or 'ADDW' and 'DELT' and 'DELW'.
(2) You're supposed to tell Browser before you go away. If you always do
this, but check that *it* hasn't gone away before doing so, there's not
going to be a problem.
Now, as to a general message server. I don't like the idea of a reply port
being enything but a reply port, but you can always pass a port as part
of the message.
We can make the assumption that *it* isn't going to go away, ever. So:
You find it with the call FindPort("SERVER");
You can have two roles. You can be a master or a slave.
If you want to be a master, you send 'ADDM', your id, and a message port.
Server adds you to its list and replies. You are now registered. Before you
can go away you have to send it 'DELM'. It will then send 'DELM' to all
the slaves it knows you're talking to, and when it gets all their replies
it replies to you.
If you want to be a slave, you send 'ADDS', the master id, and a message port.
Server looks up the master in its list, adds you to the list of active slaves,
and forwards it your 'ADDS' message. The master's replies go back to you... the
server doesn't care. If the master doesn't exist, of course, it just replies
immediately with a failed status.
When you want to disconnect you send your master 'DELS' and it takes you out
of the list and forwards your message to the master. Once again the master
replies to you.
And of course if you ever get a 'DELM' message you reply to it and forget you
ever heard of the master.
This seems to have very little overhead. What do you all think?
--
-- a clone of Peter (have you hugged your wolf today) da Silva `-_-'
-- normally ...!hoptoad!academ!uhnix1!sugar!peter U
-- Disclaimer: These aren't mere opinions... these are *values*.peter@nuchat.UUCP (Peter da Silva) (03/09/88)
In article ... pete@violet.berkeley.edu ( Pete Goodeve ) writes: > By implication we have two ports, "his_port" and the reply port of the > first program. This should be all we need for two-way communication, as > long as the reply port is prepared to handle messages other than replies. > I don't see any difficulties here, and we don't want more ports around than > we need. I don't like the idea of using the replyport for anything but reply messages. I think there's enough semantic confusion already in the system to add more. > There's one other thing to look at first, and that's the format and content > of the messages being passed around. Well, a lot of messages could be handled by means of specific structures designed by the authors of the server programs, but here's some idea for some ids. I think it's a good representative sample, but you all will likely disagree: 'FILE' followed by a directory lock and a character pointer. 'CSTR' followed by a pointer to a C string. 'BSTR' followed by a pointer to a BCPL string. 'PORT' followed by a message port. 'FORM' followed by *a pointer to* an IFF FORM. Making it a pointer shouldn't cause anyone any problems, and should make it easier to pack stuff into a message. 'CAT ' 'LIST' followed by a pointer to IFF CAT or LIST 'MORE' followed by a byte count and a pointer to another structure of this type, that many bytes long. 'ARGV' followed by a pointer to a UNIX-style argument vector. 'STRT' followed by a pointer to a workbench startup message. 'WIN ' followed by a pointer to a window structure. 'RAST' followed by a pointer to a rastport. And so on. Making all this stuff pointers should cut down on copying, and speed things up. All parts of a message have to be MEMF_PUBLIC, of course. I don't think there's any need for EA's "MakeID" macro in this. Long characters should do fine... it's all going to be on one processor, anyway. > The receiving program should be > able to assume that if mn_Reply is NULL it should dispose of the message's > memory when done; if it is non-zero, of course, the message MUST be > replied (possibly with data and/or ID changed). I like the simplicity and security you get if you always reply to a message on its reply port. The only thing changed in the reply should be the status word. Yeh, there should be a status word in the structure somewhere, perhaps immediately before or after the ID. -- -- a clone of Peter (have you hugged your wolf today) da Silva `-_-' -- normally ...!hoptoad!academ!uhnix1!sugar!peter U -- Disclaimer: These aren't mere opinions... these are *values*.
peter@nuchat.UUCP (Peter da Silva) (03/09/88)
In article ... pete@violet.berkeley.edu ( Pete Goodeve ) writes: > [about a real neat idea for a message broker] Sounds like a much better idea than my port-server idea (which is what I now realise my last message was). You volunteering to implement it? (half-smiley: someone has to). I'm not sure the imcremental editor idea is practical, but the idea of having a "bus" for broadcast messages is a good one. If the server could be made fast enough, you could even use it for MIDI. Did I say a good one? The idea is great! Go for it. For some purposes you still need a port server. It's a lot less overhead once the connection is made, and it can do some of the same things. It could even look the same to the user. How about this idea? You have a server named "SMUS player" that you can open a channel to and send messages like 'INST', 'TRAK', and 'CLOK'. You want to play a song, you do this: FindPort("SERVER"); Send('ADDS', "SMUS player", myport); WaitPort(ReplyPort); /* this will tell the master you want to talk and wait for the player to tell you you're accepted. It will probably only allow one slave at a time. */ Send('INST', "guitar", guitar_sample); Send('TRAK', smus_track); Then you can either Send('CLOK', 0); /* Play a beat */ or Send('CLOK', 72); /* Start playing at 72 BPM */ Send('DELS'); I don't think this could be easily handles by the message broker, but then I don't think I could do the stuff the message broker likes. -- -- a clone of Peter (have you hugged your wolf today) da Silva `-_-' -- normally ...!hoptoad!academ!uhnix1!sugar!peter U -- Disclaimer: These aren't mere opinions... these are *values*.
pete@violet.berkeley.edu (03/09/88)
[This is a response to an article in comp.sys.amiga.tech. Because of the uncertain distribution of this group, I'm cross posting to comp.sys.amiga. -- Note to Ron: maybe we'd better keep any further discussion over there for now!] rminnich@udel.EDU (Ron Minnich) writes: > In article <7409@agate.BERKELEY.EDU> pete@violet.berkeley.edu () writes: > >The rest of the structure should be filled in properly, too, and we should > >be sure that the standard rules are followed. (I presume -- but haven't > >checked -- that PutMsg sets the ln_Type to NT_MESSAGE, and ReplyMsg sets it > >to NT_REPLYMSG. Anyone know for sure?) The receiving program should be > I know for sure that it does not. I found that out the hard with > internet.device, where i had to jam NT_MESSAGE into messages IN THE > DRIVER, even though the messages had been PutMsg'ed. Damn. Oh well. Thanks. We'll just have to enforce the rules ourselves, then. > Two thoughts: > 1) Seems like we also need a 'port broker'. Isn't that function served > by PIPE:, or am i missing something fundamental? I.e. rather than > use ports, just use PIPE: which strikes me as being more reliable? > Or do we lose something valuable by not using ports? I'm not sure I follow your line of reasoning there, but there are some fundamental problems I see in basing everything on pipes: they're basically byte serial, which is very inconvenient for structured data, and even more important they involve buffering and therefore COPYING of data -- a lot of overhead. Messages on the other hand don't move -- they're just inserted in a list, so making a lot of data available to another task can be very fast. For a general IPC scheme, I think we have to be concerned with speed. I'm not saying we should discard pipes: they're especially useful when the data is normally serial-file based... well, I don't need to tell you that: this all started with "tools.. etc." didn't it... > 2) As for the message broker, make it a library, then we in effect > have multiple copies running for each client ( i am thinking > as opposed to having a 'port manager' task which will be > a bottleneck) I agree totally. At least in principle. I went the port route because I thought that the concept was clearer that way, but I'm sure a library is the way to go in the long run. For a start it removes all danger of the FindPort hazard we've been talking about. On the other hand, though, I'm not sure that a manager process would in fact be a bottleneck: after all this is a one-task-at-a-time machine really (sigh)! -- Pete --
pete@violet.berkeley.edu (03/09/88)
[This is a response to an article in comp.sys.amiga.tech. Because of the uncertain distribution of this group, I'm cross posting to comp.sys.amiga. -- Note to Ron: maybe we'd better keep any further discussion over there for now!] In article <1401@louie.udel.EDU> rminnich@udel.EDU (Ron Minnich) writes: > In article <7409@agate.BERKELEY.EDU> pete@violet.berkeley.edu () writes: > >The rest of the structure should be filled in properly, too, and we should > >be sure that the standard rules are followed. (I presume -- but haven't > >checked -- that PutMsg sets the ln_Type to NT_MESSAGE, and ReplyMsg sets it > >to NT_REPLYMSG. Anyone know for sure?) The receiving program should be > I know for sure that it does not. I found that out the hard with > internet.device, where i had to jam NT_MESSAGE into messages IN THE > DRIVER, even though the messages had been PutMsg'ed. Damn. Oh well. Thanks. We'll just have to enforce the rules ourselves, then. > Two thoughts: > 1) Seems like we also need a 'port broker'. Isn't that function served > by PIPE:, or am i missing something fundamental? I.e. rather than > use ports, just use PIPE: which strikes me as being more reliable? > Or do we lose something valuable by not using ports? I'm not sure I follow your line of reasoning there, but there are some fundamental problems I see in basing everything on pipes: they're basically byte serial, which is very inconvenient for structured data, and even more important they involve buffering and therefore COPYING of data -- a lot of overhead. Messages on the other hand don't move -- they're just inserted in a list, so making a lot of data available to another task can be very fast. For a general IPC scheme, I think we have to be concerned with speed. I'm not saying we should discard pipes: they're especially useful when the data is normally serial-file based... well, I don't need to tell you that: this all started with "tools.. etc." didn't it... > 2) As for the message broker, make it a library, then we in effect > have multiple copies running for each client ( i am thinking > as opposed to having a 'port manager' task which will be > a bottleneck) I agree totally. At least in principle. I went the port route because I thought that the concept was clearer that way, but I'm sure a library is the way to go in the long run. For a start it removes all danger of the FindPort hazard we've been talking about. On the other hand, though, I'm not sure that a manager process would in fact be a bottleneck: after all this is a one-task-at-a-time machine really (sigh)! -- Pete --
rminnich@udel.EDU (Ron Minnich) (03/09/88)
In article <7543@agate.BERKELEY.EDU> pete@violet.berkeley.edu.UUCP ( Pete Goodeve ) writes: >I'm not sure I follow your line of reasoning there, but there are some >fundamental problems I see in basing everything on pipes: they're basically >byte serial, which is very inconvenient for structured data, and even more >important they involve buffering and therefore COPYING of data -- a lot of >overhead. Messages on the other hand don't move -- they're just inserted >in a list, so making a lot of data available to another task can be very >fast. For a general IPC scheme, I think we have to be concerned with >speed. Good point. What i am after is a PORTPIPE: i guess- a port manager with no copying that has the attributes of files- namely, a third party manager just as in PIPE:. Then people can safely close them, etc. without worrying about giving other people bad pointers. I am real unhappy with taking the existing port mechanism just as is- i think we have got to make this reliable, if only so Jerry Pournelle won't flame us. ron -- ron (rminnich@udel.edu)
DMasterson@cup.portal.com (03/10/88)
In previous articles, the concept of IPC (Inter Process Communications) standards has been put forward. I like the sound of the idea, but am not familiar enough with concept to debate its internals. However, there is one question I could put forth to stimulate this (I hope). What is being suggested sounds much like the concepts of IPC that are being used in SUN's version of UNIX (yeah, I know that's not too surprising). Isn't there a portable IPC/RPC package that Sun has put in the public domain to encourage people to support their system? Would it make sense to adopt this as a standard? (yeah, Amigas and Suns on a network remotely controlling each other :) I believe the package just recently went by in comp.sources... And now back to your local stations... David Masterson DMasterson@cup.portal.com <the opinions expressed here are my own. If this had been an actual emergency, you would have been informed of location of the nearest dark glasses>
hawes@dino.ulowell.edu (Bill Hawes) (03/11/88)
Newsgroups: comp.sys.amiga
Subject: Re: IPC (1): Ports & Messages
Summary:
Expires:
References: <7412@agate.BERKELEY.EDU> <752@nuchat.UUCP>
Sender:
Reply-To: hawes@dino.ulowell.edu (Bill Hawes)
Followup-To:
Distribution:
Organization: University of Lowell Productivity Center, Lowell MA.
Keywords: InterProgram Communication, tools
Newsgroups: comp.sys.amiga
Subject: Re: IPC (1): Ports & Messages
Summary:
Expires:
References: <7412@agate.BERKELEY.EDU> <752@nuchat.UUCP>
Sender:
Reply-To: hawes@dino.ulowell.edu (Bill Hawes)
Followup-To:
Distribution:
Organization: University of Lowell Productivity Center, Lowell MA.
Keywords: InterProgram Communication, tools
----- News saved at 11 Mar 88 03:26:04 GMT
I've had a chance to read and digest some of the ongoing IPC thread and
have a number of comments and suggestions to add. First an introduction:
I'm Bill Hawes, perpetrator of ARexx (and ConMan, and WShell, and ...)
have a number of comments and suggestions to add. First an introduction:
Since ARexx was my first Amy project (begun in November, 1985), I've had
much time and occasion to examine the IPC issue. ARexx was intended to
define a "macro language standard" and was implemented with the intention
that a wide variety of software could use it for macro processing (or
macro expansion, as Matt called it).
First, a couple of comments about the (REXX) language. In this forum
REXX already has three strikes against it, since it is *Not C*, but
then it was never intended as a medium-level development language.
REXX was designed to be accessible to novice (or even first-time)
programmers, but is powerful enough to catch the interest of the expert
as well. How many languages allow you to pass programs as arguments
to functions? Include source-level debugging as part of the language
definition? Provide a "content-addressable" symbol table? REXX has a
a lot of good features, even if it does DO; ... END instead of { ... }
But whether or not you like the language, the IPC protocol used by ARexx
stands by itself; there's nothing to say that an application with an
ARexx interface couldn't receive commands from a C program. It would
probably even make things run faster :-)
The IPC protocol implemented in ARexx uses public message ports to
identify the applications (hosts) that can receive and send messages.
ARexx specifies the structure and content of the messages it sends and
expects to receive (obviously), but there is plenty of room for extensions
to this part of the standard. The REXX messages are structured like this:
struct RexxMsg {
struct Message rm_Node; /* EXEC message structure */
APTR rm_TaskBlock; /* pointer to global structure */
APTR rm_LibBase; /* library base */
LONG rm_Action; /* command (action) code */
LONG rm_Result1; /* primary result (return code) */
LONG rm_Result2; /* secondary result */
STRPTR rm_Args[16]; /* argument block (ARG0-ARG15) */
struct MsgPort *rm_PassPort; /* forwarding port */
STRPTR rm_CommAddr; /* host address (port name) */
STRPTR rm_FileExt; /* file extension */
LONG rm_Stdin; /* input stream (filehandle) */
LONG rm_Stdout; /* output stream (filehandle) */
LONG rm_avail; /* future expansion */
}; /* size: 128 bytes */
ARexx marks its messages with the string "REXX" in the ln_Name field of
the message, so you can easily tell whether it comes from a REXX macro.
The two result fields are used to pass either integer codes or return
strings, depending in the options requested. The command or function name
and arguments are held in the rm_Args[] array, which looks just like the
argv[] array passed to a C program. The commands or arguments are passed
as null-terminated strings, but with the length available at negative
offsets if you're passing arbitrary data. The rm_Action field is an
integer code that specifies an action (e.g. "command") and any options.
It is quite true that ARexx does not set forth a "complete" IPC standard.
ARexx does not define a standard command set that host applications should
support, and it does not define a higher-level "name broker" to track
which applications are available or what kind of IPC protocol they support.
The issue of standardized commands is important, but imagine the howls of
outrage had I attempted to dictate what commands an editor, a database,
and a paint program should support! The command set should be decided by
the program's developer, ideally in conjunction with other developers of
related (re: competing) products.
ARexx itself can issue any sort of command; if your program wants to
receive a four-byte integer followed by a bitmap, ARexx could oblige
(though it might look ugly from the macro programmer's perspective.)
The higher-level components of the IPC standard should be fairly easy
to design and build. It would be quite straightforward to build the
"name broker", and I'd be glad to help in designing such a beast.
I'm planning to build a network demon to support remote-procedure calls
in REXX anyway, and it would offload much of that work if I could assume
that a name broker existed on the other machine. I think most of what
Matt Dillon outlined in his SARC proposal fits well within what ARexx
already does, once augmented with a global name server. For example,
to allow a host to "go away" before a macro had finished processing,
you need only have the name broker manage the replyport for your
invocation message. When your application closes down, the macro
program(s) will break cleanly and eventually reply the original message.
Anyway, I'd like those of you interested in IPC standards to at least
consider supporting the ARexx protocol. For my part, I will create a
public-domain extract from my header files giving the data structures,
definitions, and conventions used by ARexx, and propose that as the
medium-level IPC standard. We can then work on defining higher-level
components and on standardizing the command set.
-- Bill Hawes
"I have no employer, but if I did, they would share these opinions."pete@violet.berkeley.edu (Pete Goodeve) (03/11/88)
> I've had a chance to read and digest some of the ongoing IPC thread and > have a number of comments and suggestions to add. First an introduction: > I'm Bill Hawes, perpetrator of ARexx (and ConMan, and WShell, and ...) Hey Bill... Welcome to the net! And about time! Now that you're here, I have a couple of quick questions: > ARexx marks its messages with the string "REXX" in the ln_Name field of > the message, so you can easily tell whether it comes from a REXX macro. Is there a pointer to the STRING "REXX" in the ln_name field, or is this just those four characters inserted into the longword? For some reason, in all our discussions, we've forgotten about this field (though I've just been suggesting it's use to mark ports). This location could probably perfectly well be used for our proposed four-character message-type ID, rather than a pointer to a string. If you happen to do it that way, we'd have immediate compatibility, with "REXX" being one of the possible message types in a more general scheme. [And if you DON'T do it that way, would you consider changing? :-)] [Does anyone see any problem with using this field in this way? I've never seen anyone want to name a message, actually, though I think a pointer to sundry special data has sometimes been put there.] I still feel strongly that the RexxMsg format is inappropriate for a lot -- if not most -- of the message passing that interacting processes that didn't use Rexx would want to do. We want something initially as unstructured as possible, yet formalized enough so that a program can know what action to take. As to the Rexx language itself, my main feeling is that we don't want to lock out other possibilities at this stage. We all know that preferences in the computer field are largely religous, so it would be nice to have variety for people to choose their favorites from. Any you can tell from the way that the discussion has been going that a few people would like to have a crack! I have one question here too: is Rexx single threaded, or is there some way that several processes can have scripts active at once? > The issue of standardized commands is important, but imagine the howls of > outrage had I attempted to dictate what commands an editor, a database, > and a paint program should support! The command set should be decided by > the program's developer, ideally in conjunction with other developers of > related (re: competing) products. No disagreement here. (But -- repeating myself one more time -- I think the same goes for message formats, except that to be useful these must really be determined by consensus.) -- Pete --
jesup@pawl14.pawl.rpi.edu (Randell E. Jesup) (03/12/88)
In article <7590@agate.BERKELEY.EDU> pete@violet.berkeley.edu.UUCP (Pete Goodeve) writes: >Is there a pointer to the STRING "REXX" in the ln_name field, or is this >just those four characters inserted into the longword? For some reason, in >all our discussions, we've forgotten about this field (though I've just >been suggesting it's use to mark ports). This location could probably >perfectly well be used for our proposed four-character message-type ID, >rather than a pointer to a string. If you happen to do it that way, we'd ... >[Does anyone see any problem with using this field in this way? I've never >seen anyone want to name a message, actually, though I think a pointer to >sundry special data has sometimes been put there.] PLEASE don't put random data in a standard pointer position! It can really cause some annoying problems/misunderstandings down the road. Rexx uses that to say where it came from, not what it's sending. Use the command field. >I still feel strongly that the RexxMsg format is inappropriate for a lot -- >if not most -- of the message passing that interacting processes that >didn't use Rexx would want to do. We want something initially as >unstructured as possible, yet formalized enough so that a program can know >what action to take. Actually, if you don't want lots of incompatible extensions, I'd advise a fairly structured definition as early as possible. Perhaps Bill Hawes could come up with a subset REXX message that would please you, but allow stuff with rexx ports to work with this stuff (ex: TeX). Just what don't you like about the Rexx messages? >I have one question here too: is Rexx single threaded, or is there some way >that several processes can have scripts active at once? I think it's multitasking (it's a library). // Randell Jesup Lunge Software Development // Dedicated Amiga Programmer 13 Frear Ave, Troy, NY 12180 \\// beowulf!lunge!jesup@steinmetz.UUCP (518) 272-2942 \/ (uunet!steinmetz!beowulf!lunge!jesup) BIX: rjesup (-: The Few, The Proud, The Architects of the RPM40 40MIPS CMOS Micro :-)
pete@violet.berkeley.edu (Pete Goodeve) (03/14/88)
peter@nuchat.UUCP (Peter da Silva) writes: > I don't like the idea of using the replyport for anything but reply messages. > I think there's enough semantic confusion already in the system to add more. In some ways I agree, but on the other hand extra ports also add to the confusion a bit. In any general scheme, a program is going to have to expect messages from various sources anyway, so if replies and original messages are mixed up does it really confuse things much? After all there is that NT_REPLYMSG code I want to be enforced. In the case of a program requesting a "topic" from a message broker, for instance, if the reply port was only used for that -- and you had to specify another destination port -- it would sit totally idle after the initial opening of the channel. I don't think the problem is a critical one; let's leave both options open. > [suggests some possible message IDs, including:] > > 'FILE' followed by a directory lock and a character pointer. > 'CSTR' followed by a pointer to a C string. > 'BSTR' followed by a pointer to a BCPL string. > 'PORT' followed by a message port. > 'FORM' followed by *a pointer to* an IFF FORM. > Making it a pointer shouldn't cause anyone any problems, > and should make it easier to pack stuff into a message. > 'CAT ' > 'LIST' followed by a pointer to IFF CAT or LIST > [....] Great! That's the sort of starter list I was hoping for. A few minor cavils (naturally :-)). You probably shouldn't even mention BSTRs in polite messages... Let's hope that text at least can be standardized to null terminated format (with newlines allowed of course). I agree that IFFs (and other large data items) should normally be referenced by pointers, rather than being in-line in the message, though there are exceptions to this rule (which I want to get into in detail in another posting). I would prefer, though, that we just have ONE message ID for all IFFs (presumably just "IFF ") with the following pointer addressing the actual "FORM" or whatever in its usual place at the head of the data block. We can reserve IDs "FORM", "CAT" and "LIST" for the (rare) occasions they would have to be in-line. > > I like the simplicity and security you get if you always reply to a message > on its reply port. The only thing changed in the reply should be the status > word. Yeh, there should be a status word in the structure somewhere, perhaps > immediately before or after the ID. This ain't always what we want... After all one of the uses of a message is to get a Reply! The normal place to put the information being returned is into the message data block itself, so we cannot insist that the message remains unchanged. Actually the "status" return is another point that needs more discussion. There are two sorts of status as I see it: there's information about whether the server could handle the message or not (essentially YES/NO), and then there's all the other flags that might be connected to a particular message type. The second class should obviously be in the data portion of the message, but the natural place for the first is in the mn_Node.ln_Type slot that is already used for NT_MESSAGE and NT_REPLYMSG. We might even use the null type NT_UNKNOWN for a failed message, but there are lots of spare codes. The only other general status that I can think we might need at the moment is "Server no longer exists", if we implement some Port Broker process. Does anyone see any others? In aother article, Peter writes: > In article ... pete@violet.berkeley.edu ( Pete Goodeve ) writes: > > [about a real neat idea for a message broker] Uh, thanks! > .......... You volunteering to implement it? (half-smiley: > someone has to). I guess I sort of am (whenever I get time!!). I certainly would like to... > For some purposes you still need a port server. It's a lot less overhead once > the connection is made, and it can do some of the same things. It could even > look the same to the user. I've come around fairly solidly to the same point of view. It seems about the cleanest way to solve the FindPort hazard, and it should integrate nicely with the message broker (you either request a port or request a topic). I'm trying to put together a new summary of these points we've been tossing back and forth and get into other things we've left out so far, but that may take a few days.... -- Pete --