mccarrol@topaz.rutgers.edu (Mark C. Carroll <MC>) (04/03/88)
Proposal: Standard IPC protocol for Remote File Transfer Protocols
Author: Mark C. Carroll
Date: April 2, 1988
Address: mccarrol@topaz.rutgers.edu ( {backbone}!rutgers!topaz!mccarrol )
carroll@aim.rutgers.edu
Snail: Mark C. Carroll
674 Cedarbrook Rd
Bridgewater, NJ 08807
------------------------------------------------------------------------
The following is my proposal for some standard mechanism for external file
transfer protocols for terminal programs. Although this may end up being
incompatible with whatever IPC system we finally agree on, some method of
external protocols is needed now, and the IPC debate shows no signs of
coming to any consensus quickly. Since several people are already working
on putting external protocols into terminals, I think we need some sort of
system NOW. This is my attempt. Flame it if you must. But constructive
suggestions are better.
------------------------------------------------------------------------
The method that I propose is a direct extension of Execs message passing
system. It will use standard Exec message ports, and an extension of the
Exec message structure:
struct RFTP_Data
{
struct Node *RFTP_List;
long data_len;
char *data;
};
struct RFTP_Message
{
struct Message rftp_msg;
long rftp_type,rftp_len;
struct List rftp_data;
};
In this structure, the rftp_type field is one of:
RFTP_RECFILE - recieve file command. A list of filenames is in
the Data field.
RFTP_SENDFILE - send file command. A list of filenames is in the
Data field.
RFTP_DATA - character data. Data field is string of data being
recieved/transmitted.
RFTP_MESSAGE - message to be displayed to the user. Data field
is the actual message to display.
RFTP_ERR_TIMEOUT - timeout error. Data field is empty.
RFTP_ERR_MIST - other error. Data field is error message to be
displayed.
The rftp_len field is the length of the Data field.
The rftp_data field is the string of data. In a message from the terminal
to the RFTP process, this data would be the incoming data from the remote
system. In a message from the RFTP process to the terminal, this would be
either characters to transmit (RFTP_DATA), or a message to the user
(RFTP_MESSAGE, RFTP_ERR*). The rftp field is a list. This list should only
have more than one member on RFTP_RECFILE and RFTP_SENDFILE.
The terminal program would be responsible for somehow obtaining the name of
the message port owned by the RFTP process. ( <MC>Term uses a line in it's
startup file like "RFTP {Zmodem}{Zmit}{Zport}" to define an additional
protocol named Zmodem, which can be loaded with the command Zmit, and whose
message port is named Zport.)
--------------------------------------------------------------
Well, there it is. Please distribute it, and send suggestions, comments,
flames, or praise to me at any of the addresses in the header.
<MC>
--
<MC>=Mark C. Carroll,Rutgers CS Student| We try to keep ourselves detatched
mail to: ARPA: CARROLL@AIM.RUTGERS.EDU | It's clear who holds the key
UUCP: mccarrol@topaz.rutgers.edu | Drifting though the age of reason
(backbone)!rutgers!topaz!mccarrol | Now your washed up on the shore -GTRacs@amdahl.uts.amdahl.com (Tony Sumrall) (04/06/88)
In article <Apr.2.13.17.28.1988.2684@topaz.rutgers.edu>
mccarrol@topaz.rutgers.edu (Mark C. Carroll <MC>) proposes a standard IPC
protocol for file transfer programs. I'm implementing external xfer pgms
in VT100 R2.9 *now* and am *not* doing anything near what's proposed. What
I *am* doing is allowing the user to specify the complete command that is
to be used to invoke the external program (complete with replaceable host
and local file names). The reasons that I'm not doing IPC are pretty
simple:
1) I don't see what having an IPC will buy the user (in this case).
I need to be convinced that implementing such a beast will actually
benefit the end user. (Yes, I'll read follow-ups.)
2) My method won't require any modification to the external pgm. You
could write an external comm pgm in BASIC if you wanted and it
should work just fine. If a commercial comm pgm has a protocol
that you feel you need, just invoke it.
3) You don't have to learn the IPC protocol just to build your
external xfer pgm.
4) My method accomplishes the job with a minimum of effort (on my
part) and a minimum of effort on the part of the developer of the
external xfer pgm.
5) The standard doesn't exist yet.
As far as the proposal itself, I have a few questions:
1) How many bytes of data is the owner of the serial port supposed to
bundle up into the message that it sends to the external pgm?
2) How does the SP owner (serial port owner) know that the external
pgm is finished? or abended?
3) How does the SP owner know when to send the client a TIMEOUT msg?
4) How does the client know the pertinent settings of the serial port
(e.g. parity, baud rate, break time, end-of-line chars)?
5) How does the SP owner quiesce the client (in case the user decides
to abort the xfer or the user detects a loop or other failure in
the client)?
I'm not saying that I won't implement IPC in a later version...I will if
and when it becomes stable and I can discern a tangible benefit to the
end-user.
--
Tony Sumrall acs@amdahl.uts.amdahl.com <=> amdahl!acs
[ Opinions expressed herein are the author's and should not be construed
to reflect the views of Amdahl Corp. ]doug@eris (Doug Merritt) (04/06/88)
In article <26736@amdahl.uts.amdahl.com> acs@amdahl.uts.amdahl.com (Tony Sumrall) writes: > [...] I'm implementing external xfer pgms >in VT100 R2.9 *now* and am *not* doing anything near what's proposed. What >I *am* doing is allowing the user to specify the complete command that is >to be used to invoke the external program (complete with replaceable host >and local file names). The reasons that I'm not doing IPC are pretty >simple: I have a very simple counterpoint. I want to write higher-level layers on top of the transport protocol (if that's the right term). I want to do a fancy user interface that does lots of stuff (let's say somewhat like 'dnet', for purposes of discussion), including supporting some file transfer methods. If vt100 lets me tell *it* what to do via IPC, I can use it. Otherwise I can't, and have to roll my own. (Actually this came up in regard to an expert user agent that would handle mail and news on all my various Unix and BBS accounts in a really smart way.) You are taking the exact opposite tact, of calling some external program which will do the transport layer stuff for you. Fine, for adding new and obscure protocols. Probably a good idea. But certainly has nothing to do with the issue of letting some other process do the same with *you* (vt100, that is). In short, if you don't have something very similar to an IPC method, then the features being supported in vt100 have to be reinvented if we want another layer on top of it. As a matter of fact, the "right" way to do this is to go ahead and implement *all* of vt100's transfer protocols as separate tasks that vt100 communicates with via IPC, for the following reasons: 1) vt100 itself becomes much smarter, because I can still talk to it even when it is nominally "busy" doing a file transfer. With the (old) V2.6 vt100 I use, it's real easy to screw things up due to the lack of this feature. I.e. I tell a remote host kermit to send a file, then I tell vt100 to receive it...oops, accidentally selected "Kermit send"... now vt100 is confused. Another example is the apparent impossibility of telling vt100 to abort a transfer (all it has to do is send "kermit bye"!) You don't *have* to use cooperating tasks to fix these kinds of bugs individually...but I claim it's the only clean way to design-in a fix for *all* of them. 2) The transfer protocol tasks are then available to entirely different layers which don't use the vt100 user interface, which is the kind of bonus that lead to all this interest in designing an IPC interface to start with. 3) Another bonus is that I can now effectively extend vt100 just by writing another task that controls how vt100 works. This is similar to points 1 and 2 but comes up in a different context. A related point: the above applies to any kind of "slow" activity... vt100 should use IPC for sending transfered files to/from the disk, too. In this particular case it's easy to argue that pipes are better; ok, that's still a *form* of IPC to another task. The point is that you want the overseer to remain reactive even when slow I/O is happening, whether from modem, diskette, tape (maybe next year), ray tracing or whatever. So it seems to me that the only question is whether you care about how flexible your software is. Do you care about whether it's easy to use for new applications? Yes? Then use IPC. Doug Merritt doug@mica.berkeley.edu (ucbvax!mica!doug) or ucbvax!unisoft!certes!doug
acs@amdahl.uts.amdahl.com (Tony Sumrall) (04/07/88)
In article <8380@agate.BERKELEY.EDU> doug@eris.BERKELEY.EDU (Doug Merritt) writes: >In article <26736@amdahl.uts.amdahl.com> acs@amdahl.uts.amdahl.com (Tony Sumrall) writes: >> [...] I'm implementing external xfer pgms >>in VT100 R2.9 *now* and am *not* doing anything near what's proposed. What >>I *am* doing is allowing the user to specify the complete command that is >>to be used to invoke the external program (complete with replaceable host >>and local file names). The reasons that I'm not doing IPC are pretty >>simple: Whoa, hold on a sec, Doug...you left out my reasons! In particular, you left out the last on the list (but probably the biggest single reason): the interface isn't designed yet! How can I implement an IPC method if I don't know what to implement? I *did* say that I would implement it if and when it became stable and I was convinced that it was of value to the end-user. I stand by this. Now, on to your reasons that I should do it. >I have a very simple counterpoint. I want to write higher-level layers >on top of the transport protocol (if that's the right term). I want >to do a fancy user interface that does lots of stuff (let's say somewhat >like 'dnet', for purposes of discussion), including supporting some file >transfer methods. If vt100 lets me tell *it* what to do via IPC, I can >use it. Otherwise I can't, and have to roll my own. OK by me but VT100 is kind of a large "agent". It sounds like you want to replace the serial.device with VT100. This doesn't sound too good to me since VT100 is a *terminal* program. Yeah, once IPC becomes stable and I implement it and the standard includes all of the stuff you want to do I don't see why you couldn't use it as your agent but, well, it just sounds like using a pair of pliers to the job of a socket wrench. >You are taking the exact opposite tact, of calling some external program >which will do the transport layer stuff for you. Fine, for adding >new and obscure protocols. Probably a good idea. But certainly has >nothing to do with the issue of letting some other process do the >same with *you* (vt100, that is). Well, if you call ZMODEM and CIS-B obscure protocols then I agree. >In short, if you don't have something very similar to an IPC method, >then the features being supported in vt100 have to be reinvented if >we want another layer on top of it. Sounds to me like VT100 should become a layer on top of a general IPC facilitator for the serial.device--VT100 wasn't designed to be an interface to the serial port for other programs. I'd be more than happy to replace the serial port interface code in VT100 with IPC interface code to talk to the IPC serial port server! >As a matter of fact, the "right" way to do this is to go ahead and >implement *all* of vt100's transfer protocols as separate tasks that >vt100 communicates with via IPC, for the following reasons: I didn't include your reasons mainly because I think you're headed in the wrong direction. The "right" way to do this is to write an IPC serial port server and modify VT100 to use *it*. Then VT100 would become a true terminal emulator with absolutely no tranfer protocols within it. Of course the proposed IPC standard for the serial port would have to be expanded to include a "listener" request (all data coming in the serial port is given to this client as well as any other clients) in order to provide for an ASCII capture. Once this is implemented you can have whatever terminal emulator you like. They become easier to write (no transfer protocols to worry about), smaller (for the same reason), and you (the end user) only have to keep the pieces that you need (don't use kermit? fine, don't bother keeping the IPC kermit program). Anyway, like I said in my original posting, I *will* implement IPC if/when it becomes stable and I can see a tangible benefit to the end user. I believe that 2.9 will be out before the IPC becomes a standard (if I could implement a standard that doesn't exist yet I'd be *much* higher paid than I am now :-). >So it seems to me that the only question is whether you care about >how flexible your software is. Do you care about whether it's easy >to use for new applications? Yes? Then use IPC. But, Doug, your last few statements sound to me like you asking if I've stopped beating my wife. "Do you care about whether it's easy to use for new applications?" Yes, I do. "Then use IPC." How does this follow? > > Doug Merritt doug@mica.berkeley.edu (ucbvax!mica!doug) > or ucbvax!unisoft!certes!doug -- Tony Sumrall acs@amdahl.uts.amdahl.com <=> amdahl!acs [ Opinions expressed herein are the author's and should not be construed to reflect the views of Amdahl Corp. ]
doug@eris (Doug Merritt) (04/08/88)
In article <26928@amdahl.uts.amdahl.com> acs@amdahl.uts.amdahl.com (Tony Sumrall) writes: >In article <8380@agate.BERKELEY.EDU> doug@eris.BERKELEY.EDU (Doug Merritt) writes: >> [...] If vt100 lets me tell *it* what to do via IPC, I can >>use it. Otherwise I can't, and have to roll my own. > >OK by me but VT100 is kind of a large "agent". It sounds like you want to >replace the serial.device with VT100. This doesn't sound too good to me >since VT100 is a *terminal* program. [ ... ] it just >sounds like using a pair of pliers to the job of a socket wrench. Although I could use my own wishes as an example, a convenient posting from Randy Groves on the same day serves as an even better counterpoint: :In article <4742@bcsaic.UUCP> randy@bcsaic.UUCP (Randy Groves) writes: :What I need, essentially is something like VT100 with some csh- or rexx- :like language built in or able to interact with the terminal session. :I need to build complex macros to process information on the host to which :I'm attached. The macro languages built into VT100 or other PD packages :I've seen just don't cut it. This demonstrates fairly effectively that, no matter what an author *thinks* his software will be used for, somebody out there will always want some kind of extension made to it that hasn't happened yet. He can either look around for a new package, as Randy requested info about, or do his own hacking to vt100, he can plain do without, OR if vt100 submits itself to IPC control, he can write a simple program to control it, without having to modify vt100's source. Allowing such control is highly desirable due to the impossibility of any piece of software being all things to all users. Or even of foreseeing all reasonable uses. > [...] The "right" way to do this is to write an IPC >serial port server and modify VT100 to use *it*. Then VT100 would >become a true terminal emulator with absolutely no tranfer protocols >within it. Great! I agree, that's *definitely* the right way to go. >> Do you care about whether it's easy >>to use for new applications? Yes? Then use IPC. > >But, Doug, your last few statements sound to me like you asking if I've >stopped beating my wife. Sorry, no offense intended. It may sound that way, but that's because I really do regard this (IPC necessity) as an obvious conclusion. Again no offense, but I feel like arguing in favor of IPC support is like arguing in favor of multitasking. Certainly you have to give it a shot with PC/Mac/Atari/Etc folks, but sometimes they won't really understand until they get hands-on experience. A closer analogy is arguing in favor of pipes; this was something CP/M folks gave me a hard time about when I told them about UNIX pipes. They'd make similar points to yours, about how piping input to their programs was putting the cart before the horse. Sigh. I understand that it seems that way...but how do I explain the functionality gains they'd get if they just TRY being enthusiastic about pipes? (read as "IPC" here). As a matter of fact, it's true that pipes DO fail for some applications. And where they do, IPC message-passing takes over as an even more general paradigm, that gives you the same kind of improvement over pipes, that pipes gave you over, well, over batch-files/interactive-input-required/etc. > [...] I *did* say that I would implement it if and when it became stable >and I was convinced that it was of value to the end-user. [...] Glad to hear it. P.S. When I worked for Molecular some years ago (they made 64-processor z80 systems running a hacked up CP/M), it was difficult to explain the benefits of multitasking to people, because whenever someone needed a new process, they just used message passing to a whole 'nother processor! Which was great; let me tell you though, it was NOT as nicely integrated as the scheme you get from any self-respecting multitasking system. It could have been, but they didn't have experience with multitasking (or pipes) to show them the way. Let's learn from history...anything that increases the modularity of software is a big win (and adding IPC control certainly qualifies, as do pipes, multiprocessors, or multitasking). One last thing...perhaps it sounds like I think you don't believe in IPC, when you do see some benefits from it. Not at all; I just think that you have to apply it consistently as a philosophy. Ideally *any* piece of software should be drivable as easily by IPC as it is from user interaction...this allows each piece of software to become a module instead of a dead end. Look at what's been done with mouse-movement recorders and CLI/shell session savers/repeaters...very handy, but a pity you *have* to do it that way. Doug Merritt doug@mica.berkeley.edu (ucbvax!mica!doug) or ucbvax!unisoft!certes!doug
peter@nuchat.UUCP (Peter da Silva) (04/08/88)
In article ... mccarrol@topaz.rutgers.edu (Mark C. Carroll <MC>) writes: > struct RFTP_Message > { > struct Message rftp_msg; > long rftp_type,rftp_len; > struct List rftp_data; > }; Why a list? Wouldn't one packet per message be just fine for this sort of application. After all, the time is going to be dominated by the file transfer time. I'm not sure that the RFTP_DATA mode is either necessary nor desirable. You can always open a pipe and write data to it. Once again, the actual file transfer is going to be the limiting factor. In which case, here's my alternative suggestion: The file transfer program is started as a CLI program with Execute(). The command line format is: program SEND file... program RECIEVE file... The terminal program closes the serial port before Execute()ing the RFTP program, and opens it again when it's finished. > Well, there it is. Please distribute it, and send suggestions, comments, > flames, or praise to me at any of the addresses in the header. I think you need to go into a bit more detail as to exactly who sends what messages to whom, and when. Finally, here's the Pete/Peter IPC suggestion: Port name is RFTP.<protocol name>, eg: RFTP.KERMIT. Message type RCVF Items are type FILE. At most one item of type PORT may be provided. It points to a message port to recieve error messages. It may be the same as the reply port. Message type SNDF Items are type FILE. Items may be type LOCK, which the RFTP can open by using the hack of CurrentDir(Lock) and Open("", MODE_READONLY); At most one item of type PORT may be provided, as above. Message type EMSG /* Error message as described above */ Item type TEXT. Item FILE format is FILE, 8, (struct { ULONG dirlock; char *filename } *). Item PORT format is PORT, 0, (struct MessagePort *). Item TEXT format is TEXT, *, (char *). item LOCK format is LOCK, 0, (ULONG). -- -- a clone of Peter (have you hugged your wolf today) da Silva `-_-' -- normally ...!hoptoad!academ!uhnix1!sugar!peter U -- Disclaimer: These aren't mere opinions... these are *values*.
pete@violet.berkeley.edu (Pete Goodeve) (04/14/88)
In article <907@nuchat.UUCP> Peter da Silva writes: > In article ... mccarrol@topaz.rutgers.edu (Mark C. Carroll <MC>) writes: > > struct RFTP_Message > >.....[ describes his suggestion for a file transfer program message format.] [Peter makes some pertinent comments on the original suggestions.] > command line format is: > > program SEND file... > program RECIEVE file... ^^^^^ Ahem... "RECEIVE" [picky, picky..., but we'd better spell keywords right!] > I think you need to go into a bit more detail as to exactly who sends what > messages to whom, and when. My feeling as well. > Finally, here's the Pete/Peter IPC suggestion: > [...goes on to suggest a scheme using our message structure] I've no quarrel with the suggestions as such, but my basic feeling is that this whole approach is wrong. I find the current Tony Sumrall/Doug Merritt discussion much more to the point. They seem to be hashing out some good ideas there, such as a serial port IPC server. (Maybe I think they're good because they're pretty close to my original IPC suggestions from way back (:^)) -- Pete --
acs@amdahl.uts.amdahl.com (Tony Sumrall) (04/19/88)
In article <8457@agate.BERKELEY.EDU> doug@eris.UUCP (Doug Merritt) writes: >In article <26928@amdahl.uts.amdahl.com> acs@amdahl.uts.amdahl.com (Tony Sumrall) writes: >>In article <8380@agate.BERKELEY.EDU> doug@eris.BERKELEY.EDU (Doug Merritt) writes: >>> [...] If vt100 lets me tell *it* what to do via IPC, I can >>>use it. Otherwise I can't, and have to roll my own. >> >>OK by me but VT100 is kind of a large "agent". It sounds like you want to >>replace the serial.device with VT100. This doesn't sound too good to me >>since VT100 is a *terminal* program. [ ... ] it just >>sounds like using a pair of pliers to the job of a socket wrench. > >Although I could use my own wishes as an example, a convenient posting >from Randy Groves on the same day serves as an even better counterpoint: > >:In article <4742@bcsaic.UUCP> randy@bcsaic.UUCP (Randy Groves) writes: >:What I need, essentially is something like VT100 with some csh- or rexx- >:like language built in or able to interact with the terminal session. >:I need to build complex macros to process information on the host to which >:I'm attached. The macro languages built into VT100 or other PD packages >:I've seen just don't cut it. > >This demonstrates fairly effectively that, no matter what an author >*thinks* his software will be used for, somebody out there will always >want some kind of extension made to it that hasn't happened yet. He >can either look around for a new package, as Randy requested info about, >or do his own hacking to vt100, he can plain do without, OR if vt100 >submits itself to IPC control, he can write a simple program to control >it, without having to modify vt100's source. > >Allowing such control is highly desirable due to the impossibility of >any piece of software being all things to all users. Or even of foreseeing >all reasonable uses. But I interpreted Randy's request to mean that he wants to interact with the AGENT, not the terminal program. To my mind this means that the he wants to become a "listener" on the serial.device (i.e. both he and VT100 get ALL of the incoming data...both listeners can then respond). He could also talk to VT100, interjecting his own escape sequences or using VT100's resources to display info on the terminal session's windows. >> [...] The "right" way to do this is to write an IPC >>serial port server and modify VT100 to use *it*. Then VT100 would >>become a true terminal emulator with absolutely no tranfer protocols >>within it. > >Great! I agree, that's *definitely* the right way to go. Lemme see if I've got this right: you agree that VT100 should not be an agent. Rather, it is just an IPC user of the service that that the IPC serial server provides. Great! Now, let's hammer out the services that the IPC server should provide and get down to the nitty-gritty of defining the characteristics of the beast. In particular, please offer some insight on the questions that I asked in a previous posting: 1) How many bytes of data is the owner of the serial port supposed to bundle up into the message that it sends to the external pgm? 2) How does the SP owner (serial port owner) know that the external pgm is finished? or abended? 3) How does the SP owner know when to send the client a TIMEOUT msg? 4) How does the client know the pertinent settings of the serial port (e.g. parity, baud rate, break time, end-of-line chars)? 5) How does the SP owner quiesce the client (in case the user decides to abort the xfer or the user detects a loop or other failure in the client)? Don't try to spend time *convincing* me that IPC is a good thing, spend time defining the IPC interface. Once I see a definitive interface spec that has very few holes I'll be convinced. I'm willing to *work with you* on this! > Doug Merritt doug@mica.berkeley.edu (ucbvax!mica!doug) > or ucbvax!unisoft!certes!doug -- Tony Sumrall acs@amdahl.uts.amdahl.com <=> amdahl!acs [ Opinions expressed herein are the author's and should not be construed to reflect the views of Amdahl Corp. ]