timk@cs.qmw.ac.uk (Tim Kindberg) (03/15/91)
I have some questions about Mach facilties, and would be very grateful to hear from anyone with some answers: 1 Message passing. I know about the use of copy-on-write for intra-machine message passing, but I never could find any mention of using copy-on-reference for large messages sent over a network. There's an obvious problem of dependency upon some other machine for fetching pages after the message has been received. a. Is it the case that this feature hasn't been implemented - for this and perhaps other reasons - and are there any plans to include it? I note that the Chorus developers seem to have decided against including a similar feature. b. Given that copy-on-reference isn't used, how/when is buffering done for large messages sent asynchronously across the network? c. Could somebody clarify when I can/should/must use out-of-line data as opposed to in-line data? Out-of-line data is useful for scatter/gather, but does it have to be used for "large" amounts of data, and is out-of-line data always sent copy-on-write (intramachine), and therefore must be page-aligned? What, in practice, do programmers (mostly) do? d. Can (part of) a region created in receiving a message be sent in a further message? 2 Sending port rights. Is there anyone out there sending port receive rights in messages? I would very much like to hear about the application context in which you chose to do this, and how you found using this mechanism. 3 Migration. The Accent work on process migration doesn't seem to have been carried over to Mach. Are there any plans in this direction? (Accent migration also used copy-on-reference, with a residual dependency problem similar to that referred to in 1 above). -- Tim Kindberg UUCP: timk@qmw-cs.uucp | Computer Science Dept ARPA: timk%cs.qmw.ac.uk@nsfnet-relay.ac.uk | QMW, University of London JANET: timk@uk.ac.qmw.cs | Mile End Road Voice: +44 71 975 5236 (Direct Dial) | London E1 4NS
mib@geech.ai.mit.edu (Michael I Bushnell) (03/18/91)
In article <2991@redstar.cs.qmw.ac.uk> timk@cs.qmw.ac.uk (Tim Kindberg) writes:
I have some questions about Mach facilties, and would be very
grateful to hear from anyone with some answers:
I'll do my best.
1 Message passing. I know about the use of copy-on-write for
intra-machine message passing, but I never could find any mention
of using copy-on-reference for large messages sent over a network.
There's an obvious problem of dependency upon some other machine
for fetching pages after the message has been received. a. Is it
the case that this feature hasn't been implemented - for this and
perhaps other reasons - and are there any plans to include it? I
note that the Chorus developers seem to have decided against
including a similar feature.
One of the advantages of the Mach scheme over the amoeba scheme is
that the Mach microkernel knows nothing about networks and such. The
netmsgserver, which is responsible for providing inter-machine IPC
(transparently, of course) could use such a technique, but I don't
know if it's been done. Since the data is almost always used, I don't
think you save very much.
b. Given that copy-on-reference isn't used, how/when is buffering
done for large messages sent asynchronously across the network?
The current netmsgserver doesn't buffer at all.
c. Could somebody clarify when I can/should/must use out-of-line data as
opposed to in-line data? Out-of-line data is useful for scatter/gather, but
does it have to be used for "large" amounts of data, and is out-of-line
data always sent copy-on-write (intramachine), and therefore must be
page-aligned? What, in practice, do programmers (mostly) do?
You can use it whenever you want. If it isn't page aligned, or an
integral number of pages, then the receiver will be able to read some
data before/after the data you intended to send, but other than that
it works correctly. Out-of-line is faster (even if going over a
network) because it reduces the number of copies. The drawback is
that for small amounts of data, the cost of allocating memory in the
recipient may be larger than the cost of the copy. Certainly this is
true for less than a page (not to mention the data that peeks
through).
d. Can (part of) a region created in receiving a message be sent
in a further message?
Yes, with "correct" semantics.
2 Sending port rights. Is there anyone out there sending port
receive rights in messages? I would very much like to hear about
the application context in which you chose to do this, and how you
found using this mechanism.
Port rights are sent constantly. This is an essential part of the
Mach model. In the GNU multi-server, port rights represent (among
other things) files, and filesystems hand port rights to clients when
they open a file or translate a pathname to a file (for a later stat,
for example). The return port for an RPC is also sent as a port right
in the message header, and apart from some additional atomicity
guarantees, operates like a port right sent in a message.
3 Migration. The Accent work on process migration doesn't seem to
have been carried over to Mach. Are there any plans in this
direction? (Accent migration also used copy-on-reference, with a
residual dependency problem similar to that referred to in 1
above).
You'd need to implement a server that supported the task abstraction
and did process migration. This is more than I want to contemplate
doing, and it would significantly slow the system. There may be a
better way, but I don't know of one that doesn't involve kernel mods.
I know of no one who is working on this.
-mibcgy@cs.brown.edu (Curtis Yarvin) (03/19/91)
In article <MIB.91Mar18050418@geech.ai.mit.edu> mib@geech.ai.mit.edu (Michael I Bushnell) writes: >In the GNU multi-server, port rights represent (among >other things) files, and filesystems hand port rights to clients when >they open a file or translate a pathname to a file (for a later stat, >for example). Excuse my ignorance; but is this GNU server out yet? If so, where can I get it? If not, can I at least get doc somewhere?
mib@geech.ai.mit.edu (Michael I Bushnell) (03/19/91)
In article <68978@brunix.UUCP> cgy@cs.brown.edu (Curtis Yarvin) writes: In article <MIB.91Mar18050418@geech.ai.mit.edu> mib@geech.ai.mit.edu (Michael I Bushnell) writes: >In the GNU multi-server, port rights represent (among >other things) files, and filesystems hand port rights to clients when >they open a file or translate a pathname to a file (for a later stat, >for example). Excuse my ignorance; but is this GNU server out yet? If so, where can I get it? If not, can I at least get doc somewhere? HAHAHAHAHAHHA... Ahem. Sorry. [Regain composure] No, it isn't out yet. Tee hee. Docs? Documentation? Tee hee. If you're seriously interested, I can mail you the MiG interface specifications for the filesystem and such, but I can't answer any questions since I'm actually trying to write code. -mib
timk@cs.qmw.ac.uk (Tim Kindberg) (03/19/91)
In <MIB.91Mar18050418@geech.ai.mit.edu> mib@geech.ai.mit.edu (Michael I Bushnell) writes: >In article <2991@redstar.cs.qmw.ac.uk> timk@cs.qmw.ac.uk (Tim Kindberg) writes: > I have some questions about Mach facilties, and would be very > grateful to hear from anyone with some answers: >I'll do my best. Thanks for your response... >One of the advantages of the Mach scheme over the amoeba scheme is >that the Mach microkernel knows nothing about networks and such. I'm not sure that the kernel's ignorance (keeping things simple) is by itself the justification: it's more that custom network protocol choice, port protection, ecryption etc. policies can go into an extra-kernel activity. But is the performance disadvantage resulting from the extra task switch one that everyone is prepared to accept? >The >netmsgserver, which is responsible for providing inter-machine IPC >(transparently, of course) could use such a technique, but I don't >know if it's been done. Since the data is almost always used, I don't >think you save very much. The V system used to provide two calls - copyto() and copyfrom() - using which a server could selectively read from or write to buffers in the client's address space. The idea was that the server might only need to fetch or write some of the client data, but the client gives it a capability to access any part of its request/result buffer, just in case. Use of copy-on-reference would have a similar rationale. Does anyone know if V still supports this, or have any relevant experience to report? > b. Given that copy-on-reference isn't used, how/when is buffering > done for large messages sent asynchronously across the network? >The current netmsgserver doesn't buffer at all. -so is all the data sent immediately across the network and buffered at the recipient kernel, regardless of quantity and of whether there is a task ready to receive it? Mightn't copy-on-write techniques be used to buffer the data at the sending kernel, until a task does a corresponding receive? > 2 Sending port rights. Is there anyone out there sending port > receive rights in messages? I would very much like to hear about > the application context in which you chose to do this, and how you > found using this mechanism. >Port rights are sent constantly. This is an essential part of the >Mach model. In the GNU multi-server, port rights represent (among >other things) files, and filesystems hand port rights to clients when >they open a file or translate a pathname to a file (for a later stat, >for example). The return port for an RPC is also sent as a port right >in the message header, and apart from some additional atomicity >guarantees, operates like a port right sent in a message. I know about the common case of transmitting sending port *send* rights. I am interested in sending port *receive* rights. A server can send port receive rights in a message in order, for example, to transfer its client(s) to another, less loaded server. Further responses would be very welcome.. -- Tim Kindberg UUCP: timk@qmw-cs.uucp | Computer Science Dept ARPA: timk%cs.qmw.ac.uk@nsfnet-relay.ac.uk | QMW, University of London JANET: timk@uk.ac.qmw.cs | Mile End Road Voice: +44 71 975 5236 (Direct Dial) | London E1 4NS
Rick.Rashid@cs.cmu.edu (03/19/91)
Let me add a bit to Michael's note: Ed Zayas in the mid-80's did a PhD thesis in which (among other things) he implemented and use network copy-on-reference in the Accent OS (the predecessor to Mach). His work included studying network copy on reference for normal message passing and for process migration. Basically, his results were encouraging, but it was felt within the group that the problems associated with using network copy on reference in the face of network and node failures were too great to justify this as a standard feature. Ed work did lead indirectly, though, to the subsequent development of a workable interface for external pagers. The role of network copy-on-reference in Accent has essentially been replaced by the ability in Mach for any task to create a memory object for which it is the backing store. This allows data to be explicitly (as opposed to implicitly) sent by reference in messages. The effect is the same, but it is a conscious decision of the implementor rather than something done by accident. Generic network shared memory is provided by the network shared memory server. -Rick
lowry@arnor.UUCP (Andy Lowry) (03/20/91)
In article <MIB.91Mar18050418@geech.ai.mit.edu>, mib@geech.ai.mit.edu (Michael I Bushnell) writes: |> In article <2991@redstar.cs.qmw.ac.uk> timk@cs.qmw.ac.uk (Tim Kindberg) writes: |> 2 Sending port rights. Is there anyone out there sending port |> receive rights in messages? I would very much like to hear about |> the application context in which you chose to do this, and how you |> found using this mechanism. |> |> Port rights are sent constantly. This is an essential part of the |> Mach model. In the GNU multi-server, port rights represent (among |> other things) files, and filesystems hand port rights to clients when |> they open a file or translate a pathname to a file (for a later stat, |> for example). The return port for an RPC is also sent as a port right |> in the message header, and apart from some additional atomicity |> guarantees, operates like a port right sent in a message. The question here was about receive rights. It sounds to me like you've covered lots of reasons for passing around send rights. I'd also be interested in hearing of applications for passing of receive rights. -- Andy Lowry, lowry@ibm.com, (914) 784-7925 IBM Research, 30 Saw Mill River Road, P.O. Box 704, Yorktown Heights, NY 10598
mib@churchy.gnu.ai.mit.edu (Michael I Bushnell) (03/21/91)
In article <2994@redstar.cs.qmw.ac.uk> timk@cs.qmw.ac.uk (Tim Kindberg) writes:
I know about the common case of transmitting sending port *send*
rights. I am interested in sending port *receive* rights. A
server can send port receive rights in a message in order, for
example, to transfer its client(s) to another, less loaded server.
Hmm...I can't think of any case aside from task initialization where
the GNU multiserver design sends receive rights. Some may arise;
there are places where I think I'll do it, but it's a kind of esoteric
occurrence. I suspect that the rarity of sending receive rights would
justify a netmsgserver which was speedy, and sending a receive right
was potentially expensive.
-mibRichard.Draves@cs.cmu.edu (03/29/91)
> Excerpts from netnews.comp.os.mach: 18-Mar-91 Re: message passing & > migra.. Michael I Bushnell@geech (3703) > c. Could somebody clarify when I can/should/must use out-of-line > data as > opposed to in-line data? Out-of-line data is useful for > scatter/gather, but > does it have to be used for "large" amounts of data, and is > out-of-line > data always sent copy-on-write (intramachine), and therefore must be > page-aligned? What, in practice, do programmers (mostly) do? > You can use it whenever you want. If it isn't page aligned, or an > integral number of pages, then the receiver will be able to read some > data before/after the data you intended to send, but other than that > it works correctly. According to the interface spec, the receiver should not be able to read data that you did not intend to send. If out-of-line data isn't page aligned or an integral number of pages, then the excess data should show up zeroed. Michael is correct in that the correct VM copy-on-write code does not zero, but this is a bug. Rich