abbott@aero.org (Russell J. Abbott) (04/30/91)
In developing a mental model for myself of what ISIS provides, I came up
with the following. I haven't seen this in any of the ISIS literature,
although that may simply mean that I haven't read enough of it.
A nice conceptual model for a process group would seem to be
network-level object. ("Distributed object" is an alternative term, but
I prefer "network-level object" or perhaps just "network object.")
The term "object" seems appropriate since process groups include
processes that define responses to messages. In addition, each process
group has (or may be built so that it has) a single internal state.
The term "network-level" also seems appropriate since process groups are
not defined in terms of their residence on any particular processor.
Instead they (are able to) use the resources of whichever processors are
available.
Network-level objects differ from traditional objects in another way
also. They are capable of internal concurrency. Notwithstanding their
internal concurrency, however, they (are able to) maintain a consistent
state.
It seems to me that the notion of a network-level object would be quite
attractive to anyone interested in implementing network level services.
Most network resource servers (as distinct from something like an X
server on a particular display) can probably be conceptualized quite
nicely as a network object. The ISIS claim that virtual synchrony is
central to much of what goes on in distributed computing would be made
very concrete in this way.
Since I have done nothing to help in the conceptualization or
development of ISIS, I suppose it is gratutious of me make the following
suggestion, but here it is anyway. (I guess "gratuitous" applies in
both senses.)
It seems to me that the term "process group" is misleading and
unsuggestive for potential ISIS users. It would be better to present
ISIS as providing a network-level object capability as suggested above.
Process groups may then be seen as the (internal) mechanism that allows
for this capability.
With the preceding in mind, I would also recommend that ISIS do more to
enforce (or at least strongly recommend) consistency among processes
within process groups so that a process group will (unless explicitly
defined otherwise by a user) consist of processes that implement
network-level objects. I imagine, though, that the issue of
intra-process group consistency is one that the ISIS group must have
debated. I wonder why you made the decision not to go in that
direction.
-- Russ Abbott@itro3.aero.orgrcbc@cs.cornell.edu (Robert Cooper) (04/30/91)
In article <1991Apr29.205624.17754@aero.org>, abbott@aero (Russell J. Abbott) writes: >With the preceding in mind, I would also recommend that ISIS do more to >enforce (or at least strongly recommend) consistency among processes >within process groups so that a process group will (unless explicitly >defined otherwise by a user) consist of processes that implement >network-level objects. I imagine, though, that the issue of >intra-process group consistency is one that the ISIS group must have >debated. I wonder why you made the decision not to go in that >direction. > >-- Russ Abbott@itro3.aero.org Russ has some good points. I've always thought that Isis programs were a bit prone to errors of inconsistency between group members. E.g. where one group member joins the group specifying PG_LOGGED (to enable logging for total resiliency) but another member neglects to do so. Another example is where an entry must be called using abcast to ensure correct semantics of the application. One idea we've been bouncing around for a while without actually implementing it is the following: (1) Entry numbers are on a per-group basis. Currently entry numbers a global, or at least process-wide, rather like RPC procedure numbers. Instead, entry numbers would be unique only within a group. Among other things this would make management and allocation of entry numbers easier for programmers. There would continue to be a process-wide entry numbers for compatability purposes though. (2) Associated with a group entry number would be some attributes, including "Ordering" which would specify ABCAST or CBCAST (or either I suppose). Other attributes would include something I call "Extent" which says how many group members the bcast should be sent to (e.g. all, majority, one), and "Replies" which says how many replies should received before considering the bcast to have succeeded (e.g. all, one etc.). (3) Groups would also have attributes specifying options such as PG_LOGGED that are currently specified as arguments to the group join operation. Other attributes come to mind, such as a minimal and optimal number of group members. These attributes would be enforced as follows: Group attributes would be defined by the first process to join the group, in its arguments to the pg_join operation. Also there would be an isis_group_entry routine that would take a list of entry attributes. These group and entry attributes would be bundled up into a piece of group state that would be transferred to other group members as they join. If those members violated these attributes, i.e. they specified incompatible arguments to their own pg_join and isis_group_entry operations, then they would be prevented from completing their join to the group. These group and entry attributes would perhaps be specified in some group interface language. However, the above proposal does not rely on any one interface language, leaving us free to fit in with different interface languages. One example language is the IDL language developed by ANSA/ISA project in Cambridge England. Currently we have a visitor at our group from that project, so you may see some activity in this area. -- Robert Cooper
jyl@noam.Eng.Sun.COM (Jacob Levy) (04/30/91)
Russell
Robert Cooper at Cornell has been working on this issue and I
have also spent some time thinking about it. Robert has done an
initial implementation of his ideas - I'll defer to him to
describe what he has obtained so far. Here is what I've been
thinking about:
Similar to your reasoning, I also think of ISIS pgs as network
objects. There is a problem of efficiency, however. The PG
mechanism does not scale well as the number of pgs grows into
thousands. Specifically, the problem is located in the pg view
update code. Consider a process which is member of several
thousand pgs. When such a process fails, several thousand
invocations of the pg view update protocol will be run. This is a
globally synchronous (abcast) type protocol. Clearly the cost
would be prohibitive.
The solution which Robert came up with and which I embellish here
is that each process is member of several (a fixed number) of
ISIS pgs. Networked objects are mapped onto these fixed pgs and
used as a kind of "lightweight" pg. A "name service" manages the
lightweight pg namespace. As you suggest, it is important that
consistency among processes participating in a lightweight pg be
enforced so that each process indeed implements that network
object abstraction. To achieve this, we have processes register
with the name service the types of network objects they are able
to implement. The name service creates heavyweight pgs from those
processes implementing each network object, and maps network
objects to the appropriate "type" of heavyweight pg.
A lightweight pg (serving as a network object) thus consists of
the following information:
struct lightweight_pg {
/*
* Where in the heavyweight pg name space is the object?
*/
address heavyweight_pg;
/*
* A token to identify the lightweight pg
*/
char *name;
};
The problem with this solution is that it is still not scalable,
due to the presence of the name service. In order to make it
scalable, we must allow a name service to store names managed by
other name services. Thus, we store, in the lightweigth pg
structure an additional address, that of the heavyweight pg
which provides the namespace management for this network object:
struct lightweight_pg {
address heavyweight_pg;
char *name;
address namespace_manager;
};
The advantages are:
a. The namespace manager is the natural place to implement
replication, persistence, storage, activation, deactivation
etc. policies.
b. The namespace manager can be used as a transparent fallback if
communication with the heavyweight pg hosting the object is
lost. The namespace server could then reactivate the object in
another heavyweight pg.
c. Namespace managers can store names managed by other namespace
managers. This provides scalability and allows the construction
of a general graph of name spaces.
d. We can hide the notion of heavyweight pgs entirely beneath
this layer.
Does all this sound familiar?
--JYL