rcbc@honir.cs.cornell.edu (Robert Cooper) (06/10/89)
In article 22 or comp.sys.isis Ken Birman has started a discussion of
"object oriented" remote interfaces in ISIS. I'll elaborate on some of our
current plans to support the notion of a group as a "module" which
implements a well-defined "interface". This could provide a basis for a
more "object oriented" ISIS programming style, but I'll leave that subject
alone for now.
The problem:
------------
ISIS provides broadcast and other primitives with well-defined behaviour.
But when using these primitives the programmer still has many choices to
make. For instance the initiator of a broadcast must know whether to wait
for replies from ALL group members, a MAJORITY of members, or wait for just
1 reply. The initiator must know whether to use ABCAST, which provides a
totally ordered, atomic broadcast, or CBCAST which has weaker semantics but
is more efficient. The receivers of the broadcast must know whether to send
a nullreply or an abortreply to message that are not of interest to it.
One way to look at this issue is to say that an ISIS program implements
some application level protocol on top of the ISIS primitives.
These issues become problematical when doing programming-in-the-large. The
different "modules" in your application must have well-defined interfaces
so that diverse clients know exactly what the interface provides and how to
correctly use the interface. Clearly aspects such as ALL versus MAJORITY,
and the flavour of broadcast algorithm used are crucial to the overall
correctness of the resulting system, and ought to be expressed in the
interface.
Another issue is managing software upgrades. In a reliable distributed
application the lifetime of an "execution" of the application (i.e. the
time between total simultaneous failure of all system components) can be
longer the the time between modifications to the source code for that
system. If we are to produce truly reliable distributed applications we
must be able to change the implementation of its components while the
system continues to operate. And we must be able to detect when components
with incompatible versions of an interface try to communicate.
Group and entry attributes:
---------------------------
We plan to address this problem by providing viewing a process group as a
"module" which implements some "interface" and which uses some "protocol"
(defined in terms of ISIS broadcast primitives) for remote communication.
We plan to associate the following "attributes" with a group. The first two
attributes already exist but the third is new:
o size The number of member processes.
o incarnation A number incremented each time the group recovers after
a total failure.
o entries The set of messages, or entry points, that every member
of the group will accept.
Associated with each entry are the following attributes:
o entry number This identifies the function to be performed by the service,
and is the equivalent of the procedure number in many RPC
interfaces. Entry numbers range from 1 to 100, and must
be unique within a particular group.
o ordering A set chosen from TOTAL, ATOMIC, CAUSAL, or FIFO,
corresponding to GBCAST, ABCAST, CBCAST, and FBCAST
respectively, specifying the permissible broadcast
methods that a client of this group should use.
o replies Specifies the number of successful replies to be required
from a broadcast. One of ALL, MAJORITY, ONE, or NONE.
o format This specifies the number and types of data objects in
the request message and any reply message.
o version A version number associated with this entry, which is
incremented whenever some attribute of this entry
changes. When an interface changes we invisage that group
members will support both the old and new versions of the
interface while a management and configuration tool
incrementally crashes and restarts existing clients until
no obsolete clients exist.
Currently much of this kind of information about groups and entries is
encoded in the implementation of group members or in the client
application. By making these attributes explicit we can more easily ensure
that clients and group members are behaving compatibly with respect to a
group interface. The benefits for program correctness are clear.
A group member would specify these attributes when it joined a group. If
its set of attributes did not match those of the existing group members an
error would occur. On the client side, information about group attributes
would be used to automatically select the appropriate broadcast protocol.
Thus we free the application programmer from continually reimplementing common
communication patterns. And this extra information may provide opportunites
for the ISIS system to optimize some kinds of operations.
We can imagine other candidates for group and entry attributes. For
instance, whether a group uses logging to survive total failure (i.e.
simultaneous failure of all its members), or whether an entry is
idempotent, making retries upon failure easy. Since multiple groups may
accept the same entry list, perhaps we ought to define a notion of a
"group type", to capture this.
-- Robert Cooper (rcbc@cs.cornell.edu)