rsalz@bbn.com (Rich Salz) (05/18/88)
Submitted-by: libes@cme-durer.ARPA (Don Libes)
Posting-number: Volume 14, Issue 97
Archive-name: sharedmem/part04
#! /bin/sh
# This is a shell archive. Remove anything before this line, then unpack
# it by saving it into a file and typing "sh file". To overwrite existing
# files, type "sh file -c". You can also feed this as standard input via
# unshar, or by typing "sh <file", e.g.. If this archive is complete, you
# will see the following message at the end:
# "End of archive 4 (of 4)."
PATH=/bin:/usr/bin:/usr/ucb ; export PATH
if test -f 'doc/usenix/paper.trf' -a "${1}" != "-c" ; then
echo shar: Will not clobber existing file \"'doc/usenix/paper.trf'\"
else
echo shar: Extracting \"'doc/usenix/paper.trf'\" \(19292 characters\)
sed "s/^X//" >'doc/usenix/paper.trf' <<'END_OF_FILE'
X.TL
XUser-Level Shared Variables (in a Hierarchical Control Environment)
X.AU
XDon Libes
X.AI
XNational Bureau of Standards
XFactory Automation Systems Division
XGaithersburg, MD 20899
X.AB
X.PP
XWe have implemented a shared variable system for
X.UX
X4.2BSD [Joy]
Xthat emulates a shared or common memory.
X.IP \(bu
XThis system is all user level code and requires no kernel
Xmodifications. It is accessible from a variety of languages.
X.IP \(bu
XThis shared memory system provides its service
Xtransparently to remote hosts across a local-area network.
X.IP \(bu
XThis system is being used in a real application - an automated
Xfactory. Shared variables are an appropriate communication paradigm
Xin a real-time environment (in comparison to message passing or pipes).
X.AE
X.PP
X.SH
XBackground
X.PP
XThe National Bureau of Standards has been performing research in the
Xarea of robot control [Albus] for over a decade. Robot control is
Xtypified by tightly-coupled multiprocessor systems often implemented as
Xmultiple single-board computers residing in a common bus. These
Xsystems require frequent and fast transfer of small amounts of data.
X.PP
XFor example, a robot arm may require new joint angles 100 times a
Xsecond in order to move smoothly. These joint angles are very small
Xpieces of data and further, the number of joints and joint angles is
Xconstant, hence they can be stored in "well-known" locations.
X.PP
XTo support this type of application, multiported memory is used
Xso that several
Xdifferent computers can access the same locations.
XIn the robot example, the
Xcontrolling computer and the servo computers each need access to the
Xinformation.
X.PP
XCommon memory has the further advantage that as new processes are added
Xthat need information already present, extant processes do not have to
Xbe modified to deliver that information. For example, a process was
Xadded that displays the robot's actions on a graphics monitor. The
Xdisplay process was added without modification to any other part of the
Xsystem, since it uses the joint angles which are stored in well-known
Xcommon memory locations. Recently a "safety" process was added to
Xguarantee that the robot never departs its working envelope. The
Xsafety process also obtains its information from common memory.
X.PP
XIn 1981, work [Furlani] began on an automated factory.
XThis is an extension of
Xthe robot control system in many ways. It is expected that the
Xfactory model will grow over time, and become richer in the number of
Xprocesses used and the amount of information shared between processes.
XA major difference in the automated factory project
Xis that systems which have to communicate are often in
Xseparate backplanes and use different operating systems. Common
Xmemory is seen as a consistent communications methodology for this
Xdisparate collection of computers.
X.PP
XToday, as computers grow larger and become more complex, more and
Xmore layers of protocol prevent ready access to common memory. Indeed,
Xwe find that in most of our computer-to-computer interfaces, common
Xmemory is now simply a way of thinking, rather than an underlying
Ximplementation. Such is the case with
X.UX
X4.2BSD.
X.PP
XThe following section describes an implementation of the common memory
Xparadigm in the
X.UX
X4.2BSD environment.
X.SH
XI. User View
X.PP
XA server, the Common Memory Manager (CMM), acts on requests to access shared
Xvariables. Typical requests are read, write, and declare. Variables are
Xstructured and have attributes such as type, protection, lifetime, and
Xownership. Variable type checking is performed at runtime startup, since
Xprocesses are loosely connected.
X.PP
XSome of the more interesting attributes of these shared variables are:
X.PP
X.I
Xlifetime and "shelf life".
X.R
XIn a real time environment, the system must
Xsurvive a process dying or getting bogged down temporarily. When the useful
Xlifetime of a variable's value has expired, other processes are free to
Xmanipulate the variable, for instance, by redeclaring or writing it. This
Xattribute allows a "god" process to notice unexpected process deaths and
Xrestart processes or transfer control to other processes. For less
Xcritical variables (such as an infrequently updated sensor value), processes
Xcan note that a value is "stale", and make a projection that will carry them over
Xuntil the variable writer catches up to its duties.
X.PP
X.I
Xwrite count, timestamp, authorstamp.
X.R
XBy marking each variable with how many
Xtimes or when it has been written, handshaking can be performed between
Xprocesses. The archetypal example from the automation environment has
Xa supervisor sending the command "hit nail with hammer" to a
Xsubordinate. If the supervisor runs more quickly than
Xthe subordinate, without handshaking, the subordinate may never see the
Xcommand (i.e. nail is not hit). If the subordinate runs more quickly than
Xthe supervisor, without handshaking, the subordinate may execute the same
Xcommand twice (i.e. nail is hit more than once).
X.PP
XThe same problem exists when the supervisor reads the feedback from the
Xsubordinate. In particular, because there is no synchrony between
Xprocesses, the supervisor has no way of knowing what command the
Xsubordinate is responding to (without handshaking). Actual code for
Xsuch a sequence would use higher-level routines to read the shared
Xvariable. For example:
X.PP
X.DS L
X.B
X struct shared_variable *command_out, *feedback_in;
X struct shared_variable_value *expected_feedback;
X .
X .
X if (status_equal(command_out, feedback_in, expected_feedback)) . . .
X.R
X.DE
X.PP
XHere, the command to the subordinate is
X.B command_out .
XThe actual feedback received is
X.B feedback_in
Xand we are looking for
X.B expected_feedback .
XThe expected feedback is thus compared to the
Xfeedback actually received with respect to the command which generated
Xit.
X.PP
XThese handshaking and synchronization problems are thoroughly discussed
Xin [Libes].
X.PP
X.I
Xwakeup.
X.R
XVariables can be marked with a list of processes to be notified upon
Xvalue change. This was originally added for efficiency, so that
Xclients, for example, would not have to loop just waiting for new
Xvalues. It has turned out to be extremely useful for debugging.
XRather than inserting print statements in existing code and
Xrecompiling, it is possible to create a debug module that simply prints
Xout variables when they change. It has also been used for a graphics
Xmonitor that continuously displays the state of the entire system.
X.PP
XThe wakeup attribute can be
Xused to force a queuing discipline on a variable. For
Xexample, a process performing as a resource server may take requests
Xfrom a well-known common memory variable. As clients write to the
Xvariable, the CMM wakes up the resource server with each new variable
Xvalue. The resource server typically responds to each variable value in
Xturn; however it can ignore all but the most recent values if it desires.
X.PP
X.I
Xnon-exclusive write.
X.R
XVariables can be declared read or write, exclusively or
Xnonexclusively. By default, writes are exclusive, meaning that only one
Xprocess can write a variable during its lifetime. Non-exclusive write has
Xno such restriction, allowing multiple processes to write the same variable.
XOne use of this might be a server listening at a well-known location
X("socket") for a request for service.
XAny client can
Xrequest service by writing the variable associated with the socket.
X.PP
XAnother use is to send information to a log file. In our
Xapplication, we can ask all processes to log their states a given
Xnumber of
Xclock ticks. By having them all write their states to one variable,
Xa logging process can record all the states of the system through time.
XThis has proven to be invaluable during system testing.
X.PP
X.SH
XII. 4.2BSD Implementation
X.PP
XThe Shared Variables System (SVS) is written in C and uses the 4.2
Xinterprocess communication system [Leffler].
XAn interface exists for Franz Lisp.
X.PP
XThe SVS consists of three layers. The lowest level simulates a common
Xmemory facility such as is available in System V. On top of this are
Xthe procedures that transform common memory into common variables.
XThis level provides users with routines allowing access and control of
Xthe common variables.
X.PP
XSpecific to our use of the SVS is a higher level that customizes it for
Xcommunication in a hierarchical control environment. This level
Xincludes interprocess handshaking.
X.PP
X.SH
XBuilding a server on 4.2
X.PP
X4.2BSD has no common memory facility. Our system provides one via a
Xserver that stores variables in its own memory space which is private
Xto it. (See Figure 1) The server runs as a user-level process which
Xneeds no special privileges and can be run from an unrelated user-id.
X.PP
X.DS L
X
X
X
X
X
X
X
X
X
X Figure 1 goes here
X
X
X
X
X
X
X
X
X
X.DE
X.PP
XCommunications between the common memory server and clients have
Xproperties of both streams and datagrams. Communication resembles
Xvirtual circuits in that the server and clients pass many messages
Xbetween them in a conversation (like TCP). These messages are also
Xrecord-oriented (like UDP). In implementation, UDP was rejected
Xbecause it would have forced us to handle unreliability and
Xfragmentation (datagrams were
Xlimited to 1K which was smaller than the typical message).
X.PP
XIn this case, we chose TCP as the lesser of two evils. TCP had
Xthe drawback (for us) of removing record boundaries (i.e. the receiver
Xcould not tell where one write ended and the next began). To handle
Xthis, new versions of read and write were constructed to do the actual
XI/O to the sockets. These packetized and depacketized the TCP streams
Xback into records.
X.PP
XSince TCP/IP is capable of communicating across the network, clients
Xmay be distributed (figure 2) while accessing the common memory. However, the
Xsmall number of concurrent TCP connections (viz. the number of open
Xfiles) is a severe limit. Another system [Mitchell] glues distributed
Xcommon memories together throughout the factory. This design allows multiple
Xcommon memories to exist but appear to the user as one. This gluing
Xprocess bypasses the TCP restriction and allows multiple
Xinstances of the 4.2 (as well as others') common memory service to be
Xrun.
X.PP
X.DS L
X
X
X
X
X
X
X
X
X
X Figure 2 goes here
X
X
X
X
X
X
X
X
X.DE
X.PP
X.SH
XSimulating common memory on 4.2
X.PP
XThe server's own private memory serves as common memory. Access to
Xvariables, therefore, is strictly through the valid requests to the CMM
Xserver. The server accesses variables as requested and maintains
Xextra information such as variable type, owner, etc.
X.PP
XWhenever a client wants to access common memory, it does so by
Xsending a message to the CMM. Such messages are sent asynchronously
Xbut received synchronously. In particular, the server is never
Xinterrupted.
X.PP
XTypically, the server is waiting for service requests. When a request
Xis received, service is performed. This service may or may not cause
Xsending of a message in return to the client. For example, "read"
Xoperations always cause information to be returned from the server
Xwhile "write" operations don't (unless an error is encountered).
X.PP
XIt is up to the client whether it wishes to wait (even after a "read")
Xfor a response from the common memory server.
X.PP
XClient processes keep their own local version of shared variables of
Xinterest until it is convenient for them to synchronize with
Xthe CMM. By calling
X.B cm_sync() ,
Xthe user and the common memory server become synchronized.
XAn argument to
X.B cm_sync()
Xspecifies whether the client waits for a response or not.
X.PP
XWhether the client is waiting or not, an error can occur at the server
Xend, in which case an error message is returned. For example, a client
Xmay write a value but the variable may be an incompatible type. When
Xthis happens, a message is sent back to the client. The client is not
Xinterrupted, but at the next
X.B cm_sync()
Xoperation it is notified of the error.
X.PP
XInterestingly, the problem of being able to provide atomic updates to
Xcommon memory vanishes in this system. Since the server is never
Xinterrupted, client updates functionally prevent all other clients
Xaccess to common memory while their request is being serviced. This
Xremoves the classic synchronization problem which exists when multiple
Xprocesses are reading and writing real common memory with no
Xsynchronization. On the other hand, this can cause relatively long
Xwaits for service.
X.PP
XConstruction of higher-level synchronization such as semaphores is
Xeasy. Potentially stalling manipulations of semaphores and monitors
Xcan call
X.B cm_sync()
Xand then wait for a response from the server to
Xcontinue. The server acknowledges the client if and only if the client
Xsemaphore action is satisfied.
X.PP
X.SH
XIII. An application - manufacturing automation
X.PP
XWhile the SVS is a general tool, it is currently being used at the National
XBureau of Standards in the Automated Manufacturing Research Facility
X(AMRF), a testbed for research in the automation of small batch
Xmanufacturing [Simpson].
XThe project is funded by NBS and the Navy Manufacturing
XTechnology Program and significantly supported by industry through
Xdonations or loans of major components and through cooperative research
Xprograms. The objective of the project is to identify, implement and exercise
Xpotential standard interfaces between existing and future components of
Xsmall-batch manufacturing systems and to provide a laboratory for the
Xdevelopment of factory-floor metrology in an automated environment,
Xdelivering proven measurement techniques and standard reference
Xmaterials to American industry. Commercially available products are
Xbeing used to construct the facility wherever possible in order to
Xexpedite transfer of research results into the private sector.
X.PP
XTo provide a real testbed for interface standards, the AMRF is
Xintentionally composed of manufacturing and computer equipment from
Xmany vendors, thereby making its construction a major integration
Xeffort. The types of systems that must be integrated and communicate
Xwith each other include stationary and
Xmobile robots, machine tools, their controllers, and higher-level cell and factory management systems. Actual computers and operating systems
Xrun the gamut, from micros to mainframes.
X.PP
XProcesses in the AMRF are
Xorganized in a control hierarchy, much like the organization of a
Xconventional factory. At each level, tasks are broken down into
Xsimpler tasks that can be performed by subordinate processes. Control
Xis hierarchical.
X.PP
XTo support such control communication, we imagine that processes in
Xthe system communicate along hierarchical channels. Each process runs
Xasynchronously, continually "sending" commands to subordinate processes
Xand feedback to superior processes.
X.PP
XLogically however, communication is performed using the common memory model.
XFor example, a process sending commands to a subordinate writes the
Xinformation in a well-known common memory variable. The subordinate reads
Xthe information out of the common memory variable, noting whether or not it
Xcontains a new value.
X.PP
XIn reality, there are small groups of processes and processors that directly
Xcommunicate with each other, some using physical common memory and some
Xsimulating common memory. Each cluster representing a piece of common
Xmemory can choose to replicate any other portion of another cluster's common
Xmemory. An underlying network
X(figure 3) invisibly supplies this service of keeping
Xall the common memories synchronized. The result is effectively one of a
Xconsolidated common memory, with a very consistent and easy means of
Xcommunication between processes.
X.PP
X.DS L
X
X
X
X
X
X
X
X
X Figure 3 goes here
X
X
X
X
X
X
X
X
X.DE
X.PP
X.SH
XSummary
X.PP
XCommon memory is a communications methodology that is easy to use and
Xcan be efficiently implemented.
XIt can use physical common memory or be simulated by other
Xcommunications techniques such as message passing. In turn, it can be
Xused to implement any other communications techniques.
X.PP
XCommon memory has several important advantages and disadvantages. The
Xprimary advantage is the ease with which information may be shared.
XIn our application, we have required this richness of data
Xsharing between processes.
XUsing common memory also has the benefit of being able to add, remove
Xor modify processes without disturbing other processes,
X.I
Xeven if they communicate with each other.
X.R
X.PP
XThe
Xprimary disadvantage is that sharing large amounts of information in
Xthis manner can become hard to keep track of. Since there is no
Xscoping in the common memory, naming conventions must be
Xfollowed (which in effect, provides scoping). Common memory is not
Xparticularly apt for all types of communications. Picking the most
Xappropriate model (pipes, databases, message passing) will invariably
Xlead to a simpler process in the final analysis. We, of course, use
Xthese other techniques in our application when appropriate.
X.PP
XAt the lowest level of our application, efficient process-to-process
Xcommunication is necessary for real-time behavior.
XThis is
Ximplemented using real common memory with low overhead. At higher
Xlevels where common memory is simulated and the high-speed demand does
Xnot exist, the methodology extends upwards suitably.
X.PP
XIn the future, we expect to move the ideas of hierarchical control and
Xcommon memory to new applications. Currently, we are investigating the
Xpotential for control of autonomous aircraft.
X.SH
XReferences
X.XP
XAlbus, J., Barbera, T., Fitzgerald, M.L., "Hierarchical Control for
XSensory Interactive Robots", Proceedings of 31st General Assembly,
XInternational Institution for Production Engineering Research (CIRP),
XToronto, Canada, September, 1981.
X.XP
XFurlani, C., Kent, E., Bloom, H., McLean, C., "The Automated
XManufacturing Research Facility of the National Bureau of Standards",
XProceedings of the Summer Simulation Conference, Vancouver, B.C.,
XCanada, July 1983.
X.XP
XJoy, W., Cooper, E., Fabry, R., Leffler, S., McKusick, K., Mosher, D.,
X"4.2BSD System Interface Overview", Computer Systems Research Group,
XU.C. Berkeley, July, 1983.
X.XP
XLeffler, S., Fabry, R., Joy, W., "4.2BSD Interprocess Communications
XPrimer", Computer Systems Research Group, U.C. Berkeley, July, 1983.
X.XP
XLibes, D., "Handshaking in a Hierarchical Control Environment",
XInternal Memorandum, Center
Xfor Manufacturing Engineering, National Bureau of Standards,
XWashington, DC 20234.
X.XP
XMitchell, M., Barkmeyer, E.,
X"Data Distribution in the NBS Automated Manufacturing Research
XFacility", Center for
XManufacturing Engineering, National Bureau of Standards, Washington, DC
X20234, 1984.
X.XP
XSimpson, J., Hocken, R., Albus, J.,
X"The Automated Manufacturing Research Facility of the National Bureau
Xof Standards", Journal of Manufacturing Systems, Vol. 1, #1.
X.PP
X.SH
XFurther Reading
X.XP
XBloom, H., Furlani, C., Barbera, A., "Emulation as a Design Tool in the
XDevelopment of Real-Time Control Systems", 1984 Winter Simulation
XConference, Dallas, Texas, November 1984.
X.XP
XJohnson, T., Milligan, S., Fortmann, T., Bloom, H., McLean, C.,
XFurlani, C. "Emulation/Simulation of a Modular Hierarchical Feedback
XSystem", The 21st IEEE Conference on Decision and Control, Orlando, FL,
XDecember 1982.
END_OF_FILE
if test 19292 -ne `wc -c <'doc/usenix/paper.trf'`; then
echo shar: \"'doc/usenix/paper.trf'\" unpacked with wrong size!
fi
# end of 'doc/usenix/paper.trf'
fi
if test -f 'doc/user.doc' -a "${1}" != "-c" ; then
echo shar: Will not clobber existing file \"'doc/user.doc'\"
else
echo shar: Extracting \"'doc/user.doc'\" \(26953 characters\)
sed "s/^X//" >'doc/user.doc' <<'END_OF_FILE'
XThe NBS Common Memory System (CMS)
XSept 25, 1987
X
XDon Libes
XNational Bureau of Standards
XMetrology Building, Room A-127
XGaithersburg, MD 20899
X(301) 975-3535
X
XThis system emulates a shared memory system. (This system is
Xloosely based on the Hierarchical Control System Emulator (HCSE)
Xbuilt by BBN, which provided a shared memory as one of its
Xfacilities.) User processes can be distributed as well as shared
Xmemory, itself. A server, the common memory manager (CMM) handles
Xrequests to manipulate shared variables. This emulator was
Xdesigned for the purposes of providing communication between
Xhierarchically controlled processes. There is a small amount of
Xfunctions specifically for the purpose of making such communication
Xof such style easier, but the emulator is certainly not restricted
Xby this and thus it also provides communications between processes
Xwith arbitrary relationships.
X
X
XSuperficially, each process that wishes to use a shared variable,
Xdoes so as if it was not shared. Typically, one process writes
Xinto a variable, while a number of other processes may read the
Xvariable. (Other scenerios are possible.) The common memory
Xmanager (CMM) enforces this and other restrictions and also handles
Xany synchronization that may be necessary. The fact that a
Xvariable is shared is transparent to the user.
X
XThe common memory is implemented as memory private to the CMM,
Xwhich it reads and writes in response to requests by clients. The
XBBN HCSE used a distributed form of control, depending on each
Xprocess to pass control to the next. A defect of this was that the
Xunexpected death of a process would halt the emulation when control
Xwas passed to it the next time around. By centralizing control,
Xthe unexpected death of any user process will not halt the
Xemulation. In fact, this can be monitored and a new or existing
Xprocess can take over the responsibilities of a deceased process.
X
X
XProcesses
X
XThere is one special process that must always be running when an
Xemulation is in progress. It is the Common Memory Manager (CMM).
XThere will probably be other auxiliary processes such as
Xfront-ends, debuggers, and editors that are not necessary but will
Xinvariably always be running during any emulation. This is not the
Xcase now, however.
X
XThe CM manager handles requests from processes to read/write common
Xmemory variables. Various other requests are possible such as
Xdynamically changing the size, type or writer of a variable. This
Xlast possibility is very useful, if, say a process agrees to
Xdirectly take over a resource that can be passed around between
Xprocesses.
X
X
XHow to use the Common Memory System
X
XThis system emulates a shared common memory with specific features
Xfor supporting communication between hierarchically controlled
Xprocesses.
X
XThis document describes how to use the current implementation on
Xthe Sun Workstation (running Sun UNIX 1.x, 2.x and 3.x). It has
Xalso been ported to the Silicon Graphics Iris. It may be helpful
Xto read the system description and implementation notes to get a
Xbetter idea of how the system works.
X
XLet us get right to the usage details.
X
XProcesses
X
XBefore any other CMS calls, the process should identify itself to
Xthe system:
X
X int rc = cm_init("HWS","cmm_host",0); /* C */
X
X (cm_init "HWS" "cmm_host" 0) ; Lisp
X
XHere, we have declared ourselves as "HWS". This name need not be
Xdistinct, however when debugging, you will find it helpful if you
Xhave chosen different names for your cm users.
X
XThe second parameter to cm_init() is the host on which the
Xcommon memory manager is running which you wish to use. Note that
Xthere may be a CMM running on the local machine, while you use one
Xon a remote machine. The local machine may be designated either by
Xits name or by a null string.
X
XThe third parameter to cm_init() is an integer which
Xindicates the debugging level. 0 indicates no debugging. Larger
Xvalues request more debugging info. For example, 2 will give you
Xinformation about messages sent and received. 5 will generate
Xinformation about individual common memory values being
Xmanipulated. With 10, you will get a veritable flood of garbage
X(that you almost certainly don't want) including things like memory
Xallocation, variable copying, etc.
X
Xcm_init() also performs some necessary initialization of CM
Xclient data structures. cm_init() returns 0 if successful.
XAnything else is an error. A common error is that the common
Xmemory manager (cmm) is not running.
X
XBefore a second (or any further calls to) cm_init() call,
Xcm_exit() should be called. cm_exit() tells the CMM that you have
Xgone away. It also cleans up various data structures internal to
Xthe cm system.
X
XFor example:
X
X cm_exit();
X
X (cm_exit) ; Lisp
X
X
XDeclaring variables for use
X
XAll variables must be declared before use. cm_declare() is used to
Xdeclare common memory variables.
X
X cm_variable *date;
X
X date = cm_declare("date",CM_ROLE_XWRITER);
X
X (setq date (cm_declare "date" CM_ROLE_WRITER)) ; Lisp
X
Xcm_declare() returns an object that can be used when referring to
Xthis variable in the future. This object can be stored into a
Xvariable declared as type cm_variable. If cm_declare() returns
XCM_BAD_OBJECT, the declaration has failed (an error message will be
Xprinted out explaining why). Declarations can fail for a variety
Xof reasons (bad or conflicting arguments, no space left to store
Xvalues, etc).
X
XOnce cm_declare() has returned an object, this object should be
Xused whenever referring to the variable. In the case of
Xcm_declare, the first argument is almost always a string, while in
Xall other functions, the variable identifier is almost always a
Xcm_variable.
X
XThe second argument of cm_declare() specifies access rights. The
Xavailable access rights are:
X
X CM_ROLE_NONXWRITER or CM_ROLE_NONEXCLUSIVE_WRITER
X CM_ROLE_XWRITER or CM_ROLE_EXCLUSIVE_WRITER
X CM_ROLE_READER
X CM_ROLE_WAKEUP
X
XThese access rights can be combined by ORing. For example, the
Xwakeup right is always combined with at least one of the others.
X"wakeup" causes the CMM to wake the process up whenever the
Xvariable is written by someone else.
X
XObviously conflicting combinations should be avoided. If one
Xprocess has declared a variable CM_ROLE_XWRITER, other processes
Xare prohibited from any kind of write access to that variable.
XThese are the only restrictions on variable access.
X
XCommon memory values are structured in the following way.
X
XThis structure cm_value is used for this.
X
X typedef struct { /* common memory value */
X char *data;
X unsigned short msize; /* size of malloc'd space */
X unsigned short size; /* size of used space */
X char mallocable; /* 1 if space is malloc() */
X } cm_value;
X
XIf mallocable is 1, CMS will allocate space using malloc whenever the
XCMS passes a value to the user. Further, if msize is ever smaller
Xthan the incoming value, the pointer will be realloc'd and msize
Xincreased appropriately.
X
XIf you are not prepared to handle objects larger than a given size,
Xset msize yourself, and set mallocable to 0. Lisp is an example
Xwhere this must be done, as otherwise the common memory may
Xattempt to free a Lisp object, which would be a serious mistake.
X
XThe address of such a structure may be passed to cm_set_value and
Xit's relatives.
X
XA cm_value is declared in Lisp (via c-declare) along with
Xcorresponding access functions. For example, to declare and set
Xthe various elements of the structure called "foo":
X
X (setq foo (make-cm_value)) ; cm_value foo;
X (setq data (new-vectori-byte 1000)) ; char data[1000];
X (setf (cm_value->data foo) (ptr data)) ; foo.data = data;
X (setf (cm_value->size foo) 1000) ; foo.size = 1000;
X (setf (cm_value->msize foo) 1000) ; foo.msize = 1000;
X (setf (cm_value->mallocable foo) 0) ; foo.mallocable = FALSE;
X
XUser-Defined Types
X
XThe original common memory design was to support user-defined
Xtypes, but experience with other common memory systems have taught
Xus that this is "a bad thing". There is no reason why the common
Xmemory should know the type of the data that it is storing.
X
XTo provide user-defined structured values, use ASN.1 (X.409).
XThis provides for structured types which are machine independent.
X
X
XReading and Writing Variables
X
XThere are two levels of access into the common memory. Only the
Xhigher-level will be covered here. This higher-level incorporates
Xfunctions that are tailored towards communications between
Xprocesses in a hierarchically-controlled manner.
X
XVariables may be read and written with the following calls:
X
X cm_value *value = {NULL,0,0,1};
X cm_get_value(variable,value);
X cm_set_value(variable,value);
X
X (setq value (make-cm_value))
X (setf (value->data) 0)
X (setf (value->msize) 0)
X (setf (value->size) 0)
X (setf (value->mallocable) 1)
X (cm_get_value variable value)
X (cm_set_value variable value)
X
X
XSeveral additional functions exist for handling handshaking
Xbetween superior and subordinate processes in a control hierarchy.
XSpecifically, variables can be used for command or status. Status
Xvariables are identified by the system with command value they are
Xassociated with. For more information, see the technical
Xmemorandum, "Implementing Command-Feedback Variables in a
XHierarchical Environment".
X
XVariables which are command variables should be read and written
Xwith the following routines:
X
X cm_set_new_command_value(variable,value);
X cm_get_new_command_value(variable,value);
X
X (cm_set_new_command_value variable value)
X (cm_get_new_command_value variable value)
X
XOne utility routine is available for determining whether a new
Xcommand has been received. cm_new_command_pending() returns TRUE
Xor FALSE depending on whether a new command has been received.
X
X maybe = cm_new_command_pending(command_variable);
X
X (setq maybe (cm_new_command_pending command_variable))
X
XWhen a new command has been received, cm_new_command_pending will
Xreturn TRUE until cm_get_new_command_value() has been called, after
Xwhich it will return FALSE. cm_get_new_command_value() also
Xreturns TRUE or FALSE, depending upon whether it has detected a new
Xcommand. (Lisp users can expect t/nil instead of TRUE/FALSE.)
X
X
XStatus variables must be written with the routine,
Xcm_set_status_value().
X
X cm_set_status_value(command,variable,value);
X
X (cm_set_status_value command variable value)
X
XStatus (and any other) variables may be read with the routine
Xcm_get_value().
X
XTwo predicates are available that are of use to the superior
Xprocess in determining which command a subordinate process' status
Xis in response to.
X
X maybe = cm_status_equal(cmd_var,stat_var,s_value);
X maybe = cm_status_synchronized(cmd_var,stat_var);
X
X (setq maybe (cm_status_equal cmd_var stat_var s_value))
X (setq maybe (cm_status_synchronized cmd_var stat_var))
X
Xcm_status_equal() returns TRUE or FALSE, depending on whether or
Xnot the status variable, stat_var, has the value, s_value, and is
Xin response to the command specified by cmd_var.
X
Xcm_status_synchronized() returns TRUE or FALSE, depending on
Xwhether or not the status variable, stat_var, is in response to the
Xcommand specified by cmd_var. This is very helpful to the superior
Xprocess in finding out whether the subordinate process is
Xresponding the command.
X
XSynchronization
X
XSince the common memory and user processes are actually
Xunsynchronized processes, it is necessary to synchronize them
Xoccasionally. The idea of calling cm_sync(), is to force all
Xvariables in common to the user and CMM processes to have the same
Xvalue.
X
X cm_sync(behavior);
X
X (cm_sync behavior)
X
Xcm_sync() takes one argument that allows several behaviors by the
XCMM. There are three sets of options.
X
XThe first specifies whether cm_sync() should wait for at least one
Xset of variable updates (or any response from the CMM). The
Xdefault is CM_WAIT. To poll and return immediately, use
XCM_NO_WAIT.
X
XThe second option allows one the ability to examine the one set of
Xvariable updates before it has (possibly) been overwritten by an
Ximmediately following set of updates. This is very useful if you
Xhave a variable from which you expect to be written by multiple
Xprocesses.
X
XSelecting CM_WAIT_AT_MOST_ONCE allows you to read a variables value
Xbefore it is overwritten by yet another value from the CMM. This
Xis useful, for server processes, which takes requests off a queue.
Xdefault is CM_WAIT_FOR_ALL which simply returns the most recently
Xwritten value.
X
XThe third set of options controls whether the user desires an
Ximmediate reply from the common memory.
X
X cm_sync(CM_WAIT_READ);
X
XCM_WAIT_READ forces the common memory to respond with the current
Xset of variable values (whether or not they have wakeup flags set
Xon them). If you expect to poll intermittently, a variable that is
Xbeing regularly set, this is the desirable behavior.
X
XIf the CMS has already sent variable updates to you, CM_WAIT_READ
Xwill get whichever ones are appropriate depending upon the other
Xoptions you have supplied in cm_sync(). For example, specifying
Xboth CM_WAIT_READ and CM_WAIT_FOR_ALL gives an exact simulation of
Xa true common memory. Note that using CM_WAIT_READ may be slower
Xthan other forms of cm_sync() since you may have to wait for a
Xreply message from the common memory.
X
XTo combine options, bitwise-OR them together. For example, to poll
Xfor at most one new set of variable values:
X
X cm_sync(CM_NO_WAIT|CM_WAIT_AT_MOST_ONCE);
X
XHowever, it is expected that most clients will simply want to use:
X
X cm_sync(CM_WAIT);
X
Xcm_sync returns either 0 (normal completion) or negative numbers
Xdenoting an error (see the cm.h).
X
X
XOther variable information
X--------------------------
X
XVariables are more structured than in the BBN HCSE. This allows easier
Xcontrol of variables. For example, handshaking between two levels of the
Xhierarchy is automatically handled by command associations embedded in the
Xvariable structures. Variables are also tagged with their length,
Xetc...
X
XCommon memory variables have the following attributes:
X
X /* handshake.h */
X
X char name[MAXVARIABLENAMELENGTH];
X unsigned long count; /* nth definition of this value */
X unsigned long old_count;/* " " " " " " " " " " " " "when last read */
X unsigned short size; /* size in bytes of the uninterpreted data */
X struct timeval *time_stamp; /* when last written */
X command_association command_association; /* command that this variable
X is associated with */
X command_association old_command_association; /* " " when last read */
X cm_value data; /* uninterpreted data */
X
XVariables also have the following attributes (which are strictly
Xinternal to the CMM).
X
X char xwriter[PROCESSNAMELENGTH]; /* exclusive writer */
X struct {
X unsigned reader : 1;
X unsigned writer : 1;
X unsigned wakeup : 1;
X unsigned new : 1; /* written since read */
X } role[PROCESSES];
X
X
XCompiling (or interpreting) CM code
X-------------------------------
X
XTwo libraries are necessary for using common memory. cmlib.a is
Xthe common memory client code. This uses a lower-level library,
Xlibstream.a, which provides connection and packet service on top of
XTCP. Both libraries normally live in /usr/local/lib.
X
XThus, to link common memory programs:
X
X cc foo.c -lcm -lstream
X
XInclude files live in /usr/local/include/cm. Normally, it is
Xnecessary only to include /usr/include/sys/time.h and
X/usr/local/include/cm/cm_user.h as follows:
X
X #include <sys/time.h>
X #include <cm/cm.h>
X
X.lisp files are in /usr/local/lisp. There is one file provided to
Xinitialize common memory from lisp, cm_user.lisp. Thus to use
Xcommon memory, you should execute the following:
X
X (load 'cm/cm_user.lisp)
X
XThe common memory manager itself, cmm, lives in /usr/local/bin.
X
XBefore any processes are started the CMM should be started up.
XAnyone can do this (i.e. you do not have to be root or have the
Xsame uid as any other users of the CMM). Just type:
X
X /usr/local/bin/cmm
X
XNormally, nothing else is required, however the cmm can take some
Xarguments to modify the default behavior.
X
X -d [0-9]
X
XThe argument will cause the cmm to print out debugging information.
XHigher numbers evoke greater amounts of output. 0 means no
Xdebugging. 3 refers to message handling. 6 refers to slot
X(message contents) handling. 9 refers to most buffer/byte/string
Xcoping.
X
X -t seconds
X
Xblock waiting for up to this time period, if the a client's kernel
Xqueue is full while the cmm is trying to send a message to the
Xclient. After the timeout expires, the cmm goes on and will retry
Xlater. This typically implies that the client is hung, but is not
Xalways the case. The default timeout period is 5 seconds.
X
X -p port_number
X
XThe initial connection port that the common memory uses may be
Xchanged by specifying the port number. The default is 1525. This
Xis useful if you want to have multiple separate common memorys on a
Xsingle computer.
X
X
XThe cmm does not require a controlling terminal to run. Note, that
Xif the cmm is killed, all the processes using it will terminate if
Xthey are writing at the moment that the cmm is killed. This is due
Xto a SIGPIPE being sent to each of the clients. If you do not want
Xthis behavior, you should probably surround your calls to cm_sync
Xwith a setjmp/longjmp alarm, just the way one normally does with
Xblocking writes. (The common memory manager does this internally
Xto protect itself from the clients dying. You can look at it in
Xman.c.) In typical use, however, people do not do this, since the
Xcommon memory never dies of its own accord.
X
XOnce the cmm is running, you can start the user processes.
X(Specifically, the cmm process should be started before the first
Xcm_process_name() is executed.)
X
X
XAdditional Lisp notes
X---------------------
X
XIn Lisp, all functions are identical to their C counterparts.
XCommonsense dictates usage differences. A small Rosetta stone will
Xbe presented:
X
X date = cm_declare(......);
X foo = cm_declare(.....,CM_ROLE_READER);
X cm_sync(WAIT);
X cm_set_value(date,"12 Dec...");
X cm_get_value(foo,foolist);
X
X
X (setq date (cm_declare 'date CM_ROLE_XWRITER))
X (setq foo (cm_declare 'foo CM_ROLE_READER))
X
X (cm_sync WAIT)
X (cm_set_value date "12 Dec...")
X (setq foo (cm_get_value foolist))
X
X
XNote that uppercase values denote constants that should be
Xevaluated before use (i.e. unquoted). For example, to check if
Xcm_declare() returns without failure the code would look like:
X
X (cond ((eq CM_BAD_OBJECT (cm_declare ....)) nil)
X (t t))
X
X
XPorting code over to the VAX (HCSE)
X
XThe following section is only appropriate to people using the BBN's
XHCSE code on the VAX running VMS.
X
XIt is possible to run the same code on the Sun (Sun CM) and the VAX
X(BBN HCSE) if certain steps are taken.
X
XAn interface library is supported that effectively replaces the
XSun CM user calls with calls into the HCSE library. This library
Xis currently available in ~libes/cm/src.7v.
X
XCode using the the UNIX CM library with HCSE should add the
Xfollowing parameters to the link command (in a .opt file)
X
X user1:[libes.cm.src$5n7v]suncmlib/lib,
X cm_library:cmlib/lib,
X psect=cm_shrmem,page
X
XVersions of the code compiled for debugging are available by
Xspecifying:
X
X user1:[libes.cm.src$5n7v]suncmlib_debug/lib,
X cm_library:cmlib_debug/lib,
X psect=cm_shrmem,page
X
X
XThere are three restrictions upon use of the HCSE with Sun CM code.
X
XTypes
X
XThe type systems in both the Sun system and the HCSE system are
Xquite different. The HCSE types are based on Praxis. Primarily
Xthis means that they have user-definable types. Secondly, there
Xare no arbitrarily-sized data objects.
X
XTwo extra parameters on the cm_declare statement exist in the HCSE
Xversion of Sun CM to get around this.
X
XThe first is a maximum size. The second is a pointer to a type
Xstructure. If the type structure pointer is zero, the size is used
Xto automatically select a type structure. For more information
Xabout creating type structures, see the HCSE Programmers Manual.
X
XA typical call that would be portable to both the Sun and VAX
Xsystems looks like the following:
X
X if (!(date = cm_declare("data",CM_ROLE_XWRITER
X#ifdef VAX11C
X ,1000,0
X#endif
X ))) {
X printf("declare of var failed\n");
X exit(-1);
X }
X
XWhen compiled by the DEC C compiler, the additional two parameters
Xwill be included.
X
X
XThe second difference in the UNIX CM system and the HCSE is that
Xthe HCSE does not support queuing of variable updates. This means
Xthat if one process goes to sleep while a second process writes a
Xvariable several times, if the first process then wakes up, it will
Xsee only the last values written by the second process, not all the
Xintermediate ones.
X
XA different explanation of this phenomena is that there is no
Xdifference between specifying CM_WAIT_AT_MOST_ONCE and
XCM_WAIT_FOR_ALL in cm_sync().
X
X
XThe third difference is that the HCSE does not store command
Xassociations with variables in the cmm itself. (In fact, they are
Xjust dropped on the floor). Therefore, routines like
Xcm_status_equal() don't work.
X
XIf you need command associations or their effect (and most people
Xdo), you must do what other HCSE users do. That is, to stuff in a
Xnumber in front of every variable indicating who many times this
Xvariable has been written. Then create a new variable, that holds
Xthe old value of the number which you can use to compare it against.
X
XFor more information on this, read:
X
XAMRF Architectural Decision Document: Mailbox Format (2/20/85)
XAMRF Notice 85-1: Command-Status Message Structure (1/8/85)
X
XA package that implements this along with current mailbox types in
Xuse in the AMRF lives in k:~network/mailbox. An example using
Xthese functions is in the cm source directory in vws.c. vws.lisp
Xis a lisp version of the same thing.
X
X
XBasic limits of the CM system.
X
XCertain limitations exist in the CMM. It is possible to change
Xany of these and recompile. Changable limitations (and their
Xdefaults as the system is distributed are):
X
XCM_MSGSIZE 100000 /* Maximum size of any single set of variable
X updates between the cmm and a user */
XCM_SLOTSIZE 20000 /* Maximum size of any single variable */
XCM_PROCESSNAMELENGTH 20 /* Maximum length of the process name */
XCM_VARIABLENAMELENGTH 20 /* Maximum length of any variable name */
XCM_MAXVARIABLES 50 /* Maximum number of variables local to the cmm */
XCM_MAXPROCESSES 20 /* Maximum number of processes that can talk to
X the cmm simultaneously. Absolute
X maximum of 32 (or number of user
X file descriptors) under 4.2BSD. */
XCM_MAXUSERVARIABLES 100 /* Max number of variables local to a user */
X
X
XErrors
X
XMost types of errors are reported at the user program. Some
Xmessages cannot be reported back to the user, and are reported at
Xthe cmm itself. Some errors are serious enough that they are
Xreported at both the user and cmm.
X
XUser errors
X
XMost user errors can be fixed when identified. For example,
Xwriting into a variable declared to be read-only would be a
Xuser-error.
X
XSince user errors indicate a user-programming error, the CMS
Xusually prints out a message indicating the problem. It also
Xreturns an error code if possible. It is sometimes not possible to
Xdo this. For instance, the above example would not be detected
Xuntil after cm_set_value returned.
X
XThe actual message would be printed by cm_sync when it is
Xprocessing incoming messages from the cmm. Most types of errors
Xare detected by cm_sync.
X
XSystem errors
X
XSystem errors are caused by limitations in the CMS itself, the
Xenvironment it is running in and the user demands upon the system.
XOften, these cannot be avoided. For example, if the user attempts
Xto send to much data to the common memory at once, the maximum
Xmessage size can be exceeded.
X
XListing of Error Messages
X
Xcm_init:
X returns E_CM_INIT_FAILED
X initport(client): Connection refused
X Problem: cmm is not running.
X
Xcm_sync:
X failed to send msg to cmm. cmm disappeared?
X Problem: cmm died. Detected while writing to it.
X cm library (version #) is older/newer than cmm (version #)
X Problem: cmm is a different version than the
X libraries your code is compiled with. This can
X also be caused by a corrupted message. The is
X usually identifiable by wildly different version #s.
X bad slot encountered...aborting msg
X user_decode_slot: unknown slot type (#)...msg aborted
X Problem: corrupted message or internal error in CMS.
X CMM: error processing variable <name> - error message
X Problem: cmm detected error "error message" in
X processing "name". See below.
X get_slot_read_response: <name> unknown (sent from cmm)
X Problem: corrupted message or internal error in CMS
X too much data for msg!!
X output msg size = # slotsize = #
X Problem: User value is too large for CMS
X configuration. Either user error, or message size
X limit should be increased.
X cm_sd_free() called on nonmallocable object?
X Problem: internal error in CMS
X
X*:
X error: bcopy src/dest is null ptr
X Problem: internal error or user error. If user
X error, check elements of cm_value structures to see
X that they are consistent.
X
Xcmm:
X bind() failed
X initport(server): Address already in use
X failed to initialize connection socket
X Problem: another cmm is running, or a process
X already has the CMM connection socket open.
X get_variables(name) failed
X Problem: too many variables in cmm.
X process <name> is being antisocial on fd #
X Problem: process has requested wakeup service but
X is not listening to cmm updates.
X slot bad
X Problem: corrupted message or internal error in CMS
X slot error in <name> type # - error message
X Problem: corrupted message or internal error in CMS
X or user error. See error message. This message is
X sent back to the user. See below.
X
XError messages generated by the CMM and sent back to the user.
X version
X Problem: version mismatch. See above.
X bad slot type
X Problem: corrupted message or internal CMS error.
X not enough common memory to declare variable
X Problem: too many variables stored at cmm.
X cannot get nonexclusive write access
X Problem: a process has already received exclusive
X write access to this variable.
X undeclare of undeclared variable
X Problem: a nonexistent variable is being undeclared.
X variable has not been declared
X Problem: attempt to read/write variable not yet
X declared.
X not declared as writer
X Problem: attempt to write variable declared as
X read-only.
X get_slot_write: cm_flat_to_sd() failed! no space?
X Problem: cmm ran out of memory trying to read user
X message. Indicates lack of system resources or
X user sent value that was too large.
X not declared as reader:
X problem: attempt to read variable declared as
X write-only.
X
XThere are several places in the CMS where memory is dynamically
Xallocated. These may fail with an error such as
X
X func: failed malloc(object,size)
X
Xor
X
X resized failed! - out of space
X
Xwhere "func" is the CMS function calling malloc, "object" is the
Xobject being malloc'd and size is the size of the object.
XThese errors typically indicate that either:
X
X1) the user is storing or receiving incredibly lengthy values
X(probably by mistake), or
X
X2) the system is running out of internal space
X
END_OF_FILE
if test 26953 -ne `wc -c <'doc/user.doc'`; then
echo shar: \"'doc/user.doc'\" unpacked with wrong size!
fi
# end of 'doc/user.doc'
fi
echo shar: End of archive 4 \(of 4\).
cp /dev/null ark4isdone
MISSING=""
for I in 1 2 3 4 ; do
if test ! -f ark${I}isdone ; then
MISSING="${MISSING} ${I}"
fi
done
if test "${MISSING}" = "" ; then
echo You have unpacked all 4 archives.
rm -f ark[1-9]isdone
else
echo You still need to unpack the following archives:
echo " " ${MISSING}
fi
## End of shell archive.
exit 0
--
Please send comp.sources.unix-related mail to rsalz@uunet.uu.net.