[comp.os.mach] Threads and fork

samuel@nada.kth.se (Samuel Cederlind) (09/16/90)

(This question also posed to comp.sys.next)
How come forking C-threads on the NeXT does not work after
having done a unix fork() call?

As I ported the public domain XScheme to our NeXT, I made an object
extension to it, originally because we wanted to build an adventure
game.

Using the Mach operating system, objects can communicate nicely with-
out knowing the borders of their tasks. It would then be very convenient
if objects could be "forked off" into new processes.
There is a function in the NeXT Reference Manual, task_create(), very
briefly documented. In the Mach manuals it says:
> Note:  Not implemented yet. Use fork.

So I have. There are three or so threads that always need to be running
for the communication between objects. After a fork(), you have to
create all threads anew, right? Anyway, there I'm stuck. This seems
an insoluble problem. cthread_fork() yields an error, and sticks.
Is there any reason why this should be?

mbj@natasha.mach.cs.cmu.edu (Michael Jones) (09/20/90)

In article <1990Sep15.221401.25887@nada.kth.se>, samuel@nada.kth.se (Samuel Cederlind) writes:
> How come forking C-threads on the NeXT does not work after
> having done a unix fork() call?
> 
> Using the Mach operating system, objects can communicate nicely with-
> out knowing the borders of their tasks. It would then be very convenient
> if objects could be "forked off" into new processes.
> There is a function in the NeXT Reference Manual, task_create(), very
> briefly documented. In the Mach manuals it says:
> > Note:  Not implemented yet. Use fork.
> 
> So I have. There are three or so threads that always need to be running
> for the communication between objects. After a fork(), you have to
> create all threads anew, right? Anyway, there I'm stuck. This seems
> an insoluble problem. cthread_fork() yields an error, and sticks.
> Is there any reason why this should be?

Forking a multi-threaded program (and having the resultant child make sense)
presents some nearly insurmountable problems.  Two apparent solutions might
be considered:

1.  Duplicate all the threads.  While "obvious" this results in terribly
    non-intuitive behavior.  The fork() call for a single-threaded program
    largely works because all (one) threads are in a known state -- "they"
    are in the fork() call.  This isn't true for multi-threaded programs.

    For instance, some threads may be in the middle of read() or write()
    system calls during a fork().  What should a fork()ed read do?  Do two
    reads()?  Remember that the file pointer is shared between parent and
    children.  All of a sudden a simple sequential read loop in the parent
    becomes two read loops in the parent and child, returning potentially
    interleaved non-consecutive portions of the file being read.  Or you
    could return EINTR or some such thing in the child.  This also results in
    non-obvious (and non-useful) behavior.

2.  Duplicate only the fork()ing thread.  This is what Mach does.
    Unfortunately this also has problems.  Consider that other threads are
    possibly holding locks and are in the midst of critical sections.
    Interrupting their progress in mid critical section in the child means
    that the locks will be held forever and that the invariants protected by
    the locks will not be maintained.  Even if the locks are automatically
    dropped in the child the invariants are still broken.  Not pretty either.

One other non-obvious solution also exists:

3.  Duplicate only the fork()ing thread after insuring that the fork()ing
    thread has grabbed ALL the locks before forking, and then release them in
    both the parent and the child after the fork().  This insures that no
    other threads are within critical sections and allows any other needed
    threads to be explicitly created by the child.  This works, but violates
    all modularity considerations.

    I actually built a version of cthreads() here (which is now the default
    version) which does this for the cthreads() internal locks.  This version
    is distributed with Mach 2.5.

    For this approach to work every module must export routines of the form:

	MODULE_fork_prepare()	/* Grab all the module's locks */
	MODULE_fork_parent()	/* Release all locks for the parent */
	MODULE_fork_child()	/* Release locks in the child and reinit */

    All forks then have to look like:

	MODULE1_fork_prepare();
	MODULE2_fork_prepare();
	/* ... */
	if ((pid = fork()) > 0) {
	    /* Child */
	    /* ... */
	    MODULE2_fork_child();
	    MODULE1_fork_child();
	    /* Child specific code */
	} else {
	    /* ... */
	    MODULE2_fork_parent();
	    MODULE1_fork_parent();
	    if (pid < 0) {
		/* Error code */
	    } else {
		/* Parent specific code */
	    }
	}

    This requires that EVERY module follow this protocol to work.  (I suspect
    that the modules used by your program don't.)  It also presumes strong
    knowledge about lock ordering to avoid deadlock at fork prepare time.

Finally, you can do the thing you probably didn't want to do:

4.  Exec a new copy of the program, explicitly passing any necessary state
    in from the parent to the child via arguments, the environment, or
    initialization files.

Sorry to be a bearer of bad tidings.  The truth isn't always good news.

				-- Mike Jones
				CMU Mach Project