mark@intek01.uucp (Mark McWiggins) (02/23/91)
We're porting a C-tree multiuser application to the Apollo. C-tree opens files in O_RDWR mode. This works fine for multiple processes on a single machine, but the same process on another machine fails to open the file with "Text file busy". There must be a way around this, but what? I don't see anything relevant in TFM; We're running 10.3 with the BSD environment option, but the same behavior occurs with System V. Thanks in advance. -- Mark McWiggins Integration Technologies, Inc. (Intek) +1 206 455 9935 DISCLAIMER: I could be wrong ... 1400 112th Ave SE #202 Bellevue WA 98004 mark@intek.com Ask me about C++!
pcc@apollo.HP.COM (Peter Craine) (03/01/91)
In article <1991Feb23.015726.8621@intek01.uucp>, mark@intek01.uucp (Mark McWiggins) writes: |> We're porting a C-tree multiuser application to the Apollo. C-tree opens |> files in O_RDWR mode. This works fine for multiple processes on a single |> machine, but the same process on another machine fails to open the file |> with "Text file busy". |> |> There must be a way around this, but what? I don't see anything relevant |> in TFM; We're running 10.3 with the BSD environment option, but the |> same behavior occurs with System V. |> I'm afraid that I've got some bad news for you. Once you open a file for write access on a node, the node that performs the "open" is the ONLY node that will be able to access that file. "But why can it work in NFS, but not in Domain?" NFS is implemented in a client/server model [no kidding]. So, when a computer accesses a file, it is really asking a daemon on the computer where the file lives to access the file. All accesses (read/write requests) are done through the daemon, so there aren't really any contention problems. Not so with Domain. (Here we go again). Domain/OS is a "single-level store" Operating System. That means that references to parts of your virtual address space are really direct references to the disk objects that are mapped at that portion. Put another way, a disk file is "mapped" directly into your address space, and modifications to that portion of your address space are considered to be changes to that file. When you modify a file (that is, change the portion of your address space where the file is mapped), the ONLY machine that can be certain to have the most recent version of the contents of the file is the one that modified the contents. (In time, the changes will make it back across the network to the node where the file lives, but that's another story for another time.) Because you can never get access to the most up-to-date information, the designers of Domain (nay, really Aegis, for this design constraint was put in at the beginning of the OS) decided that the only node that has access to a file is the one which has it opened for write. (I have a better description of all this in a short paper I wrote for a customer once. I'll have to see if I can find it.) +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Peter Craine + "Agent Cooper, I am thrilled to pieces that Hewlett-Packard + the Dharma came to Hohoho, I really am." Chelmsford Response Center + *I* don't want my opinions. Why would HP?
mishkin@apollo.HP.COM (Nathaniel Mishkin) (03/02/91)
In article <50164f63.20b6d@apollo.HP.COM>, pcc@apollo.HP.COM (Peter Craine) writes: >In article <1991Feb23.015726.8621@intek01.uucp>, mark@intek01.uucp (Mark >McWiggins) writes: >|> We're porting a C-tree multiuser application to the Apollo. C-tree opens >|> files in O_RDWR mode. This works fine for multiple processes on a single >|> machine, but the same process on another machine fails to open the file >|> with "Text file busy". > >I'm afraid that I've got some bad news for you. Once you open a file for write >access on a node, the node that performs the "open" is the ONLY node that will >be able to access that file. > >"But why can it work in NFS, but not in Domain?" > >NFS is implemented in a client/server model [no kidding]. So, when a computer >accesses a file, it is really asking a daemon on the computer where the file >lives to access the file. ... > >Not so with Domain. (Here we go again). Domain/OS is a "single-level store" >Operating System. ... This is really not an accurate picture of what's going on or what the problems are. First off, DOMAIN accesses to remote files are done via a client/server model as well. (The server is what some of the funny processes listed by "/com/pst -l1" are.) The server is sent requests to read and write blocks of a file, just like NFS. The only significant differences between the NFS server and the one in DOMAIN is that the former uses a publicly specified protocol (good idea!) and the latter embodies and enforces certain rules about concurrent access to files (also a good idea, as far as I'm concerned). It is this second difference (and not that the DOMAIN stream I/O layer is built on top of a mapped file layer, which, as it happens, is SunOS's these days) that is the reason you can't have multiple, non-colocated writers of the same file. Note that having multiple, non-colocated writers of a file doesn't work in general with NFS either. The results of having multiple such writers is not deterministic and you're not guaranteed to see the same behavior (or ANY particular well-defined behavior) as you would if the files had been written and read locally. Yes, it works for some applications (especially if you happen to know the details about how the NFS protocol is typically IMPLEMENTED, as opposed to SPECIFIED) and, yes, maybe DOMAIN should offer the ability to let people get non-deterministic behavior; after all, it's their nickel. It just so happens that it doesn't. BTW, the OSF DCE Distributed File System, which is based on the CMU ITC Andrew File System, DOES support multiple, non-colocated writers and it does it right. It has a scheme for passing logical "tokens" around among the various network clients accessing a file. -- -- Nat Mishkin Cooperative Object Computing Division / East Hewlett-Packard Company mishkin@apollo.hp.com
rees@pisa.citi.umich.edu (Jim Rees) (03/03/91)
In article <50164f63.20b6d@apollo.HP.COM>, pcc@apollo.HP.COM (Peter Craine) writes:
I'm afraid that I've got some bad news for you. Once you open a file for write
access on a node, the node that performs the "open" is the ONLY node that will
be able to access that file...
In any distributed file system that provides both consistency and local
caching (this doesn't include NFS), you need some sort of locking protocol
to mediate access to the central object. The Domain file system was one of
the first and chose to do this locking at open time. But this gives you a
file system that doesn't look just like real Unix. The Domain file system
was actually designed and built before Aegis looked anything like Unix, so
in a way it's a miracle that it does as well as it does at imitating Unix.
In more modern Unix distributed file systems such as AFS, locking is done at
a much finer temporal resolution, usually at the individual read/write time.
This makes reading and writing slower, but makes the distributed system
look more like the central system that you're used to. In most cases this
penalty is negligible, because the normal case doesn't include multiple
writers on different clients.
In principle, you could write an ios type manager that used the single-level
store to access file system objects, but put an extra locking layer on top
to do the locking at read/write time, and make the file system appear like a
central Unix file system (modulo performance). You could even do it as a
pass-through (Apollo-speak for type inheritance) of the unstruct manager.
The open operation would open for inq-only. Read and write would re-lock
for read/write, do the io, then unlock. Other operations would simply
pass-through to unstruct.
This wouldn't be hard. Since I'm already using AFS, I'm not going to
bother.mishkin@apollo.HP.COM (Nathaniel Mishkin) (03/08/91)
In article <1991Mar1.153324@apollo.HP.COM>, mishkin@apollo.HP.COM (Nathaniel Mishkin) writes: >The only significant >differences between the NFS server and the one in DOMAIN is that the >former uses a publicly specified protocol (good idea!) and the latter >embodies and enforces certain rules about concurrent access to files >(also a good idea, as far as I'm concerned). I've been informed by reliable sources that a future release of DOMAIN has "relaxed" its locking rules to allow multiple readers and one writer spread over as many nodes as you like. (No strong promises about consistency though. Just like NFS.) I think this will make a lot of people (although not everyone, I'm sure) happier. (Among other things, it will let me read the log files on the random nodes I tend to have to watch [i.e., without crp'ing onto the nodes first].) I think it's a reasonable compromise: the kind of damage that can result from reading inconsistent data tends to be somewhat less than can result from what happens if writers collide in an uncontrolled way. -- -- Nat Mishkin Cooperative Object Computing Division / East Hewlett-Packard Company mishkin@apollo.hp.com