srinivas@ut-sally.UUCP (Srini Sankaran) (02/16/88)
I would like to know how System V.3 's RFS compares against Sun's NFS in capabilities, implementation, performance etc... If there is enough interest (and enough response!), I will summarize the replies on the net. -thanks. srini... srinivas@sally.utexas.edu ...!ut-sally!srinivas
andrew@uel.uel.co.uk (Andrew Josey) (03/07/88)
In article <10370@ut-sally.UUCP>, srinivas@ut-sally.UUCP (Srini Sankaran) writes: > I would like to know how System V.3 's RFS compares against Sun's NFS > in capabilities, implementation, performance etc... ^^^^^^^^^^^^ AT&T's RFS is fully System V compatible. RFS is media and protocol independent whereas NFS relies on a specific TCP/IP implementation. Both are data transparent, i.e. data can be accessed without the user knowing whether the data is local or remote. RFS also supports transparency to devices such as printers, tape drives, comms links etc. and has extensive administrative support for controlling access to shared resources. -- Andrew Josey, AT&T Unix Europe, a Division of AT&T (UK) Ltd. International House, Ealing Broadway, London W5 5DB, England, UK uucp:{ mcvax!ukc, attunix} uel!andrew { The usual disclaimer .... }
nowicki%rose@Sun.COM (Bill Nowicki) (03/14/88)
In article <720@uel.uel.co.uk>, andrew@uel.uel.co.uk (Andrew Josey) writes: > AT&T's RFS is fully System V compatible. RFS is media and protocol > independent whereas NFS relies on a specific TCP/IP implementation. Not true. The NFS protocol is based upon Sun RPC (recently posted to the net), which has been implemented on many different network architectures. Most of the current NFS vendors happen to use TCP/IP so that they interoperate. As for media, IP (and NFS!) runs today on everything from RS-232 dial-up, Arpanet, T1 point-to-point links, Ethernet, token rings, ProNet-80s, Hypercahnel, and FDDI. The difference often discussed is that RFS keeps state in the server about every client who has a file open. This implies that when your server goes away all I/O to open files returns with errors, while NFS allows the clients to wait until the server recovers if they wish. But the real difference is that RFS runs on System V Release 3. NFS runs on everything from MS-DOS PCs to a Cray-2: many different flavors of Unix (including BSD and SVR3), LISP Machines, and even VMS and TOPS-20 operating systems. > Andrew Josey, AT&T Unix Europe, a Division of AT&T (UK) Ltd. Obviously neither of us are biased! :-) Bill Nowicki Sun Microsystems > { The usual disclaimer .... }
dannyb@kulcs.uucp (Danny Backx) (03/14/88)
In article <720@uel.uel.co.uk> andrew@uel.uel.co.uk (Andrew Josey) writes: >In article <10370@ut-sally.UUCP>, srinivas@ut-sally.UUCP (Srini Sankaran) writes: >> I would like to know how System V.3 's RFS compares against Sun's NFS >> in capabilities, implementation, performance etc... > ^^^^^^^^^^^^ >AT&T's RFS is fully System V compatible. RFS is media and protocol independent >whereas NFS relies on a specific TCP/IP implementation. Both are data ^^^^^ You mean UDP/IP, I hope. Danny -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Danny Backx | mail: Katholieke Universiteit Leuven Tel: +32 16 200656 x 3537 | Dept. Computer Science E-mail: dannyb@kulcs.UUCP | Celestijnenlaan 200 A ... mcvax!prlb2!kulcs!dannyb | B-3030 Leuven dannyb@kulcs.BITNET | Belgium
jqj@uoregon.UUCP (JQ Johnson) (03/15/88)
The latest issue of TOCS (Feb. 1988) contains 2 papers from the 1987 SIGOPS Symposium that are directly relevant to a comparison of network file systems. A paper by John Howard et al discusses the VICE system (part of the C-MU Andrew project), and a paper by Nelson et al discusses Sprite, a network file system being developed at Berkeley. Both papers compare their system with SUN NFS, arguing that NFS is perceived as "a de facto standard" but that it has serious deficiencies when compared to their preferred system. Problems sited with NFS seem to fall into 2 categories: 1/ scalability to large systems, because of excessive network traffic, excessive server cpu loading, or difficulty in administration; 2/ problems with consistency semantics. For example, under SUN NFS if processes on different workstations perform interleaved write to a file the result is indeterminate. The two problems are argued to be related; since NFS is stateless the server cannot use any knowledge of client file caching, and hence caching must be minimized to insure consistency (that's my interpretation of the papers, not necessarily the authors'). Since these papers have been floating around for almost a year, I would think that by now SUN would have a response, either in the form of counter- arguments or draft revisions to the NFS spec. Would someone from SUN care to comment? I am particularly concerned by the difficulty in formally describing the consistency semantics of NFS. Granted that NFS is designed to work right "most of the time", it is troubling to be unable to reason formally about distributed systems built on top of NFS.
nowicki%rose@Sun.COM (Bill Nowicki) (03/16/88)
In article <1695@uoregon.UUCP>, jqj@uoregon.UUCP (JQ Johnson) writes: > The latest issue of TOCS (Feb. 1988) contains 2 papers from the 1987 > SIGOPS Symposium that are directly relevant to a comparison of network > file systems. A paper by John Howard et al discusses the VICE system > (part of the C-MU Andrew project), and a paper by Nelson et al discusses > Sprite, a network file system being developed at Berkeley. > > Problems sited with NFS seem to fall into 2 categories: > 1/ scalability to large systems, because of excessive network > traffic, excessive server cpu loading, or difficulty in > administration; Well of course the authors of systems are going to prefer their own. This is called the "Not Invented Here" syndrome. As for scalability, I would estimate about 100,000 installed systems running NFS. Last I heard, about 500 run Andrew, and perhaps a few dozen run Sprite. Don't get me wrong: these were good papers and have significant research contributions. NFS is a commercial product developed four years ago, so of course the research has to be done first. Note that Sprite explicitly gives up heterogeneity and crash recovery, and VICE assumes a local disk. Both of these projects were done on Sun workstations, so we are happy to see good results. > 2/ problems with consistency semantics. For example, under > SUN NFS if processes on different workstations perform > interleaved write to a file the result is indeterminate. Sure -- if two timesharing users perform interleaved writes on the same file the result is also indeterminate. "transparency" :-) We are working on further revisions of the NFS specification. Copies were distributed at the recent "Connectathon" for all NFS licensees (and anyone else who wanted to show up). The problem (feature?) is that we are a small group, so do not necessarily rush out and implement the fad of the week. On the other hand (please pardon the plug) we are hiring. Send resumes. -- Bill Nowicki Sun Microsytems (not an official spokespeon!)
srg@quick.COM (Spencer Garrett) (03/16/88)
Another problem with RFS is that the control structures are passed around in native binary (or so say the AT&T training folks), so you can't mount filesystems on machines with different byte orders. I'll offer this comparison. RFS is more transparent IFF you can use it. It allows access to devices, and propagates unix file system semantics more thoroughly (in fact, almost completely). NFS, on the other hand, is completely machine independent and even reasonably operating system independent. RFS runs ONLY on SYS V 3.2, and only between compatible machines. NFS lets a PC/XT mount a filesystem on a CRAY. RFS and STREAMS and such are interesting, but AT&T continues to act like their own OSG releases are all there ever was, and that networking and interoperating are only maybe going to happen someday. Fortuately, neither assertion is true.
bob@allosaur.cis.ohio-state.edu (Bob Sutterfield) (03/16/88)
In article <45660@sun.uucp> nowicki%rose@Sun.COM (Bill Nowicki) writes: >In article <1695@uoregon.UUCP>, jqj@uoregon.UUCP (JQ Johnson) writes: >> ... >> Problems sited with NFS seem to fall into 2 categories: >> 1/ scalability to large systems, because of excessive network >> traffic, excessive server cpu loading, or difficulty in >> administration; > >Well of course the authors of systems are going to prefer their own. >This is called the "Not Invented Here" syndrome. As for scalability, >I would estimate about 100,000 installed systems running NFS. >Last I heard, about 500 run Andrew, and perhaps a few dozen run Sprite. Installed base != scalability. We have encountered scaling problems in existing implementations of NFS, and we worry about what happens when we hit scaling problems in the design. We have about 130 Suns in our department so far (headed for at least 250 by September), and each NFS-mounts nine filesystems from one of our servers. We only have one server with the horsepower for that: a Pyramid 98x. All our Sun-3/180s export only one user filesystem to the rest of the world, besides taking care of their direct clients. In a steady-state network with the usual occasional client bounce, all is well. But think about when a few subnets-full of clients - say 36 or so - are taken off the air while their ND servers/IP gateways-to-the-backbone are dumped to tape. When the clients are brought back up and those mount requests hit the central server, it's not pretty. nfsd has the horsepower, particularly when there are 8 of them running. inetd and portmap have no problems directing the traffic. But each time rpc.mountd tries to service a mount request, it has to sort through its in-core cache, then flush the state to /etc/rmtab. By the time rpc.mountd has worked down its request queue, the older ones have timed out. Then those clients retry. rpc.mountd thrashes and chews up CPU time with abandon for a few days. We have documented these effects in well-controlled tests, if you're interested. We have found that killing rpc.mountd, mv'ing /dev/null to /etc/rmtab, and starting a new rpc.mountd with -d will often get us rolling again for a while, until the thrashing starts again in a few minutes. In an hour or two of babysitting, we can get all our filesystems mounted on all those clients and life goes on. The problem seems to be the bottleneck of updating /etc/rmtab. We would remove that dependency (if we had our sources by now), which would only break showmount(8). Note that this is neither a philosophical problem with NFS, nor a Pyramid-specific bug: According to conversations at UNIForum with a person from Sun's NFS portability group, all known implementations of NFS have this feature. I'm satisfied it's a problem with the current implementations of the protocol, that can be fixed by sufficient beating on vendors, or a few minutes in a quiet room with the sources. But what design misfeatures might we encounter as we scale even bigger, that aren't so easily solved? Would someone from Berkeley or CMU care to comment upon any specific scaling problems {\em in the NFS design} that Sprite or Andrew would solve in a large network of diskless workstations? -=- Bob Sutterfield, Department of Computer and Information Science The Ohio State University; 2036 Neil Ave. Columbus OH USA 43210-1277 bob@cis.ohio-state.edu or ...!cbosgd!osu-cis!bob
karl@triceratops.cis.ohio-state.edu (Karl Kleinpaste) (03/17/88)
bob@allosaur.cis.ohio-state.edu writes:
Note that this is neither a philosophical problem with NFS,
nor a Pyramid-specific bug: According to conversations at UNIForum
with a person from Sun's NFS portability group, all known
implementations of NFS have this feature.
An addendum on that: The problem is not in all known implementations;
Mark Stein of Sun's NFS development group told us in more recent
releases of NFS, the strategy using /etc/rmtab is no longer current.
I don't recall what the new strategy is that's in place, but the
/etc/rmtab bottleneck is no longer a problem.
[Bob wasn't at UniForum, and those of us who were there evidently
didn't make the point to Bob until now.]romkey@kaos.UUCP (John Romkey) (03/17/88)
In article <45660@sun.uucp> nowicki%rose@Sun.COM (Bill Nowicki) writes: :In article <1695@uoregon.UUCP>, jqj@uoregon.UUCP (JQ Johnson) writes: :: 1/ scalability to large systems, because of excessive network :: traffic, excessive server cpu loading, or difficulty in :: administration; : :As for scalability, :I would estimate about 100,000 installed systems running NFS. :Last I heard, about 500 run Andrew, and perhaps a few dozen run Sprite. Uh, 'total users in the known universe' is not what JQ Johnson was talking about. Ability to scale up a single NFS installation is... -- - john romkey ...harvard!spdcc!kaos!romkey romkey@kaos.uucp romkey@xx.lcs.mit.edu
sas@pyrps5 (Scott Schoenthal) (03/17/88)
In article <8431@tut.cis.ohio-state.edu> karl@triceratops.cis.ohio-state.edu (Karl Kleinpaste) writes: >bob@allosaur.cis.ohio-state.edu writes: > Note that this is neither a philosophical problem with NFS, > nor a Pyramid-specific bug: According to conversations at UNIForum > with a person from Sun's NFS portability group, all known > implementations of NFS have this feature. > >An addendum on that: The problem is not in all known implementations; >Mark Stein of Sun's NFS development group told us in more recent >releases of NFS, the strategy using /etc/rmtab is no longer current. >I don't recall what the new strategy is that's in place, but the >/etc/rmtab bottleneck is no longer a problem. Please get your facts straight. In the standard Sun NFS Release 3.2/4.3 (the most recent one available to vendors such as Pyramid), /etc/rmtab is updated after each mount request (successful or otherwise) by the rpc.mountd. What happens is that the mountlist is written to a temporary file. The file is then renamed to be /etc/rmtab (causing the old copy to be removed). As new mounts are added to mountlist, they are linked into the head of the list. No re-ordering of the list is done. Pyramid's OSx releases 3.1, 4.0, 4.1, and 4.4 follow this strategy. Unless Sun has changed the way that the mount daemon does service dispatch in SunOS 4.0, I would suggest that you re-consider your statement on the /etc/rmtab "bottleneck". The rest of this is in reply to Bob Sutterfield's comments ... Message-ID: <8428@tut.cis.ohio-state.edu> > Installed base != scalability. We have encountered scaling > problems in existing implementations of NFS, and we worry about what > happens when we hit scaling problems in the design. I think that this is a valid concern. One of the problems that Pyramid has had as a supplier of an NFS product is in taking algorithms which were originally designed on a microcomputer and adapting them to a large(r) multi-user (and multi-processor) environment. In some sense, this is true of a lot of code that we use to build our software products: e.g., 4.x BSD. To take the mount daemon as an example. Its list of mounted paths is kept in a linked list. Ohio State mounts some 130 Suns each with 9 NFS file systems onto its poor Pyramid 98x. Linked list traversal -- over 1000 elements -- string comparisons of pathname components to find the correct element. You be the judge. Please don't take this as a knock on Sun. NFS is a very good product and the releases that are distributed to vendors are generally of high quality. I think that it's safe to say that we (both the NFS vendor and user communities) need to understand better how NFS works (by this, I mean what are the issues involved in, say, service an NFS message and forming its reply -- timing constraints, multi-processing, multi-processor, etc). and what its bottlenecks (or implementation scalabilty issues) are. I know that Sun is looking at this problem -- an excellent presentation was given at Connectathon a few months ago by Sun on some performance measurements extracted by "diddling" with various NFS parameters (# of biods, nfsds, etc.). I'm sure that other NFS vendors are looking at this as well. Among the scalability issues I see in Unix-based implementations: socket receive queue size is limited, the current lock service is top-heavy, client timeout and re-transmit parameters are derived by magic, many of the internal data structures are kept inefficiently for large quantities of data. > [ ... ] Would someone from > Berkeley or CMU care to comment upon any specific scaling problems > {\em in the NFS design} that Sprite or Andrew would solve in a large > network of diskless workstations? I'd be interested in this as well. Of course, I'd also be interested if any of the other NFS vendors (e.g., Sun, Sequent, Gould, Alliant, Convex) had comments on NFS scalability. sas ---- Scott Schoenthal sas@pyrps5.pyramid.com Pyramid Technology Corp.
sas@pyrps5 (Scott Schoenthal) (03/17/88)
In article <17043@pyramid.pyramid.com> sas@pyrps5.pyramid.com (Scott Schoenthal) writes: >In article <8431@tut.cis.ohio-state.edu> karl@triceratops.cis.ohio-state.edu (Karl Kleinpaste) writes: >>An addendum on that: The problem is not in all known implementations; >>Mark Stein of Sun's NFS development group told us in more recent >>releases of NFS, the strategy using /etc/rmtab is no longer current. >Please get your facts straight. Ahem. Well, I should also get my facts straight. In Sun's D/NFS 1.0 release, rpc.mountd will do away with the rmtab update entirely. My apologies, Karl. My understanding is that D/NFS 1.0 is still in beta test and has not been "final released" to OEMs. Your statement should be amended to say "in the next release of NFS to Sun's NFS vendors, the strategy of using /etc/rmtab is no longer current". sas ---- Scott Schoenthal sas@pyrps5.pyramid.com Pyramid Technology Corp.
aaa@mtuni.ATT.COM (Aaron Akman) (03/17/88)
From article <184@quick.COM>, by srg@quick.COM (Spencer Garrett): > > Another problem with RFS is that the control structures are passed > around in native binary (or so say the AT&T training folks), so you > can't mount filesystems on machines with different byte orders. Not true. I'm running RFS between different machines with conflicting byte order. It works just fine. -- Aaron, mtuni!aaa, 201-957-2751
mjp@sfsup.UUCP (M.J.Purdome) (03/18/88)
In article <184@quick.COM>, srg@quick.UUCP writes: > > Another problem with RFS is that the control structures are passed > around in native binary (or so say the AT&T training folks), so you > can't mount filesystems on machines with different byte orders. > Actually, "RFS control structures" are all passed in a machine-independent format (XDR, I believe). This includes all data structures that the kernel knows about, like the stat structure, etc. RFS _does_ run across machines with different alignment and byte-ordering requirements. Data written by an application is not converted, since there is no way for RFS to know the structure of the data to convert it to a canonical form. I believe NFS works in a similar manner. This may be what the training folks meant. -- Mark Purdome -- AT&T, 190 River Road A-130, Summit, NJ 07901 [ihnp4 | allegra]!attunix!mjp disclaimer: my opinions != AT&T opinions
rees@apollo.uucp (Jim Rees) (03/20/88)
>> Problems sited with NFS seem to fall into 2 categories: >> 1/ scalability to large systems, because of excessive network >> traffic, excessive server cpu loading, or difficulty in >> administration; > >Well of course the authors of systems are going to prefer their own. >This is called the "Not Invented Here" syndrome. As for scalability, >I would estimate about 100,000 installed systems running NFS. >Last I heard, about 500 run Andrew, and perhaps a few dozen run Sprite. Installed base != scalability. We have encountered scaling problems in existing implementations of NFS, and we worry about what happens when we hit scaling problems in the design. I've never used a large NFS system, but I'm currently using a system that includes about 2000 machines sharing a single file system. Any file name can be used interchangeably by any machine. I have a hard time imagining how this would work with NFS. For example, we have a few hundred people in R&D. Almost all of them will at some time want to look at the kernel sources. If you use NFS, how does this work? Does everyone mount the file system containing the kernel sources all the time? Do you just mount it when you want to look at the sources? What happens when the sources move to a different machine? Is it possible for 2000 clients to each mount a couple dozen servers? How do you keep track of where the servers are? If I write a trip report, and want to send it to a large group of people, I put it in a file and send the file name out via email (or news). Then the people who want to look at it just read that file. How would you do this with NFS? E-mail the whole thing? Send out the file name and let people mount your disk as needed? How does uucp work with NFS? Does every machine mount the uucp lib and spool, or is there a single "uucp machine" that I rlogin in to when I want to uucp something? This is only partly meant as a criticism of NFS. I really am interested in how you folks with large NFS installations get your work done.
les@chinet.UUCP (Leslie Mikesell) (03/21/88)
>> Another problem with RFS is that the control structures are passed >> around in native binary (or so say the AT&T training folks), so you >.. RFS _does_ run >across machines with different alignment and byte-ordering requirements. >Data written by an application is not converted, since there is no way >for RFS to know the structure of the data to convert it to a canonical form. Likewise for the structs passed to ioctl() for various devices. Thus the advantage of RFS being able to access remotely-mounted devices applies only to similar machines. Has anyone mentioned that currently RFS only supports mounting of complete directories (no symbolic links to files)? This sort of precludes sharing anything from /etc .... Les Mikesell ..ihnp4!chinet!les
melohn%sluggo@Sun.COM (Bill Melohn) (03/21/88)
In article <3af44ad7.b8ab@apollo.uucp> rees@apollo.uucp (Jim Rees) writes: >I've never used a large NFS system, but I'm currently using a system >that includes about 2000 machines sharing a single file system. Any >file name can be used interchangeably by any machine. I have a hard time >imagining how this would work with NFS. I'm not sure I understand why; there is nothing intrinstically more complex about a NFS based distributed file system than one based on propritary operating systems. On the contrary, scaling up tends to show off many of NFS's design strongpoints; for example stateless becomes much more important when you have a large number of file servers which can be potentially down at any given point. >For example, we have a few hundred people in R&D. Almost all of them >will at some time want to look at the kernel sources. If you use NFS, >how does this work? Does everyone mount the file system containing the >kernel sources all the time? Do you just mount it when you want to >look at the sources? What happens when the sources move to a different >machine? Is it possible for 2000 clients to each mount a couple dozen >servers? How do you keep track of where the servers are? Typically communities of interest will mount filesystems of interest from a common set of servers. With NFS, these servers can be anything from a lowly MicroVAX up to a Cray, and the client can be a workstation, PC, or any one of some hundred odd vendor implementations. Last time I checked, the Domain file system was only available on a single family of processors from a single vendor. This is fine if you happen to have a lot of these machines around (:-)), but most big NFS users have a variety of machines from several different vendors with which they would like to build a distributed enviornment. >If I write a trip report, and want to send it to a large group of people, >I put it in a file and send the file name out via email (or news). Then >the people who want to look at it just read that file. How would you do >this with NFS? E-mail the whole thing? Send out the file name and let >people mount your disk as needed? Again, the difference between explicitly mounting a remote file system and merely referring to it in a global name space seems fairly trivial. The mount command under Unix is a super-user only command, but it would be pretty easy to write a service that would mount a remote file system for you (like the "on" command does) using NFS and the mount protocol. >How does uucp work with NFS? Does every machine mount the uucp lib and >spool, or is there a single "uucp machine" that I rlogin in to when I >want to uucp something? At Sun, many users remotely mount the file system that contains newsgroups from a central server, and then use the NNP to send posted articles to this machine for centralized distribution. It is also possible to rlogin to this machine directly.
andrew@uel.uel.co.uk (Andrew Josey) (03/21/88)
In article <184@quick.COM>, srg@quick.COM (Spencer Garrett) writes: > RFS runs ONLY on SYS V 3.2 ^^^^^^^^^^^^^^^^^^^^^^^^^^ also 3.0 and 3.1 -- Andrew Josey, AT&T Unix Europe, a Division of AT&T (UK) Ltd. International House, Ealing Broadway, London W5 5DB, England, UK uucp:{ mcvax!ukc, attunix} uel!andrew { The usual disclaimer .... }
henry@utzoo.uucp (Henry Spencer) (03/22/88)
> >... there is no way > >for RFS to know the structure of the data to convert it to a canonical form. > > Likewise for the structs passed to ioctl() for various devices. Thus the > advantage of RFS being able to access remotely-mounted devices applies > only to similar machines... Some uses of remote devices would require ioctls, but some would not. The ones that don't would work. The ones that do will have to wait until something is done about XDR encoding of ioctls (with all the difficulties that that will involve...). It's still a win.
david@ms.uky.edu (David Herron -- Resident E-mail Hack) (03/22/88)
In article <3af44ad7.b8ab@apollo.uucp> rees@apollo.uucp (Jim Rees) writes: >I've never used a large NFS system, but I'm currently using a system >that includes about 2000 machines sharing a single file system. Any >file name can be used interchangeably by any machine. I have a hard time >imagining how this would work with NFS. I'm not sure if we count as "large". We've got about 20-30 workstations sharing disk space from 6 servers -- and growing. >For example, we have a few hundred people in R&D. Almost all of them >will at some time want to look at the kernel sources. If you use NFS, >how does this work? Does everyone mount the file system containing the >kernel sources all the time? Do you just mount it when you want to >look at the sources? What happens when the sources move to a different >machine? Is it possible for 2000 clients to each mount a couple dozen >servers? How do you keep track of where the servers are? You could do it either way depending on the local preferences. Around here, we attempt to make this grand hoax that we've got one file system, and all the nfs participants keep everything that's nfs'able mounted at all times. Part of our /etc/rc stuff is a script /etc/rc.remote which does things like: if [ `hostname` != some.local.machine ]; then /etc/mount all the stuff that some.local.machine makes available to people fi It USED to be EXACTLY like that, we now have a scheme where we keep a central copy of all the configuration files. There is one file in this central place which is used to make the individual copies which actually are used on the other machines. Each line of this central file is marked for which host(s) it applies to. When things move around we edit this file, rdist it to each machine, and then reboot everybody. (If you ask me, having to reboot everybody is a bit drastic -- but then I didn't configure that end of our system). The current effect is like above. I don't think it would work to have 2000 clients mounting the same thing with NFS. I doubt that the kernal tables are able to handle this reasonably, if at all. But maybe I'm wrong about that. What's the size of Sun's internal network? >If I write a trip report, and want to send it to a large group of people, >I put it in a file and send the file name out via email (or news). Then >the people who want to look at it just read that file. How would you do >this with NFS? E-mail the whole thing? Send out the file name and let >people mount your disk as needed? Again, same deal as above. Oh, and simply tell 'em the file name. >How does uucp work with NFS? Does every machine mount the uucp lib and >spool, or is there a single "uucp machine" that I rlogin in to when I >want to uucp something? There's a "uucp machine" that you have to rlogin to when you want to uucp something. The problems are: 1. devices don't export -- when you do the uucp/uux command, it might want to immediately call up a uucico to make a call. BUT the /dev guys don't exist, so you CAN'T make the call. 2. locking -- NFS didn't support across-the-net locking of files until recent versions. Therefore, if flock()'s were being used anywhere this wouldn't actually lock anybody out. 3. different physicial machines -- NFS runs in a mixed environment. That is, our nfs is shared between uVaxIIen running Ultrix, Vaxen running Mt Xinu 4.3+NFS, and Sun's. All have different binaries and can't run each others binaries. 4. different machines have different pid's -- UUCP's lock files contain the pid of the locker so that you can check to see if the lock is still valid. (i.e. kill(0, pid) and test the return code). >This is only partly meant as a criticism of NFS. I really am interested >in how you folks with large NFS installations get your work done. Yeah. I'm interested in how it works at really large places too. I'm also curious at how Apollo's system solves those problems. For instance, it'd be wasteful to have your kernel sources physically stored on multiple machines, not to mention the headaches of maintaining consistency in those multiple copies. That means that you've got to have your 200 guys in r&d accessing the same machine to get to those kernal sources. (Substitute for kernal sources just about anything you care to name -- but needs to be accessed by a lot of people at the same time -- news for instance). How would ANY similar system avoid being bogged down considerably when lots and lots of people are accessing the same stuff? (Be it news articles or kernal sources). Around here we've got two sorts of things which are commonly accessed over NFS. One is the user files -- all our workstations are basically diskless. (The uVax2000's have a 40 meg disk, but that's used soley for the important parts of the os and for tmp and swap). The user files are stored over on servers (at the moment, 2). The other is the news articles, troff stuff (font files), and /usr/local commands. Our experience has been that the two main servers have the most consistently high load averages as well as the most consistently high error rates in the ether drivers. (These servers are uVaxIIen with DEQNA's). The correlation between load average and nfs activity is easy to see, often if the load average is 3 then 3 of the nfsd's will be active. What do other systems do to avoid bogging down the source machine for heavily used items? The other issue with across-network file systems is coordinating the placing of objects within the global file system. At least with NFS and RFS the administrators have a lot of options on how they can arrange things, but then they end up with a couple of little files to edit on each machine whenever something changes. I've described our technique for getting around the hassle of making the same change to the same file on 20+ machines. What are other methods? -- <---- David Herron -- The E-Mail guy <david@ms.uky.edu> <---- or: {rutgers,uunet,cbosgd}!ukma!david, david@UKMA.BITNET <---- <---- "Oh, I dunno -- I think Sean would be rather tasty!" -- Becky
rwhite@nusdhub.UUCP (Robert C. White Jr.) (03/26/88)
in article <184@quick.COM>, srg@quick.COM (Spencer Garrett) says: > Another problem with RFS is that the control structures are passed > around in native binary (or so say the AT&T training folks), so you > can't mount filesystems on machines with different byte orders. Having gotten out my magic book, I see that it is true that the structures are passed around in binary. I would like, however, to ask how Spencer suggests they be passed around? The network hardware and software is more than equal to the task of swapping all the bytes around while it is being processed. All the structures are aligned "long"s and/or character arays <one aray per structure MAX>. Put nicely, a x386 machine and a 3B2 and a SPARC machine may RFS freely. [I have seen the 386 and 3B2's do it, but not the SPARC as that is T.B.A.] i.e. you CAN mount filesystems on machines with different byte oreders. Rob.
rwhite@nusdhub.UUCP (Robert C. White Jr.) (03/26/88)
in article <3845@chinet.UUCP>, les@chinet.UUCP (Leslie Mikesell) says: > Likewise for the structs passed to ioctl() for various devices. Thus the > advantage of RFS being able to access remotely-mounted devices applies > only to similar machines. From part 6 of Apendix 3 [page 128] of my "magic Book", which is a loose bound set of pages which we received under the title "porting rules" but which is in fact nothing of the kind, I quote the following: "The dada structures for some system calls, like stat, fstat, statfs, fstatfs, utime, fcntl, getdents, etc., which have shorts and character strings, will involve out of place conversion." They also explain about XDR usage on the same page, and the fact that the passage of XDR [canonical form] or primitive binary is made at virtual circut setup time. The structures passed durring the above functions are difined under the approprate command/token. I don't see where it would be all that though to swizle the bytes around if necessary. Anybody who needs the bytes swizeled will take that into account when they write their driver just like for any other system/standard. > Has anyone mentioned that currently RFS only > supports mounting of complete directories (no symbolic links to files)? > This sort of precludes sharing anything from /etc .... OF COURSE RFS doesn't support symbolic links.... RFS [under SVR3+] is implemented as a "file system switch" module. The "file system switch" is the same mechanisim used to mount different types of file systems under the same kernel. As the SVR3 kernel does not support symbolic links at all, I am not supprised that the RFS standard is similarly inclined. As an added bonus, you may be interested to note that with the merger-of-effort between AT&T and SUN Microsystems, in order to produce an inclusive UNIX System Product, will be releasing an RFS/NFS merged product which will determine the request type, and act against it in accordance with the approprate standard. Both SUN and AT&T future releases and cross-grades are done with this in mind. Rob.
les@chinet.UUCP (Leslie Mikesell) (03/27/88)
In article <930@nusdhub.UUCP> rwhite@nusdhub.UUCP (Robert C. White Jr.) writes: >> Likewise for the structs passed to ioctl() for various devices. Thus the >> advantage of RFS being able to access remotely-mounted devices applies >> only to similar machines. > The structures passed durring the above functions are difined >under the approprate command/token. I don't see where it would be all >that though to swizle the bytes around if necessary. Anybody who >needs the bytes swizeled will take that into account when they write >their driver just like for any other system/standard. But my point was about passing ioctl()s to remote devices. The type of struct passed to ioctl is *device dependent*. How is a dissimilar remote machine going to prepare a canonical form of something known only to a different machine? Or alternatively, how can the device driver make sense out of something passed to it by a machine that doesn't know what is needed? In the common case of the termio struct, it might make sense to convert, but I don't see how it can be generalized. The only machines that I know of with different byte ordering and RFS are the 3B2 and the 6386. Does anyone know if they can format each others tape drives (or even put them in streaming mode)? I don't mean to imply that anything is wrong with RFS because of this, it is just a difficult problem... -Les Mikesell
bae@ati.tis.llnl.gov (Hwa Jin Bae) (03/29/88)
I am curious about the licensing policy of NFS. SUN/RPC is in public domain;
is NFS also in public domain? If not, what's the cost of getting the
source code for NFS? I seemed to remember hearing from someone that
NFS also is based on the same licensing policy as RPC package - free
for anyone to use. Does anyone know of specific details on this?
Hwa Jin Bae | I only work for LLNL and CDC; they are not responsible
Control Data Corp. | for my personal opinions.
4234 Hacienda Drive | bae@{ati,aftac,vega}.tis.llnl.gov (Internet)
Pleasanton, CA 94566 | {ames,ihnp4,lll-crg}!lll-tis!bae (UUCP)henry@utzoo.uucp (Henry Spencer) (03/30/88)
> ...The type > of struct passed to ioctl is *device dependent*. How is a dissimilar > remote machine going to prepare a canonical form of something known > only to a different machine? ... How can you prepare a canonical form of something known only to your kernel, to do an ioctl on the local machine? The answer is, you have to know how. Either you have to know what the recipient is expecting, or you have to use some standard language that will be translated automatically into what the recipient is expecting. Putting the recipient on a different machine on the other side of a network adds more ways for the two ends to misunderstand each other, but does not alter the basic nature of the problem: the sender has to know what he's doing. (This is yet another case of a general phenomenon: networks don't add any new problems, they just break all the simplistic old solutions.) -- "Noalias must go. This is | Henry Spencer @ U of Toronto Zoology non-negotiable." --DMR | {allegra,ihnp4,decvax,utai}!utzoo!henry
sxn%ingersoll@Sun.COM (Stephen X. Nahm) (03/30/88)
In article <22090@tis.llnl.gov> bae@ati.tis.llnl.gov (Hwa Jin Bae) writes: >I am curious about the licensing policy of NFS. SUN/RPC is in public domain; >is NFS also in public domain? If not, what's the cost of getting the >source code for NFS? I seemed to remember hearing from someone that >NFS also is based on the same licensing policy as RPC package - free >for anyone to use. Does anyone know of specific details on this? RPCSRC (or Sun RPC) is an unlicensed implementation of the RPC and XDR library (plus other goodies). It comes directly from SunOS code. While it's freely redistributable, I (personally) wouldn't call it public domain (see the DISCLAIMER in the package). RPCSRC 3.9 has nfs_prot.x, which is an RPCGEN-ready version of the NFS protocol. RPCGEN, which also is included in RPCSRC, is a protocol compiler that takes C-like protocol definition files and produces XDR encoding routines and stubs for writing the client and server sides of the protocol. With nfs_prot.x, the RPC library and RPCGEN you could conceivably put together your own NFS implementation. If anyone does this, I'd love to hear about it. "NFS" is a trademark of Sun Microsystems. Sun's NFS implementation is proprietary, but the protocol is not. The protocol is specified in RPCSRC both in nfs_prot.x and in a document. Sun's Portable ONC group sells a licensed reference port (called NFSSRC) of Sun's NFS implementation. It costs *alot* of money, so only companies who intend to port it to their OS and resell it usually buy NFSSRC (universities can get it cheaper). The reference is to Berkeley 4.3BSD. Sun will stop selling NFSSRC when Sys V.4 is released (the merged UNIX), since that will have ONC/NFS bundled with it. You can also buy a user level NFS implementation from us (UNFS) for less money that NFSSRC, but also *much* less performance (and it's a server only). We usually only push it as a "porting tool." If you want further details, email to "nfsnet@sun.com". Steve Nahm sxn@sun.COM or sun!sxn Portable ONC/NFS
srg@quick.COM (Spencer Garrett) (03/30/88)
In article <929@nusdhub.UUCP>, rwhite@nusdhub.UUCP (Robert C. White Jr.) writes:
) in article <184@quick.COM>, srg@quick.COM (Spencer Garrett) says:
) > Another problem with RFS is that the control structures are passed
) > around in native binary (or so say the AT&T training folks), so you
) > can't mount filesystems on machines with different byte orders.
)
) Having gotten out my magic book, I see that it is true that
) the structures are passed around in binary.
)
) I would like, however, to ask how Spencer suggests they be
) passed around? The network hardware and software is more than
) equal to the task of swapping all the bytes around while it
) is being processed. All the structures are aligned "long"s
) and/or character arays <one aray per structure MAX>.
The key word here is "native". The AT&T folks told me there was no
attempt to use network byte order and no equivalent to SUN's XDR
protocol. They appear to have been wrong, but I quoted my source
so MY statement is still correct. Binary is fine, as long as you
agree on the representation and byte order. Does the magic book
indicate how they solve this problem? I presume they use NBO
and two's complement representation, but if the standard doesn't
SAY that, they've still got a problem.rwhite@nusdhub.UUCP (Robert C. White Jr.) (03/31/88)
in article <4192@chinet.UUCP>, les@chinet.UUCP (Leslie Mikesell) says: >> The structures passed durring the above functions are difined >>under the approprate command/token. I don't see where it would be all >>that though to swizle the bytes around if necessary. Anybody who >>needs the bytes swizeled will take that into account when they write >>their driver just like for any other system/standard. > > But my point was about passing ioctl()s to remote devices. The type > of struct passed to ioctl is *device dependent*. How is a dissimilar > remote machine going to prepare a canonical form of something known > only to a different machine? Or alternatively, how can the device The RFS file system is a UNIQUE "device" READ the above! "The Structures passed durring the ABOVE FUNCTIONS are _DEFINED_ under THE APPROPRATE COMMAND/TOKEN." [in this case DUIOCTL, DUFCNTL {etc} are RFS tokens] Since only certain things are DEFINED for RFS access [just like ANY other device] there are only certain things passed.... It works like this: The ORDER of the memory structure is juggled into a pre defined patern defining the value of the first X bytes depending on which command you are issueing. [i.e. the "common stuff" comes first, and then the rest of the bits come in an otherwise-first-come-first-serve basis.] THAT means that anybody who uses the RFS stuff will know that at least the first X bytes alwars mean <whatever> the extra stuff is take-or leave. It is just like mounting a forign file system under the file system switch. If you want to do it, you figure out your constant info, and work with it. Put in a simpler form: Your kernel dosn't understand TCP/IP, RFS, NFS, IPC, SNA, BSC, or anything else until you install the drivers. The driver translates the information. That's what a driver is for. There is no great mystery in to packet translation. Your system does it every day in a thousand ways. Whe you [or whoever] writes the driver for RFS they get all the fromats [including the header files for ioctl and the like which in this case happen to be the SVR? version in pure form] and say "This is the format to assert my information on the media. Only these bytes are required for my system, so I'll have to ignore the rest on input and fake them on output" what could be simpler? It's been done a million times for TCP/IP and others... I still don't see your problem, and aparently neither do the people implementing the drivers for the byte swizeled machines. Rob.
rwhite@nusdhub.UUCP (Robert C. White Jr.) (03/31/88)
in article <724@uel.uel.co.uk>, andrew@uel.uel.co.uk (Andrew Josey) says: > In article <184@quick.COM>, srg@quick.COM (Spencer Garrett) writes: >> RFS runs ONLY on SYS V 3.2 > ^^^^^^^^^^^^^^^^^^^^^^^^^^ > also 3.0 and 3.1 My 3.1.1 System will be shocked when it finds out, All this time and it's been a criminal ;-) Rob. <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< << All the STREAM is but a page,<<|>> Robert C. White Jr. << << and we are merely layers, <<|>> nusdhub!rwhite nusdhub!usenet << << port owners and port payers, <<|>>>>>>>>"The Avitar of Chaos"<<<<<<<<<<<< << each an others audit fence, <<|>> Network tech, Gamer, Anti-christ, << << approaching the sum reel. <<|>> Voter, and General bad influence. << <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< ## Disclaimer: You thought I was serious???...... Really???? ## ## Interogative: So... what _is_ your point? ;-) ## ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
les@chinet.UUCP (Leslie Mikesell) (04/02/88)
In article <933@nusdhub.UUCP> rwhite@nusdhub.UUCP (Robert C. White Jr.) writes: >> >> But my point was about passing ioctl()s to remote devices. The type > >The RFS file system is a UNIQUE "device" > Put in a simpler form: Your kernel dosn't understand >TCP/IP, RFS, NFS, IPC, SNA, BSC, or anything else until you ... >I still don't see your problem, and aparently neither do the Do any of those other things let you mount /dev from a remote machine and proceed to open and use the special files there in the same ways that you would use local devices (transparently, as they say..)? RFS does, and with similar machines you can do things that require passing structs to ioctl(), as is required for example to set the modes on a tty line on the remote machine (see termio.h on a sysV machine). This is not at all the same as dealing with normal (non-special) files on a remote machine. In the case of normal files both machines can know how to translate each others requests. In the case of remote devices, the nature of the struct passed to ioctl() may be known only to the remote machine. The termio struct could probably be made a special case, but other devices require other types of structs... Les Mikesell ..ihnp4!chinet!les
rwhite@nusdhub.UUCP (Robert C. White Jr.) (04/04/88)
in article <4456@chinet.UUCP>, les@chinet.UUCP (Leslie Mikesell) says: > Do any of those other things let you mount /dev from a remote machine > and proceed to open and use the special files there in the same ways > that you would use local devices (transparently, as they say..)? > RFS does, and with similar machines you can do things that require > passing structs to ioctl(), as is required for example to set the > modes on a tty line on the remote machine (see termio.h on a sysV machine). > This is not at all the same as dealing with normal (non-special) files on > a remote machine. In the case of normal files both machines can know how to > translate each others requests. In the case of remote devices, the > nature of the struct passed to ioctl() may be known only to the remote > machine. The termio struct could probably be made a special case, but > other devices require other types of structs... OK, I'll Spell it out...... [This is REALLY not that tough!] Les say that I am Writing a RFS driver for somthing like a VMS system. [I dont really know anything about VMS, but I am making a relativly safe assumption that Whatever VMS uses for IOCTL-like Functions does NOT resemble System V ioctl structures.] 1) I include the RFS headder Definitions in the driver handeler. 2) I write a BIT swizzeler routine. 3) I write a "special call interpreter" to handle the IOCTLs [etc] 3a) I include the System V structures [as they will be received]. 3b) I decide any special-case restrictions for item 2. 3c) I include the structures for VMS. 3d) I translate the SV calls to VMS and dispatch them. 4) I write a dispatcher which will route special cases through item 3 Before you start screaming about the binary primitives you MUST remember that 1) Those special calls are deleivered by special means. 2) you will ALWAYS know when you are getting these special calls. 3) the contents of these special calls are NOT a mystery. 4) if the order of data on one end is different from the order on the other end, a simple and fast hack is to re-arange the order. WHENEVER ANY special file system call is issued against any "file" wether it is a device special, or normal file, will arive in a envelope which uniquely defines what the type of function actually is. In the same way that FTP or TCP/IP or any other packet type is a mystery until the kernel driver is added to the system which tells it what it means. When you add the RFS driver to our system, which tells your system about DUOPEN, DUCLOSE, DUCHDIR, DUIOCTL, DUREAD, and DUWRITE [and about 40 others] you will SIMPLY ADD THE FACT THAT IOCTL TO/FROM THIS DRIVER LOOKS LIKE X GOING TO THE KERNEL AND Y TO THE MEDIA AND MAKE THE FUNCTION F(X) AND F(Y). So, WHAT _IS_ the problem?? This is all in the standard you know, have YOU READ the standard?? [excuse me, aledged standard] Pardon me for getting a little frustrated, but none of this is unique to RFS. any packeting "standard" [like X.25] talks about forign concepts and how to translate them into something the local system can understand. The fact that you may have to swizzle bits or components into new orders dosn't really affect the reasonability of the connection. Granted, if your system needs something totally different than RFS, the guy who writes you driver may have a nightmare, but then again, if it's too forign, you probably won't be dealing with alot of the esoterics. [when was the last time you opened a device special file under MS-DOS?] <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< << All the STREAM is but a page,<<|>> Robert C. White Jr. << << and we are merely layers, <<|>> nusdhub!rwhite nusdhub!usenet << << port owners and port payers, <<|>>>>>>>>"The Avitar of Chaos"<<<<<<<<<<<< << each an others audit fence, <<|>> Network tech, Gamer, Anti-christ, << << approaching the sum reel. <<|>> Voter, and General bad influence. << <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< ## Disclaimer: You thought I was serious???...... Really???? ## ## Interogative: So... what _is_ your point? ;-) ## ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
dyer@spdcc.COM (Steve Dyer) (04/05/88)
In article <1016@nusdhub.UUCP>, rwhite@nusdhub.UUCP (Robert C. White Jr.) writes: > So, WHAT _IS_ the problem?? This is all in the standard you know, have > YOU READ the standard?? [excuse me, aledged standard] > ioctl() calls on remote devices cannot be easily handled since the ioctl argument formed in user mode can be thought of as "opaque" to the local kernel, having meaning only between the user program and the server kernel. There is no way that an arbitrary client will be able to know the form of every ioctl structure passed, since such objects are by definition arbitrarily chosen by the server machine. If the client and server memory layouts don't agree, there will be trouble. -- Steve Dyer dyer@harvard.harvard.edu dyer@spdcc.COM aka {ihnp4,harvard,husc6,linus,ima,bbn,m2c}!spdcc!dyer
les@chinet.UUCP (Leslie Mikesell) (04/07/88)
In article <1016@nusdhub.UUCP> rwhite@nusdhub.UUCP (Robert C. White Jr.) writes: >So, WHAT _IS_ the problem?? This is all in the standard you know, have >YOU READ the standard?? [excuse me, aledged standard] No, I assumed that it was proprietary to AT&T - are we talking about the same RFS? Let me give an example of what you can do with similar machines (I just tried this with 3B2s linked via starlan to be sure it works). Mount / from machine a under /a on machine b <- pretty arbitrary, actually as long as /dev is accessible log into machine b execute kermit set line /a/dev/tty23 <- happens to be a nearby terminal on machine a permissions set to rwrwrw for test set speed 1200 <- terminal was running at 9600 connect .. go to terminal, switch speed to 1200, type back and forth with terminal connected to machine b Neither kermit nor machine a's tty driver knows anything about the network or RFS; it works because the ioctl() executed on machine b delivers "struct termio" to the driver on machine a transparently. > 2) I write a BIT swizzeler routine. OK if you happen to know about the device driver in question. > 3b) I decide any special-case restrictions for item 2. This will preclude special ioctls to devices you don't know about even among identical machines. > >that 1) Those special calls are deleivered by special means. 2) you will >ALWAYS know when you are getting these special calls. 3) the contents of >these special calls are NOT a mystery. 4) if the order of data on one Umm.. Do you think the network software should know the contents of struct updata (needed to dial a 3B1 internal modem), struct fmtstruct (for formatting a 3b2 floppy), dev_dump (for the cartridge tape) ad infinitum...? >Pardon me for getting a little frustrated, but none of this is unique >to RFS. any packeting "standard" [like X.25] talks about forign concepts As far as I know, no other system even attempts transparent access to remote devices accessed directly through the file system. This does not really diminish the usefulness of RFS, since it is an additional benifit when using similar machines. However, it does point out the need for a standard way of representing the contents of a structure whose elements are not known ahead of time. Les Mikesell ...ihnp4!chinet!les
chris@mimsy.UUCP (Chris Torek) (04/07/88)
>In article <1016@nusdhub.UUCP> rwhite@nusdhub.UUCP (Robert >C. White Jr.) writes: >>So, WHAT _IS_ the problem?? In article <780@spdcc.COM> dyer@spdcc.COM (Steve Dyer) writes: >ioctl() calls on remote devices cannot be easily handled since >the ioctl argument formed in user mode can be thought of as >"opaque" to the local kernel, having meaning only between the >user program and the server kernel. Right. Here is a concrete example: I want to plug your SysV RFS 68000-based machine into my 4BSD Vax network, and I want to use a 4BSD partition from the SysV machine. To do this I need to do a DKIOWDINFO IO control. SysV does not *have* a DKIOWDINFO ioctl, or if it does, it is certain to be different from that in 4.3+. But RFS claims transparent access to remote devices! I could go add DKIOWDINFO to the SysV machine (assuming I had sources). But what if it already had one, except that its version took one `int' argument? The 4.3+ DKIOWDINFO takes a 276-byte structure. -- In-Real-Life: Chris Torek, Univ of MD Comp Sci Dept (+1 301 454 7163) Domain: chris@mimsy.umd.edu Path: uunet!mimsy!chris
ralphw@IUS3.IUS.CS.CMU.EDU (Ralph Hyre) (04/08/88)
In article <4633@chinet.UUCP> les@chinet.UUCP (Leslie Mikesell) writes: >As far as I know, no other system even attempts transparent access to remote >devices accessed directly through the file system. Well, this is a little too large a generalization. CMU CSD RFS (not to be confused with the AT&T product) supports remote device access. For example, mt -f /../a0/dev/rmt12 (where a0 is the remote hostname) works as expected. This is handy for remote backups between systems with similar architectures.) I understand that the filename syntax is similar to that of the NewCastle connection. -- - Ralph W. Hyre, Jr. Internet: ralphw@ius2.cs.cmu.edu Phone:(412)268-{2847,3275} CMU-{BUGS,DARK} Amateur Packet Radio: N3FGW@W2XO, or c/o W3VC, CMU Radio Club, Pittsburgh, PA
darrelj@formal.sm.unisys.com (Darrel VanBuer) (04/08/88)
As someone outside this debate, I would like to offer a few comments (my experience is with Xerox Lisp machines which use a combination of PUP, XNS and TCP-FTP protocols for filing, plus I have looked at the possibility of implementing RPC/NFS in this environment). Both NFS and Xerox filing protocols are built on a remote procedure call layer (RPC and Courier, respectively). This RPC layer specifies the bit patterns sent through networking layers for any possible data object, including structures. In general, only the client and the server program know how to interpret the contents of the bit stream (was that word of zero bits the integer zero, a false boolean, the color BLUE from a discreet set a character string of nulls, or even the number 5 in the subrange 5..50). Different systems have been used to drive the generation of the bit stream. Sun RPC normally uses a "RPC compiler" to generate the necessary C code to do it as a preprocessor step in C compilation. Xerox Lisp does it by run-time interpretation of an extra argument which describes the format of the current call. Neither scheme depends on the operating system understanding the semantics of the call. I infer from the discussion that RFS tries to be transparent to the Unix I/O interface. If so, ioctl does present a problem. At the OS interface, you get only a pointer to a struct and its size. Between homogeneous systems, there are no problems. Between non-homogenous systems, the problems would be almost insurmountable. Besides the byte order problems, some machines don't even pack structures the same (int is different sizes, alignment rules may force holes, etc. To have the kernel reorder to some network standard would require the kernel knowing every ioctl you are willing to pass; having the server or client reorder would require knowing the struct layout rules and byte ordering for every potential remote partner. The alternatives which provide simple solutions to structure transmission (e.g. standard presentation ala RPC) would require changing the ioctl interface to include a structure descriptor or change the declarations of ioctl structures and the way they get filled in. While these are reasonable things to do, they are not transparent to existing code. Note that having a standard for transmitting calls over the network is important. Saying it does what RFS does now (presumably a write of the ioctl structure plus some control information) is not enough -- the first port to a popular machine with a different struct layout makes such a standard counterproductive if RFS has pretentions of utility in heterogeneous environments.
citron@cit-vax.Caltech.Edu (Mark C. Citron) (01/05/89)
Could anyone summarize the differences between NSF and RFS networking systems? What are the advantages/disadvantages to each? I am trying to pick one to run on a 386pc (probably under Interactive's unix or Bell Tech's unix) with a distributed database. Any comments about either of these operating systems would also be welcome. Thanks for any help. Either post or email. Mark Citron citron@csvax.caltech.edu citron@cit-vax!UUCP
les@chinet.chi.il.us (Leslie Mikesell) (01/07/89)
In article <9018@cit-vax.Caltech.Edu> citron@cit-vax.UUCP (Mark C. Citron) writes: >Could anyone summarize the differences between NSF and RFS networking >systems? What are the advantages/disadvantages to each? I've only used RFS but as I understand things RFS preserves the unix semantics to a greater extent than NFS. For example, you can access remote devices by mounting the directory that contains them (likely to work only on similar machines), and you can do IPC between machines without special programming by creating a FIFO in a mounted directory. The standard sysV file/record locking procedures are supposed to work on remote files. RFS allows uid/gid mapping per host. NFS is available for systems other than unix. Les Mikesell
guy@auspex.UUCP (Guy Harris) (01/10/89)
>The standard sysV file/record locking procedures are supposed to >work on remote files. The NFS implementations being distributed by Sun include a separate RPC protocol that allows the standard S5 file/record locking procedures to work on remote files (except for mandatory locking, but so what?). >RFS allows uid/gid mapping per host. I think Cray has made modifications to their NFS implementation to allow this as well.
rwa@auvax.uucp (Ross Alexander) (01/10/89)
In article <7387@chinet.chi.il.us>, Leslie Mikesell writes: > I've only used RFS but as I understand things RFS preserves the unix > semantics to a greater extent than NFS. Agreed. But, sadly, the implementations I have used (RFS over WIN/TCP, 10B5 network, AT&T 3B2/600's) are slower than molasses in January. Maybe it doesn't have to be that way. It sure would be nice if wasn't. But Sun 3's and _small_ Vaxen's (Vaxstation 2000, anyone?) running NFS routinely blow the 600's out of the water, approximately 7 :: 1 is the ratio I measure. On the very same network... Ross ps. anybody out there got any tuning suggestions? I've allocated enough streams buffers to choke a horse already...
tdh@moriaMoria.Sp.Unisys.Com (Thomas Hintz) (01/12/89)
In article <437@aurora.AthabascaU.CA>, rwa@auvax.uucp (Ross Alexander) writes: > In article <7387@chinet.chi.il.us>, Leslie Mikesell writes: > > I've only used RFS but as I understand things RFS preserves the unix > > semantics to a greater extent than NFS. > > Agreed. But, sadly, the implementations I have used (RFS over > WIN/TCP, 10B5 network, AT&T 3B2/600's) are slower than molasses in Sun sells NFS as a stateless remote file system protocol over a lightweight, unreliable transport (UDP). File locking, mounting facilities are there too. AT&T sells everyone on a statefull protocol that implements full semantics running over a reliable transport (TCP). The NFS UDP transport implementation is optimistic that data packets get to their destination most of the time anyway, so the recovery in bad transmissions is in NFS. Their assertions prove true in most small LANs, so they get great performance. NFS crash recovery is also very simple because no states can get lost. The RFS TCP transport implementation incurs more overhead on every packet to ensure reliability, plus RFS incurs a lot of overhead in the way it implements the UNIX semantics and crash recovery. Personal conclusions: NFS performs better in reliable LANs where tight UNIX filesystem semantics aren't required, but RFS may perform better in unreliable networks or where tight UNIX semantics are required. Also, administration of an NFS network is a hell of a lot simpler.
les@chinet.chi.il.us (Leslie Mikesell) (01/13/89)
In article <437@aurora.AthabascaU.CA> rwa@auvax.uucp (Ross Alexander) writes: >Agreed. But, sadly, the implementations I have used (RFS over >WIN/TCP, 10B5 network, AT&T 3B2/600's) are slower than molasses in >January. Maybe it doesn't have to be that way. It sure would be nice >if wasn't. But Sun 3's and _small_ Vaxen's (Vaxstation 2000, anyone?) >running NFS routinely blow the 600's out of the water, approximately >7 :: 1 is the ratio I measure. On the very same network... Speed is relative -- what kind of times do you expect/actually get? Running RFS between 3B2's over 1Mbit starlan, I have found that loading programs or searching large directories "feels" slow but normal data file access seems reasonable considering the usual speed of the 3B2. Some sample times with the closest thing to a 1 meg file that I had laying around follow. The network should have been moderately busy with two other machines doing a tape backup. I actually ran each copy twice and the real times were within a couple of seconds on each run. Note that writing to a remote file is much slower than reading. Does ethernet/nfs really do better? I haven't used anything but the 1Mb starlan but the difference between local->local and remote->local doesn't seem bad. This isn't quite a fair sample because the local disk is the slow 32M and the remote is a faster 72M. test file: -rw-r--r-- 1 les other 1048576 Jan 11 09:43 testfile + time cp testfile testfile1 [local->local] real 1:01.7 user 0.0 sys 14.0 + time cp testfile /afbf04/tmp/testfile [local->remote] real 1:47.6 user 0.0 sys 31.0 + rm testfile1 + time cp /afbf04/tmp/testfile testfile1 [remote->local] real 1:09.4 user 0.0 sys 32.6 Strangely enough I can copy the same file over the same network onto a 6386 running DOS client software in about 45 sec. (faster than a local copy on the 3B2). Les Mikesell
tpc@bnr-fos.UUCP (Tom Chmara) (01/17/89)
In article <340@moriaMoria.Sp.Unisys.Com> tdh@moriaMoria.Sp.Unisys.Com (Thomas Hintz) writes: >In article <437@aurora.AthabascaU.CA>, rwa@auvax.uucp (Ross Alexander) writes: >> In article <7387@chinet.chi.il.us>, Leslie Mikesell writes: >> > I've only used RFS but as I understand things RFS preserves the unix >> > semantics to a greater extent than NFS. >> >Sun sells NFS as a stateless remote file system protocol over a lightweight, >unreliable transport (UDP). File locking, mounting facilities are there too. > >AT&T sells everyone on a statefull protocol that implements full semantics ^^^^^^^^^^^^^^ >running over a reliable transport (TCP). > >The RFS TCP transport implementation incurs more overhead on every packet >to ensure reliability, plus RFS incurs a lot of overhead in the way it >implements the UNIX semantics and crash recovery. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > >Personal conclusions: NFS performs better in reliable LANs where tight UNIX >filesystem semantics aren't required, but RFS may perform better in unreliable ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ I am a bit concerned about these observations. I understand that being stateless, implementing some of the behaviour of a standard UNIX FS is a problem; however, I do not see them as impossible obstacles. As I under- stand it, many of the "file locking" semantics, etc. are performed by ancilliary facilities (the "Network Lock Manager"). What UNIX semantics (aside from mandatory file locking, an admitted failing in NFS) are missing or compromised? This is a particular concern for diskless environments -- and we're seeing more and more of those. Does a diskless environment (using NFS exclusively, as with SunOS 4.0.[0-1]) exhibit SVID-compliant behaviour (minus file locking)? If not, what am I giving up? Are application programs in trouble in this environment? Thanks for taking the time...I will summarize responses if need warrants. -- I am sole owner of the above opinions. Licensing inquiries welcome. ------------------------------------------------------------------------ Tom Chmara UUCP: ..utgpu!bnr-vpa!bnr-fos!tpc BNR Ltd. BITNET: TPC@BNR.CA
hedrick@geneva.rutgers.edu (Charles Hedrick) (01/19/89)
I'm currently running on a diskless Sun 3/50. All of my files (and swapping) are done via NFS. As far as I can tell, the machine is still Unix. I think when people say that RFS follows "Unix semantics" more than NFS, they mean one or both of: - if you open /dev/tty1 on another machine over RFS and cat to it, it comes out on the remote machine's tty1. If you do this under Sun's NFS, it comes out of tty1 on your own machine. Sun's behavior is right for a protocol intended to support diskless machines. My /dev directory, like all the rest of my disk, is on my file server. I want to treat my file server as if it were my own disk. When I open /dev/tty, it want *my* /dev/tty, not the server's. (I know little about RFS. It's certainly possible that RFS has a way to get both behaviors.) - the NFS protocol itself doesn't include file locking. However Sun has a related protocol that does do locking. Sun's systems are intended to meet the full SVID, including locking. There has been a history of bugs in file locking, and it's not clear to me that even now it is totally bug free, but this isn't a protocol issue: the protocol support is present.
les@chinet.chi.il.us (Leslie Mikesell) (01/20/89)
In article <Jan.18.22.05.48.1989.264@geneva.rutgers.edu> hedrick@geneva.rutgers.edu (Charles Hedrick) writes: >I'm currently running on a diskless Sun 3/50. All of my files (and >swapping) are done via NFS. As far as I can tell, the machine is >still Unix. I think when people say that RFS follows "Unix semantics" >more than NFS, they mean one or both of: > - if you open /dev/tty1 on another machine over RFS and cat to it, > it comes out on the remote machine's tty1. If you do this > under Sun's NFS, it comes out of tty1 on your own machine. I haven't seen a diskless RFS implemementation - normally the remote disk has a mount point somewhere under a local "/". If I mount "/" from mach2 as /mach2 on mach1, then /dev/tty1 is on the local mach1, /mach2/dev/tty1 is on the remote mach2. Non-RFS disk mounts on the remote below the RFS mount are followed (although df only reports free space in the partition of the remote end of the mount point) so in the above example you could access /mach2/usr/xxx even if /usr is another disk on mach2. Is this true for NFS? > - the NFS protocol itself doesn't include file locking. However > Sun has a related protocol that does do locking. How about atomic file creation, append mode, or FIFO's? Les Mikesell
shaver@atanasoff.cs.iastate.edu (Dave Shaver) (01/20/89)
Thomas Hintz writes: >[Comparing RFS & NFS] >Also, administration of >an NFS network is a hell of a lot simpler [than an RFS network]. Oh, my. Having only worked with NFS, I'd avoid RFS based on this statement. I've found NFS ``OK,'' but flaky. Kind of like a MSCS project that got shipped before it was debugged. We're always having strange things happen. (Maybe follow-up should be sent to comp.protocols.nfs?) I don't find the administration of our NFS network that trivial. /\ Dave Shaver -=*=- CS Systems Support Group, Iowa State University \\ UUCP: {hplabs!hp-lsd,uunet!umix!sharkey}!atanasoff!shaver \/ Internet: shaver@atanasoff.cs.iastate.edu
mcm@rti.UUCP (Mike Mitchell) (01/21/89)
Here is a table that I have made for my own comparisons:
CRITERIA RFS NFS
--------------------------------------------------------------------------------
Operating multivendor hardware; multivendor hardware;
Environments System V only; BSD, System V, hybrids,
VMS server, MS-DOS client;
multiple networking protocols multiple networking protocols
--------------------------------------------------------------------------------
Implementation in-kernel; in-kernel;
stateful server; stateless server;
at inode interface at inode interface
--------------------------------------------------------------------------------
Transparency remote file access; remote file access;
remote device access (but see below)
--------------------------------------------------------------------------------
UNIX semantics preserves all UNIX does not preserve all
file system semantics UNIX file system semantics
--------------------------------------------------------------------------------
Reliability/ no single failure point; no single failure point;
Data Integrity append mode works; append mode may not work;
atomic writes work; atomic writes may not work;
open files will not be open files can be erroneously
deleted while in use; deleted while in use;
file systems cannot be file systems can be
inadvertently unmounted inadvertently unmounted
while in use while in use
--------------------------------------------------------------------------------
Concurrency supports lockf file locking not directly
file locking mechanism supported (requires separate
stateful 'lock server')
--------------------------------------------------------------------------------
Security provides tools to limit network security degenerates
access by user id to that of least secure system
--------------------------------------------------------------------------------
Much of this information is from an article in UNIX/WORLD, entitled
"AT&T's RFS and SUN's NFS: A Comparison of Heterogeneous Distributed File
Systems." The article was written by Mark J. Hatch, Michael Katz, and
Jim Rees.
I find it very strange that NFS insists that stateless servers are
the best, but then has to add statefull servers to handle locking and
other things. NFS claims to work in a heterogeneous UNIX environment,
but to do that each hardware vendor has had to re-write their operating
system. They get the template to do the re-write from SUN, so it seems
to me that NFS gets around heterogeneity problems by making everyone
run the same OS. Now everything is homogeneous as far as NFS is concerned.
RFS has some problems too. It also aproaches heterogeneity problems by
making everyone run the same code. It is not available from as many vendors
as NFS is, and it has not been out on the market as long as NFS. NFS has
had time to work some of the bugs out, while RFS is still a little new.
We don't have any machines at our shop that can run RFS, so I don't know
how easy it is to set up. We have a few machines with NFS compiled in,
but we don't use NFS either. I think that RFS is technically superior to
NFS, but NFS has a marketing advantage. NFS is available on many more
machines than RFS, so choosing which to use is easier.
There are a few companies that have chosen to go their own route with
distributed computing, and not use either NFS or RFS. Some of their
offerings are quite good, but I'm not sure how they will hold up to
the marketing pressure NFS and RFS will bring to bear.
--
Mike Mitchell {decvax,seismo,ihnp4,philabs}!mcnc!rti!mcm mcm@rti.rti.org
"There's laughter where I used to see a tear. w (919) 541-6098
It's all done with mirrors, have no fear." h (919) 361-2048guy@auspex.UUCP (Guy Harris) (01/21/89)
>I haven't seen a diskless RFS implemementation - normally the remote >disk has a mount point somewhere under a local "/". If I mount "/" >from mach2 as /mach2 on mach1, then /dev/tty1 is on the local mach1, >/mach2/dev/tty1 is on the remote mach2. Yes, but the disklessness is the whole point here; on both my machine and Charles Hedrick's machine, there *is* no equivalent of the "local mach1", so a remote file system implementation that didn't allow special files to refer to the local device, rather than the remote device, wouldn't work. (You could, at least in some UNIX implementations including the one being developed for S5R4, have an RFS mount option to specify whether special files could refer to the local or remote disk; I don't think S5R4 will have such an option, though.) Of course, the other problem with referring to remote devices is making things such as "ioctl" work if the memory layout of the argument to the "ioctl" differs between the two machines, or making "mmap" (yes, "mmap" will be in S5R4) work on a remote device.... (The former may be fixable with a fair bit of work on the drivers; the latter is probably unfixable - at least not without so much pain and overhead as to render it most likely useless.) >Non-RFS disk mounts on the remote below the RFS mount are followed >(although df only reports free space in the partition of the remote >end of the mount point) so in the above example you could access >/mach2/usr/xxx even if /usr is another disk on mach2. Is this true for >NFS? Not in the version distributed by Sun with their machines, or distributed for other people to port to their machines, although I think Doug Kingston made a change to the implementation to do just that when he was at BRL. >> - the NFS protocol itself doesn't include file locking. However >> Sun has a related protocol that does do locking. > >How about atomic file creation, append mode, or FIFO's? FIFOs work just fine over NFS, courtesy of some hackery to the protocol (code to implement said hackery is provided with recent NFS source releases, so other vendors can pick it up), although they work the same way devices do - the FIFO is strictly local, so while two processes on the same machine can communicate using it, processes on different machines cannot. The OS does its best to make append mode work; however, it can't guarantee that it works. It tries its best with atomic file creation as well, but I think that's not guaranteed either; the proposed next version of the protocol (Sun RPC includes version numbers in RPC calls, so you can have both clients and servers using multiple versions of the protocol at the same time) may have something to help it work.
karn@jupiter..bellcore.com (Phil R. Karn) (01/21/89)
>I find it very strange that NFS insists that stateless servers are >the best, but then has to add statefull servers to handle locking and >other things. I became a firm believer in NFS's stateless server approach the very first time I saw a server crash and reboot. The client applications using it just froze. When the server came back to life, they picked up and continued as if nothing had happened. From where I sit, the robustness of NFS's stateless approach is such an overwhelming practical advantage that I'm quite willing to put up with minor changes in the UNIX file system semantics to get it. And Sun is taking care of even these esoteric problems with their lock servers. Phil
prc@maxim.ERBE.SE (Robert Claeson) (01/21/89)
In article <Jan.18.22.05.48.1989.264@geneva.rutgers.edu>, hedrick@geneva.rutgers.edu (Charles Hedrick) writes: > - if you open /dev/tty1 on another machine over RFS and cat to it, > it comes out on the remote machine's tty1. If you do this > under Sun's NFS, it comes out of tty1 on your own machine. Sun's > behavior is right for a protocol intended to support > diskless machines. My /dev directory, like all the rest > of my disk, is on my file server. I want to treat my > file server as if it were my own disk. When I open > /dev/tty, it want *my* /dev/tty, not the server's. Right, but what if you mount another workstation's or server's /dev as, say, /net/server/dev (where 'server' is the name of the other machine) and cat to it under the intention that something should come out on a printer connected to the other machine? Where does the output go? > - the NFS protocol itself doesn't include file locking. However > Sun has a related protocol that does do locking. Sun's > systems are intended to meet the full SVID, including > locking. There has been a history of bugs in file locking, > and it's not clear to me that even now it is totally bug > free, but this isn't a protocol issue: the protocol support > is present. The NFS locking protocol doesn't support the SVID's mandatory locking, but only the advisory locking. I guess it will be hard to implement mandatory locking in a stateless environment like NFS. () -- Robert Claeson, ERBE DATA AB, P.O. Box 77, S-175 22 Jarfalla, Sweden "No problems." -- Alf Tel: +46 758-202 50 EUnet: rclaeson@ERBE.SE uucp: uunet!erbe.se!rclaeson Fax: +46 758-197 20 Internet: rclaeson@ERBE.SE BITNET: rclaeson@ERBE.SE
geoff@eagle_snax.UUCP ( R.H. coast near the top) (01/23/89)
In article <2735@rti.UUCP> mcm@rti.UUCP (Mike Mitchell) writes: > >I find it very strange that NFS insists that stateless servers are >the best, but then has to add statefull servers to handle locking and >other things. People always assume that there is a clean, simple dichotomy between stateful and stateless designs. The key issue to my mind is not "stateful vs. stateless" but "fault-tolerant vs. fault intolerant". RFS and many other systems are designed as fault-intolerant: if the server is lost (crashes or becomes inaccessible) client applications will fail. NFS was designed to be fault tolerant, and a variety of mechanisms were used to achieve this. For file sharing, statelessness seemed to deliver the best results. For file locking, a stateful design was obviously necessary, so a state recovery procedure was developed to handle server or network failure. > NFS claims to work in a heterogeneous UNIX environment, >but to do that each hardware vendor has had to re-write their operating >system. They get the template to do the re-write from SUN, so it seems >to me that NFS gets around heterogeneity problems by making everyone >run the same OS. Now everything is homogeneous as far as NFS is concerned. Unix is Unix is Unix... it's already fairly homogeneous, so what's your problem with sharing a little more code? There's no real difference between NFS and RFS in this respect. But you are ignoring the various non-Unix implementations - MS-DOS (multiple), AmigaDOS, VMS, MVS, VM, etc. - most of which were implemented straight from the spec. I won't waste network bandwidth with the full list of NFS licensees. It's more than marketing hype, my friend. -- Geoff Arnold, Sun Microsystems Inc. | "It is well known that the longer one PC Dist. Sys. Group (home of PC-NFS) |postpones a pleasure, the greater the UUCP: {hplabs,decwrl...}!sun!garnold |pleasure when it arrives. Therefore, if ARPA: garnold@sun.com |one postpones it forever..." (Smullyan)
mjm@attibr.UUCP (Mike Matthews) (01/25/89)
> We don't have any machines at our shop that can run RFS, so I don't know > how easy it is to set up. We have a few machines with NFS compiled in, > but we don't use NFS either. I think that RFS is technically superior to > NFS, but NFS has a marketing advantage. NFS is available on many more > machines than RFS, so choosing which to use is easier. We use RFS at ATT International and have found it quite easy to setup. We use it for the following applications at this time: 1 - Facilitate HDB uucp transfer over Ethernet (STARLAN). 2 - Do high speed backups for a net of 3B2/600's using one device remotely mounted. Our network is growing everyday and hopefully new applications will be added ( There is a strange tech culture at ATT to buy and implement new technology before finding any use for it ). A database application would be a interesting test. It's true that RFS only works for V.3 but UNIX consolodation under V.4 might pave the way for wider acceptance/availability. > -- > Mike Mitchell {decvax,seismo,ihnp4,philabs}!mcnc!rti!mcm mcm@rti.rti.org > Mike Matthews Tech Support ATT International Basking Ridge, NJ
gertjan@atcmp.nl (Gertjan Vinkesteyn) (02/07/89)
In article <463@maxim.ERBE.SE>, prc@maxim.ERBE.SE (Robert Claeson) writes: > In article <Jan.18.22.05.48.1989.264@geneva.rutgers.edu>, hedrick@geneva.rutgers.edu (Charles Hedrick) writes: > The NFS locking protocol doesn't support the SVID's mandatory locking, > but only the advisory locking. I guess it will be hard to implement > mandatory locking in a stateless environment like NFS. Mandatory locking is part of SystemV Release 3, not in Berkeley releases -- UUCP and other network )\/( ..!hp4nl!kunivv1!atcmpe!gertjan connections via mcvax )/\( gertjan@atcmp.nl (internet) This note does not necessarily represent the position of AT Computing BV
guy@auspex.UUCP (Guy Harris) (02/08/89)
>> The NFS locking protocol doesn't support the SVID's mandatory locking, >> but only the advisory locking. I guess it will be hard to implement >> mandatory locking in a stateless environment like NFS. > >Mandatory locking is part of SystemV Release 3, not in Berkeley releases So? That's totally irrelevant. *Neither* mandatory *nor* advisory *record* locking is part of Berkeley releases; all they currently have now is "flock". Most systems implementing NFS aren't "Berkeley releases"; they may have "started with 4.xBSD" but they've added a number of other functions - mandatory locking would be just another function. In fact, some systems implementing NFS *are* System V Release 3 systems.... NFS isn't a Berkeleyism. The only hard-core Berkeleyism in the current protocol is that many of the error numbers happen to be 4.xBSD "errno" values (a lot of them also happen to be S5 "errno" values, since they're really V7 "errno" values); the current NFS version 3 protocol spec changes that - they're now not "errno" values for any UNIX version.