dsamperi@Citicorp.COM (Dominick Samperi) (02/26/91)
We have observed a significant performance hit when links are done
over the network (via NFS). At first I thought this was due to the
fact that we are linking against several large libraries. So I
copied the libraries to a local disk, and was surprised to find
that it made very little difference. Further investigation
revealed that what made all of the difference was whether or
not the user's home directory was local or NFS-mounted. More
precisely, the performance hit resulted from the need to write
the output executable file (about 10Megs) to an NFS-mounted
directory. Modifying the makefile so that the output executable
file is written to a local ("cache") partition removed the
performance hit.
Surely this must be a problem that others have encountered. Does
anybody have a better solution? We are using SunOS 4.1.
It appears that NFS works well when one is making many small
requests, say, to fetch object files from a library over the
network, or to include header files from NFS-mounted directories.
But there is a significant hit when NFS is used to copy a large
file (or create a large executable file). One observation that
I cannot explain is the fact that large executables that are
located in remote (NFS-mounted) directories seem to start up
quickly (faster than a straight copy). Any ideas?
Thanks for any feedback on this.
--
Dominick Samperi -- Citicorp
dsamperi@Citicorp.COM
uunet!ccorp!dsamperimjr@hussar.dco.dec.com (Marcus J. Ranum) (02/26/91)
dsamperi@Citicorp.COM (Dominick Samperi) writes: >[...] Further investigation >revealed that what made all of the difference was whether or >not the user's home directory was local or NFS-mounted. More >precisely, the performance hit resulted from the need to write >the output executable file (about 10Megs) to an NFS-mounted >directory. This is because NFS writes are done synchronously - IE: the calling process waits until the write has been performed on the server before it returns. You can somewhat offset the damage by running the block I/O demon on the client (biod(8)) but as long as you have to do NFS writes you'll see some performance loss. Another approach some take is to add battery-backed RAM caches to machines, to speed up synchronous writes on the servers, thereby speeding up the clients as well. Legato's PrestoCache (available for a variety of machines, available as a built-in on some DEC servers) can speed NFS performance up to 300% in some cases. mjr.
jik@athena.mit.edu (Jonathan I. Kamens) (02/26/91)
(Note: The article to which I am replying was posted separately to the three
newsgroups in my Newsgroups: line; the References: line of this message
indicates the Message-IDs under which it was posted in those newsgroups.)
It is likely that the reason linking goes slowly when creating an executable
in an NFS filesystem is that the linker has to seek back and forth to various
points in the file while linking. Because of that, it isn't just a matter of
reading in the sequential blocks of a file or writing out the sequential
blocks of a file -- the same blocks have to be read in over and over again
each time the linker seeks to them.
A possible work-around to avoid this problem is to create a symbolic link
in the directory in which you are compiling to force the linking to take place
in a local directory like /tmp or /usr/tmp (or just to specify such a
directory when specifying the output file name to the linker), and then mv the
file onto the NFS partition when it's done linking. You'll probably get a
significant speed improvement that way.
In fact, I just linked emacs (my emacs sources are on NFS) into a local
file, and then did the same link in the emacs source directory. The output of
/bin/time from the local link:
102.9 real 11.1 user 13.6 sys
The output of /bin/time from the NFS link:
260.4 real 10.7 user 14.6 sys
--
Jonathan Kamens USnail:
MIT Project Athena 11 Ashford Terrace
jik@Athena.MIT.EDU Allston, MA 02134
Office: 617-253-8085 Home: 617-782-0710mouse@thunder.mcrcim.mcgill.edu (der Mouse) (02/27/91)
In article <1991Feb26.033503.12885@Citicorp.COM>, dsamperi@Citicorp.COM (Dominick Samperi) writes: [ stuff about linking when the a.out is NFS-remote ] > But there is a significant hit when NFS is used to copy a large file > (or create a large executable file). One observation that I cannot > explain is the fact that large executables that are located in remote > (NFS-mounted) directories seem to start up quickly (faster than a > straight copy). One plausible reason for this is that the executable is probably a ZMAGIC executable, so it's not read in all at once. Starting it up is a matter of the following (I'm glossing over things here; let's not get excessively nit-picky): - allocating swap space for the data segment (data+bss in the executable file) - loading the initialized data - creating demand-zero pages for the bss - branching to the entry point Notice that the text segment is not brought over - yet. Instead, it is brought over on demand using the normal paging mechanisms: trying to execute an instruction that happens to not be in core at the moment causes a page fault, and that page of the executable is brought over. Try linking your executable as an OMAGIC executable instead, so the text segment isn't paged out of the a.out file, and see if the performance difference goes away.... der Mouse old: mcgill-vision!mouse new: mouse@larry.mcrcim.mcgill.edu
zink@panix.uucp (David Zink) (03/01/91)
dsamperi@Citicorp.COM (Dominick Samperi) writes: > We have observed a significant performance hit when links are done > over the network (via NFS). At first I thought this was due to the Actually I have observed some related interesting lil' things. First and foremost, though was when I was working on a compression program. I would execute commands like (where /bin is local and ~/work is mounted from a twin machine with NFS): $ cd work $ cat /bin/* | ncmpress > ./filename & $ ls -l The ls would hang until the ncmpress program completed (about five minutes). In scheduling theory we call this 'starvation'. I don't know if this is true for all versions, ours was HP-UX between two HP-PA boxes. -David Zink