remaker@icarus.amd.com (Phillip Remaker) (02/05/91)
Once again I thank USENET for prompt response to my queries on NFS traffic
over links slower than 10 Mbit/sec. I got several replies, but I am only
including some in this post since a number of users asked that I not
post their mail directly to usenet. So I will take a stab at summarizing.
0) If you don't have to do it, don't. You can expect significant performance
degradation. Over a T1, the performace would be reduced by a factor of
approximately 3, not counting the effects on other non-NFS traffic
sharing the line. Also, I am told that the performance degrades sharply
as more NFS traffic is added to the link.
But, if you insist on doing it:
1) Using a T3 between routers is an option, if you have the $$$ to rent such a
beast and if it is tarrifed in your area. This is an extreme solution,
but may be a good one if a local microwave hop is in the budget.
2) If you *must* run NFS over a slow link, there are some neat tricks you
can use to make the whole process less painful. What follows is a summary
of hints. There is no warranty, no guarantee, and your milage may vary.
(So please don't harass the nice people who proposed these hints 8-)).
o Set rsize and wsize to lower values. 2k is a good size, as opposed
to the default 8k. This prevents the NFS from stomping on the inter-
active traffic while file transfer goes on at a "usable" rate. A
fellow from Sun Technical Marketing suggested 1024 for a size so that
one NFS operation is a single UDP packet. This keeps monster packet
trains from hosing your long lines.
SPECIAL NOTE: PC-NFS users will need to get a patch that allows
them to modify the rsize and wsize parameters.
o Don't use NFS over slow links to launch executables from the remote
disks. Several users said this was painful. (Although they do it
anyway 8-)).
o Set UDP checksums on if IP headers are stripped off anywhere. This
costs in performace but ensures data integrity.
o NFS over sattelite links is explicitly not recommended.
The delay is too substantial and can dramatically reduce
performance to under 100kbits/sec.
3) One respondent points out that NFS over TCP is available in 4.3bsd Reno
and in upcoming 4.4bsd. He tells me that the throughput for NFS
over TCP will outstrip the NFS over UDP in mounts over slow, unreliable
lines (He cited a speaker at USENIX for that fact).
4) More programming wizards are working on some slick mods to NFS to make
it more "WAN-friendly." I cheer their efforts, and await the wide-
spread implementation.
My conclusion:
As a policy, we have decided not to allow NFS over the slow lines, simply
because the performance hit is too significant, and the amount of time
required to configure and test the machines with these modifications is
not available. Also, other interactive traffic on our T1 is too valuable
to interrupt just because some yo-yo misconfigures rsize and wsize.
If we do NFS over T1, I suspect we will simply change rsize and wsize to
1024 in the mount tables for all remotely mounted drives over the T1.
We will also get the PC-NFS patches where appropriate.
Thanks for all of the input everyone. Direct any further questions to me
and I will try to help. Flames to /dev/null, all standard disclaimers apply.
=================================================================
Included messages
=================================================================
From ames!cisco.com!daly Tue Jan 29 22:04:31 1991
To: tli@cisco.com
Cc: remaker@brahms.amd.com, cs@cisco.com
Subject: Re: NFS through cisco routers
(Re: T3 lines)
I'm not sure of the official tariff status for PacBell - I know ATT
has it for interexchange. In any case you'll find the price
breathtaking and so will merit a visit from the PacBell salesperson.
These deals are often done on a special case basis, rather than by
tariff. Also, you'll want 'non-subrated' t3 service, as the
subrated variety is really just an array of 1-28 T1s.
How far do you want to go? radio might be an alternative.
-tom daly
cisco
From: ckollars@East.Sun.COM (Chuck Kollars - Sun Technical Marketing)
To: remaker@icarus.amd.com
Subject: Re: NFS traffic over links < 10Mbit /sec
Newsgroups: comp.protocols.nfs
Organization: Sun Microsystems, Billerica MA
In article <1991Jan30.023832.24723@amd.com> you write:
>Greetings! Bureaucrats and other network illiterate types are demanding to
>run NFS traffic over a T1 between campuses (about a 6 mile run).
We have a T1 link between Massachusetts and California. It's subdivided
into three sections: one for a video link, one for the PBX telephones, and
the third for TCP/IP traffic. We routinely run NFS over this link, and
it works fairly well.
>This link is already congested with LAT, LAST, FTP, and DECENT traffic.
>Well, not really congested, about 10-20% utilized.
>
>Since NFS is UDP based, I *know* that it is not a "good idea" to run
>it over low bandwidth lines, but I need references, testimonials, etc. so
>that I can present my case without having to "burn to learn."
In this context I'd take "low-bandwidth" to mean 4800 baud. T1 is fairly fast.
>Also, if there are ways to sanely do NFS over slow links, I am
>interested. Kernel mods, secrets, tweaks, black art, all welcomed.
Our experience is that you _must_ change the rsize= and wsize= parameters on
the mount to be only 1024 rather than the default 8192. This makes a single
NFS operation == a single UDP packet. It also prevents NFS from blasting out
huge "packet trains" that effectively shut down Telnet trafic.
It makes a night and day difference. With the default mount size, it barely
works at all, and any other traffic on the link is severely degraded. With
the reduced mount size, it works quite well (except performance is a little
slow), and other traffic on the link is virtually unaffected.
>Another alternatibe is to install a >= 10 Mbit link in the form of
>T3 or dedicated fiber/microwave. The distance is about 6-7 miles.
>We have bandwidth now, but will latency then be an issue?
>
>We are using cisco routers on both ends of the link, so there is more
>delay.
>
>I am under the impression that NFS is for machines only in geographically
>close proximity. Correct me if I am wrong.
It was, it was. Lots of folks, for lots of good reasons, refuse to continue
to accept this limitation. We're working on it ...hard! And in fact we've
already made some progress. We can mount stuff in the UK. We've experimented
with mounting the X archive server at MIT from a machine at JPL. etc.
>All tips, pointers, hints, flames, tweaks, twiddles, and frobs welcome.
>
>I will post a summary if there is sufficient interest.
>
>
>--
>Phillip A. Remaker A.M.D. M/S 167 P.O. Box 3453 Sunnyvale, CA 94088-3000
>TCP/IP internetworking from hell. DoD #185 remaker@amd.com 408-749-2552
> Things to do today: 1) Get a clue. 2) Get a job. 3) Get a life.
--
chuck kollars <ckollars@East.Sun.COM>
Sun Technical Marketing, located in Sun's Boston Development Center
To: remaker@icarus.amd.com (Phillip Remaker)
Subject: Re: NFS traffic over links < 10Mbit /sec
From: Gregory Bond <gnb@bby.oz.au>
>>>>> On 30 Jan 91 02:38:32 GMT, remaker@icarus.amd.com (Phillip Remaker) said:
Phillip> Also, if there are ways to sanely do NFS over slow links, I am
Phillip> interested. Kernel mods, secrets, tweaks, black art, all welcomed.
We run NFS over a 48k link to another city (1000km). It is not used
for serious access (e.g. loading binaries is a REAL PITA), but for
acessing infrequently-needed data files it is magic. The link also
takes terminal traffic and print jobs, so is also say 20% used.
A NEAT trick is to adjust the cross-link mounts to use 2k request
sizes (rsize=, wsize= mount options) so the latency for each packet is
not too large and the retrys are much reduced. This also helps if the
routers/bridges don't have huge packet buffers (as ours don't).
Phillip> I am under the impression that NFS is for machines only in
Phillip> geographically close proximity. Correct me if I am wrong.
Well, it works best for low latency nets. An FDDI link across town
would be no worse than an FDDI link across the room.
Greg.
From: peterh@usit.uio.no (Peter Hausken)
To: remaker@icarus.amd.com (Phillip Remaker)
Cc: peterh@gollum.uio.no (Peter Hausken)
Subject: Re: NFS traffic over links < 10Mbit /sec
In Norway we have used NFS between two Universities without any serious
problems. The connection between the University of Oslo and the University
of Tromso goes via Trondheim. The configuration is as follows:
1Mbps 256Kbps
UiO LAN --- cisco ----------- cisco ---------------cisco --- UiTo LAN
(UiO-GW) (NO-GW) (UiTo-GW)
There are a number of PC's at the University of Tromso using a public
domain server at UiO over the WAN. They all have PC-NFS 3.0.1. The
performace for a PC is actually not much worse than for NFS servers
on the LAN. And as far as I have experienced there has been no errors
in the the transmission. There is no guarantee for error-free transmission,
but for starting programs from Oslo has been no problem.
We actually use NFS over 64Kbps lines between various sites of the
University of Oslo. With equal success. The one thing that has to be
done is to reduce the write size from the 8K default down to 1K or
lower when mounting. Sun provided a patch to PC-NSF for that. In the
UNIX mount command it can be set as an option. Within the University
there is mainly Mac-level bridges (VitaLink). 64Kbps is slow and it
takes time to read a file or start a program, but I have not encountered
any problems, even on lines with high load.
I'm not encouraging people in Tromso to load the lines with NFS trafic,
but for picking up programs and the like it is convenient.
Your T1 link (1.5Mbps) should have more than enough power to take the
trafic....
Please tell me if you get numbers of letters saying that it does not
work, or you have other experiences. We are planning to upgrade the
national network to considerably larger bandwith and I hope we will
be able to use NFS on a regular basis between our four Universities.
----------------------------------------------------------------------------
/////// // // Peter Hausken, University of Oslo
// // // // PB. 1059, Blindern, N-0316 OSLO 3, Norway
/////// ////// Voice: +47-2-453524 Fax: +47-2-455770
// // // Internet: Peter.Hausken@USE.UiO.NO (peterh@ifi.uio.no)
// // // X.400 SA: G=Peter;S=Hausken;OU=USE;O=UiO;P=UNINETT;C=NO
From: Toerless Eckert <Toerless.Eckert@Informatik.Uni-Erlangen.de>
Message-Id: <9101311912.AA10426@faui43.informatik.uni-erlangen.de>
To: remaker@icarus.amd.com
Subject: Re: NFS traffic over links < 10Mbit /sec
Status: R
4.3reno and upcoming 4.4bsd contain a freely available implementation
of NFS with support for NFS over TCP.
In <1991Jan30.023832.24723@amd.com> remaker@icarus.amd.com (Phillip Remaker) writes:
>Since NFS is UDP based, I *know* that it is not a "good idea" to run
>it over low bandwidth lines, but I need references, testimonials, etc. so
>that I can present my case without having to "burn to learn."
The numbers shown at USENIX by a speaker from the University of Guelph
in Ontario, Canada showed that unmodified NFS over UDP would manage
about a quarter the throughput of TCP over a 56Kbit line. Your losses
should be less severe, but you'll still have pretty awful performance.
Bandwidth is also less important than latency a lot of the time; I
some of our servers in-house get beaten to death because of duplicate
requests.
>Also, if there are ways to sanely do NFS over slow links, I am
>interested. Kernel mods, secrets, tweaks, black art, all welcomed.
That's easy to hand-wave about. There are mods to make at the client
and at the server end that really make a difference. First, the client
should keep a record of how long each operation from each server takes,
and adjust its retransmission timeouts based on that delay. Doing this
right keeps the line clear of retransmission traffic and lets the
server do more real work. On the server side, the NFS dispatch routine
should keep track of duplicate requests and either send back cached
responses or drop the duplicates for a finite amount of time. My
feeling is that caching the replies is unnecessary since a lot of the
retransmissions are due to client impatience and UDP socket buffer
overflows on busy servers, not due to lost replies. With just a
smart client implementation, the USENIX speaker (damnit, I wish I
had my proceedings here!) got UDP performing better than TCP due to
lighter computation overhead on his Microvax II.
>I will post a summary if there is sufficient interest.
Please do. I think a lot of people are playing with these issues; I
know I hope to test this exact code at Connectathon this year.
Rob T
--
Rob Thurlow, thurlow@convex.com or thurlow%convex.com@uxc.cso.uiuc.edu
"The Canadian rock singer, Ronnie Hawkins, has it all figured out. 'Believe
in God?' he says, "Man, I believe in God like nobody else. It's the fucking
ground crew I don't trust." - "Running Risks", Angela Issajenko
From ames!ica.philips.nl!geertj Mon Feb 4 14:58:20 1991
Subject: Re: NFS traffic over links < 10Mbit /sec
Organization: Philips TDS, Innovation Centre Aachen
In article <1991Jan30.023832.24723@amd.com> you write:
>Greetings! Bureaucrats and other network illiterate types are demanding to
>run NFS traffic over a T1 between campuses (about a 6 mile run).
>
>This link is already congested with LAT, LAST, FTP, and DECENT traffic.
>Well, not really congested, about 10-20% utilized.
>
>Since NFS is UDP based, I *know* that it is not a "good idea" to run
>it over low bandwidth lines, but I need references, testimonials, etc. so
>that I can present my case without having to "burn to learn."
I have no personal experience, but know about the NFS implementation
of Cayman systems, who implemented Van Jacobsen's 'slow start' algorithm
and reports possible operation across 9600 bauds.
If this is true, it isn't the NFS protocol that counts, it's the implementation.
If you happen to make a summary, I don't mind receiving a copy.
Cheers,
Geert Jan
--8<--nip-nip---------------------------------------------------------------
"We trained hard - but it seemed that every time we were beginning to form up
into teams, we would be reorganized. It was to learn later in life that we tend
to meet any new situation by reorganizing, and a wonderful method it can be
for creating the illusion of progress while producing confusion, inefficiency,
and demoralisation." - Petronius, 100 BC
Geert Jan de Groot, Philips ICA, Weisshausstrasse 1, 5100 Aachen, Germany
Email: geertj@ica.philips.nl or ..!hp4nl!philica!geertj
Phone: +49 241 6003 714 FAX: +49 241 6003 709
==============================================================================
--
Phillip A. Remaker A.M.D. M/S 167 P.O. Box 3453 Sunnyvale, CA 94088-3000
TCP/IP internetworking from hell. DoD #185 remaker@amd.com 408-749-2552
Things to do today: 1) Get a clue. 2) Get a job. 3) Get a life.brent@terra.Eng.Sun.COM (Brent Callaghan) (02/06/91)
In article <1991Feb5.031603.2969@amd.com>, remaker@icarus.amd.com (Phillip Remaker) writes: > > o Set UDP checksums on if IP headers are stripped off anywhere. This > costs in performace but ensures data integrity. Actually, set UDP checksumming if the packets go off-ethernet. Without UDP checksums you only have the ethernet CRC to protect your data. IP headers have a checksum, but it only protects the header itself. -- Made in New Zealand --> Brent Callaghan @ Sun Microsystems Email: brent@Eng.Sun.COM phone: (415) 336 1051
guy@auspex.auspex.com (Guy Harris) (02/07/91)
> A fellow from Sun Technical Marketing suggested 1024 for a size > so that one NFS operation is a single UDP packet. Presumably he meant "fits in a single link-level packet" or "fits in a single IP datagram"; even with 8K reads and writes, one NFS operation fits into a single UDP packet - the UDP packet just happens to be turned into a fragmented IP datagram on most if not all networks.
meissner@osf.org (Michael Meissner) (02/08/91)
In article <7371@exodus.Eng.Sun.COM> brent@terra.Eng.Sun.COM (Brent Callaghan) writes: | In article <1991Feb5.031603.2969@amd.com>, remaker@icarus.amd.com (Phillip Remaker) writes: | > | > o Set UDP checksums on if IP headers are stripped off anywhere. This | > costs in performace but ensures data integrity. | | Actually, set UDP checksumming if the packets go off-ethernet. | | Without UDP checksums you only have the ethernet CRC to | protect your data. IP headers have a checksum, but it | only protects the header itself. Always set UDP checksums, even over ethernet. Back in a previous life, I was on a local ethernet where I could not reliably build GCC without one .o being corrupted. Whomever in Sun made the choice not to checksum NFS packets should have his/her/its head examined.... -- Michael Meissner email: meissner@osf.org phone: 617-621-8861 Open Software Foundation, 11 Cambridge Center, Cambridge, MA, 02142 Considering the flames and intolerance, shouldn't USENET be spelled ABUSENET?
brent@terra.Eng.Sun.COM (Brent Callaghan) (02/08/91)
In article <MEISSNER.91Feb7191607@curley.osf.org>, meissner@osf.org (Michael Meissner) writes: > Always set UDP checksums, even over ethernet. Back in a previous > life, I was on a local ethernet where I could not reliably build GCC > without one .o being corrupted. Whomever in Sun made the choice not > to checksum NFS packets should have his/her/its head examined.... I too have heard of instances where NFS data has been corrupted by bad ethernet hardware or flakey routers that drop a bit now and then. When NFS first shipped on Suns, we had just one or two MIPS of CPU to play with. UDP checksumming was a significant overhead. Bit errors on most ether networks was (and is) very low. Now, with most workstations provide 10 - 30 MIPS, the performance impact of checksumming is not an issue - particularly when implemented as part of a memory copy operation. BTW: you can turn on checksums just by setting udp_cksum to 1. -- Made in New Zealand --> Brent Callaghan @ Sun Microsystems Email: brent@Eng.Sun.COM phone: (415) 336 1051
jim@cs.strath.ac.uk (Jim Reid) (02/08/91)
In article <7645@exodus.Eng.Sun.COM> brent@terra.Eng.Sun.COM (Brent Callaghan) writes: In article <MEISSNER.91Feb7191607@curley.osf.org>, meissner@osf.org (Michael Meissner) writes: > Always set UDP checksums, even over ethernet. Back in a previous > life, I was on a local ethernet where I could not reliably build GCC > without one .o being corrupted. Whomever in Sun made the choice not > to checksum NFS packets should have his/her/its head examined.... When NFS first shipped on Suns, we had just one or two MIPS of CPU to play with. UDP checksumming was a significant overhead. Bit errors on most ether networks was (and is) very low. Now, with most workstations provide 10 - 30 MIPS, the performance impact of checksumming is not an issue - particularly when implemented as part of a memory copy operation. As I recall, the reason why UDP checksumming was not switched on for NFS was nothing to do with performance (though it was bound to be a small contributory factor). In the early days of Sun and NFS, most vendors were using the 4.2 BSD TCP/IP code. This didn't compute UDP checksums properly, so it would sometimes accept corrupted packets and reject valid ones. Disabling UDP checksumming meant that protocols on top of UDP like NFS had a better chance to communicate with each other, especially when they ran on machines with different byte ordering. This BSD bug has long since been fixed and hopefully most vendors have picked it up by now. Jim
tmt@osf.org (Tom Talpey) (02/11/91)
In article <JIM.91Feb8134359@baird.cs.strath.ac.uk>, jim@cs.strath.ac.uk (Jim Reid) writes: |> In article <7645@exodus.Eng.Sun.COM> brent@terra.Eng.Sun.COM (Brent Callaghan) writes: |> In article <MEISSNER.91Feb7191607@curley.osf.org>, meissner@osf.org (Michael Meissner) writes: |> > Always set UDP checksums, even over ethernet. |> When NFS first shipped on Suns, we had just one or two MIPS |> of CPU to play with. UDP checksumming was a significant overhead. |> In the early days of Sun and NFS, most vendors were using the 4.2 |> BSD TCP/IP code. This didn't compute UDP checksums properly The 4.2 code botched UDP checksums on big-endian machines. Suns, being of that ilk, had obvious reasons to preserve interoperability. However, the "performance" benefit was another reason. UDP checksums should generally be enabled these days, but remember that doing so affects both outgoing and incoming checksum computation on all UDP packets, not just NFS. The best solution is to turn off checksums on the broken systems and enable everything else, but sometimes it's not feasible. Tom Talpey tmt@osf.org