09998WAS%MSU@PUCC.PRINCETON.EDU ("Bill.Simpson") (08/23/90)
A month ago, I wrote a very short message to this group concerning NIS.NSF.NET, saying "It certainly has a *terrible* implementation of TCP/IP." I quickly received the following private message: > From: "Bill Rubin" <RUBIN%YKTVMZ.BITNET@CUNYVM.CUNY.EDU> > To: 09998was@ibm.cl.msu.edu > Subject: Re: How to get RFC documents > > I am probably asking for trouble here, but would you like to tell me > exactly what you find so distasteful about our TCP/IP product? We have > a very large number of customers who I believe are very satisfied with > it. If you don't like our anonymous FTP implementation, I can believe > that, it's nothing to be proud of, but it's better than nothing, which > is what we offered before we added the feature. Perhaps you should > have just said we have a *terrible* implementation of anonymous FTP. Well, I spent about 15 hours benchmarking NIS.NSF.NET against other machines on the internet, carefully documenting the packet traces, and passing my observations on to IBM. I did this at my own expense, as a matter of curiousity, because I thought that the folks who were sending the messages were "real programmers" who were really interested in getting it right (a Jay Elinsky was also involved). Instead, my messages were sent to Merit, the operator of NSF.NET, causing a political uproar. I personally do not appreciate finger pointing as a substitute for action, and in this case it was grossly unjustified. Let's keep the record straight: the folks at Merit *are* "real programmers". In the past, when I have uncovered a problem they have stayed up until 5:30 in the morning tracking it down (and I suspect without overtime). The internet has been vastly improved during the tenure of their operation of the NSF net. The NIS.NSF.NET hardware was contributed by IBM, and I understand that the software is the latest IBM release. My benchmark was carefully conducted for a direct comparison of the TCP/IP implementations, under the worst possible circumstances to make implementation flaws stand out clearly. The failures demonstrated are plainly of the host, and not of the environment. As to the "very satisfied" customers, I will point out that MSU has not yet installed the VM TCP/IP product, even though it came "free" with other products. And I occasionally read "IBMTCP-L" on bitnet. 'Nuf said. On to the results.... Bill Simpson 09998was@ibm.cl.msu.edu 09998was@msu.bitnet
09998WAS%MSU@PUCC.PRINCETON.EDU ("Bill.Simpson") (08/24/90)
The benchmark was conducted over the noisiest line I know of, at slow
speeds, with small packets, with both large and small windows, to
encourage packet loss and demonstrate transmission and syncronization
handling. With the exception of NIC.NSF.NET, all testing was done
on a single connection, one test at a time, to eliminate modem
or time of day variability.
The packet size was 232 bytes. Maximum transfer bandwidth was:
at 9600 bps: 768 bytes/second
at 1200 bps: 96 bytes/second
NIS.NSF.NET [35.1.1.48] is an IBM 4381P02 running VM/HPO-5 FAL 1.2.1
(according to Merit -- there is no host DNS record).
I understand it to be a lightly loaded machine. The traces show
less than MSS sized packets, too many retransmissions, timeouts of
20 seconds or more, and failure to combine ACKs with bidirectional data.
The 10 minute transmission timeout causes many RFCs to be unavailable
over serial lines (but may be configurable). The dir LIST facility
is excrutiatingly slow.
at 9600 bps: 320, 353, 299, 382 bytes/second
at 1200 bps: 54, 49
MERIT.EDU [35.1.1.42] is a Sun running "FTP server (version 4.115 ..."
(again, no host record)
I understand it to be a heavily loaded machine and therefore to
be running rather slow in its response times. It is located
on the same ethernet as NIS.NSF.NET, so I thought that it would
make a good comparison for round trip baseline.
at 1200 bps: 77, 85
TERMINATOR.CC.UMICH.EDU [35.1.33.8] is a Sun-3/160 running
"FTP server (version 4.172 ..."
It is located a little farther away (6 hops) in another building at UMich.
The distance didn't seem to have any effect.
at 1200 bps: 92
So I thought that I'd try a little farther!
THUMPER.BELLCORE.COM [128.96.41.1] is 8 hops father away than NIS.NSF.NET,
over NSFnet and JVNCnet. The host record says it is a VAX-8650
running BSD4.2, but the FTP message was "SunOS 4.1", so who knows?
at 1200 bps: 84
The next week, just for jollies, I tried NIC.DDN.MIL.
It's often difficult to get to, through some incredibly congested
gateways. I did this at 16:00 EDT, which should be right in the
middle of their working day out west. I pinged it for 10 minutes,
in 20 sec intervals for a srtt of 2.420 sec and mdev of 0.605 sec;
not counting that 2/3 of the packets were lost! The host record says
it's a DEC 2060 running Tops20.
at 1200 bps: 73, 78, 74, 76
CONCLUSION:
IBM-FAL for VM runs at 1/2 the speed of many of its competitors.
It fails to meet several host requirements, not just for TCP/IP
but for FTP and SMTP as well.
Unfortunately, the only product which I am aware of with worse performance
is Spartacus KNET, also for VM.
For the good of the user community, I would recommend to Merit that
the RFC mirror be moved to a more capable machine.
Bill Simpson
09998was@ibm.cl.msu.edu
09998was@msu.bitnet09998WAS%MSU@PUCC.PRINCETON.EDU ("Bill.Simpson") (09/19/90)
For those of you who were wondering what in the world Doug Nelson's
spiteful message was referring to, here is the old message, revised
by the helpful comments sent privately (thanks all). I had given up
on the thread, but it seems the issue has more life than I thought!
This benchmark was conducted over the noisiest line I know of, at slow
speeds, with small packets, with both large and small windows, to
encourage packet loss and demonstrate transmission and syncronization
handling. With the exception of NIC.DDN.MIL, all testing was done
on a single connection, after midnight -- to eliminate line, route, load,
or congestion variability.
The packet size was 232 bytes. Optimal transfer bandwidth is:
at 9600 bps: 768 bytes/second
at 1200 bps: 96 bytes/second
NIS.NSF.NET [35.1.1.48] is an IBM 4381P02 running VM/HPO-5 FAL 1.2.1
(according to Merit -- there is no host DNS record).
I understand it to be a lightly loaded machine. The traces show
less than MSS sized packets, too many retransmissions, timeouts of
20 seconds or more, and failure to combine ACKs with bidirectional data.
The 10 minute transmission timeout causes many RFCs to be unavailable
over serial lines (but may be configurable). The dir LIST facility
is excruciatingly slow.
at 9600 bps: 320, 353, 299, 382 bytes/second
at 1200 bps: 54, 49
MERIT.EDU [35.1.1.42] is a Sun 3/150 running SunOs 3.5
the banner says "FTP server (version 4.115 ..."
(again, no host record)
I understand it to be a heavily loaded machine and therefore to
be running rather slow in its response times. It is located
on the same ethernet as NIS.NSF.NET, so I thought that it would
make a good comparison for round trip baseline.
at 1200 bps: 77, 85
TERMINATOR.CC.UMICH.EDU [old address deleted] is a Sun-3/160.
the banner says "FTP server (version 4.172 ..."
It is located a little farther away (6 hops) in another building at UMich.
The distance didn't seem to have any effect (perhaps because it's a T1 link),
and I attribute the improved performance to a lighter load.
at 1200 bps: 92
So I thought that I'd try a little farther!
THUMPER.BELLCORE.COM [128.96.41.1] is a Sun 4/490 running SunOs 4.1.
It is 8 hops farther away than NIS.NSF.NET, over NSFnet and JVNCnet.
None-the-less, the transfer rate is quite respectable.
at 1200 bps: 84
The next week, just for jollies, I tried NIC.DDN.MIL.
The host records says it's a DEC 2060 running Tops20.
It's often difficult to get to, through some incredibly congested
gateways. I did this at 16:00 EDT, which should be right in the
middle of their working day out west. I pinged it for 10 minutes,
in 20 sec intervals for a srtt of 2.420 sec and mdev of 0.605 sec;
not counting that 2/3 of the packets were lost! Even under such
adverse conditions, it outperforms the IBM.
at 1200 bps: 73, 78, 74, 76
CONCLUSION:
IBM-FAL for VM runs at 1/2 the speed of many of its competitors.
It fails to meet several host requirements, not just for TCP/IP
but for FTP and SMTP as well.
Unfortunately, the only product which I am aware of with worse performance
is Spartacus KNET, also for VM.
For the good of the user community, I would recommend to Merit that
the RFC mirror on NIS.NSF.NET be moved to a more capable machine.
Bill Simpson
09998was@ibm.cl.msu.edu
09998was@msu.bitnetcliff@garnet.berkeley.edu (Cliff Frost) (09/20/90)
In article <9009190321.AA14869@ucbvax.Berkeley.EDU>, 09998WAS%MSU@PUCC.PRINCETON.EDU ("Bill.Simpson") writes: |> For those of you who were wondering what in the world Doug Nelson's |> spiteful message was referring to, here is the old message, revised |> by the helpful comments sent privately (thanks all). I had given up |> on the thread, but it seems the issue has more life than I thought! |> |> This benchmark was conducted over the noisiest line I know of, at slow On this list we've seen all this before. We're still waiting for you to substantiate your accusations. For those of you who aren't familiar with the product Bill.Simpson so gratuitously slams, his "benchmarks" show one thing unambiguously: Data transfer rates from nis.nsf.net are low. Wow. In his postings Bill.Simpson has said that he has proven that this is due to the software running on nis.nsf.net--specifically as opposed to the configuration of the software and hardware on that machine (which he admits he knows nothing about). He has never publically listed these failings, nor the alleged host requirements the sw fails to meet. I've asked him twice in private mail what specific software failings he has found, and several others have asked him this publically, and he has never responded. Perhaps he has found flaws, perhaps not. It's looking like we'll never know. Cliff Frost <cliff@Berkeley.EDU> Central Computing Services UC Berkeley