tcp-ip@ucbvax.ARPA (06/01/85)
From: mills@dcn6.arpa Folks, Responding to Vint's request, here are some relevant data covering the ARPANET/MILNET gateway performance. The data have been extracted from the lastest weekly report produced by BBN and cover only the ARPANET/MILNET gateways, which represent only seven out of the 38 operational BBN core gateways. (Who knows how many non-core gateways there are out there...) The period covered by these data cover just short of a six-day period and detail the average and peak throughputs and loss rates. The totals shown are for all of the 38 gateways. Comments follow the tables. Total Throughput GWY RCVD RCVD IP % IP DEST % DST NAME DGRAMS BYTES ERRORS ERRORS UNRCH UNRCH ---------------------------------------------------------------------- MILARP 4,169,046 306,185,112 273 0.00% 7,153 0.17% MILBBN 4,638,747 272,396,860 458 0.00% 30,045 0.65% MILDCE 3,952,555 280,374,422 372 0.00% 23,747 0.60% MILISI 5,282,635 624,869,302 779 0.01% 20,353 0.39% MILLBL 2,896,764 175,123,126 143 0.00% 6,639 0.23% MILSAC 2,765,136 157,981,916 1,122 0.04% 10,588 0.38% MILSRI 2,133,985 117,968,018 169 0.00% 13,832 0.65% ---------------------------------------------------------------------- TOTALS 92,368,009 5,768,504,913 1,556,736 1.69% 190,545 0.21% GWY SENT SENT DROPPED % DROPPED NAME DGRAMS BYTES DGRAMS DGRAMS ------------------------------------------------------- MILARP 4,146,989 295,751,188 101,471 2.39% MILBBN 4,669,813 276,807,235 157,068 3.25% MILDCE 3,942,271 284,077,034 59,404 1.48% MILISI 5,138,585 577,311,096 247,222 4.59% MILLBL 2,877,744 174,574,553 55,537 1.89% MILSAC 2,792,073 165,159,590 13,393 0.48% MILSRI 2,156,255 127,256,463 53,483 2.42% ------------------------------------------------------- TOTALS 92,523,789 5,721,526,805 1,466,274 1.56% Note that the load balancing, while not optimal, is not too bad. The data do now show, of course, the extent of the double-hop inefficiencies pointed out previously. The ARPANET/MILNET gateways see fewer IP errors than average, but somewhat more broken networks and dropped packets than average. ====================================================== Mean Throughput (per second) and Size (bytes per datagram) GWY RCVD RCVD IP AVG BYTES NAME DGRAMS BYTES ERRORS PER DGRAM -------------------------------------------------------- MILARP 8.14 597.90 0.00 73.44 MILBBN 9.06 531.92 0.00 58.72 MILDCE 7.72 547.50 0.00 70.93 MILISI 10.32 1220.21 0.00 118.29 MILLBL 5.66 341.97 0.00 60.45 MILSAC 5.40 308.50 0.00 57.13 MILSRI 4.17 230.36 0.00 55.28 GWY SENT SENT DROPPED AVG BYTES NAME DGRAMS BYTES DGRAMS PER DGRAM ------------------------------------------------------- MILARP 8.10 577.53 0.20 71.32 MILBBN 9.12 540.53 0.31 59.28 MILDCE 7.70 554.73 0.12 72.06 MILISI 10.03 1127.34 0.48 112.35 MILLBL 5.62 340.90 0.11 60.66 MILSAC 5.45 322.51 0.03 59.15 MILSRI 4.21 248.50 0.10 59.02 These values are way below the maximum throughput of the LSI-11 gateways (about 200 packets/sec); however, the average size is very small relative to the maximum ARPANET/MILNET packet size of 1007 octets. One would expect the resource crunch to be the limited buffer memory available in the present LSI-11 implementation. Note that BBN is working actively toward a dramatic increase in available memory, as noted previously. ====================================================== Peak Throughput (sum of datagrams/sec, input + output, time is time of data collection) GWY TOTAL TIME DROP TIME NAME T'PUT OF DAY RATE OF DAY ------------------------------------------------------------------------ MILARP 47.28 5/24 09:16 27.26% 5/25 22:04 MILBBN 39.53 5/23 15:32 20.70% 5/24 02:18 MILDCE 36.67 5/24 08:02 26.12% 5/24 17:59 MILISI 44.45 5/23 15:02 32.39% 5/21 16:08 MILLBL 37.76 5/22 19:43 34.91% 5/24 12:02 MILSAC 36.91 5/23 13:03 5.75% 5/21 08:53 MILSRI 22.78 5/24 08:47 24.89% 5/21 16:08 Even under peak loads the gateway horsepower is not particularily taxed; however, the buffering is obviously suffering a good deal. The times of peak throughputs do not seem to correlate with the times of peak drop rates, which tends to confirm that most of the drops occur in bunches under conditions approaching congestive collapse. The instrumentation in our gateway beteen the ARPANET and four local nets, some of which are connected by medium-speed (4800/9600 bps) lines, tends to support the above observations and conclusions. We see intense spasms on the part of some hosts (names provided upon request) which clearly are to blame for almost all of the congestion observed here. These hosts apparently have been optimized to operate well on local Ethernets with small delays and tend to bombard the long-haul paths with vast numbers of retransmissions over very short intervals. I would bet a wadge of packets against a MicroVAX-II that the prime cause for the braindamage is ARP and the unfortunately common implementation that loses the first data packet during the address-resolution cycle. If this is fixed, I bet the tendency to err on the low side of retransmission estimates would go away. There are other causes of packet spasms that have been detailed in many of my previous messages. Happily, some have gone away. Those remaining symptoms indicate continuing inefficiencies in piggybacking and send/ack policies leading to tinygram floods (with TELNET, in particular). The sad fact is that these problems have been carefully documented and are not hard to fix; however, it takes only a few bandits without these fixes to torpedo the entire Internet performance. Dave -------
tcp-ip@ucbvax.ARPA (06/03/85)
From: CERF@USC-ISI.ARPA Dave, Thanks very much for a most helpful summary. One thing which I note about the MILISI gateway, in addition to its having far more traffic that most others is that its packet size (per datagram) is larger than a single packet message, assuming the avg bytes you show do NOT include all the TCP/IP header bytes which must fit inside the text of a single packet message. If memory serves, these messages have room for at most 1008 bits or 126 bytes, some of which have to be given over to IP or TCP header. If I'm correct in this analysis, the MILISI gateway is injecting a good deal of multi-packet traffic which puts a potential strain on the current imp end/end protocol. I note that MILISI has a higher rate of datagram dropping - one might guess this is a result of increased buffer demands and possibly increased incidence of timeouts for multipacket transmission permission? Dave is right about the need to fix those wayward implementations which try to treat all nets as if they are identical in performance. To borrow from Jack Haverty: You can fool some of gateways all of the time and all of the gateways, some of the time, but you can't... The whole SYSTEM has to be thought of as a system and all parts need to meet some standards of performance and adaptation. If this is not achievable (and it's a big challenge) then nets and gateways need to find ways to cut off identifiable abusers. Vint
tcp-ip@ucbvax.ARPA (06/03/85)
From: MILLS@USC-ISID.ARPA In response to the message sent 2 Jun 1985 05:04-EDT from CERF@USC-ISI.ARPA Vint, The datagram sizes shown in my data include the IP and TCP headers, so a lot more gateways than just ISI are chirping multi-packet messages. From previous reports, I would not count the ISI data as typical - there might be something special going on there. I'll alert the field operatives... Dave -------
tcp-ip@ucbvax.ARPA (06/04/85)
From: Marianne Gardner <mgardner@BBNCCY.ARPA> Dave, I must disagree with both you and Vint. It is typical for ISI to have an average datagram size over 100 bytes. ISI does not have a disproportionate amount of traffic and drops fewer packets than most of the mailbridges. Marianne
tcp-ip@ucbvax.ARPA (06/04/85)
From: MILLS@USC-ISID.ARPA In response to your message sent Mon, 3 Jun 85 14:33:44 EDT Marianne, Your comments do not jibe with the data in my message - MILISI leads the pack in throughput, size and almost all categories of breakage. You might have taken umbrage if I tattled on ISI (as against MILISI), but my message clearly addressed the mailbridges only. Dave -------
tcp-ip@ucbvax.ARPA (06/04/85)
From: James O'Toole <james@gyre> Vint, Your message said "1008 bits or 126 bytes" but the IMP really handles 1008 *bytes*. I didn't notice that the average packet sizes were ever higher than that, which would surprise me. Am I missing the point here, or did you mix your units? --Jim P.S. Hmmm, isn't that really 1006, not 1008?
tcp-ip@ucbvax.ARPA (06/04/85)
From: Marianne Gardner <mgardner@BBNCCY.ARPA>
Dave,
You data suffers taking too small a sample. Yes, last week MILISI had
more traffic than the other bridges, but this is not the usual case as the
summaries below will show. Let me say again, that if you look at the trap
reports and throughput summaries for the last few months, you will not find
that MILISI has more problems than the other mailbridges.
Sorry if my shortening the name of MILISI confused you.
The data that follows covers 7 day periods, from Monday to the following
Sunday. The date indicated is the Sunday that ends the collection period.
date: 4/14 (Sun)
GTWY HOURS RCVD DST RCVD PPS SENT PPS AVG
COVERED DGRAMS UNRCH DGRAMS DGRAMS BYTES
MILARP 162.00 5,491,493 0.09 9.42 9.24 129.35
MILDCE 162.00 5,445,388 0.72 9.34 9.24 70.18
MILISI 162.00 4,976,827 0.58 8.53 8.48 117.84
MILLBL 162.25 3,800,470 2.64 6.51 6.11 65.16
MILSAC 162.00 2,591,494 0.17 4.44 4.48 63.04
MILSRI 162.00 2,162,412 2.08 3.71 3.70 59.10
date: 4/21 (Sun)
GTWY HOURS RCVD DST RCVD PPS SENT PPS AVG
COVERED DGRAMS UNRCH DGRAMS DGRAMS BYTES
MILARP 155.50 4,184,138 0.29 7.47 7.44 79.09
MILISI 158.75 4,914,047 0.58 8.60 8.66 114.79
MILLBL 158.75 3,934,242 0.59 6.88 6.76 78.86
MILSAC 158.75 2,667,978 0.81 4.67 4.70 61.66
MILSRI 158.75 2,051,680 1.59 3.59 3.62 59.14
date: 5/5 (Sun)
GTWY HOURS RCVD DST RCVD PPS SENT PPS AVG
COVERED DGRAMS UNRCH DGRAMS DGRAMS BYTES
MILARP 164.25 4,550,474 1.40 7.70 7.75 74.89
MILBBN 164.25 5,622,661 0.67 9.51 9.56 60.07
MILDCE 161.75 4,467,564 0.94 7.67 7.66 77.32
MILISI 164.25 5,190,179 0.46 8.78 8.64 115.16
MILLBL 164.25 3,797,215 0.41 6.42 5.94 63.03
MILSAC 164.25 3,120,124 0.49 5.28 5.21 68.68
MILSRI 164.25 2,058,651 0.42 3.48 3.57 65.51
date: 5/12 (Sun)
GTWY HOURS RCVD DST RCVD PPS SENT PPS AVG
MILARP 161.25 4,765,258 0.34 8.21 8.19 74.53
MILBBN 161.25 4,809,526 0.81 8.29 8.38 60.88
MILDCE 161.25 4,643,647 0.56 8.00 7.96 78.24
MILISI 161.25 5,059,553 0.50 8.72 8.65 96.73
MILLBL 161.25 3,501,986 0.30 6.03 5.93 62.55
MILSAC 161.25 3,073,519 0.49 5.29 5.22 66.44
MILSRI 161.25 1,994,121 0.17 3.44 3.54 66.18
date: 5/19 (Sun)
GTWY HOURS RCVD DST RCVD PPS SENT PPS AVG
COVERED DGRAMS UNRCH DGRAMS DGRAMS BYTES
MILARP 162.00 4,442,049 0.23 7.62 7.62 72.99
MILBBN 161.75 5,030,583 0.81 8.64 8.78 61.53
MILDCE 159.50 4,428,993 0.73 7.71 7.67 90.27
MILISI 162.00 4,866,970 0.55 8.35 8.37 105.50
MILLBL 162.00 3,103,610 0.15 5.32 5.23 64.42
MILSAC 162.00 2,990,150 0.50 5.13 5.17 64.61
MILSRI 162.00 2,046,546 0.74 3.51 3.59 57.72
date: 5/26 (Sun)
GTWY HOURS RCVD DST RCVD PPS SENT PPS AVG
COVERED DGRAMS UNRCH DGRAMS DGRAMS BYTES
MILARP 142.25 4,169,046 0.17 8.14 8.10 71.32
MILBBN 142.25 4,638,747 0.65 9.06 9.12 59.28tcp-ip@ucbvax.ARPA (06/04/85)
From: MILLS@USC-ISID.ARPA In response to your message sent Mon, 3 Jun 85 16:13:14 EDT Marianne, I surely didn't mean to start a bluster here. Vint's original comment addressed the curiousity that MILISI datagrams were fatter than the others. Your data show this to be true in four of the five samples. My comment that MILISI exibited more breakage than the others was not intended as a blanket indictment and, in fact is not justified in view of your data. My suspicion that ISI is indeed special is confirmed both because of the consistently high datagram size, as well as the increasing share of the system load, going from number four at the beginning of your sample history consistently upwards to number one at the end. MILISI is again number one this week in both departments. My aside to Vint that I would "alert the field operatives" has come to pass and you have broken cover. Your next assignment, should you choose to accept it, is to tell us why. Dave -------
tcp-ip@ucbvax.ARPA (06/04/85)
From: Ron Natalie <ron@BRL.ARPA> One observation: MILISI is the gateway that EGP tells my gateways to use for all the nets on the other side of the bridges. Assuming this is true for both sides of the house, it would seem that this is the gateway used for peoples little ethernets to talk through to get to the otherside. Most local nets have higher MTU's than the IMP 1008 (typically 1536). This results in their gateways sending lots of fragmented datagrams all over the place. Since most of the LANs derive choose the MIL/ARPA bridge from the EGP information, ISI may bear the brunt of the fractured ethernet traffic. IP fragmentation is admittedly a bad thing to do on a regular basis. BRL avoids this by keeping most of our LAN's at 1008 (for historical reasons mostly, the gateways originally didn't know how to fragment). -Ron
tcp-ip@ucbvax.ARPA (06/05/85)
From: Mark Crispin <Crispin@SUMEX-AIM.ARPA>
Jim -
An 1822 packet is 1008 bits, which is not only 126 bytes
but also 28 36-bit words. Hosts never see packets, they see
messages which can be up to 8159 bytes; that is, a 96 bit leader
plus 8 packets full of data minus 1 bit.
Many people erroneously confuse messages with packets, and
unfortunately a lot of host software does the same.
-- Mark --
-------tcp-ip@ucbvax.ARPA (06/05/85)
From: Ron Natalie <ron@BRL.ARPA> Actually IMP packets are 1008 bits, IMP regular messages (what we know as packets) are from 96 to 8159 bits. Subtracting 96 bits off for the IMP leader leaves 8063 bits which is 1007 octets and 7 bits left over (a septet, I suppose). -Ron
tcp-ip@ucbvax.ARPA (06/08/85)
From: CERF@USC-ISI.ARPA Marianne, I don't understand exactly what you are disagreeing with; I based my comments on the tables that Dave published. Are you disagreeing with the numbers he shows? It seemed to me that those tables reflected the ISI gateway bearing a consdierably larger amount of traffic than any other gateway by at least a factor of two (if memory serves) and that it dropped a higher percentage of packets. Please elaborate on your recent short note so it is clearer to me, at least, how you interpret Dave's tables. thanks, Vint
tcp-ip@ucbvax.ARPA (06/08/85)
From: CERF@USC-ISI.ARPA Jim, unless things have changed dramatically when I wasn't looking, the IMP takes in up to 1008 bytes but breaks them into 1008 bit packets. The X.25 versions of the IMP take in 1024 bytes, so I suspect the single IMP packet is probably 1024 bits long in data field rather than 1008, for those IMPs. The IP datagram sizes therefore tell you whether you are dealing with single or multi-packet messages in the network. Any datagram larger than 126 bytes is therefore a multi-packet message and subject to the end/end protocol for multipacket messages. The 8 messages outstanding problem is usually exacerbated when there are a lot of small messages - short datagrams carrying interactive telnet traffic are the worst case. It's possible I wasn't very clear in what I was trying to say, but I do believe I'm correct that the IMP still breaks up messages larger than 126 bytes into multiple small packets of up to 1008 bits. I could be wrong about the 1008 bits, it could have gone up a little, but not much beyond 1024 bits, I bet. Vint
tcp-ip@ucbvax.ARPA (06/08/85)
From: CERF@USC-ISI.ARPA Ron, Hmmmm. the average byte statistics don't tell us about maxima or distribution of 1822 message (see, I am being careful to distinguish 1822 MESSAGE from IMP PACKET!) sizes. Perhaps Marianne Gardner has available more detailed statistics which would tell us whether there is any significant amount of IP level fragmentation going on. Presumably one would not see the fragmenting going on at MILISI, for instance. Rather, it would occur at the gateway which interfaces the local area net to the ARPANET. If there were a significant number of 1536 byte IP messages being sent, these would fragment into two IP datagrams of roughly 1000 bytes and 536 bytes. [aside: I am no longer sure whether the present preferred gateway fragmentation algorithm at IP level is to make datagrams that just fit in the next net, in which case 1000 and 536 would be more or less right for ARPANET, neglecting headers and stuff; or whether the policy is to fragment to 576 maximum or to make all IP fragments uniform in size and less than or equal to 576 or less than or equal to the next net's maximum message size. Can a knowledgeable source please set me straight on that? ] It seems plain that something is causing the average IP datagram size to be larger at MILISI than at other gateways. Vint
tcp-ip@ucbvax.ARPA (06/08/85)
From: Andrew Malis <malis@BBNCCS.ARPA> Vint, You are completely correct about 1822 messages greater than 126 bytes being broken up into packets up to 126 bytes long each, and that the IMP end-to-end protocol imposes greater overhead (destination IMP buffer space preallocation, to be precise) for multipacket messages. For X.25 traffic, the IMP uses 134-byte packets, so that a full-sized X.25 message (1024 bytes + some overhead) still fits in 8 packets. The same multipacket overhead applies. Andy Malis
tcp-ip@ucbvax.ARPA (06/09/85)
From: Mike Brescia <brescia@bbnccv> Vint, Ron, Dave, Marianne, &al. The statistics collected from the BBN gateways include only total packet counts and total byte counts. The average reported is (you guessed it) byte-count divided by packet count. There's no facility for histograms, maxima, standard deviation, or other interesting statistics. The gateways currently fragment in the manner you recall, Vint, with the first fragment(s) being as large as possible, and the last one containing the leftovers. The idea of splitting a datagram as close to the half-way (or 1/N if N fragments are needed) was not deemed useful for the arpanet, because the end-to-end overhead is the same for a 130 byte or 576 byte packet as for a 1000 byte packet. The best policy for the arpanet was to get the packets large as possible. My intuition about MILISI carrying larger packets than others is that ISI has large hosts (TOPS20) on both sides of the mil-arpa divide, and they may do lots of file transfers (ftp big packets) or screen outputs (telnet big packets). One of the ISI wizards may be able to shoot holes in that speculation... Good hunting, Mike
tcp-ip@ucbvax.ARPA (06/25/85)
From: Marianne Gardner <mgardner@BBNCCY.ARPA>
Vint,
Sorry to take so long to answer your message. My question was not with the
interpretation of the Dave's data, but with the fact that you only saw one
week's data. I have been looking at throughput data for the mailbridges
every day for almost a year. I saw a different picture. In fact, the week
covered by Dave's data did show a rise in the proportion of traffic
going to MILISI. Before that week MILARP, MILBBN, MILDCEC, and MILISI
all received about the same amount of traffic; MILLBL, MILSAC, and MILSRI
received less. We saw MILISI receiving more than its share of traffic all
month, but last week the traffic distribution again looked even. Such
fluctuations in traffic are common. They are worth attention only when
they persist and cause problems.
In any case, your memory increased the disparity in the traffic
distribution. The weekly averages are given below. The drop rates varied.
Sometimes they were as high as 4%, sometimes they were low. We, at BBN,
are looking into this problem.
DATAGRAMS RECEIVED PER SECOND, averaged over one week
6/2 6/8 6/16 6/23
MILARP 7.21 6.59 5.95 6.17
MILBBN 9.09 9.82 8.71 9.02
MILDCE 7.62 8.24 7.12 8.79
MILISI 9.31 12.08 10.13 9.68
MILLBL 6.06 6.25 5.41 5.26
MILSAC 4.92 6.24 5.76 4.48
MILSRI 3.88 4.31 3.67 3.26
Perhaps, the people at ISI, who were so good about admitting to their
penchant for cross-network ftping, will have an explanation for the extra
traffic last month. Actually, the answer is more likely to come from
across the network, since the increase in MILISI's traffic was accompanied
by a drop in everyone else's traffic.
Marianne