bryan@uhura1.uucp (Bryan Curnutt) (03/08/91)
I've also heard (though I don't know how true it is) that Ethernet network performance degrades drastically if too many nodes are added to the network, and that the number of nodes to do this is relatively small. Is there any truth to this? -- Bryan Curnutt bryan%uhura1@uunet.uu.net
spurgeon@atget.cc.utexas.edu (Charles Spurgeon) (03/08/91)
In article <1991Mar7.221028.9883@uhura1.uucp>, bryan@uhura1.uucp (Bryan Curnutt) writes: |>I've also heard (though I don't know how true it is) that Ethernet |>network performance degrades drastically if too many nodes are added |>to the network, and that the number of nodes to do this is relatively |>small. Is there any truth to this? Every Ethernet, indeed every LAN no matter what the technology used, has a different equipment mix and load profile, so your mileage will vary. On the other hand, there are no special limits to host population/traffic on an Ethernet other than those listed in the spec (for example, 100 transceivers per thick Ethernet segment, 30 connections per thin Ethernet segment, 1024 addressable stations per Ethernet system linked with repeaters). There have been a number of papers written about performance on various simulations of "Ethernet like" networks. These simulations seem to be the ones cited when people are describing "hard limits" in the Ethernet protocol. For some empirical tests of Ethernet performance that show what the real Ethernet protocol can deliver, check out the DEC Technical Report listed below: Measured Capacity of an Ethernet: Myths and Real- ity David R. Boggs, Jeffrey C. Mogul, Christopher A. Kent. Proceedings of the SIGCOMM '88 Symposium on Com- munications Architectures and Protocols, ACM SIGCOMM, Stanford, CA., August 1988, 31 pps. From the Abstract: "Ethernet, a 10 Mbit/sec CSMA/CD network, is one of the most successful LAN technologies. Considerable confu- sion exists as to the actual capacity of an Ethernet, especially since some of the theoretical studies have examined operating regimes that are not characteristic of actual networks. Based on measurements of an actual implementation, we show that for a wide class of appli- cations, Ethernet is capable of carrying its nominal bandwidth of useful traffic, and allocates the bandwidth fairly." This paper is also available over the Internet via electronic mail from the DEC Western Research archive server. Send a message to the following address with the word "help" in the Subject line of the message for detailed instructions. The address is WRL- Techreports@decwrl.dec.com. You may also request a copy of the report through the U.S. postal system by writing to: Technical Report Distribution DEC Western Research Laboratory, UCO-4 100 Hamilton Avenue Palo Alto, California 94301
karl@ddsw1.MCS.COM (Karl Denninger) (03/11/91)
In article <1991Mar7.221028.9883@uhura1.uucp> bryan@uhura1.UUCP (Bryan Curnutt) writes: >I've also heard (though I don't know how true it is) that Ethernet >network performance degrades drastically if too many nodes are added >to the network, and that the number of nodes to do this is relatively >small. Is there any truth to this? >-- >Bryan Curnutt bryan%uhura1@uunet.uu.net Oh, I don't know about that. Does 300 nodes count, spread between two buildings? :-) If you route or bridge packets intelligently, you can do much better. -- Karl Denninger (karl@ddsw1.MCS.COM, <well-connected>!ddsw1!karl) Public Access Data Line: [+1 708 808-7300], Voice: [+1 708 808-7200] Copyright 1991 Karl Denninger. Distribution by site(s) which restrict redistribution of Usenet news PROHIBITED.
mart@csri.toronto.edu (Mart Molle) (03/12/91)
spurgeon@atget.cc.utexas.edu (Charles Spurgeon) writes: >In article <1991Mar7.221028.9883@uhura1.uucp>, bryan@uhura1.uucp (Bryan >Curnutt) writes: >|>I've also heard (though I don't know how true it is) that Ethernet >|>network performance degrades drastically if too many nodes are added >|>to the network, and that the number of nodes to do this is relatively >|>small. Is there any truth to this? >Every Ethernet, indeed every LAN no matter what the technology used, has >a different equipment mix and load profile, so your mileage will vary. On >the other hand, there are no special limits to host population/traffic on an >Ethernet other than those listed in the spec (for example, 100 transceivers >per thick Ethernet segment, 30 connections per thin Ethernet segment, >1024 addressable stations per Ethernet system linked with repeaters). It is not as simple as you suggest. There is indeed very strong evidence to show that network performance can degrade drastically when the number of active hosts reaches a threshold that is significantly below 1024. >There have been a number of papers written about performance on various >simulations of "Ethernet like" networks. These simulations seem to be >the ones cited when people are describing "hard limits" in the Ethernet >protocol. For some empirical tests of Ethernet performance that show >what the real Ethernet protocol can deliver, check out the DEC Technical >Report listed below: > Measured Capacity of an Ethernet: Myths and Reality > David R. Boggs, Jeffrey C. Mogul, Christopher A. Kent. > Proceedings of the SIGCOMM '88 Symposium on Com- > munications Architectures and Protocols, ACM > SIGCOMM, Stanford, CA., August 1988, 31 pps. > From the Abstract: > "Ethernet, a 10 Mbit/sec CSMA/CD network, is one of the > most successful LAN technologies. Considerable confu- > sion exists as to the actual capacity of an Ethernet, > especially since some of the theoretical studies have > examined operating regimes that are not characteristic > of actual networks. Based on measurements of an actual > implementation, we show that for a wide class of appli- > cations, Ethernet is capable of carrying its nominal > bandwidth of useful traffic, and allocates the > bandwidth fairly." Why are you citing this paper? Their study was limited to 24 hosts, and hence its findings are irrelevant for the question at hand. Let me give you three reasons to support my claim: 1. Careful examination of the truncated binary exponential backoff algorithm. Consider a worst-case scenario, where a large number (~=1024) of active hosts is beating on the network continuously. According to the back-of- the-envelope ``1/Q'' model introduced in the original 1976 CACM paper by Metcalfe and Boggs, all hosts will quickly spread out their retries until they hit the maximum backoff delay (i.e., uniform retries over the next 1024 ``slot times'') and so the system settles down to stable operation with (roughly) one retry per slot time, giving an asymptotic utilization of (packet length)/ [(packet length) + (e-1)* (slot time)] independent of the number of hosts, as long as its no more than 1024 due to the truncation of the backoff interval. The flaw in this ``1/Q'' model is that the backoff counter gets reset after 15 unsuccessful retries, and thereafter the next several attempts occur very rapidly (as the host ``relearns'' about the congestion problem). It can be shown that in the limit of very high collision rates, the average time between successive retries BY A SINGLE NODE is only about 225 slot times -- so a 1024 host network would have an average of =four= hosts attempting to transmit at every ``slot''. Plugging this value of the traffic generated by each station into an analytical model (e.g., Sohraby, et al., "Comments on `Throughput Analysis for Persistent CSMA Systems'", IEEE Trans. Comm., Feb. 1987) indicates that, depending on the packet size, adding too many hosts causes the percentage of dropped frames to skyrocket and the Ethernet utilization to plummet. For 1024 byte frames, the loss probability shoots up between 10 and 100 hosts, and the utilization drops off between 800 and 1200 hosts; for 64 byte frames the loss probability shoots up between 100 and 600 hosts and the utilization drops off between 100 and 500 hosts. 2. Using simulation (gasp!) to extrapolate the results of the `Myths and Reality' paper to large numbers of hosts. We wrote yet another Ethernet simulator, including all the grubby details we could think of (like the 9.6 usec interpacket gap, 12 byte preamble/ checksum) and configured it to mimic the network configuration reported by Boggs, et al., in the paper above. After validating our simulator by reproducing their experimental data for 1-24 hosts, we increased the number of hosts into the hundreds and observed the same effects at roughly the same numbers of hosts as described above. 3. Cite the experience of someone who has (apparently) actually tried to run an Ethernet with that many hosts. The following article appeared about a year ago in comp.dcom.lans, in response to a similar query about Ethernets with large numbers of hosts. Evidently, the experience at Livermore indicates that 1024 hosts on a single Ethernet `collision domain' is too many... +---------------------------------------------------------------------- | >From: oberman@rogue.llnl.gov (Oberman, Kevin) | Newsgroups: comp.dcom.lans | Subject: Re: 1024 stations on 1 ethernet? | Keywords: ethernet,lan | Message-ID: <51316@lll-winken.LLNL.GOV> | Date: 7 Mar 90 11:15:42 GMT | Sender: usenet@lll-winken.LLNL.GOV | Reply-To: oberman@rogue.llnl.gov | Organization: Lawrence Livermore National Laboratory-Engineering | Lines: 25 | | In article <1990Mar7.012322.18918@granite.cr.bull.com>, piacenti@granite.cr.bull.com (Paul Piacentini) writes... | >my 1st post ... I hope it comes out right | > | > I have read somewhere more than once that up to 1024 stations are allowed | >on 1 ethernet lan. Does this mean 1024 ACTIVE stations, or 1024 adapter | >conections? Does anyone know why this is a limitation? We are rapidly | >approaching this figure, but we have several ethernets connected via | >level 2 bridges. Does this mean we can have up to 1024 stations on each | >side of the bridges? Any info would be appreciated, thanks. | | This limit is technically right, but practically wrong. | | The cause of this limit is the collision backoff alogorithm. When a collision | is detected by a device, the device backs off for a random time. The backoff is | on of 1024 discrete time intervals. If you have more than about 800 active | devices on the Ethernet, thing tend to go foobar. A bridge starts the count | over again. A repeater does not. | | R. Kevin Oberman | Lawrence Livermore National Laboratory | Internet: oberman@icdc.llnl.gov | (415) 422-6955 | | Disclaimer: Don't take this too seriously. I just like to improve my typing | and probably don't really know anything useful about anything. +---------------------------------------------------------------------- Mart L. Molle Computer Systems Research Institute University of Toronto Toronto Canada M5S 1A4 (416)978-4928
haas%basset.utah.edu@cs.utah.edu (Walt Haas) (03/12/91)
In article <1991Mar11.133033.25283@jarvis.csri.toronto.edu> mart@csri.toronto.edu (Mart Molle) writes: >spurgeon@atget.cc.utexas.edu (Charles Spurgeon) writes: > >>In article <1991Mar7.221028.9883@uhura1.uucp>, bryan@uhura1.uucp (Bryan >>Curnutt) writes: >>|>I've also heard (though I don't know how true it is) that Ethernet >>|>network performance degrades drastically if too many nodes are added >>|>to the network, and that the number of nodes to do this is relatively >>|>small. Is there any truth to this? > >>Every Ethernet, indeed every LAN no matter what the technology used, has >>a different equipment mix and load profile, so your mileage will vary. On >>the other hand, there are no special limits to host population/traffic on an >>Ethernet other than those listed in the spec (for example, 100 transceivers >>per thick Ethernet segment, 30 connections per thin Ethernet segment, >>1024 addressable stations per Ethernet system linked with repeaters). > >It is not as simple as you suggest. There is indeed very strong evidence >to show that network performance can degrade drastically when the number >of active hosts reaches a threshold that is significantly below 1024. A realistic limit for an Ethernet is probably on the order of 200 hosts, at least in our environment. We limit router legs to 100 to 200 hosts, and usually run into loading issues before we get to that number. I know of sites that have built larger bridged Ethernets, and the broadcast traffic became unmanageable. One such site is now introducing routers into the net to manage broadcasts etc. In view of this experience the exponential backoff issue seems pretty academic. -- Walt Haas haas@ski.utah.edu
moleres@druco.ATT.COM (Rick Moleres) (03/13/91)
> From: bryan@uhura1.uucp (Bryan Curnutt) > Newsgroups: comp.dcom.lans > Subject: Ethernet performance degradation? > Message-ID: <1991Mar7.221028.9883@uhura1.uucp> > Date: 7 Mar 91 22:10:28 GMT > Reply-To: bryan@uhura1.UUCP (Bryan Curnutt) > I've also heard (though I don't know how true it is) that Ethernet > network performance degrades drastically if too many nodes are added > to the network, and that the number of nodes to do this is relatively > small. Is there any truth to this? > -- > Bryan Curnutt bryan%uhura1@uunet.uu.net As I understand it, it is the amount of traffic that degrades performance, not the number of nodes. One could argue that increasing the number of nodes increases the amount of traffic on the ethernet, but that is not necessarily true. An ethernet uses CSMA/CD (Carrier Sense Mutliple Access/ Collision Detection) to gain access to the bus. When a collision is detected, an exponential backoff algorithm is used by a node to determine when to try again. If the amount of traffic on the ethernet is heavy, then collisions occur often and the interval before retries becomes exponentially long. Pretty soon you have an under-utilized bus because the nodes are waiting for their next retry time. Thus, performance on the ethernet has degraded due to heavy traffic. -Rick
spurgeon@.uucp (Charles E. Spurgeon) (03/15/91)
In article <1991Mar11.133033.25283@jarvis.csri.toronto.edu> mart@csri.toronto.edu (Mart Molle) writes: >spurgeon@atget.cc.utexas.edu (Charles Spurgeon) writes: > >>In article <1991Mar7.221028.9883@uhura1.uucp>, bryan@uhura1.uucp (Bryan >>Curnutt) writes: >>|>I've also heard (though I don't know how true it is) that Ethernet >>|>network performance degrades drastically if too many nodes are added >>|>to the network, and that the number of nodes to do this is relatively >>|>small. Is there any truth to this? > >value of the traffic generated by each station into an analytical model >(e.g., Sohraby, et al., "Comments on `Throughput Analysis for Persistent >CSMA Systems'", IEEE Trans. Comm., Feb. 1987) indicates that, depending >on the packet size, adding too many hosts causes the percentage of dropped >frames to skyrocket and the Ethernet utilization to plummet. For 1024 >byte frames, the loss probability shoots up between 10 and 100 hosts, >and the utilization drops off between 800 and 1200 hosts; for 64 byte >frames the loss probability shoots up between 100 and 600 hosts and the >utilization drops off between 100 and 500 hosts. The Boggs, Mogul and Kent paper I cited in my reply to the original post also contains some good advice for Ethernet implementation in which they point out that you should not install long repeated networks (use bridges or routers instead), and that you should try to keep the number of hosts per cable segment down to a low roar. Most campuses I'm aware of have used routers to limit the size of Ethernet segments and to isolate themselves from the effects of misconfigured hosts and high level protocols emitting lots of broadcasts. Chopping the Ethernets up into smaller chunks definitely pays dividends in terms of reliability and manageability. On the other hand, back in the old days before high performance bridges and routers were readily available, large repeated networks were built on campuses that got involved with networking early on. At Stanford there was a large 3 megabit experimental Ethernet that linked a goodly number of hosts. The main problem I recall with that system was that every winter, when it rained in California, the old abandoned cable plant that had been expropriated for this network would get water in its conduits causing high levels of packet loss. Once things dried out, it tended to work fine. :-) At the University of Texas, another major campus that used Ethernet early on, the old campus network ran on a large fiber optic system built using repeaters. That too linked a good number of hosts and ran for some time with lots of repeaters and long segments. >3. Cite the experience of someone who has (apparently) actually tried >to run an Ethernet with that many hosts. Ok. I found an old copy of a paper written in 1986 at DEC in my piling system, in which they monitor the operation of the Ethernet at their Spit Brook Road, New Hampshire facility. At the time this network supported hundreds of nodes on segments linked to a backbone with repeaters. The backbone itself had a couple of LAN Bridges in it. For the test they removed the LAN Bridges and ran the whole thing as one big repeated system. They were running 467 nodes, including 23 tips supporting 1,528 terminal lines. The system was virtually all DECnet, SCA, and LAT traffic and most of the packets were LAT packets and fairly small in size. This is pretty typical for a network back then - lots of tip/terminal traffic. During working hours their percentage of utilization fluctuated between 15-22% with short spikes, sample period was .5 seconds. From the sounds of it, this network ran fine for them. I hasten to add that I wouldn't recommend that anyone set out to build large repeated, or even bridged, networks. With everything managed by one group, running DECnet, and using multicasts like the network described above, it obviously worked. But anyone trying to manage a heterogeneous network of that size, especially in a campus environment where there is a lot of freedom and people tend to (mis)configure hosts on their own, will find themselves dealing with a real challenging job. Dealing with the havoc that misconfigured or buggy hosts can cause on a large repeated or bridged network can be pretty hair raising. Routers, with their ability to block the propagation of broadcasts on a network system, are known as "network firewalls" for good reason.