m1rcd00@fed.FRB.GOV (Bob Drzyzgula) (10/05/88)
Here at the Federal Reserve Board, we have a recently installed
ethernet with over 400 connection points, serving over 100 (Sun) CPUs
and over 300 users on about 50 workstations and 240 VT-220 Clone
Terminals. What with faster and faster workstations, increased database
access and (probably) mainframe (MVS) NFS coming in the next year,
we are beginning to think a bit more seriously about network bandwidth.
Maybe worry is more the word. This article is a solicitation for
comments, ideas and suggestions from the usenet community for
modifications (the less drastic, the better, of course) that might
be made to this installation to increase capacity. I think that our
network configuration is typical enough so that discussion of this
problem may be of interest to a large number of ethernet sites.
Sadly I feel that I must add the following statement: THIS IS NOT TO
BE MISTAKEN FOR AN RFP, RFQ, RFI, OR ANY OTHER FORMAL PROCUREMENT
INSTRUMENT. THIS IS ONLY A SUGGESTION THAT WE TOSS SOME IDEAS AROUND.
SALESMEN ARE WELCOME TO TELL ME ABOUT THEIR PRODUCTS, BUT THERE IS NO
PROMISE HERE OF FORMAL CONSIDERATION OF OR RESPONSE TO ANY PROPOSALS
RECEIVED AS A RESULT OF THIS POSTING.
So anyway, as I was saying, the network is configured as below:
Thick-net Thick-net
+-+---------+--+--+ (10Mbps) +--+----------+---+
| | | Fiber Inter-Building Run | |
|<--AUI-->| +---------------------------+ | AUI
| | +----+----+
| | | Gateway |
| | +----+----+
+---+---+ +--+---+ RS-232 |
| IB/2 | | LS/1 |------------* (Connection Point) | Bus &
+---+---+ +------+ | | Tag
| | |
|<--AUI |<--Thinnet +-----+-----+
| | | Mainframe |
+---+---+ AUI Cable +---+----+ +-----------+
| MT800 |------------------| MR9000 |
+-------+ +--------+
The IB/2 is a MAC-Level bridge from Bridge Communications. The MT-800
is an 8-port 802.3 Transceiver (~DELNI) from Cabletron, while the
Cabletron MR9000 is an 8-port Thinnet Repeater (~DEMPR). The LS/1 is a
16-64 port TCP/IP terminal server from Bridge Communications. The
Mainframe is an Amdahl 580-5840 running VM/CMS, connected via a Sun
Channel-Attach Gateway, running Sunlink Local 3270. That mainframe is
connected to two other Amdahl 580-5890 Mainframes running MVS/XA.
(There is a great deal of coveted data on the mainframes;
the introduction of mainframe NFS will unleash a great burden on the
backbone.) There are 9 IB/2s, 7 on one side of the fiber, 2 on the
other. There are 14 LS/1s.
Connection points thus have both RS-232 and ethernet capability. Up to
5 connection points are daisy-chained on each thinnet-run. The
wallplates have two BNC connectors, with a short (6-inch) loop of
thinnet provided for use as a shunt when the connection is not in use.
50-conductor trunk cables run from the LS/1s to office suites, where
they are split into 3-pair runs to the connection points in a star
configuration. Terminals run at 9600 baud.
So far I have a few ideas about how to upgrade the network without
major reconstruction. First would be to change the load on the
network. Right now we have a lot of diskless workstation traffic.
Rather than have those workstations page over the net, We could
install paging packs on them. We are considering a couple of sources
of 90MB ESDI/SCSI external disks. This would seem to be an ideal size;
enough for root & /tmp and lots of paging space. This change would
free up quite a bit of the local thinnet bandwidth for NFS, terminal
and mainframe traffic.
Also, you will notice that the LS/1s are directly on the backbone.
(Actually, there is a thinnet run off an MR-9000 involved in most LS/1
connections. That would have been a little too much for the diagram,
however.) What this means is that the packets/sec count on the backbone
is driven up by all those typists. This is a harder problem,
since the LS/1s serve geographically contiguous collections of office
suites, (necessarily, because of an empirically derived 200-ft limitation
for our RS-232 trunk cables), whereas the areas served by all thinnet on
the local side of a single IB/2 are associated only by the organizational
structure of the divisions; putting the LS/1s on the local side of an IB/2
would result in some number (estimated at 30-40%) of users' keystrokes
going across *two* bridges (and, alas, the backbone) by default. I am
really at a loss to know what to do about this.
In the way of hardware upgrades, there is a measure that may improve
overall network performance, but not add that much to the total backbone
bandwidth. Our IB/2s use MC68000 processors, which only operate at
something like 20-30% of normal ethernet capacity. Bridge does now have
MC68020 processor boards for these boxes; in fact they do not sell the
MC68000 version anymore. We plan as a first step to install some of
these, but that isn't going to do us any good when the backbone is
saturated anyway.
The next thing would be to rip out the backbone and replace it with
a Proteon or FDDI ring and replace the IB/2s with some sort of
fiber-to-ethernet bridge. In our installation, the thicknet backbone
is coiled up on racks in single rooms on either end of the fiber run,
which is just point-to-point between two repeaters. So this shouldn't
be too difficult.
This is where it gets a little vague. I am pretty sure that the gear
to install an 80-100 Mbps backbone should be available now or "Real
Soon Now". I am concerned about the following questions:
1. Is it now, or will it eventually be honest-to-gosh FDDI?
(Whatever that is)
2. If the fiber-to-ethernet bridge has multiple ethernet
ports, are these ports logically isolated from one another,
or are they there just for convenience?
3. Does the bridge filter traffic at the MAC level or below?
Is it transparent to TCP/IP? (We're one big class B internet
and want to stay that way.) Must the routing information be
programmed in or does it use some dynamic-adaptive routine?
4. Can we use our current (100/140 um) inter-building fiber run?
5. Are there fan-out units that would allow us to build a star
topology fiber sub-network (eg to extend the backbone to heavily
used machines)?
6. Are controllers available for Sun VME bus? Is there software
available for such controllers? Could one imagine running a Sun
right on the backbone (idling the ethernet controller on that
machine)?
7. Are network management and diagnostic tools provided, or is
this just a "black box"?
8. Other things that I eventually learn are important
considerations.
Please feel free to send me email about this, or post it if you feel
it is of general interest. Thank you all in advance for you help.
--Bob
-----------------------------------------------------------------------
Bob Drzyzgula uucp: uunet!fed!rcd
Mail Stop 96 internet: rcd@fed.frb.gov
Federal Reserve Board bitnet: fed!rcd@uunet.uu.net
20th & C Streets, NW (I know, but last I checked,
Washington, DC 20551 bitnet hadn't found us)
-----------------------------------------------------------------------
--
Bob Drzyzgula
Federal Reserve Board, Washington, DC, 20551; uunet!fed!rcdhedrick@athos.rutgers.edu (Charles Hedrick) (10/06/88)
First, you would need far more than 240 terminal users before the traffic from your terminal servers becomes a bandwidth issue. So I wouldn't worry about where your terminal servers are. The obvious first place to attack is your diskless workstations. I would avoid putting more than 30 or so on one Ethernet, though it depends upon how much memory they have, what sort of applications they run, etc. So the idea is to put them in clusters separated from the rest of the network by at least a bridge, if not a router. Assuming you're using Unix sytems as file servers, the cheapest way to do this is to add an Ethernet card to your file server. Then you create a subnet just for the clients of one or more file servers, and use one of the file servers to gateway to the main Ethernet. I don't much like this, because I don't really think Unix software is quite optimal for acting as a gateway, but maybe I'm spoiled. I'd still split things into groups of a few dozen machines with their file servers, but use a real router (e.g. cisco or Proteon) to connect to a backbone Ethernet. It's likely to be less expensive to put your diskless clusters on separate subnets than to go to local disks on everything. It's also likely to be easier to administer. (Do you *really* want to have to worry about corrupted file systems due to a power failure on every one of your workstations?) Also, you should take a look at broadcast rates on your networks. You've got a big network using only bridges. Since you have a limited number of vendors, and probably one group responsible for maintaining the stuff, this may be OK, but you're beginning to be near the upper bound of the size network I'd construct without some real routers to keep the broadcast traffic confined. Make sure you don't run rwhod or anything else that routinely broadcasts. Beyond this, your ability to gain bandwidth depends upon how well you can predict who talks to whom. If you can predict that certain groups talk only to certain other groups, then you can try to keep them on a single Ethernet, and use bridges or routers to isolate the groups. The fact that everybody wants to talk to the mainframe may not alone sink you. It's unlikely that any NFS implementation is going to be able to deliver more than the a few Mbits/sec. No matter how many people might potentially want to use your mainframe data, I'll bet you won't overrun an Ethernet. However you will certainly want to be careful how you arrange things so that not much else is going on with that Ethernet. If the mainframe is the main resource that every department is using, then put it directly on your highest level backbone. My intuition is that with some careful thought you can probably survive using just Ethernet technology, by moving some cables around and adding a few bridges or routers at critical locations. You don't have any supercomputers or interactive 24-bit network graphics, so my intuition is that you should be able to survive without FDDI bandwidths for the moment. I understand that you'd feel safer with bridges that could handle most of the bandwidth of an Ethernet. On the other hand, if they can really handle 20 to 30%, that may well be good enough. The whole point of a bridge is to increase overall capacity by separating local and non-local traffic. You'd hope that even on a 100% loaded Ethernet, no more than 20 to 30% of the traffic would be going to destinations off the Ethernet. I'm betting it's going to be about a year before FDDI is practical except for the adventuresome, and it's going to be very expensive at the start.
billp@unet.UUCP (Bill W. Putney) (10/07/88)
In article <Oct.5.22.19.11.1988.27612@athos.rutgers.edu> hedrick@athos.rutgers.edu (Charles Hedrick) writes: > >Also, you should take a look at broadcast rates on your networks. >You've got a big network using only bridges. Since you have a limited >number of vendors, and probably one group responsible for maintaining >the stuff, this may be OK, but you're beginning to be near the upper >bound of the size network I'd construct without some real routers to >keep the broadcast traffic confined. Make sure you don't run rwhod or >anything else that routinely broadcasts. > Just a short note re: my experience with a net much like the one described. I have about 250 Suns (mostly diskless stations), 300 P.C.'s running TCP/IP and about 300 terminals on LS/1's. I have concidered changing some of our bridges to routers because of possible broadcast traffic load. When I recently got a Sniffer and set it up to look only at broadcast/multicast packets I was surprised to find out how little bandwidth and packet rate was attributable to these packet types. I haven't seen the numbers, but I bet routers have more delay and a lower (in general) throughput than would a bridge from the same supplier. I also have a couple of devices around that arn't TCP/IP and MAC layer bridges means never having to say your sorry. -- Bill Putney
ron@ron.rutgers.edu (Ron Natalie) (10/07/88)
> When I > recently got a Sniffer and set it up to look only at broadcast/multicast > packets I was surprised to find out how little bandwidth and packet rate > was attributable to these packet types. Ah, but is that the whole story. Sun had to come up with another policy for rwho on their own inhouse Ethernet for a very interesting reason. What would happen is every three minutes a broadcast rwho packet would go out. This, in itself, is innocuous. Then every diskless client on the net would immediately go page in the rwhod to answer the packet swamping the fileservers. It only takes a few broadcasts to spur some real neat network crashes. You can also try pinging the broadcast address sometime to find out what really is going on on your net. -Ron