HEDRICK@RED.RUTGERS.EDU (Charles Hedrick) (08/30/85)
We just got our IP network into a loop. It's clearly my fault, but the problem is subtle enough that I thought it might be useful for me to point it out to others. It is important to understand our network configuration. We have two Ethernets, 128.6.3 and 128.6.4. For readability, I am going to drop the 128.6 and refer to them as networks 3 and 4. They are connected by two gateways. One of them is a Pyramid 90x (4.2 Unix). It is configured to act as a normal IP gateway. Because not all of our systems know about subnetting, it also does the "ARP hack". Suppose host 128.6.4.2 (which I will refer to as 4.2, since I am dropping 128.6) wants to talk to host 3.1. It will send out an ARP: from 4.2, broadcast, who is 3.1? The gateway will see this ARP request. It will check its tables, realize that the sender needs gatewaying to talk to the target, and that it is prepared to act as that gateway. So the gateway will respond from gateway, to 4.2, 3.1 is <gateway's Ethernet address> This is a lie, since it is claiming that its own Ethernet address is 3.1's. But the lie is useful, since it will cause 4.2 to send its packets to the gateway, which will deliver them. The second gateway is an Applitek Ethernet bridge. This is a broadband cable system onto which you can put Ethernet bridges. They are not IP gateways. Instead, they are "transparent" Ethernet-level gateways. Each bridge runs in promiscuous mode, looking at every packet on its Ethernet. When it sees any packet addressed to someone on a different Ethernet, it sends the packet over the broadband to the bridge on the appropriate Ethernet. It makes no changes to the packet. This is totally invisible to the hosts. The hosts think we just have one big Ethernet. Now, consider what happens when 4.2 wants to talk to 3.1. It will send out an ARP from 4.2, broadcast, who is 3.1? The bridge on network 4 will pass this to the bridge on 3, which will then broadcast it. 3.1 will see it, and respond from 3.1 to 4.2, 3.1 is <its Ethernet address> The bridge will pass this packet back to network 4, whose bridge will send it to 4.2. The connection is now set up, and all of the packets will follow this same path. Now for the fun. Consider what will happen when a host wants to talk to another host on the same network: from 4.2, broadcast, who is 4.3? First, 4.3 will see this, and respond from 4.3 to 4.2, 4.3 is <its Ethernet address> However the Applitek bridge will pick up the original broadcast and repeat it on all of the other subnets. (Since the bridges are protocol-independent, they know nothing about Internet addresses. They have no way to know that the ARP will be satisfied locally. Thus they forward all broadcasts to all subnets. This is moderately reasonable.) In particular, it will repeat it on network 3. Our Pyramid gateway will see this request. Since it is an ARP on network 3 looking for a host on network 4, the Pyramid will offer to gateway: from gateway to 4.2, 4.3 is <gateway's Ethernet address on network 3> The Applitek bridge will pick this on on network 3 and pass it back to 4, where it gets sent back to 4.2. If 4.2 is a Unix system, it will believe the last ARP reply that it sees. So we now have an ARP entry: 4.3 gateway's address on network 3 The funny thing is, this may work. Packets destined for 4.3 will be sent to the gateway's other side. The Applitek bridge will see an Ethernet address on the other subnet, and forward it. The gateway will get the packet, and forward it back to network 4, where it will presumably be delivered to the correct place. However the gateway itself is not immune from this problem (at least not with the code I have in it now). When the gateway attempts to talk to 4.3, the ARP packet will again be forwarded to the other subnet, and the gateway itself will respond. Thus the gateway will end up with an ARP table entry containing 4.3 gateway's own address on network 3 At this point we have a loop. Obviously the simplest answer is that as long as the Applitek system is working, we turn off the Pyramid gateway code. However we would like that code to be available as a backup should the Applitek system go down. It begins to appear that we are going to have to check specifically for this sort of situation. However in a complex network topology, it may not be entirely clear who one should and should not be willing to gateway for. It is possible that the real moral is that "transparent" gateways and IP gateways may have trouble coexisting. Let my hasten to point out that both the subnetting implementation and the "ARP hacking" code on the Pyramid are mine. They should not be blamed on either Berkeley or Pyramid. -------
geof@su-shasta.ARPA (08/30/85)
Actually, the real moral of the story is that transparent gateways and transparent gateways have trouble coexisting. If you had been using a real IP gateway (no criticism intended) there would have been no problem. - Geof
root%bostonu.csnet@csnet-relay.arpa (BostonU SysMgr) (08/30/85)
Re: Packet loops caused by dumb broad/base band gateways This is one of the major reasons why at Boston University we chose to put our broad/base band gateways into our 4.2 hosts, we use Ungermann/Bass Net/1 DR11-W (soon SUN also) NIUs. The 4.2 hosts then are just typical gateways and are intelligent about IP and our broadband interfaces are completely analagous to our baseband (eg. they support ARP.) For reliability we have more than one 4.2 host on our baseband with broadband interfaces. Routed (with the bugs removed) takes care of transitions for us, it all works quite smoothly and I have not yet detected it to cause any load. Further, those hosts with broadband interfaces just talk to each other directly under this scheme, rather than routing over ethernets. I was always very suspicious about this mindless gateway approach for this and a few other reasons, like what happens to your broadcast packets? I am sure there is a reasonable solution to this, but utilizing the broadband directly seemed the way to go. If I needed to add PUP or XNS I am pretty sure it would just be a 'typical' job to recognize those packets need to be forwarded in the drivers. If I needed to gateway DECNET, I would tell them too bad, I told you to stay away from it (I am anticipating a response viz a viz the need to support other protocols on the system.) We in fact can give our DECNET hosts 19.2kb synchronous p-p interfaces on our broadband cable which is about all they can expect. -Barry Shein, Boston University
mogul@Navajo (Jeff Mogul) (08/30/85)
Obviously the simplest answer is that as long as the Applitek system
is working, we turn off the Pyramid gateway code. However we would
like that code to be available as a backup should the Applitek system
go down. It begins to appear that we are going to have to check
specifically for this sort of situation. However in a complex network
topology, it may not be entirely clear who one should and should not
be willing to gateway for. It is possible that the real moral is
that "transparent" gateways and IP gateways may have trouble
coexisting.
What is going on here is that the two gateways (transparent and IP)
are based on different models of the world; the transparent gateway
(bridge) assumes that it should act like a piece of wire, i.e. that
the cables connected to it should appear to be electrically connected.
In this model, forwarding the broadcasts (ARP messages) is not only
legal but required.
The IP gateway does not assume this; specifically, the "ARP hack"
assumes that the cables are NOT electrically connected. In this
model, forwarding of undirected broadcasts is a no-no.
I believe the moral is that one should not mix metaphors. If you
don't want to use your Pyramid as a full-time gateway, buy a
gateway box from someone. Scrap the Applitek, or use it to
physically extend a subnet and pray that it doesn't fail.
Question of the day: is it possible to put two Appliteks (or
any other transparent bridges) in parallel, for reasons of
reliability, without getting into a loop? If not, then you
shouldn't expect that putting a bridge in parallel with a
gateway should work any better.