@DCN7.ARPA:mills@dcn6.arpa (10/18/85)
From: mills@dcn6.arpa Folks, Following is a longish report on current status and plans of my ongoing project to bring up a demonstration of the DoD Internet protocol suite operating at the network layer above AX.25. I first tried to hack the original TAPR-1 firmware to do this in both connection and connectionless modes, but gave up due to ornery problems in transparency and flow control. I next tried the WA8DED firmware and found some joy, pleasantries of which are reported below. The report is rather technical and intended for those with some protocol background and implementation experience, expecially in the areas of connectionless (datagram-oriented) networks and gateways. Overview I rebuilt what has become a monsterous driver for the LSI-11 "fuzzball" system, which is an old grizzly used for network protocol development, testing and performance evaluation and supports just about every protocol sung in the IP/TCP protocol suite. The LSI-11/73 I used for the present purpose (DCN6.ARPA) belongs to me personally and is located in my home along with 30 megabytes of disk, a swamp of radios, towers, a leased digital circuit to the office (hence the Internet) and related mud creatures. The driver supports both the ordinary connection-mode, stream-ASCII operation commonly used to access BB stations, etc., as well as both connection and connectionless modes for Internet Protocol (IP) datagrams and its client Transmission Control Protocol (TCP), upon which application protocols like the TELNET virtual-terminal, FTP file-transfer and SMTP mail-transfer protocols are built. The station complement, consisting of TAPR-1, fuzzball and VHF transceiver, is organized as a full-function Internet gateway between subnet 128.4.1, representing the radio channel, and DCNET (128.4), which is itself gatewayed to the Internet via the ARPANET. It should be noted that no traffic capability between the radio channel and the ARPANET is expressed or implied in this document and that all traffic between the radio channel and any other nets reachable via nongovernment paths (Ford, U Michigan and U Maryland at present) is experimental and strictly noncommercial in nature. The WA8DED Firmware The WA8DED firmware ex box provides one connectionless-mode channel and five connection-mode channels, which can also be operated in connectionless mode. The TAPR is operated in host mode; that is, the fuzzball has to poll for everything, which it does at one-second intervals on all five channels. This handles the flow-control problem from the TAPR to the fuzzball. Since my TAPR is fully populated with memory (about 700 buffers) and TCP manages end-end flows anyway, I didn't bother with flow control from the fuzzball to the TAPR. Host messages to the TAPR consist of the sequence <channel number> <control code> <count> <text> , where <channel number> is an octet in the range 0-4, <control code> is an octet with value 0 for data and 1 for command, <count> is an octet representing the number of octets in <text> reduced by one and <text> is a string of up to 256 octets with no restriction on code combinations. Commands are in ASCII and are interpreted the same way as in ordinary (i.e. non-host) mode, except that the terminating <CR> is not included. No more than one command can be sent in a message and commands may not be split between messages. In the case of IP datagrams, the raw datagram itself is simply encapsulated in the <text> field as-is. TAPR messages to the host have the same format as above, except that <control code> is an octet with values in the range 0-7, indicating success/failure, as well as status information, headers and data. 0 normal response to command (no <count> or <text> field) 1 normal response to command 2 error response to command 3 link status information 5 monitor header 6 monitor data 7 user data In all except codes 0,6-7 the <text> field is <NUL>-terminated. Codes 0-2 occur only in response to commands, while the remainder can occur spontaneously (but only in response to a poll, of course). Codes 5-6 apparently occur only on channel 0, while code 7 occurs only on channels 1-4. This creates problems in connectionless mode as described later. In the case of code 5 the <text> field consists of an internally formatted representation of the AX.25 header, which is followed immediately by the data (if present) as a code 6. Upon initializatin the driver sends the sequence <NAK><ESC>JHOST 1<CR> , which toggles the TAPR into host mode, if not already there, and is grumpily ignored if it is. The polling sequence, which is sent every second, consists of the five commands <0><1><0>G <1><1><0>G <2><1><0>G <3><1><0>G <4><1><0>G , which empties the TAPR queues at a rate limited to one message per second on each channel. When lots of things are going on, such as when monitoring all channel traffic and managing a connection or two, the TAPR queues start to back up. The polling strategy and rate matching can obviously be improved. Connection Management The WA8DED firmware assigns incoming connection requests serially starting at channel 1 and continuing to channel 4 or a limit which can be set by a command. Completion of the handshake sequence results in a link status (code 3) reply on the assigned channel. Outgoing connection requests can be made on any channel as specified by the connect "C" command. In the case of channels 1-4, the request results in the AX.25 connection-setup procedure, while in the case of channel 0, the request specifies the route for subsequent connectionless (UI) frames. If a data message is sent on channel 0 or on any other channel not preceded by a connect command, the data is sent as a UI frame. Connection data (I frames) appear as code-7 replies on the assigned channel and can be interleaved with data from other channels. Connectionless data (UI frames) can apparently be received only as monitoring data when enabled by the monitor "M" command and are preceded by a code-5 message indicating the route and protocol id. The protocol id can be changed by a command, but only on a global basis. An AYT (ARPANETese for "are you there" aka "keep-alive") consisting of the RR supervisory message, is sent every two minutes on every active connection. If the max-retransmission counter is exceeded the TAPR attempts to reset the connection while holding on to the channel queues; however, if that fails the connection is aborted with an appropriate link status message. This can result in delays up to several minutes before the link-level protocol gives up and obviously will result in congestive collapse in retransmissions with conventional transport-level implementations. The W3HCF Fuzzware The TAPR driver constructed for the fuzzball can operate with both ASCII streams or IP datagrams in either connection or connectionless modes. The driver has an interface to the stream-oriented terminal I/O, so that local terminals and remote network (TELNET) terminals can connect via the driver and TAPR to AX.25 stations in the ordinary way. Using this interface, if the first character in a line terminated by <CR><LF> is <ESC>, the line is interpreted as a command, rather than data. If the first character of the command itself is a digit, the selected channel is changed accordingly and remains in effect. In addition, a rather ad-hoc set of editing rules for <CR> and <LF> had to be implemented for compatibility with local BB stations, so this interface is necessarily code sensitive. In the case of IP datagrams, a full-bore IP-address/callsign dynamic mapping cache was implemented using two tables, the Channel Table and the Route Table. The Channel Table latches the current status (state and time-to-live (TTL) counter), last non-data message received and callsign. The callsign is used as the local-network address in the IP model. Following is an example (slightly edited) display captured during normal operation: CID Callsign Status Message ------------------------------------------------------------------------- 0 W3HCF 0 0 5 fm W3IWI to KA3DBK via WB4JFI-5* ctl I17 pid F0 1 W4HCP 0 0 3 (1) DISCONNECTED fm W4HCP via WB4JFI-5 2 0 0 0 3 W3HCF 2 120 0 @PF0 4* 0 0 1 CHANNEL NOT CONNECTED The CID field indicates the channel number, with the asterisk indicating the currently selected one, while the Status field is coded as first the state and then the TTL. The intent is that, if the TTL counts down to zero the channel is closed. The reason for this will be described momentarily. The Message field begins with the <code> field (codes 1-5 only) of the last message from the TAPR followed by the <text> field as-is. as received. A code of zero indicates the last message was a command sent to the TAPR. In the above, channel 2 has never been used. The Route Table is maintained as a cache using mechanisms similar to the Address Resolution Protocol (ARP) used with Ethernets. It consists of an associative array of two-field entries, one field containing the IP address and the other the associated digipeater route. The table is presently handcrafted; however, plans are to dynamically manage the entries as the result of monitored channel activity, perhaps augmented by ARP-stype broadcasts. Obviously, this is where the fun lies in the protocol-development area. Following is an example display of the current version: IP address Route ------------------------------ [128.4.1.1] W3HCF WB4JFI-5 [128.4.1.2] W4HCP WB4JFI-5 [128.4.1.3] WD5DBC WB4JFI-5 [0.0.0.0] W3HCF WB4JFI-5 Note that the contents of the Route field can be used as-is as the argument in a TAPR connect command. In this particular case, the local radio channel is configured as subnet 128.4.1 of net 128.4, which is the swamp in which our own fuzzballs sleaze. Also note the default [0.0.0.0], which matches all other IP addresses, used for loopback and testing. Connection Mode The whole contraption can operate in either connection or connectionless mode. In connection mode it operates like this. Assume no channels are connected and a datagram originates at the fuzzball or arrives from another link destined for 128.4.1.2, previously determined to belong to W4HCP and to be reachable via the WB4JFI-5 digipeater. The Route Table is searched for 128.4.1.2 and yields the route string W4HCP WB4JFI-5. Next the Channel Table is searched for an active channel matching the destination callsign W4HCP in that string, keeping track of the highest-numbered inactive channel in the process and avoiding the monitor channel (0) and the currently selected ASCII channel (4 in the above display). If the match succeeds, the datagram is immediately sent on the associated channel; if not, a connection command specifying the route string is sent on the higest-numbered inactive channel and that channel is marked active. The datagram starting all this is sent following the command and is queued in the TAPR. Note that all datagrams are sent with protocol id CC (hex), which is assigned to the IP network layer in AX.25. Datagrams arriving from the TAPR on an active channel are simply tossed into the normal fuzzball routing algortihm just as if they arrived from other links. A transmitted or received datagram causes the TTL field in the Channel Table entry to be reset to its maximum. The TTL fields of all active channels are decremented once in a while and, if one of them counts down to zero, a disconnect command is sent to the channel, allowing it to be used for possibly other destinations. Note that the AYT mechanism mentioned above is also operative and would normally catch a broken station before the TTL mechanism did. The TTL mechanism is intended primarily as a garbage collector when channels lie dormant for unreasonable periods. Presumably as the result of monitoring activity, amendments are continuously being made to the Route Table. If a connection breaks with datagrams queued, the queue will be discarded and a new connection attempted according to the latest information in the Route Table. The lost datagrams will be recovered through the ordinary retransmission mechanisms of the client transport protocol. Obviously, the key to the success of this scenario lies in the ability to effectively collect the route information and maintain the Route Table. Further development is anticipated in this area. Connectionless Mode In connectionless mode all IP datagrams are sent and received on channel 0. This requires considerable heroics in separating the desired IP traffic from other traffic on the channel, not to mention monitoring headers and data. A datagram originating at the fuzzball or arriving from another link initiates the same operations as in the connection mode, as described above, except that channel 0 is used and it is never marked active. This implies some overhead, since a connection command is always sent preceeding every datagram. It is easy to avoid this by the simple expedient of saving the last argument and suppressing this if the argument is unchanged. At present the only mechanism to capture IP datagrams from the radio channel is by turning on UI-frame monitoring (Ethernetters know this as promiscuous mode) and carefully filtering IP datagrams from the general slosh. The Thursday-night implementation (which should be obsolete by the time you read this) is to toss everything at the IP-header validation routine, which verifies correct IP header format and checksum and with high probability chucks everything but IP datagrams on the floor. Initial Experiments The driver is built so that ordinary monitoring headers and data are sent to the system log file, which allows channel activity to be observed in real time, as well as captured for later analysis. During initial testing it was obvious that a fair number of other protocol implementations active in the local community are defective, inefficient or worse. A report on that will be presented at another time. Specific IP tests were conducted in connection and connectionless mode via the WB4JFI-5 digipeater, which is on a high tower and has a very strong signal in the W3HCF local area. It can then be assumed that almost all packet losses are in the direction from W3HCF to WB4JFI-5 and those due to collisions aggravated by the hidden-terminal problem. Initial experiments used virtual-terminal (TELNET), file-transfer (FTP) and raw-datagram (ICMP Echo) protocols. In general, the tests were successful and demonstrated that the Internet protocols worked handily via the lashup. However, several problems were immediately apparent, among them: 1. The speed of the fuzzball-TAPR connecting line is only 1200 bps, so that nontrivial queuing delays due monitoring and status information place exceptional strain on the TCP retransmission-delay estimation algorithm. The result is an unneccessarily high incidence of TCP retransmissions. Obviously, the cure for this is to hop up the speed, which will have been done before you read this. 2. In connection mode when times get tough the TAPR tries heroically to get the traffic through, which may involve several minutes of retransmissions, resets, etc., during which TCP is retransmitting and possibly abandoning the connection. Even if the traffic does get through, the result is gobs of useless duplicates and end-end ACKs which clutter up the channel. The cure for this is to reduce the max-retransmission limits and short-circuit the reset procedure. 3. I have observed that, on average, between one packet in four and one packet in ten is lost on the local radio channel. This is a high loss rate even for TCP, which is designed for a more lossy environment even than HDLC, let alone AX.25. Casual inspection of the monitoring log shows inspired braindamage on the part of some local AX.25 implementations using excessively large window sizes, which lead to excessive retransmissions at such high loss rates. This problem has nothing intrinsically to do with IP or TCP and can be cured by judicious intervention by the Protocol Police. Conclusions and Plans The "Nagle Algorithm" in conjunction with other refinements developed for the fuzzball TCP implementation was designed specifically to minimize spurious packet traffic and in principal could result in much improved performance. While the packetization and estimation algorithms designed for the present generation of TCP implementations have been designed for an extremely wide range in delays, speeds and loss rates, they have not been optimized for the extreme regime represented by a 1200-bps packet channel with loss rates in the neighborhood of one in four. The fuzzball TCP implementation, which has been designed and heavily instrumented for protocol development, is a useful tool to explore these issues. From a purely engineering perspective the use of ARQ techniques in such a regime is flawed at best and should in any case be augmented by some form of FEC coding. In addition, the use of compression techniques (e.g. the SLIP protocol) should be explored as a mechanism to reduce packet-header overheads. Exploration of these issues may lead to revised packet formats and checksumming procedures. One of the first results expected to come out of this investigation is a requirement for separate checksums on the link-level header (AX.25 address, route and control bits) and data portions of the frame. The expectation is that residual "errors" alleged at the AX.25 level would be corrected by its client while nevertheless preserving the integrity of the routing and framing functions. Probably the most effective realization of connectionless mode in the AX.25 paradigm would involve implementation of at least the IP-datagram support in the TAPR or equivalent. This is not really hard at all and amounts to adding a feature, keyed on the presence of CC in the protocol-id field together with a specific destination callsign, to indicate an IP message to the client host, which should be delivered with AX.25 header intact in order to support dynamic route cacheing as discussed previously. It should be possible to specify a list of such callsigns, which could then be used to implement a multicast/broadcast capability similar to Ethernets. This feature should be completely independent of the monitoring facility. Finally, it will be important in future to implement a filtering mechanism, based on the protocol-id field, which allows the casual user to avoid screen clutter due to unprintable code combinations while monitoring non-vanilla protocols. Dave -------