narten@cs.purdue.EDU (Thomas Narten) (11/11/88)
In article <30939@think.UUCP> barmar@think.COM (Barry Margolin) writes: >I am considering modifying an Ethernet implementation (specifically, >the Symbolics implementation) to support reception of trailer packets. The idea behind trailers is simple. Let's minimize the amount of data copying that takes place when applications on different machines communicate with each other. In UNIX (and other systems), data is copied when it crosses the boundary between user process space and kernel space. When performing bulk data transfer, packet headers are small relative to the data, and data rather than header copying dominates the cost of packet processing. If your machine supports virtual memory, it might be possible to eliminate some data copying by switching page table pointers. Specifically, if the data that crosses the kernel/process boundary starts on a memory frame boundary, and the date completely fills the frame, the OS could remap page table entries instead of copying data. Page table twiddling is semantically equivalent to copying so long as the data covers a complete frame. Programs can take steps to issue write operations that operate on frame-aligned, frame-sized amounts of data. With library procedures, this is fairly easy to do. To permit the reverse case -- copy data from OS to user space -- special steps are required to force the data portion of an incoming packet to reside on a frame boundary. By default, the data portion of incoming packets won't be alligned properly. The trailer protocol attempts to insure that data will be aligned properly for page table twiddling. In trailer packets, the 14 byte ethernet header is followed by the *user* data. Any other headers (e.g. IP,TCP) are appended at the end of the packet. Device drivers are set up in such a way that the network device DMA's incoming packets into a memory location beginning 14 bytes to the left of the start of a memory frame. Thus, the start of "data" (from the ethernet point of vew) begins on a frame boundary. On receipt of a trailer packet, the device driver rearranges the packet contents so the header logically appears before the data portion. This may involve copying the header to the beginning of the packet, but most importantly, the *data* part does not move. If an mbuf-like data structure is used, no copying is needed. In any case, the header is small relative to the data. As has been pointed out before on this list, the value of trailers is dubious at best. The key ideas behind trailers are: - remap page table entries instead of copying - arrange data in such a way that user data is aligned on frame boundaries - transfer user data in multiples of the machine frame size - the network carries frame-sized chunks of data On VAXen, frame sizes are small (.5k). On Suns, they are 8K. Thus, Suns would need to transfer 8k chunks of data in a single packet to get any benefit. Ethernets limit packets to 1.5k. Thus, Suns get *no* benefit from trailers. Other new machine architectures favor large frame sizes If your systems emphasize one protocol family (e.g. TCP/IP), your system can get the same performance gain without having to actually use trailer encapsulation. One simply directs the network device driver to assume that most packets will have OFFSET = 14+20+20 bytes of headers (Ethernet/IP/TCP) prepended to the data. The driver then arranges to DMA incoming packets starting at memory location -OFFSET from the begining of a memory frame. Even if the performance win is big (I know of no hard evidence that it is), each machine must have smarts that allow it to determine which other machines on the LAN run trailers and which do not. If one machine uses trailers to communicate with a machine that does not, things just don't work. This is one of famous network features that (like broadcast storms), each site seems to re-discover independently, after several hours of hair pulling. The relevent trailer question is not "how does one implement trailers", but "why implement them at all". -- Thomas Narten narten@cs.purdue.edu or {ucbvax,decvax}!purdue!narten
barmar@think.COM (Barry Margolin) (11/12/88)
In article <5380@medusa.cs.purdue.edu> narten@cs.purdue.EDU (Thomas Narten) writes: [Summary of RFC-893 ommitted.] > This may involve copying the header to the beginning of the >packet, but most importantly, the *data* part does not move. That is the crux of my question. How do you copy the header to the beginning of the packet without moving the data to make room? As I said, remapping page tables is not an option, since the Ethernet driver is implemented in a process running in virtual mode on Symbolics machines (packets are just Lisp byte arrays). >The relevent trailer question is not "how does one implement >trailers", but "why implement them at all". That issue is not important to me. The point is, some of our machines like to send trailer packets, and we lose when they try to do it to Symbolics machines. The Suns and 4.3bsd machine use trailer negotiation, but Ultrix requires us to specify "-trailers" in the ifconfig command in /etc/rc.local, and sometimes this gets omitted. Sometimes I'll realize the problem right away, but other times people go for days without figuring it out. I see no reason why the Lispms shouldn't be able to receive anything the Vaxen like to send (within reason). I have NO plans to teach the Lispms how to SEND trailers, though. I see this as an instance of "be liberal in what you accept, conservative in what you send" rule. Barry Margolin Thinking Machines Corp. barmar@think.com {uunet,harvard}!think!barmar
thomas@uplog.se (Thomas Hameenaho) (11/15/88)
In article <31039@think.UUCP> barmar@kulla.think.com.UUCP (Barry Margolin) writes: >That is the crux of my question. How do you copy the header to the >beginning of the packet without moving the data to make room? (cons header packet) :-) The mbuf strategy is actually something along that line. -- Real life: Thomas Hameenaho Email: thomas@uplog.{se,uucp} Snail mail: TeleLOGIC Uppsala AB Phone: +46 18 189406 Box 1218 Fax: +46 18 132039 S - 751 42 Uppsala, Sweden