sob@harvisr.harvard.edu (Scott Bradner) (10/19/90)
Well here it is, the results of the 3rd round of router testing. This round consists of local Ethernet to Ethernet routers, all of them I could get. This document is copyright 1990 by Scott Bradner. You can repoduce this posting in any way you want as long as the copy is complete with no modifications. A set of files for the slides that I used in the InterOp report are on husc6.harvard.edu in pub/rtests (the old stuff is in "old"), lots of nice graphs etc. Scott Bradner, Harvard University, 33 Kirkland St., Cambridge MA 02138 (617)495-3864, sob@harvard.edu First some history: I got a call from Wellfleet back in August wondering if I was going to go through another round of tests. I said that I did not have the required equipment, since many of the vendors now had equipment that was faster than the test setup we had used last year. "We will give you some equipment to do your tests." he said. "Well..." said I, "how about a loan?", "Sure" he said. So after some discussion they agreed to loan me some of the equipment that they were using for in house testing. They also agreeed to provide access to Terry Bradley who had done all the programming on the test bed. (I, naturally, wanted some changes in the test software.) Terry spent quite a while making the changes I requested and getting out a number of small bugs, and I sent out a call for equipment to be tested. I got back quite a few devices and herein report the results of the tests. All but one of the vendors sent a person along with the router to do setup. (So I would not have to read all those manuals and set the things up wrong in the end.) The tests for each of the devices were done over one or two days at a lab at Harvard. The test setup: A modified Wellfleet router was used a a test source. This device had two separate channels of test traffic. Each channel consisted of one packet source port and one "keep alive" port. The "keep alive" port sent out groups of frames designed to convince the router that a node existed on this "net" that spoke some specific protocols. The supported protocols were: IP ( keep alive was a ARP response) DECNET (hello packet) AppleTalk II (AARP packet) IPX (IPX-RIP? response) bridge mode ( a packet with a specific source MAC address) The device-to-be-tested was attached so that an "input" port was attached to one of the test frame sources and an "output" port was attached to the corresponding "keep alive" port. If the device had four or more ports, both sets of test channels were used. The modified Wellfleet router generated a stream of test frames consisting of a specific number of frames, in a selected protocol, of a selected size and with a selected interframe gap (ifg). One of the test channels (channel 3) was used for all of the tests and the 2nd channel was used only for the dual stream test. test source frame rate: size theoretical channel "3" channel "4" 64 14880 14489 14549 128 8445 8324 8340 256 4528 4494 4498 512 2349 2340 2341 1024 1197 1195 1195 1518 812 812 812 (The Wellfleet test code can do quite a bit more than I made use of, it can generate mixed protocol packet streams for example, additional work needs to be done to create a set of test specifications that better emulate a "real" data network.) The counter: A HP 4972A Lan Protocol Analyzer was used to find the frame rates and to count the output frames. The HP ran the "stats" program to monitor the network. The router configuration: The router was configured at the start of the test suite and the configuration was not changed throughout the first set of tests. The configuration was then altered to add a single "filter" condition, tests were run on that, then the configuration was changed to add 9 more filter conditions and tests were run on that. The bridge tests were configured for and run separately. The tests: Note: The design of these tests comes from the work of the IETF Benchmarking Methodology Working Group. single channel: "delay" Aim - find delay through router. A Tektronix 2337 scope was attached to both the input and output router ports. A stream of frames with a large ifg was sent through the router. The scope was used to measure the time delay between the end of the input frame and the start of the output frame. "raw rate" Aim - find out impact of max rate input data rate. The HP was connected to the "output" port of the router. A stream of frames of a specific protocol and a specific size with the minimum ifg, was sent to the input port on the router. The number of frames was selected to produce about 30 sec of data stream. The "stats" program was reset after the start of the data stream and the "10 sec avg" value was watched to get the "raw rate". "max rate" Aim - find highest rate at which router passes 100% of input frames. The HP was connected to the "output" port of the router. A stream of frames of a specific protocol and a specific size with a selected inter frame gap, was sent to the input port on the router. The number of frames was selected to produce about 20 sec of data stream. The "stats" program was reset before the data stream was started. The "total frames" value was used to see if all of the offered frames were forwarded. If not, the ifg was increased, if yes the ifg was reduced. This process was followed until the point at which a reduction of "1" in the ifg would not pass all the frames. The HP was then attached to the frame source and the test rerun with the determined ifg to get the frame rate. "dual stream" Aim - find effect of more than one data stream through router. "raw rate" test run but with both frame sources sending frames with the minimum ifg. The HP was first connected to one "output", a test was run to find the rate, then the HP was moved to the other output and the test rerun. NOTE: this test is flawed, the "right" way to do this is the same as for the "max rate" test, putting the same frame rate into both inputs and counting the outputs. But this was just to much work, strong need for automation. Some nomenclature: "one filter" permit traffic from ip address 192.32.100.1 to 192.32.200.1 "10 filter" permit traffic from ip address 192.32.100.11 to 192.32.200.11 permit traffic from ip address 192.32.100.12 to 192.32.200.12 permit traffic from ip address 192.32.100.13 to 192.32.200.13 permit traffic from ip address 192.32.100.14 to 192.32.200.14 permit traffic from ip address 192.32.100.15 to 192.32.200.15 permit traffic from ip address 192.32.100.16 to 192.32.200.16 permit traffic from ip address 192.32.100.17 to 192.32.200.17 permit traffic from ip address 192.32.100.18 to 192.32.200.18 permit traffic from ip address 192.32.100.19 to 192.32.200.19 permit traffic from ip address 192.32.100.1 to 192.32.200.1 "between interface cards" for the single channel tests means a data stream that goes in a port on one Ethernet interface card and out a port on a 2nd Ethernet interface card. "within interface card" for the single stream tests means a data stream that goes in a port on an Ethernet interface card and out on another port on the same card. "between interface cards" for the dual stream tests means two data streams, both of which come in ports on one Ethernet interface card and both of which exit from ports on a 2nd Ethernet interface card. "within interface card" for the dual stream tests means one data stream that comes in a port on one Ethernet interface card and exits via another port on the same card, and a 2nd stream that enters on one port on a separate Ethernet interface card and exits via a 2nd port on this separate card. The results: (I have checked these numbers but there may be transcription or typing errors.) 3com NETBuilder - TCP/IP - within interface card size theo test ip_max ip_fld 64 14880 14489 6404 6401 128 8445 8324 6648 6553 256 4528 4494 4004 3953 512 2349 2340 2202 2183 1024 1197 1195 1010 1153 1518 812 812 799 792 3com NETBuilder - Bridge Mode - within interface card size theo test br_max br_fld 64 14880 14489 9830 9750 128 8445 8324 6735 6564 256 4528 4494 3994 3953 512 2349 2340 2202 2183 1024 1197 1195 1159 1153 1518 812 812 805 792 BBN T20 - TCP/IP - within interface card size theo test ip_max ip_fld 1f_max 1f_fld 10f_max 10f_fld 64 14880 14489 4315 4664 2620 2778 2620 2775 128 8445 8324 4339 4654 2433 2777 2433 2777 256 4528 4494 4493 4493 2231 2772 2231 2774 512 2349 2340 2340 2340 2340 2340 2340 2340 1024 1197 1195 1194 1194 1194 1194 1194 1194 1518 812 812 811 811 811 811 811 811 cisco AGS+ - TCP/IP - between interface cards size theo test ip_max ip_fld 1f_max 1f_fld 10f_max 10f_fld 64 14880 14489 14488 14488 14488 14488 14488 14488 128 8445 8324 8324 8324 8324 8324 8324 8324 256 4528 4494 4493 4493 4494 4494 4494 4494 512 2349 2340 2340 2340 2340 2340 2340 2340 1024 1197 1195 1194 1194 1195 1195 1195 1195 1518 812 812 811 811 812 812 812 812 cisco AGS+ - AppleTalk II - between interface cards size theo test at2_max at2_fld 64 14880 14489 14488 14488 128 8445 8324 8324 8324 256 4528 4494 4494 4494 512 2349 2340 2340 2340 cisco AGS+ - DECNET - between interface cards size theo test dn_max dn_fld 64 14880 14489 13097 13270 128 8445 8324 8324 8324 256 4528 4494 4494 4494 512 2349 2340 2340 2340 1024 1197 1195 1195 1195 1518 812 812 812 812 cisco AGS+ - IPX - between interface cards size theo test ipx_max ipx_fld 64 14880 14489 14488 14488 128 8445 8324 8324 8324 256 4528 4494 4494 4494 512 2349 2340 2340 2340 1024 1197 1195 1195 1195 1518 812 812 812 812 cisco AGS+ - Bridge Mode - between interface cards size theo test br_max br_fld 64 14880 14489 14488 14488 128 8445 8324 8324 8324 256 4528 4494 4494 4494 512 2349 2340 2340 2340 1024 1197 1195 1195 1195 1518 812 812 812 812 cisco AGS+ - Dual IP Streams - between interface cards size theo test1 test2 d1_fld d2_fld 64 14880 14489 14549 9682 9700 128 8445 8324 8340 8324 8340 256 4528 4494 4498 4493 4498 512 2349 2340 2341 2340 2341 1024 1197 1195 1195 1194 1195 1518 812 812 812 812 812 cisco AGS+ - TCP/IP - within interface card size theo test ip_max ip_fld 1f_max 1f_fld 10f_max 10f_fld 64 14880 14489 14488 14488 14488 14488 14488 14488 128 8445 8324 8324 8324 8324 8324 8324 8324 256 4528 4494 4494 4494 4494 4494 4494 4494 512 2349 2340 2340 2340 2340 2340 2340 2340 1024 1197 1195 1195 1195 1194 1194 1194 1194 1518 812 812 812 812 811 811 811 811 cisco AGS+ - AppleTalk II - within interface card size theo test at2_max at2_fld 64 14880 14489 14488 14488 128 8445 8324 8324 8324 256 4528 4494 4494 4494 512 2349 2340 2340 2340 cisco AGS+ - DECNET - within interface card size theo test dn_max dn_fld 64 14880 14489 13097 13256 128 8445 8324 8323 8323 256 4528 4494 4494 4494 512 2349 2340 2340 2340 1024 1197 1195 1195 1195 1518 812 812 812 812 cisco AGS+ - IPX - within interface card size theo test ipx_max ipx_fld 64 14880 14489 14488 14488 128 8445 8324 8324 8324 256 4528 4494 4494 4494 512 2349 2340 2340 2340 1024 1197 1195 1195 1195 1518 812 812 812 812 cisco AGS+ - Bridge Mode - within interface card size theo test br_max br_fld 64 14880 14489 14488 14488 128 8445 8324 8324 8324 256 4528 4494 4494 4494 512 2349 2340 2340 2340 1024 1197 1195 1195 1195 1518 812 812 812 812 cisco AGS+ - Dual IP Streams - within interface card size theo test1 test2 d1_fld d2_fld 64 14880 14489 14549 9689 9698 128 8445 8324 8340 8323 8340 256 4528 4494 4498 4494 4498 512 2349 2340 2341 2340 2341 1024 1197 1195 1195 1194 1195 1518 812 812 812 812 812 Network Systems Corporation EN640-8 - TCP/IP - between interface cards size theo test ip_max ip_fld 1f_max 1f_fld 10f_max 10f_fld 64 14880 14489 6327 6233 4636 4413 1718 1670 128 8445 8324 5105 5189 4111 3875 1629 1576 256 4528 4494 3755 3752 3745 3756 1599 1536 512 2349 2340 2124 2123 2122 2123 1565 1493 1024 1197 1195 1137 1136 1138 1138 1141 1135 1518 812 812 786 784 787 786 788 784 Network Systems Corporation EN640-8 - AppleTalk II - between interface cards size theo test at2_max at2_fld 64 14880 14489 827 0 128 8445 8324 808 0 256 4528 4494 770 0 512 2349 2340 702 269 Network Systems Corporation EN640-8 - DECNET - between interface cards size theo test dn_max dn_fld 64 14880 14489 919 0 128 8445 8324 886 0 256 4528 4494 825 0 512 2349 2340 727 280 1024 1197 1195 588 452 1518 812 812 443 381 Network Systems Corporation EN640-8 - Dual IP Streams - between interface cards size theo test1 test2 d1_fld d2_fld 64 14880 14489 14549 3224 3396 128 8445 8324 8340 2832 2975 256 4528 4494 4498 2638 2651 512 2349 2340 2341 2335 2123 1024 1197 1195 1195 1138 1195 1518 812 812 812 786 812 Network Systems Corporation EN640-8 - TCP/IP - within interface card size theo test ip_max ip_fld 1f_max 1f_fld 10f_max 10f_fld 64 14880 14489 5986 6604 4775 4630 1732 1702 128 8445 8324 5259 5732 4605 4164 1662 1629 256 4528 4494 3727 3757 3729 3641 1541 1523 512 2349 2340 2128 2123 2122 2123 1508 1450 1024 1197 1195 1140 1138 1139 1135 1139 1135 1518 812 812 788 784 788 783 780 782 Network Systems Corporation EN640-8 - AppleTalk II - within interface card size theo test at2_max at2_fld 64 14880 14489 828 0 128 8445 8324 805 0 256 4528 4494 767 0 512 2349 2340 701 266 Network Systems Corporation EN640-8 - DECNET - within interface card size theo test dn_max dn_fld 64 14880 14489 919 0 128 8445 8324 885 0 256 4528 4494 826 0 512 2349 2340 719 278 1024 1197 1195 580 453 1518 812 812 439 381 Novell NetWare 386 - TCP/IP - between interface cards size theo test ip_max ip_fld 64 14880 14489 3275 3270 128 8445 8324 3207 3202 256 4528 4494 2892 2856 512 2349 2340 1822 1764 1024 1197 1195 1050 1030 1518 812 812 744 731 Novell NetWare 386 - IPX - between interface cards size theo test ipx_max ipx_fld 64 14880 14489 3295 3240 128 8445 8324 3274 3172 256 4528 4494 2953 2852 512 2349 2340 1861 1768 1024 1197 1195 1078 1031 1518 812 812 787 731 Proteon P4200 - TCP/IP - between interface cards size theo test ip_max ip_fld 1f_max 1f_fld 10f_max 10f_fld 64 14880 14489 3616 3576 2675 2054 2675 2050 128 8445 8324 3081 3058 2247 1674 2258 1668 256 4528 4494 1667 1529 1493 1253 1493 1248 512 2349 2340 948 896 875 815 876 814 1024 1197 1195 508 473 465 447 465 447 1518 812 812 348 325 324 313 325 313 Proteon P4200 - DECNET - between interface cards size theo test dn_max dn_fld 64 14880 14489 1747 1469 128 8445 8324 1595 1453 256 4528 4494 1505 1401 512 2349 2340 968 804 1024 1197 1195 539 452 1518 812 812 405 310 Proteon P4200 - IPX - between interface cards size theo test ipx_max ipx_fld 64 14880 14489 1881 1593 128 8445 8324 1718 1560 256 4528 4494 1608 1503 512 2349 2340 970 812 1024 1197 1195 519 450 1518 812 812 411 310 Proteon P4200 - Dual IP Streams - between interface cards size theo test1 test2 d1_fld d2_fld 64 14880 14489 14549 1085 1181 128 8445 8324 8340 U U 256 4528 4494 4498 825 833 512 2349 2340 2341 665 1107 1024 1197 1195 1195 1109 665 1518 812 812 812 270 433 Proteon rig - TCP/IP - between interface cards size theo test ip_max ip_fld 1f_max 1f_fld 10f_max 10f_fld 64 14880 14489 12802 12142 4622 4513 4605 4530 128 8445 8324 7896 7998 4233 4212 4222 4204 256 4528 4494 4351 4350 2964 2944 2964 2943 512 2349 2340 2287 2294 1850 1835 1850 1837 1024 1197 1195 1186 1181 1060 1047 1060 1048 1518 812 812 812 805 749 741 749 742 Proteon rig - Dual IP Streams - between interface cards size theo test1 test2 d1_fld d2_fld 64 14880 14489 14549 8516 7749 128 8445 8324 8340 8027 6989 256 4528 4494 4498 4494 3936 512 2349 2340 2341 2333 4098 1024 1197 1195 1195 1193 1192 1518 812 812 812 810 809 Timeplex TIME/LAN 100 - TCP/IP - between interface cards size theo test ip_max ip_fld 64 14880 14489 5480 4822 128 8445 8324 4865 4162 256 4528 4494 3287 3253 512 2349 2340 1969 1949 1024 1197 1195 977 973 1518 812 812 691 687 Wellfleet Link Node - TCP/IP - between interface cards size theo test ip_max ip_fld 1f_max 1f_fld 10f_max 10f_fld 64 14880 14489 14473 14473 11107 10778 9708 9998 128 8445 8324 8322 8322 8322 8322 8324 8324 256 4528 4494 4493 4493 4494 4494 4494 4494 512 2349 2340 2340 2340 2340 2340 2340 2340 1024 1197 1195 1196 1196 1193 1193 1195 1195 1518 812 812 811 811 811 811 812 812 Wellfleet Link Node - AppleTalk II - between interface cards size theo test at2_max at2_fld 64 14880 14489 12676 13629 128 8445 8324 8324 8324 256 4528 4494 4493 4493 512 2349 2340 2341 2341 Wellfleet Link Node - DECNET - between interface cards size theo test dn_max dn_fld 64 14880 14489 10151 10404 128 8445 8324 8322 8322 256 4528 4494 4494 4494 512 2349 2340 2340 2340 1024 1197 1195 1194 1194 1518 812 812 812 812 Wellfleet Link Node - IPX - between interface cards size theo test ipx_max ipx_fld 64 14880 14489 10420 10642 128 8445 8324 8324 8324 256 4528 4494 4493 4493 512 2349 2340 2340 2340 1024 1197 1195 1194 1194 1518 812 812 812 812 Wellfleet Link Node - Bridge Mode - between interface cards size theo test br_max br_fld 64 14880 14489 14488 14488 128 8445 8324 8324 8324 256 4528 4494 4494 4494 512 2349 2340 2340 2340 1024 1197 1195 1195 1195 1518 812 812 812 812 Wellfleet Link Node - Dual IP Streams - between interface cards size theo test1 test2 d1_fld d2_fld 64 14880 14489 14549 7790 7780 128 8445 8324 8340 7186 7196 256 4528 4494 4498 4491 4493 512 2349 2340 2341 2340 2339 1024 1197 1195 1195 1194 1194 1518 812 812 812 811 811 Wellfleet Link Node - TCP/IP - within interface card size theo test ip_max ip_fld 1f_max 1f_fld 1f_max 1f_fld 64 14880 14489 11086 12646 8983 9636 8930 9655 128 8445 8324 8322 8322 8324 8324 8324 8324 256 4528 4494 4493 4493 4494 4494 4494 4494 512 2349 2340 2340 2340 2340 2340 2340 2340 1024 1197 1195 1196 1196 1193 1193 1193 1193 1518 812 812 811 811 811 811 811 811 Wellfleet Link Node - AppleTalk II - within interface card size theo test at2_max at2_fld 64 14880 14489 10707 11390 128 8445 8324 8322 8322 256 4528 4494 4493 4493 512 2349 2340 2339 2339 Wellfleet Link Node - DECNET - within interface card size theo test dn_max dn_fld 64 14880 14489 8634 8982 128 8445 8324 8322 8322 256 4528 4494 4494 4494 512 2349 2340 2340 2340 1024 1197 1195 1193 1193 1518 812 812 811 811 Wellfleet Link Node - IPX - within interface card size theo test ipx_max ipx_fld 64 14880 14489 8780 9109 128 8445 8324 8322 8322 256 4528 4494 4494 4494 512 2349 2340 2340 2340 1024 1197 1195 1194 1194 1518 812 812 811 811 Wellfleet Link Node - Bridge Mode - within interface card size theo test br_max br_fld 64 14880 14489 14488 14488 128 8445 8324 8324 8324 256 4528 4494 4494 4494 512 2349 2340 2340 2340 1024 1197 1195 1195 1195 1518 812 812 812 812 Wellfleet Link Node - Dual IP Streams - within interface card size theo test1 test2 d1_fld d2_fld 64 14880 14489 14549 12652 13024 128 8445 8324 8340 8324 8337 256 4528 4494 4498 4491 4497 512 2349 2340 2341 2334 2341 1024 1197 1195 1195 1195 1195 1518 812 812 812 811 811