sob@harvisr.harvard.edu (Scott Bradner) (10/19/90)
Well here it is, the results of the 3rd round of router testing.
This round consists of local Ethernet to Ethernet routers, all of them I
could get.
This document is copyright 1990 by Scott Bradner.
You can repoduce this posting in any way you want as long as the copy
is complete with no modifications.
A set of files for the slides that I used in the InterOp report are on
husc6.harvard.edu in pub/rtests (the old stuff is in "old"), lots of
nice graphs etc.
Scott Bradner, Harvard University, 33 Kirkland St., Cambridge MA 02138
(617)495-3864, sob@harvard.edu
First some history:
I got a call from Wellfleet back in August wondering if I was going
to go through another round of tests. I said that I did not have the
required equipment, since many of the vendors now had equipment that was
faster than the test setup we had used last year. "We will give you some
equipment to do your tests." he said. "Well..." said I, "how about a loan?",
"Sure" he said. So after some discussion they agreed to loan me some of
the equipment that they were using for in house testing. They also agreeed
to provide access to Terry Bradley who had done all the programming on the
test bed. (I, naturally, wanted some changes in the test software.)
Terry spent quite a while making the changes I requested and getting out a
number of small bugs, and I sent out a call for equipment to be tested. I
got back quite a few devices and herein report the results of the tests.
All but one of the vendors sent a person along with the
router to do setup. (So I would not have to read all those manuals and set
the things up wrong in the end.) The tests for each of the devices were
done over one or two days at a lab at Harvard.
The test setup:
A modified Wellfleet router was used a a test source. This device
had two separate channels of test traffic. Each channel consisted
of one packet source port and one "keep alive" port. The "keep alive"
port sent out groups of frames designed to convince the router that a
node existed on this "net" that spoke some specific protocols. The
supported protocols were:
IP ( keep alive was a ARP response)
DECNET (hello packet)
AppleTalk II (AARP packet)
IPX (IPX-RIP? response)
bridge mode ( a packet with a specific source MAC address)
The device-to-be-tested was attached so that an "input" port
was attached to one of the test frame sources and an "output" port
was attached to the corresponding "keep alive" port. If the device had
four or more ports, both sets of test channels were used.
The modified Wellfleet router generated a stream of test frames
consisting of a specific number of frames, in a selected protocol, of a
selected size and with a selected interframe gap (ifg). One of the test
channels (channel 3) was used for all of the tests and the 2nd
channel was used only for the dual stream test.
test source frame rate:
size theoretical channel "3" channel "4"
64 14880 14489 14549
128 8445 8324 8340
256 4528 4494 4498
512 2349 2340 2341
1024 1197 1195 1195
1518 812 812 812
(The Wellfleet test code can do quite a bit more than I made use of,
it can generate mixed protocol packet streams for example, additional
work needs to be done to create a set of test specifications that better
emulate a "real" data network.)
The counter:
A HP 4972A Lan Protocol Analyzer was used to find the frame rates
and to count the output frames. The HP ran the "stats" program
to monitor the network.
The router configuration:
The router was configured at the start of the test suite and the
configuration was not changed throughout the first set of tests.
The configuration was then altered to add a single "filter"
condition, tests were run on that, then the configuration was
changed to add 9 more filter conditions and tests were run on that.
The bridge tests were configured for and run separately.
The tests:
Note:
The design of these tests comes from the work of the IETF
Benchmarking Methodology Working Group.
single channel:
"delay"
Aim - find delay through router.
A Tektronix 2337 scope was attached to both the input and
output router ports.
A stream of frames with a large ifg was sent through the router.
The scope was used to measure the time delay between the end
of the input frame and the start of the output frame.
"raw rate"
Aim - find out impact of max rate input data rate.
The HP was connected to the "output" port of the router.
A stream of frames of a specific protocol and a specific size
with the minimum ifg, was sent to the input port on
the router. The number of frames was selected to produce about
30 sec of data stream.
The "stats" program was reset after the start of the data stream
and the "10 sec avg" value was watched to get the "raw rate".
"max rate"
Aim - find highest rate at which router passes 100% of input frames.
The HP was connected to the "output" port of the router.
A stream of frames of a specific protocol and a specific size
with a selected inter frame gap, was sent to the input port on
the router. The number of frames was selected to produce about
20 sec of data stream.
The "stats" program was reset before the data stream was started.
The "total frames" value was used to see if all of the offered
frames were forwarded. If not, the ifg was increased, if yes
the ifg was reduced. This process was followed until the point
at which a reduction of "1" in the ifg would not pass all the
frames.
The HP was then attached to the frame source and the test rerun
with the determined ifg to get the frame rate.
"dual stream"
Aim - find effect of more than one data stream through router.
"raw rate" test run but with both frame sources sending frames
with the minimum ifg.
The HP was first connected to one "output", a test was run to
find the rate, then the HP was moved to the other output and the
test rerun.
NOTE: this test is flawed, the "right" way to do this is the same
as for the "max rate" test, putting the same frame rate into both
inputs and counting the outputs. But this was just to much work,
strong need for automation.
Some nomenclature:
"one filter"
permit traffic from ip address 192.32.100.1 to 192.32.200.1
"10 filter"
permit traffic from ip address 192.32.100.11 to 192.32.200.11
permit traffic from ip address 192.32.100.12 to 192.32.200.12
permit traffic from ip address 192.32.100.13 to 192.32.200.13
permit traffic from ip address 192.32.100.14 to 192.32.200.14
permit traffic from ip address 192.32.100.15 to 192.32.200.15
permit traffic from ip address 192.32.100.16 to 192.32.200.16
permit traffic from ip address 192.32.100.17 to 192.32.200.17
permit traffic from ip address 192.32.100.18 to 192.32.200.18
permit traffic from ip address 192.32.100.19 to 192.32.200.19
permit traffic from ip address 192.32.100.1 to 192.32.200.1
"between interface cards" for the single channel tests means
a data stream that goes in a port on one Ethernet interface
card and out a port on a 2nd Ethernet interface card.
"within interface card" for the single stream tests means
a data stream that goes in a port on an Ethernet
interface card and out on another port on the same card.
"between interface cards" for the dual stream tests means two data
streams, both of which come in ports on one Ethernet
interface card and both of which exit from ports on a 2nd
Ethernet interface card.
"within interface card" for the dual stream tests means one data
stream that comes in a port on one Ethernet interface card
and exits via another port on the same card, and a 2nd
stream that enters on one port on a separate Ethernet
interface card and exits via a 2nd port on this separate
card.
The results:
(I have checked these numbers but there may be transcription or
typing errors.)
3com NETBuilder - TCP/IP - within interface card
size theo test ip_max ip_fld
64 14880 14489 6404 6401
128 8445 8324 6648 6553
256 4528 4494 4004 3953
512 2349 2340 2202 2183
1024 1197 1195 1010 1153
1518 812 812 799 792
3com NETBuilder - Bridge Mode - within interface card
size theo test br_max br_fld
64 14880 14489 9830 9750
128 8445 8324 6735 6564
256 4528 4494 3994 3953
512 2349 2340 2202 2183
1024 1197 1195 1159 1153
1518 812 812 805 792
BBN T20 - TCP/IP - within interface card
size theo test ip_max ip_fld 1f_max 1f_fld 10f_max 10f_fld
64 14880 14489 4315 4664 2620 2778 2620 2775
128 8445 8324 4339 4654 2433 2777 2433 2777
256 4528 4494 4493 4493 2231 2772 2231 2774
512 2349 2340 2340 2340 2340 2340 2340 2340
1024 1197 1195 1194 1194 1194 1194 1194 1194
1518 812 812 811 811 811 811 811 811
cisco AGS+ - TCP/IP - between interface cards
size theo test ip_max ip_fld 1f_max 1f_fld 10f_max 10f_fld
64 14880 14489 14488 14488 14488 14488 14488 14488
128 8445 8324 8324 8324 8324 8324 8324 8324
256 4528 4494 4493 4493 4494 4494 4494 4494
512 2349 2340 2340 2340 2340 2340 2340 2340
1024 1197 1195 1194 1194 1195 1195 1195 1195
1518 812 812 811 811 812 812 812 812
cisco AGS+ - AppleTalk II - between interface cards
size theo test at2_max at2_fld
64 14880 14489 14488 14488
128 8445 8324 8324 8324
256 4528 4494 4494 4494
512 2349 2340 2340 2340
cisco AGS+ - DECNET - between interface cards
size theo test dn_max dn_fld
64 14880 14489 13097 13270
128 8445 8324 8324 8324
256 4528 4494 4494 4494
512 2349 2340 2340 2340
1024 1197 1195 1195 1195
1518 812 812 812 812
cisco AGS+ - IPX - between interface cards
size theo test ipx_max ipx_fld
64 14880 14489 14488 14488
128 8445 8324 8324 8324
256 4528 4494 4494 4494
512 2349 2340 2340 2340
1024 1197 1195 1195 1195
1518 812 812 812 812
cisco AGS+ - Bridge Mode - between interface cards
size theo test br_max br_fld
64 14880 14489 14488 14488
128 8445 8324 8324 8324
256 4528 4494 4494 4494
512 2349 2340 2340 2340
1024 1197 1195 1195 1195
1518 812 812 812 812
cisco AGS+ - Dual IP Streams - between interface cards
size theo test1 test2 d1_fld d2_fld
64 14880 14489 14549 9682 9700
128 8445 8324 8340 8324 8340
256 4528 4494 4498 4493 4498
512 2349 2340 2341 2340 2341
1024 1197 1195 1195 1194 1195
1518 812 812 812 812 812
cisco AGS+ - TCP/IP - within interface card
size theo test ip_max ip_fld 1f_max 1f_fld 10f_max 10f_fld
64 14880 14489 14488 14488 14488 14488 14488 14488
128 8445 8324 8324 8324 8324 8324 8324 8324
256 4528 4494 4494 4494 4494 4494 4494 4494
512 2349 2340 2340 2340 2340 2340 2340 2340
1024 1197 1195 1195 1195 1194 1194 1194 1194
1518 812 812 812 812 811 811 811 811
cisco AGS+ - AppleTalk II - within interface card
size theo test at2_max at2_fld
64 14880 14489 14488 14488
128 8445 8324 8324 8324
256 4528 4494 4494 4494
512 2349 2340 2340 2340
cisco AGS+ - DECNET - within interface card
size theo test dn_max dn_fld
64 14880 14489 13097 13256
128 8445 8324 8323 8323
256 4528 4494 4494 4494
512 2349 2340 2340 2340
1024 1197 1195 1195 1195
1518 812 812 812 812
cisco AGS+ - IPX - within interface card
size theo test ipx_max ipx_fld
64 14880 14489 14488 14488
128 8445 8324 8324 8324
256 4528 4494 4494 4494
512 2349 2340 2340 2340
1024 1197 1195 1195 1195
1518 812 812 812 812
cisco AGS+ - Bridge Mode - within interface card
size theo test br_max br_fld
64 14880 14489 14488 14488
128 8445 8324 8324 8324
256 4528 4494 4494 4494
512 2349 2340 2340 2340
1024 1197 1195 1195 1195
1518 812 812 812 812
cisco AGS+ - Dual IP Streams - within interface card
size theo test1 test2 d1_fld d2_fld
64 14880 14489 14549 9689 9698
128 8445 8324 8340 8323 8340
256 4528 4494 4498 4494 4498
512 2349 2340 2341 2340 2341
1024 1197 1195 1195 1194 1195
1518 812 812 812 812 812
Network Systems Corporation EN640-8 - TCP/IP - between interface cards
size theo test ip_max ip_fld 1f_max 1f_fld 10f_max 10f_fld
64 14880 14489 6327 6233 4636 4413 1718 1670
128 8445 8324 5105 5189 4111 3875 1629 1576
256 4528 4494 3755 3752 3745 3756 1599 1536
512 2349 2340 2124 2123 2122 2123 1565 1493
1024 1197 1195 1137 1136 1138 1138 1141 1135
1518 812 812 786 784 787 786 788 784
Network Systems Corporation EN640-8 - AppleTalk II - between interface cards
size theo test at2_max at2_fld
64 14880 14489 827 0
128 8445 8324 808 0
256 4528 4494 770 0
512 2349 2340 702 269
Network Systems Corporation EN640-8 - DECNET - between interface cards
size theo test dn_max dn_fld
64 14880 14489 919 0
128 8445 8324 886 0
256 4528 4494 825 0
512 2349 2340 727 280
1024 1197 1195 588 452
1518 812 812 443 381
Network Systems Corporation EN640-8 - Dual IP Streams - between interface cards
size theo test1 test2 d1_fld d2_fld
64 14880 14489 14549 3224 3396
128 8445 8324 8340 2832 2975
256 4528 4494 4498 2638 2651
512 2349 2340 2341 2335 2123
1024 1197 1195 1195 1138 1195
1518 812 812 812 786 812
Network Systems Corporation EN640-8 - TCP/IP - within interface card
size theo test ip_max ip_fld 1f_max 1f_fld 10f_max 10f_fld
64 14880 14489 5986 6604 4775 4630 1732 1702
128 8445 8324 5259 5732 4605 4164 1662 1629
256 4528 4494 3727 3757 3729 3641 1541 1523
512 2349 2340 2128 2123 2122 2123 1508 1450
1024 1197 1195 1140 1138 1139 1135 1139 1135
1518 812 812 788 784 788 783 780 782
Network Systems Corporation EN640-8 - AppleTalk II - within interface card
size theo test at2_max at2_fld
64 14880 14489 828 0
128 8445 8324 805 0
256 4528 4494 767 0
512 2349 2340 701 266
Network Systems Corporation EN640-8 - DECNET - within interface card
size theo test dn_max dn_fld
64 14880 14489 919 0
128 8445 8324 885 0
256 4528 4494 826 0
512 2349 2340 719 278
1024 1197 1195 580 453
1518 812 812 439 381
Novell NetWare 386 - TCP/IP - between interface cards
size theo test ip_max ip_fld
64 14880 14489 3275 3270
128 8445 8324 3207 3202
256 4528 4494 2892 2856
512 2349 2340 1822 1764
1024 1197 1195 1050 1030
1518 812 812 744 731
Novell NetWare 386 - IPX - between interface cards
size theo test ipx_max ipx_fld
64 14880 14489 3295 3240
128 8445 8324 3274 3172
256 4528 4494 2953 2852
512 2349 2340 1861 1768
1024 1197 1195 1078 1031
1518 812 812 787 731
Proteon P4200 - TCP/IP - between interface cards
size theo test ip_max ip_fld 1f_max 1f_fld 10f_max 10f_fld
64 14880 14489 3616 3576 2675 2054 2675 2050
128 8445 8324 3081 3058 2247 1674 2258 1668
256 4528 4494 1667 1529 1493 1253 1493 1248
512 2349 2340 948 896 875 815 876 814
1024 1197 1195 508 473 465 447 465 447
1518 812 812 348 325 324 313 325 313
Proteon P4200 - DECNET - between interface cards
size theo test dn_max dn_fld
64 14880 14489 1747 1469
128 8445 8324 1595 1453
256 4528 4494 1505 1401
512 2349 2340 968 804
1024 1197 1195 539 452
1518 812 812 405 310
Proteon P4200 - IPX - between interface cards
size theo test ipx_max ipx_fld
64 14880 14489 1881 1593
128 8445 8324 1718 1560
256 4528 4494 1608 1503
512 2349 2340 970 812
1024 1197 1195 519 450
1518 812 812 411 310
Proteon P4200 - Dual IP Streams - between interface cards
size theo test1 test2 d1_fld d2_fld
64 14880 14489 14549 1085 1181
128 8445 8324 8340 U U
256 4528 4494 4498 825 833
512 2349 2340 2341 665 1107
1024 1197 1195 1195 1109 665
1518 812 812 812 270 433
Proteon rig - TCP/IP - between interface cards
size theo test ip_max ip_fld 1f_max 1f_fld 10f_max 10f_fld
64 14880 14489 12802 12142 4622 4513 4605 4530
128 8445 8324 7896 7998 4233 4212 4222 4204
256 4528 4494 4351 4350 2964 2944 2964 2943
512 2349 2340 2287 2294 1850 1835 1850 1837
1024 1197 1195 1186 1181 1060 1047 1060 1048
1518 812 812 812 805 749 741 749 742
Proteon rig - Dual IP Streams - between interface cards
size theo test1 test2 d1_fld d2_fld
64 14880 14489 14549 8516 7749
128 8445 8324 8340 8027 6989
256 4528 4494 4498 4494 3936
512 2349 2340 2341 2333 4098
1024 1197 1195 1195 1193 1192
1518 812 812 812 810 809
Timeplex TIME/LAN 100 - TCP/IP - between interface cards
size theo test ip_max ip_fld
64 14880 14489 5480 4822
128 8445 8324 4865 4162
256 4528 4494 3287 3253
512 2349 2340 1969 1949
1024 1197 1195 977 973
1518 812 812 691 687
Wellfleet Link Node - TCP/IP - between interface cards
size theo test ip_max ip_fld 1f_max 1f_fld 10f_max 10f_fld
64 14880 14489 14473 14473 11107 10778 9708 9998
128 8445 8324 8322 8322 8322 8322 8324 8324
256 4528 4494 4493 4493 4494 4494 4494 4494
512 2349 2340 2340 2340 2340 2340 2340 2340
1024 1197 1195 1196 1196 1193 1193 1195 1195
1518 812 812 811 811 811 811 812 812
Wellfleet Link Node - AppleTalk II - between interface cards
size theo test at2_max at2_fld
64 14880 14489 12676 13629
128 8445 8324 8324 8324
256 4528 4494 4493 4493
512 2349 2340 2341 2341
Wellfleet Link Node - DECNET - between interface cards
size theo test dn_max dn_fld
64 14880 14489 10151 10404
128 8445 8324 8322 8322
256 4528 4494 4494 4494
512 2349 2340 2340 2340
1024 1197 1195 1194 1194
1518 812 812 812 812
Wellfleet Link Node - IPX - between interface cards
size theo test ipx_max ipx_fld
64 14880 14489 10420 10642
128 8445 8324 8324 8324
256 4528 4494 4493 4493
512 2349 2340 2340 2340
1024 1197 1195 1194 1194
1518 812 812 812 812
Wellfleet Link Node - Bridge Mode - between interface cards
size theo test br_max br_fld
64 14880 14489 14488 14488
128 8445 8324 8324 8324
256 4528 4494 4494 4494
512 2349 2340 2340 2340
1024 1197 1195 1195 1195
1518 812 812 812 812
Wellfleet Link Node - Dual IP Streams - between interface cards
size theo test1 test2 d1_fld d2_fld
64 14880 14489 14549 7790 7780
128 8445 8324 8340 7186 7196
256 4528 4494 4498 4491 4493
512 2349 2340 2341 2340 2339
1024 1197 1195 1195 1194 1194
1518 812 812 812 811 811
Wellfleet Link Node - TCP/IP - within interface card
size theo test ip_max ip_fld 1f_max 1f_fld 1f_max 1f_fld
64 14880 14489 11086 12646 8983 9636 8930 9655
128 8445 8324 8322 8322 8324 8324 8324 8324
256 4528 4494 4493 4493 4494 4494 4494 4494
512 2349 2340 2340 2340 2340 2340 2340 2340
1024 1197 1195 1196 1196 1193 1193 1193 1193
1518 812 812 811 811 811 811 811 811
Wellfleet Link Node - AppleTalk II - within interface card
size theo test at2_max at2_fld
64 14880 14489 10707 11390
128 8445 8324 8322 8322
256 4528 4494 4493 4493
512 2349 2340 2339 2339
Wellfleet Link Node - DECNET - within interface card
size theo test dn_max dn_fld
64 14880 14489 8634 8982
128 8445 8324 8322 8322
256 4528 4494 4494 4494
512 2349 2340 2340 2340
1024 1197 1195 1193 1193
1518 812 812 811 811
Wellfleet Link Node - IPX - within interface card
size theo test ipx_max ipx_fld
64 14880 14489 8780 9109
128 8445 8324 8322 8322
256 4528 4494 4494 4494
512 2349 2340 2340 2340
1024 1197 1195 1194 1194
1518 812 812 811 811
Wellfleet Link Node - Bridge Mode - within interface card
size theo test br_max br_fld
64 14880 14489 14488 14488
128 8445 8324 8324 8324
256 4528 4494 4494 4494
512 2349 2340 2340 2340
1024 1197 1195 1195 1195
1518 812 812 812 812
Wellfleet Link Node - Dual IP Streams - within interface card
size theo test1 test2 d1_fld d2_fld
64 14880 14489 14549 12652 13024
128 8445 8324 8340 8324 8337
256 4528 4494 4498 4491 4497
512 2349 2340 2341 2334 2341
1024 1197 1195 1195 1195 1195
1518 812 812 812 811 811