pat@cs.cornell.edu (Pat Stephenson) (05/03/90)
As Ken has mentioned here before, ISIS V2.0 will include a new option, "bypass" mode. When bypass mode is enabled, communication within groups (the common case) will use new, faster, protocols which bypass (hence the name) the protos process. The purpose of this note is to outline the performance improvements we've seen using these protocols. (Full details of the protocols can be found in a paper we've just written, available via anon ftp from cu-arpa.cs.cornell.edu, file pub/bypass.ps). We've examined two styles of using the system, which we call "RPC" and "streaming". In RPC mode, a process sends a message to the group and waits until it receives a reply from everyone; in streaming mode, a process simply blast messages at the group as fast as possible, without waiting for any replies. The summary is that bypass is giving us a speedup of 3-10. Most remote RPC's are 3 or 4 times faster, local rpc's are 7-10 times faster, and where I could make a streaming comparison, bypass is 4-8 times faster. (The non-bypass system is not really configured to support heavy streaming.) With or without bypass, performance falls off more or less linearly as the number of processors increases. We know of ways to improve these figures further, mainly involving the use of network-level or hardware-level multicasts, and we are experimenting with this. The perceptive will notice one or two slight anomalies in the figures below; we're working on these, too. RPC times are in milliseconds; streaming throughput times are in kbytes/sec or msgs/sec. All broadcasts were with excluded CBCAST, except the RPC's to *every* member of a group, which were done with normal CBCAST. Every member of a group is on a different physical processor (all SUN 3/60's running SUNOS 4.0.3). RPC's were repeated 1000 times. Small packets are 1 byte, medium packets are 1100 bytes, large packets are 7k bytes. ----------------------------------------------------------------------- BYPASS RPC Small Packets Medium Packets Large Packets nreplies = g g-1 g g-1 g g-1 g=1 2.8 2.8 2.8 g=2 13 13 15 15 27 27 g=3 16 16 20 20 38 40 g=4 20 20 25 26 50 52 NON-BYPASS RPC Small Packets Medium Packets Large Packets nreplies = g g-1 g g-1 g g-1 g=1 22 22 31 g=2 54 46 59 49 76 72 g=3 65 56 71 60 91 88 g=4 68 62 78 66 100 97 BYPASS STREAM Small Packets Large Packets (nreplies = 0) msgs/sec throughput/sec (msgs/sec) g=1 1250 7000k (1000) g=2 132 426k (61) g=3 140 211k (30) g=4 131 155k (22) NON-BYPASS STREAM Small Packets Large Packets (nreplies = 0) msgs/sec throughput/sec (msgs/sec) g=1 150 513k(73) g=2 * 105k (15) g=3 * * g=4 * * * Normal mode is not really configured to handle streaming, so these tests did not complete.