ruseng@iesd.dk (Carsten Ruseng Jakobsen) (05/10/89)
Major change of TCP performance when sending packets with sizes close to 4096 and 8192 bytes. While evaluating the performance of SUN-RPC, I detected the TCP protocol slowing dramatical down when sending packets with sizes close to the magical 4096 and 8192 bytes. The tests I ran was a simple RPC-program with a client calling it's server with an array of some specified size. The server routine was a null routine, doing nothing. When the array-size was between (4045 - 4952) and (8090 - 8960) the time used for the communication was 5 - 10 times slower than expected. It was only the TCP protocol that suddenly had this overhead, the UDP protocol seemed unaffected, which makes me suspect the implementation of TCP. I tried to figure out what happened, by monitoring the net with "traffic". I observed that 1] there was NO collisions on the ethernet (10Mbit) 2] there were NO errors detected. 3] none of the cpu's were working very much 4] none of the machines were swapping 5] the number of context-switches weren't very high. 6] #packets/sec was 25% - 50% of what I saw when sending eg 512 bytes. >From a profile of the program it can be seen that it is in the function _write called from write_tcp that 78% of the time is used. In the "normal" cases i expected 10% there. The experiment was performed on 3 sun3/60 workstations running SUNOS-4.0.1. They were totally isolated from anything else in the world. They were not running yp, neither syslog or cron. Except from the operating system, the only processes running was the server and the client processes. The tests I performed was to measure the time used for 100 RPC calls. For each argument size the time for the 100 calls were measured 20 times, and an average calculated. The RPC-call was made using TCP and UDP protocols. Further was it performed with server and client on the same machine, and server and client on two different machines. The client called the server with an argument of N bytes. The results I got was: - -------------------------------------------------------------- argument size| T C P | UDP N | local remote | local remote - -------------------------------------------------------------- 0 | 6,9 7,1 ||| 5,6 5,8 512 | 8,2 9,1 ||| 6,7 7,6 1024 | 10,6 11,6 ||| 7,9 9,1 2048 | 14,9 15,7 ||| 10,7 12,8 4000 | 23,6 24,2 ||| 15,6 20,0 * 4096 |174,5 199,8 ||| 16,1 20,3 8000 | 40,2 40,8 ||| 26,2 34,5 * 8192 |182,8 199,9 ||| 26,8 35,1 | ||| - -----------|---------------------|||------------------------------------- local = server-process and client-process reside on the same machine. remote= server and client reside on 2 different nodes. If you have any suggestions which explains these results, then please *please* e-mail them to me.
sxn@sun.com (Stephen X. Nahm) (06/07/89)
In article <8905101313.AA04157@iesd.dk> in Sun-Spots-Digest: Volume 7, Issue 304, ruseng@iesd.dk (Carsten Ruseng Jakobsen) writes: >While evaluating the performance of SUN-RPC, I detected the TCP protocol >slowing dramatical down when sending packets with sizes close to the >magical 4096 and 8192 bytes. I've emailed Carsten previously, but I'm posting here for the information of those reading this list. The performance anomalies reported are a result of the default buffer sizes used by the RPC library. The buffer sizes can be changed by the user as described below: ----- Begin Included Message ----- >From sxn Wed May 17 12:04:07 1989 To: ruseng@iesd.dk Subject: Re: Elbows in RPC/TCP performance You can fix things by setting the "send_buf_size" and/or "recv_buf_size" parameters of clnttcp_create() and svctcp_create(). For your example, where the client sends big packets and gets back small acks, use: remote = clnttcp_create(&sin, YOURPROG, YOURVERS, &sock, 24000, 0); (where sock can be RPC_ANYSOCK, and sin is set correctly.) For the server side, which is receiving big packets and sending back small acks, use: transp = svctcp_create(RPC_ANYSOCK, 0, 24000); I set up an RPC program that repeated your scenario. Before I made the above changes, I got similar results as you got. After I adjusted the appropriate parameters correctly (and without any setsockopt manipulation), I got: Size: 0 time: 0.94 Seconds Size: 512 time: 0.97 Seconds Size: 1024 time: 1.16 seconds Size: 2048 time: 1.50 Seconds Size: 4000 time: 1.99 Seconds Size: 4096 time: 2.28 seconds Size: 8000 time: 3.37 Seconds Size: 8192 time: 3.81 seconds Size: 8300 time: 3.70 seconds Size: 16300 time: 6.30 Seconds Size: 16392 time: 8.72 seconds Size: 16400 time: 10.54 seconds See rpc(3) in the SunOS manual, or in the RPCSRC 4.0 man directory. Steve Nahm ----- End Included Message ----- -- Steve Nahm sxn@sun.COM or sun!sxn