joshua@athertn.Atherton.COM (Flame Bait) (07/18/89)
.\"
.\" Paper comparing commercial RPC systems for speed and dependability.
.\" By Joshua Levy
.\" This is in troff format. On a laser printer it is 8 pages long.
.\" The abstract is human-readable and about 40 lines down.
.\"
.\" To print use tbl(1), me(4), and ?roff(1):
.\" tbl file | nroff -me | col | lpr
.\" or
.\" tbl file | ptroff -me -Plw
.\"
.de (D \" start definition
.(q
.i
..
.de )D \" end definition
.r
.)q
..
.de SH \" section header macro
.sp 1
.sh 1 "\\$1"
..
.\"
.\" initialization
.\"
.po 1.0i
.sz 12
.na
.hy 14
.ll 6.5i
.he ''Comparative Commercial RPCs''
.fo 'NCF'%'\*(td'
.hx
.sv 1i
.(l C
.sz +2
.b "A Comparison of Commercial RPC Systems"
.sz -2
.sp
.i "Joshua Levy*"
.(f
* Author's full addresses:
.br
.in +7
joshua@atherton.com {hpda|decwrl|sun}!athertn!joshua
.br
Atherton Technology, 1333 Bordeaux Dr., Sunnyvale, CA, 94089.
.br
work: 408 734-9822, FAX: 408 744-1607, home: 415 968-3718
.in -7
.)f
.sp .5v
Atherton Technology\(dg
.(f
\(dg Nothing in this document should be interpreted as a promise of
a future feature in any Atherton product.
.)f
.sp 1.5
.i ABSTRACT
.)l
.(q
\fRRPCs (Remote Procedure Calls) represent a convenient and increasingly
common method for distributing systems among networked computers.
This paper compares RPC systems from Apollo, Netwise,
and Sun for speed and dependability.
.sp 0.5
Speed tests showed Sun's and Apollo's RPC products performed
comparably for small packets, but Apollo's system degraded for
increasingly large packets. Netwise's RPC system performed below
either Sun or Apollo in all situations.
Dependability tests showed both Sun (using UDP) and Apollo
performed dependably at machine loads under 5, but Sun lost
dependability at higher loads.
Netwise and Sun (using TCP) are built on reliable protocols,
and so were assumed reliable.
.sp 0.5
The author is not connected with any RPC vendors, and undertook
this comparison to determine which products were suitable
building blocks for Atherton Technology's products. All data was
gathered using the commercially available versions of each vendor's
software as of November 1988.
.)q
.sp 10
.lp
\fB\(co 1989 Joshua Levy\fP
.lp
Permission is granted to distribute this paper in its entirety.
.br
Permission is granted to distribute parts of this paper
as long as this copyright is included and the
the distributing person and/or company does not also distribute a
commercial RPC system. The one exception is the RPC Performance Table
(page 8), which can be distributed separately by RPC system vendors.
.bp
.SH "Introduction"
.pp
RPCs (Remote Procedure Calls) represent a convenient and increasingly
common method for distributing systems among networked computers.
This paper compares RPC systems available from Apollo,
Netwise, and Sun. These RPC systems are compared in
two ways, speed and dependability.
The author wrote and
ran test programs to compare speed and dependability of each product.
.pp
Before gathering data it was assumed, since
each RPC product was solving the same problem, they would have
similar performance and dependability; the major differences would be in
ease of use. This belief was mistaken and quickly corrected by the data.
Slight differences in design goals lead to major differences in
implementation and performance.
.pp
The data for this paper was gathered between November 1988 and
January 1989 using the commonly available versions of each RPC
system. Apollo's RPC was Multivendor version
1.00 and documentation "Revision 00". Netwise's RPC was
version 2.01 and documentation "Version 1.00". Finally,
Sun's RPC was the version shipped with SunOS 3.2 and documentation
"Revision B of 17 February 1986".
.SH "Network Environment"
.pp
Network software is implemented in layers, each layer being built
on the one below with several different protocols available
on each layer. A common configuration has (from lowest to highest)
ethernet, IP, UDP and TCP, and finally, an RPC protocol.\(dg.
.(f
\(dg Network terminology
is full of SLA (Silly Little Acronyms). In fact, I have tried
to minimize their usage in this paper. For example,
Apollo's RPC system could be called NCS, Netwise's data representation system
could be called PDU, while Sun's could be called XDR, etc. \fIAd Nausium\fP.
.)f
.pp
UDP is an unreliable packet oriented protocol, while TCP
is a reliable connection oriented protocol. "Unreliable" means the
a data packet may not arrive, may arrive after a packet sent
before it (ie. in any order), or may arrive more than once; the only
guarantee is that every time a packet arrives, the
data in it is uncorrupted. "Reliable" means data arrives exactly once,
in the right order, and uncorrupted. "Packet oriented" communications
are similar to a letter since each piece of mail is addressed and routed
separately. "Connection oriented" communications are similar to a phone
call, because a connection is established, messages are sent back and
forth, and the connection is broken.
.pp
UDP packets have a maximum size, which varies from machine to machine,
but TCP connections can handle any amount of data. However, in
UNIX, each TCP connection requires a file descriptor, therefore,
an application can have only a limited number of TCP connections
open at any one time. Conversely, all UDP communications are sent
through one file descriptor, allowing an unlimited number of UDP
``connections''.
.SH "Speed"
.(D
How fast can a stub procedure be executed?
.)D
.pp
Timing tests were run on four different RPC configurations: Apollo,
Netwise, Sun (udp) and Sun (tcp). Sun allows the programmer to
specify the transport layer, resulting in different levels of performance.
Therefore, each transport layer was timed separately.
.pp
The analysis was performed on two Sun 3/140s. Each had 8 megabytes
of main memory, running SunOS 3.2 (a 4.2 BSD derivative).
The test program was the only application running on the Sun(s)
during the test. All four sets of software were tested on the
same hardware configuration.
.ig
Sun, for example, ships their UDP with checksumming turned off. This
speeds it up, but also introduces the usually impossible problems of
data corruption within a UDP packet.*
Sun's rational seems to be that, since their UDP is almost always used
on top of ethernet (which does its own checksumming) there is no need for
the UDP protocol to checksum also.
..
.pp
The test is a shell script which repeatedly ran a client program.
This client program made 100 remote procedure
calls, and printed how long it took.
Each time this client ran, its RPCs had different sized arguments, were sent
to different machines, or had a different number of other clients running
in parallel. The clients called a server on the local machine, or
a foreign machine. The size of the arguments passed to the RPC (and the
size of the return
value) was 12, 512, 8K, or 16K bytes. The number of clients
running in parallel was 1, 2, or 4. All 24 combinations of these
parameters were run, using the four different RPC implementations,
Apollo, Netwise, Sun (udp), and Sun (tcp), except that Sun (udp) and
Netwise each have maximum packet sizes (8K and 2K bytes, respectively).
.lp
A table attached to the end of this report summarizes the results of this
timing, specifically:
.np
Sun (udp) is the fastest RPC, when it is usable.
.np
Sun (tcp) is the fastest RPC, when Sun (udp) is not usable.
.sh 2 Analysis
.pp
These differences in performance are understandable when a few facts
are known about (1) which
transport layer each RPC system uses, and (2) how it is used. Apollo
uses UDP, making sure each packet is 1K or less. Sun (udp) uses
UDP, using the maximum packet size*,
.(f
* The Maximum UDP packet size is usually 8K or 2K bytes.
.)f
while Sun (tcp) and Netwise use TCP.
.pp
First, consider the two UDP based systems: Apollo and Sun (udp).
Since both use the same communications layer they should be the same
speed. An explanation for their speed discrepancy is the UDP
packet size they use. Since UDP is unreliable, an RPC system
using it must wait for an acknowledgement that its packet arrived.
This means sending 2 1K byte packets take longer than sending 1 2K byte
packet. The more data which must be sent (up to the maximum UDP
packet size) the worse things will get for the Apollo implementation.
.pp
Second, consider the two TCP based systems: Netwise and Sun (tcp).
Again, since both use the same communications layer they should have the same
speed. One explanation for Netwise's slower performance, is
data conversion (which is fully discussed below in
\fIData Representation\fP). Briefly, Netwise must translate all data
into a standard format (ASN.1). Sun does too; however, Sun uses
MC68000 as their standard data format. So, in fact, Sun did no
data conversion, while Netwise did twice: from MC68000 format to
ASN.1 and back.
.pp
Third, compare UDP and TCP based RPC systems. Setting up and
tearing down a TCP connection requires 6 small packets. Once
the connection is set up, however, TCP can move data more
economically than UDP.
The result: UDP will always be faster for RPC calls which require
a small number of UDP packets to move the arguments. TCP will become
faster as more UDP packets are required. The exact number of UDP packets
which must be sent to make TCP more economical depends on each
protocol's implementation and the type of network being used.
Empirical tests (the results on page 8) show for 8 UDP packets worth
of data (8K bytes), Apollo is slower than Sun (tcp).
.sh 2 "Biases"
.pp
It is not possible to write a totally neutral test program or run
it in a totally neutral environment, since any program and any
environment implicitly helps some RPC systems and hurts others.
These test ran in a common environment,
so the results will have wide applicability. Nevertheless,
this section discusses the particular biases introduced by the test
program and its environment.
.pp
The test program was written so the TCP connection between the
client and server was setup and torn down for each RPC call. This
ensured that the application could never run out of file descriptors,
but it slowed the TCP based protocols.
If it is known ahead of time that there will be a relatively small
number of clients per server, then a TCP connection could be set up between
each client and the server(s) it uses. Such a scheme would allow TCP
based RPCs to attain speeds similar to the UDP based RPCs, and maintain
the high reliability available from TCP. Netwise ships "skeletal servers"
with their RPC system. By changing three macros in these files,
servers are created which keep track of open TCP connections, to make
the scheme described above easier to implement.
.pp
As a side effect of the decision to open and close the port of each
RPC call, address resolution\(dg
.(f
\(dg Address resolution maps the text name of a machine, "olive" to a
network address. Users like to use text names, but network protocols
use numerical addresses, so a translation must be done.
.)f
was redone for each RPC call. This can take a while.
.pp
Applications which use the client/server model fall into two categories.
There are some applications which have a small number of clients
and servers (often the exact number is known in advance) and there
are those applications which have a large number of clients and
servers (with the exact number unknown in advance). For the first
type of application, opening some permanent connections is not a problem,
since there are few of them. The second type of application, however,
requires that connections be opened and closed for each RPC call. If this
is not done, the application may use too many connections. The timing
tests assumed an application of the second type, which gave an advantage to the
UDP based RPC systems, Apollo and Sun (udp).
.pp
Some experiments showed that opening a connection and keeping it
open for many RPC calls resulted in faster response for the TCP based
RPC systems. Typically, keeping connections opened caused Netwise and Sun
(tcp) to execute functions in 10% to 30% of the time previously needed.
.pp
The data sizes used in the test could be prejudicial. Twenty and
512 were chosen because it was felt that most RPC calls would have
about that much data;
8K because that is overwhelmingly the most common size used here at
Atherton; and 16K because that is the maximum data used here. The choice
of 8K is especially prejudicial, since it is the maximum UDP packet size
on the test network. For reasons described above, this is the worst
place for Apollo to be compared to Sun (udp).
.pp
The test network was a homogeneous collection of MC68000 machines
which gave Sun an advantage. The reasons for this are complicated,
and deserve a subsection of their own:
.sh 2 "Data Representation"
.pp
Since all the machines in a heterogeneous network do not
have the same data representation, an RPC system must have a way to
translate data between different machine architectures. There are two
ways to do this: tag each transmission with its architecture and
have the receiving machine check the tag and convert as needed; or translate
each transmission into a standard form so each sender converts to standard
form and each receiver converts back to machine form. Apollo chose the
first method, while Netwise and Sun chose the second. Netwise uses an
OSI standard (ANS.1) as its standard form, while Sun uses the MC68000
data representation.
.pp
Apollo uses a tagged data representation, which has two disadvantages.
Some architectures simply cannot be represented
using the Apollo tag. Examples include machines with 9 bit bytes, or machines
which use any floating point representation except IBM, Cray, IEEE, or VAX.
Also, every machine must know how to decode data sent from every other
possible machine.
.pp
Both Netwise and Sun use a standard data representation. This means
they can support any architecture, no matter how obscure.
There are two differences between Netwise's and Sun's data
representation. Netwise has a flag, which can be used if the programmer
knows both machines have the same architecture. It causes Netwise
to use the native machine representation.
Sun's use of the MC68000 representation means if both machines are MC68000
based, no data translation is done, and if one machine is MC68000 based,
then only one translation is required. For comparison, Netwise always
performs two data translations, and Apollo performs none if both machines
have the same architecture, or one if they are different.
.SH "Dependability"
.(D
What are the chances an RPC call will be silently ignored or
executed more than once?*
.)D
.(f
* Note: this is not a measure of RPC errors, but rather errors
which are not reported to the program issuing the RPC.
.)f
.pp
If an RPC system is implemented on top of TCP, then its dependability
is guaranteed by TCP. Netwise and Sun (tcp) were therefore assumed
dependable, and were not tested.
If an RPC system is implemented on top of UDP, however, then the RPC
system must guarantee each server call is executed exactly once,
since UDP packets can be lost or duplicated. In order to test
dependability a simple client was written which sent an integer to the server,
and then incremented it. The server compared the integer sent by the
client to its own counter, printing a message if they were different, and
incrementing its internal counter. Losing or duplicating an RPC call
would cause the counters to become out of sync, triggering an error
message from the server.
.pp
This program was run on the same network on which the speed tests were
run under, except only the server was a 3/140. The clients were either
3/140s or 3/60s with 12 megabytes of RAM. The load on the server machine was
artificially raised. This was done by running small programs which were
infinite loops and had no disk operations.
.(b
.TS
center ;
cB s s s s
cB s s s s
rI cI cI cI cI
r c c c c.
Dependability
(errors/1,000,000 calls)
=
server load average: ~0.1 ~5.0 ~10.0 ~20.0
Apollo 0 0 0 \(dg
Sun (udp) 0 0* 10* 300
.TE
.)b
.(f
\(dg Apollo's RPC can not operate at this load. It fails,
causing a core dump after 2,000 to 30,000 RPC calls.
.)f
.(f
* At one point there was a network error while running the test suites and
all of the servers died. (Both Sun and Apollo with loads of 5, 10, and 20
were running at the time.) The Apollo servers died "clean" but the Sun
servers returned 3 or 4 errors. These were not counted.
.)f
.sh 2 "Discussion"
.pp
The major danger is that a server routine will be executed twice. This
can happen in two situations. If the data returning from the server to
the client is lost (which can happen with UDP), the client will time
out and resend the request, causing the server to execute its
routine twice. Or, the initial data from the client to the server
could arrive twice (this can also happen with UDP), also causing
the server to execute the service routine twice.
.pp
To see how dependability can affect an application,
consider a simple banking system.
Assume the average user will have 10 bank transactions
a week, and each transaction requires 20 RPC calls. For
this application an RPC system which fails every million calls
will cause one failure per customer per 95 years, which sounds
inconsequential. But, if the bank has five million customers,
that becomes over fifty thousand errors a year. Another example
would be an engineering application which makes 1000 RPC calls a
day and is used every working day (250 in a year). A failure rate
of one in a million translates to one failure every four years.
.lp
With Sun (udp) it is important to realize that these silent
errors can occur and take appropriate action.
.np
Switch to a dependable RPC system.
.np
Decide the rate of errors is acceptable and does not need to be fixed.
.np
Have the server routine check for duplicates (add an ID number to each RPC
request).
.np
Define the server routine(s) to be idempotent\(dd.
.(f
\(dd An idempotent routine can be executed more than once with out disturbing
the system Examples: gethostname, fseek. Counter Examples: unlink, lockf.
.)f
.pp
An example of ``tuning'' an application to be more dependable
is Sun's Network File System (NFS). Obviously, a file
system must be very reliable, even under overloaded conditions, or
its users will revolt. Sun achieved this while using their UDP
based RPC by making most NFS server routines idempotent, and having
the server check the rest to make sure they could not be
executed twice.
.bp
.SH "Conclusions"
.pp
The proper conclusion for a paper of this sort is an algorithm, which,
given an application and its implementation constraints, returns the
"best" RPC system to use in implementing that application. Unfortunately,
generalizing such an algorithm is difficult. Here are
some general rules to keep in mind while evaluating an RPC product:
.np
Determine what is important to your application: speed,
dependability, ease of use, a particular feature, etc. Each
RPC system has its own strengths and weaknesses. They are not
all the same.
.np
Distributed programs can be modified to use any RPC system very quickly,
in as little as a week. It may be worthwhile, therefore, to experiment
with several systems before deciding on one.
.np
For small packets (below 1K bytes), Sun (udp) and Apollo are the same
speed. As the amount of data increases, however, Sun outperforms Apollo.
.np
Once an application is using RPCs, the overhead of having the service
routine(s) on a foreign machine (as compared to the local machine) is
small.
.np
For lightly (under 5) loaded machines, Sun (udp) and Apollo are equally
dependable, but as the load increases, Sun becomes less reliable.
.np
If an application has a relatively small number of servers and clients,
it may be better to open a permanent TCP connection between them. In
this case the speeds reported in this paper will be largely irrelevant,
since they assumed that permanent connections could not be made.
.bp
.uh Updates
.pp
Ideally, this paper should be updated each time a new RPC system is
released. Unfortunately, this is not feasible, and there must be
a lapse between when a paper is written and when it is read. The RPC
world is changing quickly now, so this gap is sure to create some
inaccuracies.
Between the time I write this, and the time you read it, all the vendors
plan to come out with major improvements. I anticipate the following
improvements:
.np "Apollo"
Apollo SR 10.1 should become wide spread. A block mode
will be added to speed the movement of large amounts of data; this
mode will not require an acknowledgement for each 1K packet.
The bug which occurs running on heavily loaded servers will be fixed.
.np "Netwise"
Netwise will come out with version 2.0 of their software which allows
packets larger than 2K bytes.
.np "Sun"
SunOS 4.0 should become wide spread, and the RPC libraries shipped with
it should become the standard.
.sp 2
.uh "Code Availability"
.pp
Some of the programs used to test speed and dependability are available
from the author, as are machine readable copies of this paper.
.sp 2
.uh "Thanks and Acknowledgements"
.pp
Thanks to Darrel Strom (Coherent Thought), Tom Scoville (HP),
Nat Mishkin (Apollo), Mat Applebaum (Netwise), Steve Nahm (Sun),
Bob Lyon (Legato)
and everyone at Atherton Technology who supported and helped me.
.sp 2
.uh "Bibliography"
.lp
\fBC Language RPC Tool, User Guide\fP. Netwise Inc.
(Release 1.0 June 1988)
.lp
\fBNetwork Computing System (NCS) Reference\fP. Apollo Computer Inc.
(Order No. 010200 Revision 00)
.lp
\fBNetworking on the Sun Workstation\fP. Sun Microsystems Inc.
(Part No. 800-1324-03 Revision B)
.lp
\fIApollo NCA and Sun ONC: A Comparison\fP. Apollo Computer Inc.
.lp
RFC 1057 and RFC 1014. The Network (ARPA) Information Center.
These documents define Sun's RPC protocol and its XDR data representation
facility for the ARPA net community.
.lp
RFC 793. Describes the TCP protocol.
.lp
RFC 768. Describes the UDP protocol.
.lp
Bergan, Eric S. and Tolchin, Stephen G. \fIUsing \fP[Sun's]\|\fPRemote
Procedure Calls (RPC) for a Distributed Clinical Information System\fP,
in \fBUniForum 1986 Conference Proceedings\fP.
.lp
Cozzolino, Thomas J. and Pierce, Thomas H., \fIUsing the Network Computing
System in a Molecular Modeling Application\fP, Rohm and Haas Company.
.sp 5
.pp
The author received his degree in Computer Science Engineering from
the Moore School (University of Pennsylvania) in 1987. He was
introduced to UNIX as a summer intern at Ford Aerospace. After
graduating he worked for Teknowledge (now: Cimflex Teknowledge)
and Atherton Technology. He is interested in portability, computer
languages, networks and has dabbled in AI. He does not believe
in the \fIKobiyoshi Maru\fP scenario.
.\"
.\" change format for tables
.\"
.bp
.sz 10
.TS
center ;
cB s s s s
cB s s s s
c s s c c
r c c c c .
\s+2RPC Timing Performance\s0
The Raw Speed Numbers
=
location of server and client(s): same machine different machine
Implementation Clients Size Speed Speed
Sun (udp) 1 12 4.2 4.3
Sun (tcp) 1 12 5.6 6.0
Apollo 1 12 4.0 4.4
Netwise 1 12 11.1 11.9
Sun (udp) 2 12 6.4 6.0
Sun (tcp) 2 12 11.7 9.0
Apollo 2 12 7.2 7.5
Netwise 2 12 15.7 15.3
Sun (udp) 4 12 13.9 12.3
Sun (tcp) 4 12 22.6 17.3
Apollo 4 12 14.1 12.1
Netwise 4 12 30.1 25.7
Sun (udp) 1 512 4.3 4.7
Sun (tcp) 1 512 6.0 6.7
Apollo 1 512 5.3 5.7
Netwise 1 512 13.4 14.2
Sun (udp) 2 512 7.9 6.4
Sun (tcp) 2 512 15.8 10.3
Apollo 2 512 11.0 8.0
Netwise 2 512 20.2 19.3
Sun (udp) 4 512 14.0 12.5
Sun (tcp) 4 512 27.4 19.9
Apollo 4 512 19.3 15.9
Netwise 4 512 38.0 53.4
Sun (udp) 1 8K 5.9 9.6
Sun (tcp) 1 8K 11.4 13.4
Apollo 1 8K 42.0 43.2
Sun (udp) 2 8K 16.8 23.2
Sun (tcp) 2 8K 23.1 23.4
Apollo 2 8K 81.6 70.8
Sun (udp)* 4 8K 38.0 34.6
Sun (tcp) 4 8K 46.3 42.0
Apollo 4 8K 152.9 126.1
Sun (tcp) 1 16K 18.2 21.8
Apollo 1 16K 91.5 84.0
Sun (tcp) 2 16K 34.2 34.4
Apollo 2 16K 161.4 147.2
Sun (tcp) 4 16K 68.2 69.2
Apollo 4 16K 298.6 248.5
.TE
.(f
* One timeout error occurred during this test.
.)f
.sp 2
.(l L
\fBClients\fP is the number of processes accessing the server at once.
\fBSize\fP is the packet size, in bytes.
\fBSpeed\fP is seconds, rounded to the nearest tenth, for 100 remote procedure calls.
\fB\(co 1989 Joshua Levy. This page may be reprinted only in its entirety.\fP
All tests run on Sun 3/140s with 8 Megs of memory on a lightly loaded ethernet.
.)l
.brlars@salt.acc.com (Lars J Poulsen) (07/20/89)
In article <6567@joshua.athertn.Atherton.COM> on comp.misc
joshua@Atherton.COM (Flame Bait) writes in a long, worthwhile technical
review article comparing 4 different RPC protocols:
Second, consider the two TCP based systems [vendor names
omitted]: Again, since both use the same communications layer
they should have the same speed. One explanation for [vendor
A]'s slower performance, is data conversion (which is fully
discussed below in "Data Representation"). Briefly, [vendor A]
must translate all data into a standard format (ASN.1).
[vendor B] does too; however, [vendor] uses MC68000 as their
standard data format. So, in fact, [vendor B] did no data
conversion, while [vendor A] did twice: from MC68000 format to
ASN.1 and back.
[I have omitted vendor names to comply with the spirit of Joshua's
copyright notice.]
You now have a sense of why those of us who have been around long enough
to remember when 1 MIPS was a campus mainframe are so opposed to the
parking garage sized ISO/OSI protocol stacks. OSI mandates an ASN.1
presentation layer, while TCP/IP usually punts this by specifying a
standard representation equal to the natural representation on either a
VAX or an MC68K or soon to be iAPX386. This ensures that some group of
users get an efficient implementation, while the ISO model guarantees
equal and equally bad performance for all.
/ Lars Poulsen <lars@salt.acc.com> (800) 222-7308 or (805) 963-9431 ext 358
ACC Customer Service Affiliation stated for identification only
My employer probably would not agree if he knew what I said !!fwb@demon.siemens.com (Frederic W. Brehm) (07/20/89)
In article <951@anise.acc.com> lars@salt.acc.com (Lars J Poulsen) writes: >... OSI mandates an ASN.1 >presentation layer, while TCP/IP usually punts this by specifying a >standard representation equal to the natural representation on either a >VAX or an MC68K or soon to be iAPX386. This ensures that some group of >users get an efficient implementation, while the ISO model guarantees >equal and equally bad performance for all. You forgot that the Session layer can negotiate the presentation encoding. The basic encoding for ASN.1 is required for conformance, and therefore interoperability. Private encoding methods (like the TCP "punt") may be used if both participants agree. This ensures that some group of users get an efficient implementation, while all users may communicate. -- Frederic W. Brehm Siemens Corporate Research Princeton, NJ fwb@demon.siemens.com -or- princeton!siemens!demon!fwb
carla@boulder.Colorado.EDU (Carla Mowers) (07/25/89)
The results presented in Joshua Levy's RPC paper are based on data that is now nearly nine months old. An article I've posted to comp.misc contains some performance measurements based on new releases of both the Netwise and Apollo products. A new Sun product has not been released since Joshua's measurements were made. The article contains data collected in early June of 1989 by Netwise. All source code used for these measurements may be obtained by sending a request to "tony@wldrdg.UUCP". Please direct all other questions and comments to this account as well. This article is being posted from this account due to technical difficulties with the USENET feed at Netwise. Of course, you should probably take any numbers presented here with a large grain of salt. What one party measures doesn't necessarily correlate well to the environment another is interested in. The leap-frog nature of the game means that any measurement won't be valid long. Internally, we have a next- release prototype that improves on the performance of our currently shipping product as, I'm sure, do the other vendors. The suggestion from Lars Poulsen that ASN.1 encodings lead to poor performance isn't necessarily true. Our performance numbers show us approaching Sun's RPC in several areas (opaque buffers still need work), and our current prototype is, at worst, 2% slower than Sun in the categories we tested. The flexibility of ASN.1 can actually be an advantage. Our prototype exploits this for large buffers. By using a longer length form (one byte longer) we can get the ASN.1 buffer aligned such that bcopy() can make the transfer from the user's buffer to the transmission buffer much more quickly. I believe you'll find the posted information basically lacking in hype. If you view this as a commercial use of the net, I'm sorry. I'm simply responding to an article that contained outdated information regarding one of our products. Tony Andrews Netwise, Inc. 2477 55th St. Boulder, CO 80301 Phone:303-442-8280 UUCP: onecom!wldrdg!tony
carla@boulder.Colorado.EDU (Carla Mowers) (07/25/89)
The following paper contains more recent performance data for Sun, Apollo, and Netwise. Format as described below. If you don't have access to troff, or the mm macros, send me email to receive a plain text output or a hard copy by mail. Tony Andrews Netwise, Inc. 2477 55th St. Boulder, CO 80301 UUCP: onecom!wldrdg!tony .\" Sun/Apollo/Netwise RPC Performance .\" 6/9/89 .\" .\" Tony Andrews .\" Netwise, Inc. .\" 2477 55th St. .\" Boulder, CO 80301 .\" .\" Format with: .\" .\" tbl file | nroff -mm | col | lpr .\" .nr Hu 2 .nr Cl 5 .nr Hb 5 .nr Ej 0 .TL Remote Procedure Call Performance .sp A Review of Current Product Offerings .AU "Netwise, Inc." .AU "June 9, 1989" .AF "" .MT 4 .PH "''''" .PF "'Netwise, Inc.''June 9, 1989'" .HU "Introduction" This paper presents preliminary results of an ongoing performance study of the major commercial remote procedure call (RPC) products. The testing thus far has measured round-trip times for remote procedure calls using a variety of simple data types, as well as the basic code size overhead for a simple remote procedure call. .P The products used for this comparison are: .BL .LI Netwise C Language RPC TOOL, version 2.02 .LI Apollo NCS, version 1.1 .LI Sun/RPC, version 3.9 .LE .HU "Testing Environment" All tests were performed on a Sun 3/280 system running SunOS 3.5 in single-user mode. Client and server processes executed within the same system. Each test run measured the elapsed real time for 500 calls. The times shown represent the average of five such test runs. .P We attempted to measure only the remote procedure call overhead itself. In each case, client/server binding overhead was minimized by using the most optimal binding method and excluding any set-up calls from the measured time. In addition, all procedures were declared as "idempotent" for the Apollo tests. .P The tests used are as follows: .VL 20 5 .LI void void procedure with no parameters .LI add adds two integers and returns the result .LI "integer 100" integer array of 100 elements as the sole input parameter; returns an integer .LI "integer 1000" same as above but with 1000 elements .LI "string" concatenates two null-terminated string parameters and returns the result; input strings are 15 and 18 characters long .LI "opaque 4000" passes and returns a buffer of 4000 bytes; also returns an integer .LI "opaque 8000" passes and returns a buffer of 8000 bytes; also returns an integer .LI "opaque 16000" passes and returns a buffer of 16000 bytes; also returns an integer .LE .P The tests were implemented as uniformly as possible across the three products tested. In some cases an output parameter was used instead of a return value due to RPC compiler restrictions, but no significant changes were required for any of the RPC compilers. .P Testing was done for both TCP and UDP transports wherever possible. .HU "Timing Results" The following table shows the measured round-trip per-call times (in milliseconds) as measured in the client process, excluding extraneous binding and set-up overhead. .DS .TS center,expand,tab(%); l 2 | ce 2 s 2 s 2 | ce 2 s 2 s 2 | l 2 | ce 2 s 2 s 2 | ce 2 s 2 s 2 | l 2 | ce 2 ce 2 ce 2 | ce 2 ce 2 ce 2 | l 2 | ne 2 | ce 2 | ne 2 | ne 2 | ne 2 | ne 2 |. %TCP Transport%UDP Transport %Round-Trip Time (ms)%Round-Trip Time (ms) %Sun%Apollo%Netwise%Sun%Apollo%Netwise _ Void%5.01%-%5.34%4.11%5.95%4.37 _ Add%5.22%-%5.47%4.22%6.82%4.48 _ Integer 100%10.37%-%11.42%8.60%10.38%9.99 _ Integer 1000%57.03%-%65.35%46.41%81.04%59.01 _ String%6.38%-%7.11%5.17%11.49%6.00 _ Opaque 4000%28.30%-%72.38%15.38%131.66%61.15 _ Opaque 8000%62.05%-%140.40%25.09%353.06%116.82 _ Opaque 16000%105.27%-%272.05%-%431.31%- _ .TE .DE .P Some of the RPC times above warrant further explanation. No TCP times are presented for Apollo because NCS supports only datagram transports such as UDP. Times are also missing for Netwise and Sun for UDP in the "opaque 16000" test case. This is because of the size limit for UDP messages. Apollo has avoided this limit through additional protocols implemented above UDP. .P The Netwise UDP times were measured using a pre-release product. All other times represent currently available products. .P The slower performance of Netwise with respect to Sun in the opaque tests is due to the use of a suboptimal ASN.1 constructed encoding. Future releases will use a simpler ASN.1 encoding, bringing the numbers more in line with Sun. .HU "Code Size Results" Code sizes for a minimal distributed application (the void test from above) were measured. The sizes were broken down into the following categories: .BL .LI Application Code .br User-written application code. On the client side, this includes any binding calls required by the RPC system. .LI Server Initialization .br This is the server program's main routine, which generally performs some network initialization and begins waiting for RPC requests. In the case of Apollo's NCS, this code is written by the user. .LI Generated Code .br Code generated by the RPC compiler for client/server stubs, data transfer procedures, and so forth. .LI Libraries .br Code obtained by linking with runtime libraries provided by the RPC vendor. This does not include any code from the standard C runtime library. .LE .SP .P The observed code sizes are as follows: .DS .TS center,expand,tab(%); l 2 | ce 2 se 2 se 2 | ce 2 se 2 se 2 | l 2 | ce 2 se 2 se 2 | ce 2 se 2 se 2 | l 2 | ce 2 ce 2 ce 2 | ce 2 ce 2 ce 2 | l 2 | ne 2 ne 2 ne 2 | ne 2 ne 2 ne 2 |. %Client%Server %Object Code Size (bytes)%Object Code Size (bytes) Code Type%Sun%Apollo%Netwise%Sun%Apollo%Netwise _ Application%304%452%264%16%20%20 _ Server Init.%-%-%-%300%356%1108 _ Gen'd Code%100%132%132%276%52%984 _ Libraries%12216%38836%9964%17884%38836%9396 _ TOTAL%12620%39420%10360%18476%39264%11508 _ .TE .DE