gary@dvnspc1.Dev.Unisys.COM (Gary Barrett) (08/21/89)
I would be interested in hearing from anyone on the net who is wrestling with the problem of benchmarking TCP/IP and OSI software across equipment from various vendors. I have posted a similar inquiry on this subject about a month ago, questioning whether there are any standard benchmarking programs to do such an evaluation. I had very few responses. The only benchmark tool seemingly available (for TCP/IP) is ttcp. Certainly, others "out there" must be wrestling with the problem of of "standard" networking software. How can it be analyzed across systems such that users can fairly decide whether one product is clearly superior to another? Standard benchmarks are evolving for Transaction Processing. So why not for standard network services like FTAM or TELNET sessions? I know that vendor Marketeers often quote numbers, but unless those numbers are backed by agreed-upon benchmarks, those numbers amount to snake-oil (at worst) and apples-to-oranges analyses (at best). I welcome any comments anyone may have on this subject. I would be especially interested in hearing from "users", how you expect to evaluate Open-Network offerings as part of your formal purchasing process. Thanks. Gary L. Barrett Unisys Devon Engineering Facility Wayne, PA
worsley@ditmela.oz (Andrew Worsley) (08/24/89)
In article <758@dvnspc1.Dev.Unisys.COM>, gary@dvnspc1.Dev.Unisys.COM (Gary Barrett) writes: > > I would be interested in hearing from anyone on the net who is > wrestling with the problem of benchmarking TCP/IP and OSI software > across equipment from various vendors. > > I have posted a similar inquiry on this subject about a month ago, > questioning whether there are any standard benchmarking programs to do > such an evaluation. I had very few responses. The only benchmark > tool seemingly available (for TCP/IP) is ttcp. > > Certainly, others "out there" must be wrestling with the problem of > of "standard" networking software. How can it be analyzed across > systems such that users can fairly decide whether one product is clearly > superior to another? Standard benchmarks are evolving for Transaction > Processing. So why not for standard network services like FTAM or > TELNET sessions? I know that vendor Marketeers often quote numbers, > but unless those numbers are backed by agreed-upon benchmarks, those > numbers amount to snake-oil (at worst) and apples-to-oranges analyses > (at best). > > I welcome any comments anyone may have on this subject. I would be > especially interested in hearing from "users", how you expect to > evaluate Open-Network offerings as part of your formal purchasing process. I agree that it is important to have reliable methods of comparing different implementations and if this often means relying on some one else performing measurements. But these protocols you mention are really appropriate for benchmarking. For instance FTAM the main things I would be looking at are size of implementation, number bytes/sec for a binary/ASCII file. Usually the size is not very important and the speed is limited by the medium between them (e.g. X.25 baud rate or for ethernet the network implementation). Probably the MOST important thing at the moment is reliability and conformance to the standard. There is a lot of work going on to construct conformance testing standards especially in the U.K. but testers seem to be several times more complex than implementations themselves. People will charge a fortune to verify that a product is conforment. At the moment the best bet is to try to interwork with a known implementation like ISODE. I think this is the policy that NISC (is that the U.S. Gov.'s standards Institutes name now?) are leaning towards. Similar comments go for Telnet. The throughput of characters isn't really likely to be a problem with people typing speeds (prehaps the CPU load might be important though) but options available and how likely it is able to talk to different versions is very important. This is related to conformance but certainly not gaurented by it. Andrew Worsley Division of Information Technology (Melbourne), Phone +61 3 347 8644 C.S.I.R.O. Fax +61 3 347 8987 55 Barry St. Telex AA 152914 Carlton, Vic, 3053, Australia E-mail: worsley@ditmela.oz.au -- Division of Information Technology (Melbourne), Phone +61 3 347 8644 C.S.I.R.O. Fax +61 3 347 8987 55 Barry St. Telex AA 152914 Carlton, Vic, 3053, Australia E-mail: worsley@ditmela.oz.au
karl@asylum.sf.ca.us (Karl Auerbach) (08/24/89)
I disagree with some of your points:
1. Typical file transfer rates are not usually limited by the medium
speed, but by a number of other factors. Besides the efficiency of
the underlying protocol stack major limits are imposed by the
burden of converting file formats from host form to network form and
vice versa. For FTAM binary, this may not be too bad. But as the
file structure becomes more complex the burden increases rapidly.
(Even simple "line image" handling will slow things noticably.)
2. Telnet throughput *is* important, even for people typing. "Sticky"
keyboard response can severly damage the productivity of the user.
(This is more of a round-trip time issue than it is a "throughput"
issue.)
--karl--dcrocker@AHWAHNEE.STANFORD.EDU (Dave Crocker) (08/25/89)
To reinforce Karl's first point about data conversion cost, the TCP/IP FTP program can transfer files in a standardized network-ASCII or in a regular binary. For virtually all systems, the former requires character-at-a-time scan and convert, whereas the latter involves only block copies. It is not unusual for the throughput different to be as large as a factor of two. Dave