gary@dvnspc1.Dev.Unisys.COM (Gary Barrett) (08/21/89)
I would be interested in hearing from anyone on the net who is wrestling with the problem of benchmarking TCP/IP and OSI software across equipment from various vendors. I have posted a similar inquiry on this subject about a month ago, questioning whether there are any standard benchmarking programs to do such an evaluation. I had very few responses. The only benchmark tool seemingly available (for TCP/IP) is ttcp. Certainly, others "out there" must be wrestling with the problem of of "standard" networking software. How can it be analyzed across systems such that users can fairly decide whether one product is clearly superior to another? Standard benchmarks are evolving for Transaction Processing. So why not for standard network services like FTAM or TELNET sessions? I know that vendor Marketeers often quote numbers, but unless those numbers are backed by agreed-upon benchmarks, those numbers amount to snake-oil (at worst) and apples-to-oranges analyses (at best). I welcome any comments anyone may have on this subject. I would be especially interested in hearing from "users", how you expect to evaluate Open-Network offerings as part of your formal purchasing process. Thanks. Gary L. Barrett Unisys Devon Engineering Facility Wayne, PA
snorthc@RELAY.NSWC.NAVY.MIL (08/23/89)
> From: dvnspc1!gary@burdvax.prc.unisys.com (Gary Barrett) > Organization: Unisys Corporation, Devon, PA > Subject: Comparing Performance of TCP/IP or OSI on Multivendor Equipment > I would be interested in hearing from anyone on the net who is > wrestling with the problem of benchmarking TCP/IP and OSI software > across equipment from various vendors. Here at the Naval Surface Warfare Center we are facing the same problem! > The only benchmark tool seemingly available (for TCP/IP) is ttcp. I get the impression that tools like nfsstones and the X benchmark sw available on expo aren't what you are after. I have two subnets assigned for testing purposes only. I have measurement tools (Excelan Lanalyser and FTP SW Lanwatch) stationed on these subnets. By isolating the hardware and software being tested I am getting reproducible results. The problem with this approach is that it is very manual, because the collection points are MS-DOS machines. I am currently exploring using SNMP agents on routers between the subnets to collect data and using the SNMP client software from CMU to get the data from the agents for analysis. YES, there HAS to be a BETTER way! I am trying to answer questions like these: Given two protocols/applications with similar functionality such as: - TCP's FTP; - OSI's FTAM. Which is more efficient? This has to be considered over a wide range of tests: establishing the connection; xfer small, large files; various hw/sw vendor implementations; file management capabilities..... Given a single protocol or application which vendors implementation is "best": - meet standards such as RFC? - interoperate with other major vendors implementations? (of course the first case should ensure the second :-) ) - handle exception conditions. This last can probably never be tested properly with a pure standard benchmark. There are zillions of things worth testing; sticking to file xfer, try ftping a fairly large file from a noisy subnet that has a Bridge/3com GS/3 as a router. Many FTPs will break here because large efficient packets often won't get through. FTP SW's FTP has an option that allows you to change the window size. (anyone know how to do this with SUNOS or Ultrix?) Given identical protocol/application/implementation what is the effect of changing the network adapter or the router? What are the protocols/applications required to support a: - OA user; - "business (lots of database access) user; - scientific user? I am making some slow progress in collecting this data, the really hard task is to build a model from the empirical base. If anyone has a good recommendation for a good simulation for a non-deterministic network (UNIX minis on ethernet with PC/MAC/workstation clients) I am quite interested. Any suggestions, comments are solicited. Flames are OK, I used to be a potter and own asbestos, fiberfrax, and kevlon protective gear. thank you for your support stephen northcutt (snorthc@relay.nswc.navy.mil) My management is too busy trying to give my office to someone else to worry about what I say.