caf@omen.UUCP (Chuck Forsberg WA7KGX) (01/03/88)
Given the interest in "real world" protocol performance recently expressed in this newsgroup, I'm posting the raw results of the Omen Technology torture tests. Study them if you wish and do your own editorializing. Of necessity, not all programs were tested. If I get enough requests, I'll test some different programs. If the subject program is not commonly available shareware, please mail me a current copy (or pair if necessary) for testing. ------------------------------- cut here: ptest.doc ---------------- Recent reviews of communications programs in PC Magazine and elsewhere failed to test the performance of the various programs under the stress of line disturbances, slow computers, or background operation. To remedy this situation, here are the results of Omen Technology's Protocol Torture Tests run from time to time on various file transfer protocols and programs. Anyone who questions the honesty or validity of these tests is invited to send a knowledgeable representative to Omen to monitor a repeat of the test(s) in question, and/or submit newer released versions of software for retest. The test conditions are clearly described and easy to reproduce for the benefit of those who wish to repeat the tests. Chuck Forsberg 503-621-3406 Omen Technology INC 17505-V NW Sauvie IS RD Portland OR 97231 Protocol Stress Tests 7-5-87 Omen Technology is connected to a rural telephone exchange in Sauvie Island, a suburb of Portland Oregon. This variability of phone line quality has led to an emphasis on protocol robustness. The Omen Technology Protocol Stress Test simulates noise patterns seen on marginal telephone calls remarkably closely, given the random nature of line noise. Sending Computer (TeleGodzilla BBS system): IBM PC, V-20, 8087, Maynard 10 MB hard disk, IBM CGA, B&W composite monitor, AST MegaPlus II multifunction board. 2400 bps modems. Receiving Computer: QIC Labs AT Clone, 8mHz 0ws, Seagate 4051, Genoa Super EGA, Thompson UltraScan monitor. The receiving modem was dialed into the sending modem (TeleGodzilla). Both lines were on the same exchange, and line hits in this configuration are rare. Line noise was generated by a Radio Shack Duophone 145 connected in parallel with the receiving modem, pulse redialing the number 1-800-123-4567. This generates "line hits" simulating those seen on marginal phone connections. There has been some discussion about the relevance of such a test setup and the distribution of "line hits" it generates. Line hits at short periodic intervals of less than a second or so (generated by clock slippage) are not well simulated, and only protocols with an extremely short minimum block length ( << 128) can function in such environments. This test is not that demanding; properly written XMODEM or XMODEM based protocols succeed most of the time, and the protocols that have the reputation among the bulletin boards of reliable operation under stress rarely fail this test. The test file used in these tests was BBS_TEE.ARC (a BBS program that operates under Pro-YAM or ZCOMM), selected for its convenient 29960 byte length. It is available for downloading from TeleGodzilla, but any .ARC file of about that length should give the same results. Software: YAMK version 16.74 (ZMODEM, YMODEM), GT PowerComm Version 1221 (MEGAlink) Each computer was running the same communications software. No TSRs were present. Test# PROT Throughput Remarks 1 MEGALI Failed RX reported success at block 136 2 MEGALI Failed TX reported success at block 40, RX slow to abort 3 MEGALI Failed TX reported success at block 95 4 MEGALI 44 cps Redials exhausted before transfer finished 5 MEGALI Failed TX reported success at block 88 1 YMODEM Transfer successful, throughput not recorded 2 YMDM-k 86 cps Transfer finished before redials exhausted 1 ZMODEM 122 cps (z pt30 on RX) Transfer finished first 2 ZMODEM 91 cps (z pt20 on RX) Transfer finished first 1 SK 82 cps Pro-YAM to Pro-YAM Transfer finished first 2 SK 94 cps (rx: k pt2) Pro-YAM to Pro-YAM XFER finished first 3 ZMODEM 120 cps (z pt20 pW600 on RX) Transfer finished first 1 SK 50 cps XTALK Mark IV to X4 Transfer finished first Throughputs were reported by the receiving program. SK=Kermit Sliding Windows Protocol Stress Tests Performed 12-13-87 Configuration: As above except AT clone had a CGA clone board instead of EGA clone. Same BBS_TEE.ARC file. As before, transfers were at 2400 bps. Software: YAM.EXE 17.02, Crosstalk Mark IV 1.01, GT Power 13.00, Procomm 2.42. As of 12-14-87 these are believed to be the latest released versions of these programs. A new Procomm was downloaded from Compuserve to verify the correctness of the PROCOMM.EXE file. YMODEM is a batch protocol, called YMODEM Batch on Procomm and Crosstalk Mark IV. Timings by stopwatch. S/W PROT Transfer Time / Remarks Pro-YAM ZMODEM 3:53 Tight timing: 2 sec, 1 sec within packets Pro-YAM ZMODEM 3:59 Tight timing: 2 sec, 1 sec within packets Pro-YAM ZMODEM 3:39 Tight timing: 2 sec, 1 sec within packets Pro-YAM ZMODEM 3:54 Tight timing: 2 sec, 1 sec within packets Pro-YAM ZMODEM 4:54 Standard timing Pro-YAM ZMODEM 4:36 Standard timing Pro-YAM ZMODEM 5:03 Standard timing Pro-YAM Kermit 6:14 Standard timing Pro-YAM YMODEM FAILED Tight timing: 2 sec, 1 sec within packets Pro-YAM YMODEM 5:44 Standard timing Pro-YAM YMODEM 5:20 Standard timing Pro-YAM YMDM-k 6:07 Standard timing Pro-YAM YMDM-k 5:06 Standard timing GT Pwr Megali 8:30 GT Pwr Megali 9:30 GT Pwr Megali FAILED TX reported success GT Pwr Megali FAILED TX reported success GT Pwr Megali 10:23 Procm YMDM FAILED Stack overflow, lockup Procm YMDM FAILED Stack overflow, lockup XTALKm4 DART 9:84 XTALKm4 DART 5:06 XTALKm4 DART 4:15 XTALKm4 DART 4:16 XTALKm4 DART 5:59 XTALKm4 DART 5:54 XTALKm4 DART 5:08 XTALKm4 DART FAILED RX:Too many Errors TX: locked up XTALKm4 DART 4:45 XTALKm4 DART 5:11 XTALKm4 YMDM FAILED XTALKm4 YMDM FAILED XTALKm4 YMDM FAILED XTALKm4 YMDM 2:13 Normal transfer time without line hits XTALKm4 YMDM FAILED XTALKm4 YMDM FAILED