kaigler@ihlpa.ATT.COM (Kaigler) (01/30/88)
Hi All,
I've sent this message before, so if you've seen it, just ignore it, okay. I'm
relatively new at working with TCP-IP and would like some information. I'll be
testing TCP-IP and VAX, UTS, and 3B2 machines and would like to know if there
exists some sort of test tools which would be of some help. By the way this is
streams based TCP-IP. Any information would be greatly appreciated.
Ronnie Kaigler
ihlpa!kaigler
IH 1B-232
(312) 979-1762hrp@ishmael.CRAY.COM (Hal Peterson) (02/01/88)
The TCP/IP community does not have a strong tradition of formal
testing from specifications. Historically (see RFC-1025, ``TCP and IP
Bake Off''), testing a TCP/IP implementation has meant structured
plugging it in and trying it, with as many other implementations as
possible.
Over the last few years, there has been a movement toward more formal
testing from within the DCA, and they are currently beta testing
(metatesting?) a suite covering every conceivable aspect of all the
MIL-STD protocols: IP, TCP, FTP, TELNET, and SMTP. The tests are
comprehensive and do find bugs, but require a significant investment
of time and effort to run.
There was a note on the tcp-ip list last year from Bob Jones
(bobj@kauai.mcl.unisys.com) in which he wrote:
For those who are interested, the DCA is setting up an administrative
agreement with the National Bureau of Standards to provide for the
certification of the TCP/IP suite of protocols. NBS is expected to
announce for vendors to participate under the National Voluntary
Laboratory Accreditation Program (NVLAP). This program is expected
to be in place sometime in early 1988. Any questions please call
DCEC, Jim Tontonoz, 703-437-2038.
This certification would probably involve passing the aforementioned
DCA test suite.
--
Hal Peterson / Cray Research / 1440 Northland Dr. / Mendota Hts, MN 55120
hrp%hall.CRAY.COM@umn-rei-uc.ARPA ihnp4!cray!hrp (612) 681-5884karn@thumper.bellcore.com (Phil R. Karn) (02/03/88)
> The TCP/IP community does not have a strong tradition of formal > testing from specifications. Historically (see RFC-1025, ``TCP and IP > Bake Off''), testing a TCP/IP implementation has meant structured > plugging it in and trying it, with as many other implementations as > possible. Actually, I think there is a lot to be said for this approach. Formal verification is difficult, slow and expensive, while the informal testing of a new implementation on the real Internet quickly shows whether it's likely to work in actual use. The real world has a way of revealing things that remain stubbornly hidden in the lab. The TCP/IP experience has shown that finding clear-cut protocol violations is not difficult. The *real* issues are a) getting rid of gaps and ambiguities in the specs themselves that can allow different implementations to "conform" yet not be interoperable, and b) figuring out how to armtwist the vendor of a broken implementation to fix it once a problem has been identified. Formal verification suites attack neither of these problems. Phil
DCP@QUABBIN.SCRC.SYMBOLICS.COM (David C. Plummer) (02/05/88)
Date: 2 Feb 88 19:32:51 GMT
From: thumper!karn@faline.bellcore.com (Phil R. Karn)
a) getting rid of gaps and ambiguities in the specs themselves that can
allow different implementations to "conform" yet not be interoperable, and
You forgot (that goes between a and b):
Convincing a vendor that his implementation is broken.
I remember when I was testing the TCP I wrote for Symbolics LispMs. We
were one of the first to use segments with overlapping sequence numbers
in a big way (mostly for telnet). It didn't work with Unix machines.
The response was "Well, Unix is right, so you must be doing something
wrong." We still have this problem with Unix TCP/FTP giving a reply
code for some operation, DELETE I think, that is not what the FTP spec
says it should be. The typical answer from Unix people is "You should
only look at the first digit; the others are secondary information."
And we respond with "The spec says it should return <xyz>." And it goes
'round and 'round.
b) figuring out how to armtwist the vendor of a broken implementation to
fix it once a problem has been identified.DCP@QUABBIN.SCRC.SYMBOLICS.COM (David C. Plummer) (02/05/88)
Date: 2 Feb 88 19:32:51 GMT
From: thumper!karn@faline.bellcore.com (Phil R. Karn)
a) getting rid of gaps and ambiguities in the specs themselves that can
allow different implementations to "conform" yet not be interoperable, and
You forgot (that goes between a and b):
Convincing a vendor that his implementation is broken.
I remember when I was testing the TCP I wrote for Symbolics LispMs. We
were one of the first to use segments with overlapping sequence numbers
in a big way (mostly for telnet). It didn't work with Unix machines.
The response was "Well, Unix is right, so you must be doing something
wrong."
We then got calls from people trying to communicate between our machines
and Apollo. Apollo said "We have no trouble talking to Unix machines."
We still have a similar problem with Unix TCP/FTP giving a reply code
for some operation, DELETE I think, that is not what the FTP spec says
it should be. The typical answer from Unix people is "You should only
look at the first digit; the others are secondary information." And we
respond with "The spec says it should return <xyz>." And it goes 'round
and 'round.
I'm not saying we don't occaisionally pass the buck without looking; I'm
just relaying anecdotes of how difficult things can be.
b) figuring out how to armtwist the vendor of a broken implementation to
fix it once a problem has been identified.karn@THUMPER.BELLCORE.COM (Phil R. Karn) (02/05/88)
I agree with you except for the part where you place all of the blame for the FTP problem on UNIX. The specs say that you should be conservative in what you send, liberal in what you accept. So in the case of the FTP reply code, you're *both* at fault -- UNIX for not generating the correct code, your code for not looking at just the first digit. Phil
swanson@EDN-VAX.ARPA (John Swanson) (02/05/88)
Phil, Having helped produce the verification suite for DoD I can't keep quiet any longer. I think you are confusing the tools with their usage. First I wonder how much detailed thought went into the ad-hoc testing you are talking about. Are the authors concerned with compliance to the standards or "getting their Problem fixed". When we examined the specifications for each protocols we documented those descrepancies for DCA and asked for clarification and factored any response into the test suite. Obviously since the verification suite is a tool I doubt if it can twist anyones arm. However, if a vendor is required to pass the verification suite of tests perhaps there would not be as many "broken" implementations. Therefore I would think those people who really want testing would be trying to support the NVLAP process the Government is trying to start up, rather than reinvent the testing wheel. John
brian@wb6rqn.UUCP (Brian Lloyd) (02/06/88)
Phil,
Although a protocol test suite is not perfect, it will tend to catch
errors before the software is released (hopefully). Perhaps this will
reduce the amount of ad-hoc testing required. I agree with you that
there is no test like trial-by-fire but perhaps the VS can shake out the
more flagrant bugs ahead of time.
In any case, I have a feeling that your stuff will get tested against
the suite anyway.
Brian Lloyd, President
Sirius Systems, Inc.
(301) 540-2066
{bellcore, syscad, cp1, irs3, n3dmc}!wb6rqn!brianchris@GYRE.UMD.EDU (Chris Torek) (02/07/88)
>Although a protocol test suite is not perfect, it will tend to catch >errors before the software is released (hopefully). On the other hand, it can also induce the `well we passed the protocol test suite so if it fails to work it must be your fault' syndrome. I think the only cure for this is to state clearly that an implementation must not only pass the verification tests, but also interoperate with other such implementations, and that the end users reserve the right to change the test suite as needed and retest. >Brian Lloyd, President >Sirius Systems, Inc. Share and enjoy, Share and enjoy, Journey though life with a plastic boy Or girl by your side, Let your pal be your guide. . . . (sorry) Chris