KASTEN@MITVMA.MIT.EDU (Frank Kastenholz) (02/13/88)
Hal Peterson (hrp%windsor.CRAY.COM@uc.msc.umn.edu) at Cray wrote >I should stress, though, that a verification suite is only part of a >comprehensive testing strategy, and that bake-offs can be extremely >valuable for finding bugs. A verification suite is software, and like >all software it is imperfect. It is entirely possible that two >implementations both pass the suite but don't interoperate. Bake-offs >can catch that, and when they do they have found as many as FOUR bugs: > >1. a bug in one of the implementations. >2. a bug in the other implementations; after all, they could both be > at fault. >3. a bug in the specification. It may be unclear or inconsistent or > incomplete and so have misled the implementors. >4. a bug in the verification suite. It should have caught the > problem, and next time it will, since it's a simple matter to add > a test to a well-designed suite. The suite gets better as time > goes on. His last two points bring two simple(?) questions to mind - How do the Specification Bugs get resolved? Who Verifies the Verification Suite???? (Qui Ipsos Custode Custodiet) Frank Kastenholz Atex Inc.