newbie@inmet (08/21/89)
We have been working with both an in-house compiler crunch suite, as well as the comercial Plum-Hall ANSI validation suite. The Plum-Hall tests are *VERY* complete. They test things so arcane and remote that when your done, if you can get your compiler to pass all of the tests successfully, you can be assured that it works as perfectly as possible. The tests are self-checking so all you have to do is check the output from the programs to see if anything went wrong. Error messages contain file and line-number references so locating problems is a snap, even if fixing them isn't. Two include files can be customized for both your compiler and host-system. I don't have an address for the company handy, but if your interested, mail me and I'll see if I can find it.
paulr@sequent.UUCP (Paul Reger) (08/23/89)
I was wondering what experiences anyone has had with test suites for
C. To date we are using a commercial suite for C but we are finding
the coverage of it lacks in many areas particulary when it comes to
'oddball' expressions found often in kernel code. For example the
following code construction:
if ((struct xyz *) 0x2000 ->element == 29)
Is not in the test suite, neither is:
int
func(i)
register int i;
{
...
if (((short) i = f(45)) < 0)
/* the line containing the problem... */
Is not in our suite either.
We are embellishing the suite when we do find the bugs, but the bugs
are not found in a 'scientific' way. Rather, we just 'build the
world' (Operating system on down), and then testing the built binaries
often reveals bugs in the compiler (arrrgh!!!).
The process is time consuming and therefore costly.
What I would be interested in is responses from your experiences with
testing a C compiler:
1. What suites do you consider the best ones ??
2. Are there automatic tools available which produce and evaluate
random sequences of 'oddball' C expressions (this assumes they also
contain an 'Oracle' which will evaluate the correctness of the code
produced by the compiler...).
3. How successful are such automatic test case generators ???
paulr (Paul Reger)
Sequent Computer Systems Beaverton, OR.
... {sun,ucbvax!rutgers!ogccse,uunet}!sequent!paulrwalter@hpclwjm.HP.COM (Walter Murray) (08/26/89)
Paul Reger writes: > I was wondering what experiences anyone has had with test suites for > C. To date we are using a commercial suite for C but we are finding > the coverage of it lacks in many areas particulary when it comes to > 'oddball' expressions found often in kernel code. For example the > following code construction: > if ((struct xyz *) 0x2000 ->element == 29) > Is not in the test suite, neither is: > int > func(i) > register int i; > { > ... > if (((short) i = f(45)) < 0) > /* the line containing the problem... */ It's not clear what you are asking for. Neither of these constructs is legal, either by K&R or ANSI C, though I have no doubt there are some compilers which accept them. An ANSI-conforming compiler will be required to produce a diagnostic on a program that contains either of these examples. Are you asking for a validation suite that guarantees that a compiler will produce the required diagnostic, or do you want it to ensure that a compiler will actually accept such bogus constructs? Walter Murray --
bright@Data-IO.COM (Walter Bright) (08/26/89)
In article <20500@sequent.UUCP> paulr@crg3.UUCP (Paul Reger) writes: >I was wondering what experiences anyone has had with test suites for C. >What I would be interested in is responses from your experiences with >testing a C compiler: >1. What suites do you consider the best ones ?? >2. Are there automatic tools available which produce and evaluate >random sequences of 'oddball' C expressions. >3. How successful are such automatic test case generators ??? Test suites come in 3 flavors: 1. Testing from the user's point of view. This means going through the manual and trying out all the combinations of each behavior described in the manual. I haven't seen the Plum Hall suite, but I assume this is the essence of it. 2. Testing that the optimizer is really functioning, that is, testing that specific code is generated for specific input (this is not the same as 1, which only specifies that the output code *behaves* correctly). 3. Testing every line of code in the compiler. Coverage testers are normally used to verify that every line of code was executed at least once by some function. Unfortunately, only (1) is available as a product, and those general test suites cannot hope to test compiler-specific features. Another major failing in commercial test suites is lack of testing of floating point and transcendental function accuracy. In their effort to make the test portable, the details of floating point are not addressed. I've also seen 'test suites' which consist of simply feeding millions of lines of C code into the compiler. This is largely ineffective because: 1. The C code needs a test suite itself to verify that it compiled correctly, not just that it compiled. 2. Different people have different styles. A million lines of C written by Joe Programmer probably won't test the compiler any better than 10,000 lines written by him, and any atypical things he does could be boiled down to <100 lines. What I mainly rely on is that when I fix a bug, I add the bug report as a test case to my test suite. Over the years, this has resulted in a very comprehensive test suite for C and C++. (Please don't ask me for it, I regard that suite as a major competitive advantage for me!)