rfg@MCC.COM (Ron Guilmette) (07/07/89)
c++-suite-1.00 -- now available for anonymous FTP
As many readers of this group already know, I have for some
time been developing a test/validation suite for C++
language processors.
My employers (the Microelectronic and Computer Technology
Corp.) have seen the wisdom in making this suite generally
available, and it has been officially donated to the Free
Software Foundation.
The suite is now available for anonymous FTP and can be
obtained (as a compressed tar file) from yahi.stanford.edu
in the file called ~ftp/pub/c++-suite-1.00.tar.Z. The size
of this file is 92409 bytes. There is currently no documen-
tation for the suite.
The intent of the suite is to test for conformance to the
"current definition" of the C++ language. Although there
was no "standard" available to guide its construction, the
suite only tests for features which have been described by
Bjarne Stroustrup (in published writings) as official parts
of "the language". This includes newer features available
in Cfront 2.0 (and G++) but does *not* include such things
as templates and delegation-by-pointer (which are specula-
tive at this point in time).
The suite really needs a great deal of additional work but,
to the best of my knowledge, no other compiler test suites
for C++ currently exist, so this suite is (by definition)
better than anything else you can get for this purpose at
this time.
As you can tell, I'm not entirely proud of this suite yet,
but I'm making it available now because many people have
asked for it and because it may be useful for some people
until something better comes along.
User are encouraged to add additional tests to the suite.
Each such added test will bear the name of the person who
developed the test.
Note that this suite comes without warranty or support of
any kind. Be sure to read the file called COPYING in the
top level directory for an explanation of your rights and
responsibilities as a user of the suite.
The suite consists of a set of "driver" routines (written as
Bourne shell scripts) and a set of actual test files. The
scripts are currently setup to parse GNU-style error mes-
sages. They have to be altered slightly to get them to
parse other styles of error messages (e.g. cfront). I don't
know how to do this right now since I don't even have a copy
of cfront that I can actually execute myself.
There are about 150+ individual test files in the suite, and
each of these files generally tries to (torture) test one
feature in many different ways. Thus, the total number of
individual "tests" made by the suite is several times the
number of "files".
The suite is currently patterned loosely on the Ada Compiler
Validation Suite (ACVC). I may modify this organization
drastically at some point in the future. There are simple
tests that ought to pass compilation, and if they do, they
are generally considered to have "PASSED". These are the
class A tests. The class B tests check that various errors
which SHOULD be detected at compile time ARE detected at
compile time. These tests seem to give C++ language proces-
sors the most trouble currently. Like the ACVC, each of the
B tests contains certain lines with compile-time detectable
errors in them and these lines also contain the string "//
ERROR". The test driver (script) attempts to see that all
lines so marked are flagged by the compiler with at least
one error. If each such line in a given test file is
flagged by the compiler as having an error, then the test is
said to have "PASSED". Extra errors and/or warnings are not
significant in evaluating the results of B tests. The class
C tests are executable tests. These tests must compile and
link and execute without errors and they must return a zero
exit code or else they are assumed to have "FAILED".
Other classes of tests exist, but they are currently unused.
See the ACVC for more information.
Note that the rules of the Ada Validation Organization (AVO)
are rather liberal when it comes to evaluating the results
of class B tests. Specifically, the AVO allows these tests
to be broken up into smaller pieces if a compiler under test
cannot recover from one type of error sufficiently well to
detect other errors later in the same file. My intent is
that this "escape clause" (which allows for a total absence
of reasonable error recovery) should NOT be applied when
testing C++ language translators. Thus, C++ language pro-
cessors SHOULD be able to pass all class B tests without
breaking up the test files.
The (current) C++ language is roughly comparable to the Ada
language in complexity and number of features. Thus, a
"good" test suite would include about the same number of
test files (i.e. 3000+). Based on this (very rough) esti-
mate, I have to conclude that my current suite is no more
that about 5% of what a good suite would be.
I would like to continue to work on this suite, but I would
need to find some person or organization willing to sponsor
such activity. If you or your organization would like to
provide such sponsorship (in whole or in part) please con-
tact me. Perhaps the completion of this suite could be done
as a "joint venture" with several sponsors. The completion
of the suite would benefit users more than vendors, but the
whole C++ community would benefit. I plan to be begin a PhD
program in Computer Science in the Fall, but I believe that
I could make time for additional work on the suite if
encouraged to do so.
Note that the intent of the suite is *not* to encourage con-
troversy about language features (there is plenty of that
already). Rather, its intent is similar to that of the
current (draft) ANSI C standard, i.e. to codify existing
practice.
One last word about the testing of C++ language processors
(or the lack thereof). If you can find ANY C++ language
processor that passes all of these tests, BUY IT! You may
want to run the suite against your favorite language proces-
sor just for the thrill of finding out about a whole lot of
things that it *won't* do. I hope that people will run the
suite on each of the C++ language processors now available
and publish the (terse) results here.
Please send comments and suggestions regarding the suite to
the E-mail address given below.
// Ron Guilmette - MCC - Experimental Systems Kit Project
// 3500 West Balcones Center Drive, Austin, TX 78759 - (512)338-3740
// ARPA: rfg@mcc.com
// UUCP: {rutgers,uunet,gatech,ames,pyramid}!cs.utexas.edu!pp!rfgkearns@read.columbia.edu (Steve Kearns) (07/07/89)
Re the test suite: What is the standard tested... g++ or c++? LALR parsing or BS parsing? A test suite like this can go a long way toward DEFINING a standard, if vendors perceive a value in passing it.