rfg@pink.ACA.MCC.COM (Ron Guilmette) (07/07/89)
c++-suite-1.00 -- now available for anonymous FTP As many readers of this group already know, I have for some time been developing a test/validation suite for C++ language processors. My employers (the Microelectronic and Computer Technology Corp.) have seen the wisdom in making this suite generally available, and it has been officially donated to the Free Software Foundation. The suite is now available for anonymous FTP and can be obtained (as a compressed tar file) from yahi.stanford.edu in the file called ~ftp/pub/c++-suite-1.00.tar.Z. The size of this file is 92409 bytes. There is currently no documen- tation for the suite. The intent of the suite is to test for conformance to the "current definition" of the C++ language. Although there was no "standard" available to guide its construction, the suite only tests for features which have been described by Bjarne Stroustrup (in published writings) as official parts of "the language". This includes newer features available in Cfront 2.0 (and G++) but does *not* include such things as templates and delegation-by-pointer (which are specula- tive at this point in time). The suite really needs a great deal of additional work but, to the best of my knowledge, no other compiler test suites for C++ currently exist, so this suite is (by definition) better than anything else you can get for this purpose at this time. As you can tell, I'm not entirely proud of this suite yet, but I'm making it available now because many people have asked for it and because it may be useful for some people until something better comes along. User are encouraged to add additional tests to the suite. Each such added test will bear the name of the person who developed the test. Note that this suite comes without warranty or support of any kind. Be sure to read the file called COPYING in the top level directory for an explanation of your rights and responsibilities as a user of the suite. The suite consists of a set of "driver" routines (written as Bourne shell scripts) and a set of actual test files. The scripts are currently setup to parse GNU-style error mes- sages. They have to be altered slightly to get them to parse other styles of error messages (e.g. cfront). I don't know how to do this right now since I don't even have a copy of cfront that I can actually execute myself. There are about 150+ individual test files in the suite, and each of these files generally tries to (torture) test one feature in many different ways. Thus, the total number of individual "tests" made by the suite is several times the number of "files". The suite is currently patterned loosely on the Ada Compiler Validation Suite (ACVC). I may modify this organization drastically at some point in the future. There are simple tests that ought to pass compilation, and if they do, they are generally considered to have "PASSED". These are the class A tests. The class B tests check that various errors which SHOULD be detected at compile time ARE detected at compile time. These tests seem to give C++ language proces- sors the most trouble currently. Like the ACVC, each of the B tests contains certain lines with compile-time detectable errors in them and these lines also contain the string "// ERROR". The test driver (script) attempts to see that all lines so marked are flagged by the compiler with at least one error. If each such line in a given test file is flagged by the compiler as having an error, then the test is said to have "PASSED". Extra errors and/or warnings are not significant in evaluating the results of B tests. The class C tests are executable tests. These tests must compile and link and execute without errors and they must return a zero exit code or else they are assumed to have "FAILED". Other classes of tests exist, but they are currently unused. See the ACVC for more information. Note that the rules of the Ada Validation Organization (AVO) are rather liberal when it comes to evaluating the results of class B tests. Specifically, the AVO allows these tests to be broken up into smaller pieces if a compiler under test cannot recover from one type of error sufficiently well to detect other errors later in the same file. My intent is that this "escape clause" (which allows for a total absence of reasonable error recovery) should NOT be applied when testing C++ language translators. Thus, C++ language pro- cessors SHOULD be able to pass all class B tests without breaking up the test files. The (current) C++ language is roughly comparable to the Ada language in complexity and number of features. Thus, a "good" test suite would include about the same number of test files (i.e. 3000+). Based on this (very rough) esti- mate, I have to conclude that my current suite is no more that about 5% of what a good suite would be. I would like to continue to work on this suite, but I would need to find some person or organization willing to sponsor such activity. If you or your organization would like to provide such sponsorship (in whole or in part) please con- tact me. Perhaps the completion of this suite could be done as a "joint venture" with several sponsors. The completion of the suite would benefit users more than vendors, but the whole C++ community would benefit. I plan to be begin a PhD program in Computer Science in the Fall, but I believe that I could make time for additional work on the suite if encouraged to do so. Note that the intent of the suite is *not* to encourage con- troversy about language features (there is plenty of that already). Rather, its intent is similar to that of the current (draft) ANSI C standard, i.e. to codify existing practice. One last word about the testing of C++ language processors (or the lack thereof). If you can find ANY C++ language processor that passes all of these tests, BUY IT! You may want to run the suite against your favorite language proces- sor just for the thrill of finding out about a whole lot of things that it *won't* do. I hope that people will run the suite on each of the C++ language processors now available and publish the (terse) results here. Please send comments and suggestions regarding the suite to the E-mail address given below. -- // Ron Guilmette - MCC - Experimental Systems Kit Project // 3500 West Balcones Center Drive, Austin, TX 78759 - (512)338-3740 // ARPA: rfg@mcc.com // UUCP: {rutgers,uunet,gatech,ames,pyramid}!cs.utexas.edu!pp!rfg