wood@uunet.uu.net (Tom Wood) (08/27/88)
The following is based on an inquiry from Dec 1987. I've had several inquiries concerning my findings on language validation, verification, and test suites, so I'm posting my current references. I've organized this according to language. I'm still in the process of tracking down these leads, and I cannot attest to the accruacy of this information. I have sent away for more information on six of the C suites. If you can add any information to this posting, please let me know. =============================== ADA =============================== ACVC - Ada Compiler Validation ACEC - Suite to test Ada performance. The University of Michigan has a suite of Ada tests and benchmarks. Don't forget the DoD Ada validation suite! It's supposed to be one of the most strenuous around, and is applied to all Ada compilers before they are allowed to use the name (since Ada is a registered trademark of the DoD, they can do that). ============================= ALGOL68 ============================= There used to be an ALGOL68 test set. This set was maintained by Dick Grune while working at the CWI [Amsterdam]. His current address is dick@cs.vu.nl. He can give you more details about the suite. ================================ C ================================ SVVS - System V Verification Suite also tests the C compiler, especially the libraries. (From AT&T) ACE C VALIDATION SUITE - Excerpts from a press release below. Checks the compliance of a C compiler to the ANSI X3J11 draft standard. Also tests library functions for conformance to the X/OPEN standard or the SVID where these differ from their draft standard counterparts. The suite consists of over 50000 lines of source in about 600 different programs together performing over 2000 tests. The ACE C Validation suite is priced at 15500 dutch guilders (currently approximately $7500). Contact Ge Gaal at info@ace.nl or ..!uunet!mcvax!ace!info ACE Associated Computer Experts bv Phone: +31 20 646416 Van Eeghenstraat 100 Telex: 11702 (ace nl) 1071 GL Amsterdam Fax: +31 20 750389 The Netherlands MetaWare sells a small C validation suite. $2,000 903 Pacific Ave, Suite 201 Santa Cruz, CA 95060 (408) 429-6382 The Plum Hall Validation Suite for C $10,000 Plum Hall 1 Spruce Ave. Cardiff, NJ 08232 (609) 927-3770 The PERENNIAL Validation Suite for C Compiler Validation PERENNIAL 4677 Old Ironsides Drive, Suite 450 Santa Clara, CA 95054 (408) 727-2255 C Compiler Torture Test - Checks a compiler against K&R. $20 The Austin Code Works 11100 Leafwood Lane Austin, TX 78750-3409 (512) 258-0885 HCR offers a C Test Suite in various forms (50,000 tests to 350,000 tests) HCR Corporation Phone: (416) 922-1937 130 Bloor Street West Telex: 06-218072 HCR TOR Suite 1001 Fax: (416) 922-8397 Toronto, Ontario Canada M5S 1N5 Unix had SQ and CQ (Grampst) for Unix and C, but these are AT&T internal. ============================= FORTRAN ============================= Though it was not available outside Hewlett-Packard when last I knew, I thought you might like to know about it for completeness. I helped to design a testing suite for HP FORTRAN 77 aproximately 4 years ago. Back then there were ~6000 tests and about 750K lines of FORTRAN code in the test suite. Inquiries could be made to the Computer Language Lab, 11000 Wolfe Road, Cupertino, CA. There was a smaller testing suite for Pascal. The Federal Govt validation suites for Fortran and Cobol are not available to the general public. The last time I heard, a single test run cost a vendor about $15K, making it useless for those of us who develop language-processing tools. Even the reports of the tests may be of little use to users; I'm told they consist mainly of pass/fail tables. Fortran: US Department of Commerce National Technical Information Service Springfield, VA ============================== PASCAL ============================= PVS - BSI Pascal Compiler Validation Service (BS 6192/ISO 7185) This is a third party validation service. It includes access to the validation suite by the vendor. Developed in part by the University of Tasmania and the National Physical Labratory. Adopted by the Federal Software Management Support Center (part of the GSA) in 1986. Assessment Department Phone: 0442 3111 British Standards Institution Telex: 82424 Maylands Avenue Hemel Hempstead, Herts HP2 4SQ Federal Software Management Support Center General Services Administration Two Skyline Place, Suite 1100 5203 Leesburg Pike Falls Church, VA 22041 Pascal: Richard J Cichelli, President Software Consulting Services 3162 Bath Pike Nazareth, PA (215) 837-8484 The University of Tasmania in Australia distributed a Pascal Validation Suite as of 1981. Arthur Sale of UNSW did the Pascal Validation Suite available from ANSI (indirectly, they can tell you who to get it from). ============================== SNOBOL ============================= Algorithms in SNOBOL - Actually, this is a book, but the programs discussed are sufficiently complex (and I mean COMPLEX) that they are accepted as a good SNOBOL compiler test suite. --- Tom Wood Data General, Research Triangle Park, NC {the known world}!mcnc!rti!xyzzy!wood [For Fortran, there are two test suites available from the NTIS. Both were for Fortran 66 the last time I tested, which was quite a while ago. The simpler is from the Navy, which is a small set that is quite easy to pass. The other was developed by the NBS and did a good job of testing grotty things like arithmetic statement functions with complex results. -John] -- Send compilers articles to ima!compilers or, in a pinch, to Levine@YALE.EDU Plausible paths are { ihnp4 | decvax | cbosgd | harvard | yale | bbn}!ima Please send responses to the originator of the message -- I cannot forward mail accidentally sent back to compilers. Meta-mail to ima!compilers-request
eachus@harvard.edu (Eachus) (08/30/88)
>If you can add any information to this posting, please let me know. >=============================== ADA =============================== >ACVC - Ada Compiler Validation This is the official validation test suite. Currently about 4000 files, my guess is about 20000 actual tests. (Some tests must be in multiple files, but many of the test programs contain multiple tests. A real fun experience -- if you are into S&M. >ACEC - Suite to test Ada performance. This is being developed by the Boeing Military Airplane Company under government contract. First release is scheduled to occur any day now. It will be distributed by the Data and Analysis Center for Software (DACS) at Griffiss AFB, Rome NY. >The University of Michigan has a suite of Ada tests and benchmarks. These are useful, but for the most part require intrumentation to Get accurate timings. A better set of benchmarks is the ACM SIGAda Performance Issues Working Group (PIWG) test suite, which is the suite most vendors use when publishing perfomance figures. >Don't forget the DoD Ada validation suite! It's supposed to be one >of the most strenuous around, and is applied to all Ada compilers >before they are allowed to use the name (since Ada is a registered >trademark of the DoD, they can do that). There is no question that the ACVC is the most strenuous around, and the standards are MUCH stricter than for other validations. Before the validation you submit a list of tests which you think don't apply to your validation and why. (Most of these are usually for floating point precision beyond what the compiler supports.) With this you provide a listing from running all of the tests, and a few other things. (This is the infamous "box of paper".) If the validation agency approves your submission, they come on site to verify the results themselves. Failing any test means that you fail the validation. No second chances, no fixing bugs on the spot. If you pass you get to do it all again next year. Incidently, Ada is no longer a registered trademark of the DoD. The trademark was intentionally allowed to lapse last year as having accomplished its purpose. Robert I. Eachus -- Send compilers articles to ima!compilers or, in a pinch, to Levine@YALE.EDU Plausible paths are { decvax | harvard | yale | bbn}!ima Please send responses to the originator of the message -- I cannot forward mail accidentally sent back to compilers. Meta-mail to ima!compilers-request
wood@uunet.uu.net (Tom Wood) (06/14/91)
[Questions on validation suites arrive all the time. This message from August 1988 is the most complete report on validation suites that I have seen. It's nearly three years old, so I expect that some of the information is out of date. Additions and corrections are welcome. I'll summarize the updates in a few weeks. -John] The following is based on an inquiry from Dec 1987. I've had several inquiries concerning my findings on language validation, verification, and test suites, so I'm posting my current references. I've organized this according to language. I'm still in the process of tracking down these leads, and I cannot attest to the accruacy of this information. I have sent away for more information on six of the C suites. If you can add any information to this posting, please let me know. =============================== ADA =============================== ACVC - Ada Compiler Validation ACEC - Suite to test Ada performance. The University of Michigan has a suite of Ada tests and benchmarks. Don't forget the DoD Ada validation suite! It's supposed to be one of the most strenuous around, and is applied to all Ada compilers before they are allowed to use the name (since Ada is a registered trademark of the DoD, they can do that). ============================= ALGOL68 ============================= There used to be an ALGOL68 test set. This set was maintained by Dick Grune while working at the CWI [Amsterdam]. His current address is dick@cs.vu.nl. He can give you more details about the suite. ================================ C ================================ SVVS - System V Verification Suite also tests the C compiler, especially the libraries. (From AT&T) ACE C VALIDATION SUITE - Excerpts from a press release below. Checks the compliance of a C compiler to the ANSI X3J11 draft standard. Also tests library functions for conformance to the X/OPEN standard or the SVID where these differ from their draft standard counterparts. The suite consists of over 50000 lines of source in about 600 different programs together performing over 2000 tests. The ACE C Validation suite is priced at 15500 dutch guilders (currently approximately $7500). Contact Ge Gaal at info@ace.nl or ..!uunet!mcvax!ace!info ACE Associated Computer Experts bv Phone: +31 20 646416 Van Eeghenstraat 100 Telex: 11702 (ace nl) 1071 GL Amsterdam Fax: +31 20 750389 The Netherlands MetaWare sells a small C validation suite. $2,000 903 Pacific Ave, Suite 201 Santa Cruz, CA 95060 (408) 429-6382 The Plum Hall Validation Suite for C $10,000 Plum Hall 1 Spruce Ave. Cardiff, NJ 08232 (609) 927-3770 The PERENNIAL Validation Suite for C Compiler Validation PERENNIAL 4677 Old Ironsides Drive, Suite 450 Santa Clara, CA 95054 (408) 727-2255 C Compiler Torture Test - Checks a compiler against K&R. $20 The Austin Code Works 11100 Leafwood Lane Austin, TX 78750-3409 (512) 258-0885 HCR offers a C Test Suite in various forms (50,000 tests to 350,000 tests) HCR Corporation Phone: (416) 922-1937 130 Bloor Street West Telex: 06-218072 HCR TOR Suite 1001 Fax: (416) 922-8397 Toronto, Ontario Canada M5S 1N5 Unix had SQ and CQ (Grampst) for Unix and C, but these are AT&T internal. ============================= FORTRAN ============================= Though it was not available outside Hewlett-Packard when last I knew, I thought you might like to know about it for completeness. I helped to design a testing suite for HP FORTRAN 77 aproximately 4 years ago. Back then there were ~6000 tests and about 750K lines of FORTRAN code in the test suite. Inquiries could be made to the Computer Language Lab, 11000 Wolfe Road, Cupertino, CA. There was a smaller testing suite for Pascal. The Federal Govt validation suites for Fortran and Cobol are not available to the general public. The last time I heard, a single test run cost a vendor about $15K, making it useless for those of us who develop language-processing tools. Even the reports of the tests may be of little use to users; I'm told they consist mainly of pass/fail tables. Fortran: US Department of Commerce National Technical Information Service Springfield, VA ============================== PASCAL ============================= PVS - BSI Pascal Compiler Validation Service (BS 6192/ISO 7185) This is a third party validation service. It includes access to the validation suite by the vendor. Developed in part by the University of Tasmania and the National Physical Labratory. Adopted by the Federal Software Management Support Center (part of the GSA) in 1986. Assessment Department Phone: 0442 3111 British Standards Institution Telex: 82424 Maylands Avenue Hemel Hempstead, Herts HP2 4SQ Federal Software Management Support Center General Services Administration Two Skyline Place, Suite 1100 5203 Leesburg Pike Falls Church, VA 22041 Pascal: Richard J Cichelli, President Software Consulting Services 3162 Bath Pike Nazareth, PA (215) 837-8484 The University of Tasmania in Australia distributed a Pascal Validation Suite as of 1981. Arthur Sale of UNSW did the Pascal Validation Suite available from ANSI (indirectly, they can tell you who to get it from). ============================== SNOBOL ============================= Algorithms in SNOBOL - Actually, this is a book, but the programs discussed are sufficiently complex (and I mean COMPLEX) that they are accepted as a good SNOBOL compiler test suite. --- Tom Wood Data General, Research Triangle Park, NC {the known world}!mcnc!rti!xyzzy!wood [For Fortran, there are two test suites available from the NTIS. Both were for Fortran 66 the last time I tested, which was quite a while ago. The simpler is from the Navy, which is a small set that is quite easy to pass. The other was developed by the NBS and did a good job of testing grotty things like arithmetic statement functions with complex results. -John] -- Send compilers articles to compilers@iecc.cambridge.ma.us or {ima | spdcc | world}!iecc!compilers. Meta-mail to compilers-request.