worley@compass.UUCP (Dale Worley) (08/31/88)
The idea of an Ada validation suite always struck me as a great idea. However, several programmers here have been using several Ada compilers, and we keep finding differences between validated compilers. When we look up the differences, we usually find that one or the other compiler is deviating from the Reference Manual in some minor way. This leads me to believe that the validation suite is pretty weak. (This may be a little harsh -- Ada is a huge and complicated language, and an exhaustive validation suite would be very difficult to build.) Does the DoD intend the validation suite to verify compiler compliance? If so, are they actively upgrading the suite as compiler bugs are found? Etc., etc. Dale compass!worley@think.com Compass, Inc. -- the usual disclaimers apply
karl@grebyn.COM (Karl Nyberg) (08/31/88)
... This leads me to believe that the validation suite is pretty weak. Compared to other languages, I think you'd be surprised at just how strong the Ada Compiler Validation Capability is! (I'll forward you some material on other compiler validation suites in private mail.) Does the DoD intend the validation suite to verify compiler compliance? If so, are they actively upgrading the suite as compiler bugs are found? Etc., etc. Yes, they are actively upgrading the ACVC suite. The ACVC is currently at version 1.10, and is updated, I believe, on an annual basis (it may be changing to a bi-annual basis). I seem to recall that the latest set of tests to be added to the suite concerns checking of Chapter 13 features (see below). Apparently, there's also a document known as the DoD Ada Implementation Guidelines (give or take), which describes the objectives intended to be covered by the ACVC. The August 29th issue of Government Computer News has an article titled "DOD Issues Revised Ada Compiler Validation Tests" (p. 53), which makes the following points regarding the latest release of the ACVC: Contains more than 400 changes from the previous release most were "clarifications" 16 corrected inconsistencies in prior tests 27 were rewritten to prevent optimization of the function being tested The current suite contains about 75% of the test objectives mentioned above. Furthermore, the latest issue of the Ada IC Newsletter, July 1988 states: Recognizing the importance of Chapter 13 of the ANSI/MIL-STD 1815A, the AJPO encourages the Ada community to review ACVC 1.10 as soon as possible, and to raise issues regarding this aspect of the Ada language. Send comments to the AVO so we can use them to improve future releases. Ms. Audrey Hook Institute for Defense Analysis 1801 Beauregard St. Alexandria, VA 22311 So, if you have comments about or suggestions for inclusions in the ACVC, there's your ticket! -- Karl --
jon@june.cs.washington.edu (Jon Jacky) (09/02/88)
> (Dale Worley writes) we keep finding differences between validated > compilers. ... we find that one or the other is deviating from the > Reference Manual. ... This leads me to believe that the validation > suite is pretty weak. Limitations in the validation suite are candidly discussed in a very informative paper by one of the people who built it: John B. Goodenough. Ada compiler validation: an example of software testing theory and practice. In: A.N. Haberman and U. Montinari (Eds.), System Development and Ada, Proceedings of the CRAI Workshop on Software Factories and Ada, Capri Italy, May 26 - 30, 1986. Lecture Notes in Computer Science No. 275. New York, Springer-Verlag 1986, pps. 195 - 232 Goodenough wrote: "The test suite today (Version 1.8) represents 2400 programs. ... The test suite is not yet complete. We have identified approximately 1400 additional tests that need to be written just to cover all aspects of the language adequately." - Jonathan Jacky, University of Washington