perk@infmx.UUCP (Paul Perkovic) (02/22/90)
In <1990Jan26.093013.3379@log-sv.se> gunnar@log-sv.se (Gunnar Bostr|m) asks: >Are there any tools or validation suites for testing SQL-databases >for how they conform to the SQL-standard? In the United States, the National Institute of Standards and Technology (NIST), a component of the United States Department of Commerce, has responsibility for establishing standards for Federal Government use. Computer-related standards are adopted as Federal Information Processing Standards (FIPS) and are often cited in government procurements. Early this year (February 2, 1990), NIST published in the Federal Register a revision of FIPS 127, which establishes ANSI X3.135-1989 (Database Language SQL with Integrity Enhancement) for Federal procurements. (The ANSI text is, I believe, identical to the ISO text in 9075:1989.) "The testing of language processors to determine the degree to which they conform to the Federal Standards is required by Government departments and agencies in accordance with [citations omitted]. As a part of its mission, the National Computer Systems Laboratory (NCSL) is responsible for providing language processor validations for Federal Information Processing Standards (FIPS) in support of Government departments and agencies procurement requirements." (from a NIST draft document) Currently, the Software Standards Validation Group within NCSL at NIST has available for licensing a suite of several hundred tests designed to cover the syntax rules and general rules of the SQL standard in any of nine environments: embedded COBOL, FORTRAN, Pascal, or C; module language COBOL, FORTRAN, Pascal, or C; or interactive SQL. For further information, contact: Joan Sullivan National Computer Systems Laboratory Building 225, Room A266 National Institute of Standards and Technology Gaithersburg, Maryland 20899 UNITED STATES OF AMERICA 1-301-975-3258 Joan distributed Software Validation Request Forms for SQL at the most recent ANSI X3H2 (Database) standards committee meeting. NIST expects to begin their validation service in April of this year. Some observations: o These tests are designed to be machine independent; they use standard language constructs for the host languages but may need to be modified for a particular hardware/software environment. o These tests attempt to verify that basic language formats and syntax rules are enforced, but they are not exhaustive tests by any means. The test database contains only a few tables, they each contain only a few rows, and most tests are very simple. (Apparently they contain too many tables for at least one vendor, though, based on a comment Joan made during a recent presentation!) They do not attempt to test how many levels of subquery nesting an implementation supports, or how many columns in a table, or how many joined tables in a query, etc. o NIST originally planned to give a certification even if a vendor was unable to pass all of the tests, allowing a grace period of one year to come into full compliance. They now seem to have changed their approach, and are talking about requiring 100% compliance for certification. (A Certificate of Validation gets your product listed in the quarterly Validated Compiler List.) Most recently they have begun to talk about a certified test report (they run the tests at your site and produce a report), with certification postponed until after a trial period. o Even though the test suite is by no means exhaustive, every DBMS vendor with whom I have spoken has raised questions about one or more of the NIST tests, suggesting that each product currently has at least one deficiency. So don't count on buying a fully conforming, certified product off the shelf anytime too soon. o Note that even if two implementations are both certified as passing the validation test suite (and thus presumably conforming to the ANSI and ISO standards), an identical application program run against the two implementations might well obtain different results as a consequence of the many implementor-defined or undefined aspects of the standard (e.g., character sets, collating sequence, precision and scale of arithmetic results). Since I work for a vendor, not NIST, I don't know what their policy would be on licensing the validation suite outside the U.S. Here, anyone with about $1795 can purchase a license to use the test suite on a single CPU, and (for only slightly more money) a license to run the tests on many CPUs. For a company like us, with more than 150 platforms supported, the multi-CPU license was a good deal. / perk perk@informix.com ...{pyramid|uunet}!infmx!perk Paul Perkovic (415) 591-7700 anytime Informix Software, Inc. (415) 926-6821 17 Rinconada Cir./ Belmont, CA 94002 4100 Bohannon Dr./ Menlo Park, CA 94025 This is an informed but not an official statement of Informix Software, Inc.