rws@EXPO.LCS.MIT.EDU (Bob Scheifler) (01/12/90)
Request For Proposals: Xlib and X Protocol Test Suite Development MIT X Consortium January 12, 1990 For Immediate Release The X Testing Consortium has produced a Test Suite covering the Xlib and X Protocol standards of the MIT X Consortium. The current version of this suite is Release T7 (Alpha). The MIT X Consortium would like to see this test suite maintained and enhanced, and is soliciting contract bids for this purpose. The components of Release T7 pertinent to this RFP are the directories Xlib_test.7.0 and Xproto_verf in the distribution, covering Xlib and the X Protocol. These components need a detailed review, to form a better understanding of how complete the coverage really is and what specific areas are not adequately handled. Here are some specific issues that need to be addressed: A detailed review of the existing written test specifications must be performed, and cross-checked with the suite's implementation to discover major discrepancies and omissions. Analysis should include not only "in-the-small" issues, such as whether all possible values for a given parameter are tested, but also more global issues, such as testing of complex window hierarchies and multi-client interactions. Actual code coverage analysis should be performed by running the test suite against an instrumented MIT X11R4 implementation of Xlib and the server, preferably in a configuration supporting multiple screens, depths, and visuals. The algorithms and code used to produce "good pixmaps" (to be supplied by MIT under the contract) need to be evaluated for correctness, and compared with the X11R4 algorithms. The particular pixmap images used in the current suite need to be evaluated to see if they provide appropriate coverage. Test coverage of off-screen graphics, multiple screens, multiple visuals, multiple window depths, and multiple pixmap depths must be reviewed. Test code and test specifications must be reviewed against the X11R4 Xlib and protocol specifications, to find missing or incomplete tests. Some coherent mechanism for relating error reports back to specific test specifications needs to be designed and integrated in to the suite. The output of the suite needs to be reviewed with goal of making the suite suitable for both verification and regression testing. The overall structure of the suite, and the impact of having used an automated test generator to produce the code, must be evaluated to determine the best strategy for long-term maintenance. Documentation of the test suite must be reviewed for completeness and utility. The first major deliverable of the contract will be a report, covering the evaluation performed. The report should: establish what is known to be good about the test suite; detail the major problems found and their proposed solutions; describe the tasks remaining to complete the suite and the time and resources required; and establish an initial prioritization of the work. MIT will review this report and, in cooperation with the subcontractor and interested members of the X Consortium, will establish final priorities and identify additional deliverables, with a staged release plan. The resulting set of deliverables will be a revised version of the test suite, including source code, data files, specifications, and documentation. Bids should be for a fixed time period, covering not only production of the initial report but also the work estimated to arise from the review, based on current knowledge of the test suite. Bids should not exceed two calendar years in duration. If more time is believed to be required, the bid should justify this but still scale back. Negotiation for additional funding may be possible if the actual work exceeds the resources covered by the original contract. A basic breakdown of the budget should be provided with the bid. Bids must include a proposal for when the initial report will be ready, and what resources will be employed to produce it. Bids should specify the number and types of people to be used on the project, their qualifications (particularly with respect to X server internals, Xlib programming, and graphics/windowing testing), the percentage of their time devoted to the project, what areas they will be responsible for, and whether they already exist in the organization or need to be hired for the project. Bids should also specify the hardware and software platforms to be used for development and testing, and whether those platforms already exist in-house. The subcontractor must have reliable electronic mail connectivity to the Internet. MIT will own title to all intellectual property created under the contract. MIT will protect all original property produced under the contract with copyrights and patents as necessary, but will allow eventual unrestricted use and redistribution of materials provided that ownership and credit notices are retained. The subcontractor can hold joint copyright with MIT, subject to these terms. Bids must be received by March 16, 1990. Bids should be sent in hardcopy form (not FAX) to the Director of the MIT X Consortium: Robert W. Scheifler MIT Laboratory for Computer Science 545 Technology Square Cambridge, MA 02139 Questions about this RFP should be addressed to the Director, preferably by electronic mail, but failing that by phone or FAX: Internet address: rws@expo.lcs.mit.edu Phone: (617) 253-0628 FAX: (617) 258-8682 It is expected that a subcontractor will be selected, and contracts signed, on or before May 15, 1990. If your firm does not already have a copy of Release T7, please contact the Director to obtain one.