Kimbrough%dsg@ti-csl.CSNET (Kerry Kimbrough) (06/19/87)
Who can tell me about the availability of testing tools for Xv11? Of particular interest are tools to test a Xv11 server implementation. Are there any server certification suites (hopefully in the form of language-binding-independent protocol) available or planned? Will anything like HP's v10 tmon exist soon? (Jon Brewster, are you out there?!!) Also of interest would be tools for testing or debugging X clients.
larry@hpcvlo.HP.COM (Larry Woestman) (06/22/87)
In a response following this one are the minutes of an X testing Consortium meeting. Larry Woestman Member of Technical Staff Corvallis Workstation Operation Hewlett Packard Co. 1000 NE Circle Blvd. Corvallis, Or. 97330 email: hplabs!hp-pcd!larry phone: (503) 750-4111
larry@hpcvlo.HP.COM (Larry Woestman) (06/22/87)
This is message 1 from the X testing distribution list. If you are missing
messages, contact:
Larry Woestman
Hewlett Packard Co.
Corvallis Workstation Operation
1000 NE Circle Blvd.
Corvallis, Or. 97330
email: hplabs!hp-pcd!larry
phone: (503) 750-4111
These are the notes for the X testing meeting held on May 19, 1987 at
Wilsonville, Or. They are being sent to the contact persons identified at that
meeting. I am also mailing paper copies of the slides that were presented by
Jon Brewster and Larry Jones to the contact persons.
Martha Zimet (of Sun) has written a test plan document for her company that
incorporates much of what went on at the meeting. I am including it at the end
of this message so that we can use it as an additional basis for discussion.
--------------------------- start of notes --------------------
The meeting was organized to cover the following topics:
1. How the various companies can work together to create a broad range
of test suites.
2. discuss HP's and DEC's approach to testing X
3. structure of test suites
4. X version 11 protocol enhancements required to support testing
5. test suite distribution
It was held in the HP sales office in Wilsonville, Oregon on May 19, 1987. It
was attended by representatives of the following companies (hereafter called
"The Consortium") that are using X:
Apollo
Brown University
Data General
Digital Equipment Corporation
Electro Scientific Industries (an HP OEM)
Hewlett Packard
Integrated Solutions
Masscomp
Mentor
National Semiconductor
Sequent
Sun Microsystems
Tektronix
meeting agenda:
9:00 - 9:15 opening remarks
9:15 - 9:45 introductions
9:45 - 10:30 HP approach to X testing
10:30 - 10:45 break
10:45 - 11:15 DEC approach to X testing
11:15 - 11:45 brainstorm problems/issues
11:45 - 12:15 brainstorm objectives statement
12:15 - 1:30 lunch / "tmon" (HP test program) demo
1:30 - 2:30 resolve some of the issues
2:30 - 2:40 break
2:40 - 3:30 solutions
3:30 - 4:00 action items / wrapup
4:00 - 5:00 social hour
opening remarks:
objectives:
definition - what
methodology - how
participation - togetherness
HP approach to X testing:
50/50 QA/implementation effort
explained resource locking
device drivers could be modified instead of X server hooks
against:
no transportibility
only X group affected
color map set up in X11 is critical for screen match
sampling - color map and pixels
server tells a lot - interrogate the server
DEC approach to X testing:
testing xlib (and X server indirectly)
transportable code (C) over different OS's (VMS, sysV, BSD, ...)
verify bad as well as good parameters
granularity of tests - subsets needed
manpower - 8 full time developers writing tests (july 87)
concentration on xlib, not clients
test generator - produces testing database and C test source
automatic insertion of test cases
performance test(s) included in test suite
Bruce Coorpender - Tektronix is developing a testing tool for uwm
problems/issues:
client test methodology
connectivity - communications, path verification
methods for supplying portable clients
language binding testing - standardization of various bindings
what do we want to test?
creation of generalized performance criteria for server implementation
continued participation by vendors, how do we proceed?
do we share freely in the results of this group? (concensus was yes)
method for supplying portable tests
maintenance of suite(s)
handling of optional features/ extentions to protocol/server
who/how to provide the baseline (who is going to count the pixels?)
how do we resolve device dependent areas of X?
input synthesis extention needed
cannonical pixmaps needed
partitioning of testing
what level of stress testing do we do?
what level of commitment do the various parties have in this effort?
establishment of technical contact list for each participating company
schedule
X10 or X11 or both?
what do we provide as tools?
server verifier to verify protocol messages (both ways)
xlib verifier (with byte order permutations)
objectives statement (this is preliminary):
The impetus to develop an X Version 11 verification suite results from
three questions posed by all parties involved in implementing X
servers:
Does my own implementation work according to the standard?
I've changed my implementation, does it still work?
Will my implementation work across other hardware platforms?
An X Version 11 verification suite will help ensure the capability for
application importation and help ensure server interoperability.
By promoting cooperation among Consortium members and minimizing a
duplication of effort, more portable test programs will be available
than any one company can afford to produce. Test programs will be
available for X version 11 clients, servers, and the C language
interface to xlib.
other objectives agreed to by the consortium:
focus on X Version 11
favor public distribution via MIT if possible
test the xlib level and the protocol level (bidirectionally)
communication between companies will be via email distribution list
next meeting time and agenda will be discussed via distribution list
There was little interest shown in sharing test suites for clients,
except for xterm and uwm. However, there was a great deal of interest
shown in sharing tools to use to test clients.
action items:
Hewlett Packard has agreed to set up and maintain a electronic mail
distribution list that will serve as the forum for discussion about X
testing by the consortium members.
Hewlett Packard has also agreed to prepare a proposal for an extension
to the X version 11 protocol to allow a client to create events inside
the server so that it looks to any client like the user hit a key or
moved the mouse. The proposal will be reviewed by the consortium
before being proposed to the general X community.
Tektronix has agreed to attempt to get MIT to accept the deliverables
listed below for public domain distribution along with the rest of the
X version 11 code.
Integrated Solutions has agreed to make plans for a "connectathon",
where any company in the consortium can bring their equipment and test
interoperability with other company's equipment.
Sun, Apollo, and any other company in the consortium that wishes to
have agreed to provide to DEC test cases that will help expose any
possible problems with the graphics implementation in an X version 11
server.
deliverables:
Hewlett Packard has agreed to prepare an extension to the X version 11
protocol to allow a client to create events inside the server so that
it looks to a specified client like the user hit a key or moved the
mouse.
Hewlett Packard has also agreed to prepare generic client exerciser and
session capture and repeat tools derived from "tmon", a test monitor
that has been developed at HP.
Digital Equipment Corporation has agreed to prepare a set of functional
tests for the X version 11 xlib and server combination. This set of
tests will be written in C, and will be as operating-system independent
as possible.
Mentor has agreed to prepare an X version 11 server stress tester.
Sequent has agreed to prepare an X version 11 protocol verifier.
GSS has agreed to prepare the definition and code needed for comparison
of cannonical pixmaps. Cannonical pixmaps are information from a
display that is in a form that allows comparisons with information from
other displays that may differ in resolution, number of color planes,
or other details. This is needed so that tests can be written that are
independent of (or at least dependent only on a small number of
parameters) the particular display being tested.
Tektronix has agreed (I think) to prepare a set of functional tests for
xterm (a VT102 terminal emulator) and uwm (a window manager).
what's next?
The members of the Consortium have agreed to have a "first pass" of
their assigned deliverables completed by June 15, 1987. They will be
distributed by the email distribution list for review. At that time,
the second Consortium meeting will be scheduled.
------------------- start of Martha Zimet's document ------------------------
DATE : 20 May 1987
TITLE : X Version 11 Validation Test Plan
REVISION: Draft 1.0
AUTHOR : Martha Zimet (mzimet@sun.COM)
Member of Technical Staff
Sun Microsystems
1. INTRODUCTION
1.1. BACKGROUND
This document is a FIRST DRAFT definition of the strategy for the
X Version 11 validation test, as discussed by a Consortium of vendors
and other interested parties, on 19 May 1987.
This document describes the test objectives, the components to be tested,
the organizations responsible for and participating in the validation test
effort, the methodology and techniques to be applied, and will be updated
and revised based on requests by the Consortium to do so.
It is the intent of the Consortium to place the X Version 11 validation
test suite into the Public Domain.
1.2. OBJECTIVES
The impetus to develop a comprehensive X Version 11 validation
suite results from three questions posed by all parties involved
in implementing X servers:
1.) Does my own implementation work according to the standard?
2.) I've change my implementation, does it still work?
3.) Will my implemenation work across hardware platforms?
A comprehensive X Version 11 validation suite will ensure the capability
for foreign application importation and ensure server interoperability.
By promoting the cooperation among Consortium members and minimalizing
a duplication of effort, portable test programs will be produced to validate
the conformance of X Version 11 server implementations to the X Version 11
Protocol.
1.3. WHAT TO DO WITH THIS DOCUMENT
Please read this document and reply to the author, by electronic mail, with
your comments, questions, concerns, and other input.
1.4. DISTRIBUTION
o The Consortium:
Apollo
Brown University
Data General
DEC
ESI
HP
Integrated Solutions
Masscomp
Mentor
National Semiconductor
Sequent
Sun Microsystems
Tektronix
o Interested parties at Sun Microsystems:
Jack Collins
Smita Deshpande
Steve Evans
Don Lundquist
Darrell Miller
David Rosenthal
Warren Teitelman
Martin Wong
2. STRATEGY
2.1. COMPONENTS TO BE TESTED
The validation test suite will include test cases for the following
X Version 11 components:
1. XLIB
2. Cononical pixmaps
3. Input synthesis extensions
4. X protocol
5. Xterm
In addition, the validation test suite will include test cases derived
from the following methodologies:
1. Stress testing
2. "Guarantee" the generation of graphics anomalies
3. Generic client excerciser
4. Session capture and repeat
2.2. COMPONENTS NOT TO BE TESTED
Other than the "generic client" excerciser mentioned in the previous
section, test cases for server clients will not be generated by the
Consortium.
2.3. STANDARDS and CONVENTIONS
No testing standards and conventions have been defined by the Consortium
at the present time.
However:
1. Each Consortium member particpating in test development has
agreed to develop "comprehensive" test suites, given their time
and resource limitations.
2. Each member will determine their own test design methodology,
or combination of methodolgies, to be utilized (eg. path or decision
coverage, random-input, boundary-value analysis, etc.)
3. Each member should attempt to develop a subset of all possible
test cases that has a high probability of detecting errors.
4. Each member should attempt to develop portable test cases.
3. APPROACH
This section will be completed as each Consortium member develops and
documents test designs and test cases for their specific area of
responsibility.
4. TEST SUITE CERTIFICATION
As test suites are developed, each member of the Consortium has agreed to
certify other members' test cases. An area of special concern to all
members is the "graphical" correctness of test cases.
5. TEST DELIVERABLES and RESPONSIBILITIES
For each test deliverable, the responsibility of the member includes
test design, implementation and documentation:
1. Functional tests for XLIB - DEC
2. Stress tester - Mentor
3. X protocol verifier - Sequent
4. Cononical pixmaps - GSS
5. Input synthesis extensions - HP
6. Test certification/graphical correctness - et al.
7. Sample test cases for "breaking" graphics - Sun, Apollo, et al.
8. Generic client excerciser (tmon) - HP
9. Session capture and repeat - HP
10. Xterm - Tektronix
11. Validation test plan - Sun
6. SCHEDULE
The members of the Consortium have agreed to have a "first pass" of their
assigned tests completed by 15 June 1987. At that time, the second
Consortium meeting will be scheduled.
7. DEPENDENCIES and UNRESOLVED ISSUES
At the end of the first Consortium meeting, the following issues were
still unresolved:
1. Connectivity communcations
2. Language bindings test standards
3. Generalized performance criteria
4. Method for supplying portable tests
5. Maintenance of test suite
6. Handling of "optional" features
7. Resolution of device dependent areas of X
8. Verification of vendor committments
9. Schedule
------------------- end of Martha Zimet's document ------------------------larry@hpcvlo.HP.COM (Larry Woestman) (06/22/87)
An additional note: Future meetings of the X testing consortium are planned. If your company is interested in commiting resources to help develop X tests, you may want to send a someone to the next meeting. I will post a notice of the time and place of the next meeting when I know what they are. Larry Woestman Member of Technical Staff Corvallis Workstation Operation Hewlett Packard Co. 1000 NE Circle Blvd. Corvallis, Or. 97330 email: hplabs!hp-pcd!larry phone: (503) 750-4111
jdm@gssc.UUCP (John D. Miller) (06/25/87)
Pardon me, but our representatives were at the Consortium meeting in
Wilsonville. I believe we have signed up for cannonical pixmap testing.
--
-- jdm "Caution, sleep turbulence."
in real life: John D. Miller, Graphic Software Systems (GSS), Beaverton, OR
...!{tektronix!verdix}!sequent!gssc!jdm (503) 641-2200larry@hpcvlo.HP.COM (Larry Woestman) (06/26/87)
oops! I did it again! I forgot to include GSS in the original report, and then I forgot to correct the report before posting it to this group. I apologize again to GSS. Larry Woestman Member of Technical Staff Corvallis Workstation Operation Hewlett Packard Co. 1000 NE Circle Blvd. Corvallis, Or. 97330 email: hplabs!hp-pcd!larry phone: (503) 750-4111