marick@m.cs.uiuc.edu (07/12/89)
Here's a position paper I wrote for a workshop. I'd be interested in
seeing what you think about it.
Bidirectional Technology Transfer and Testing
A Position Paper for the
Workshop on Directions in Software Analysis and Testing
Brian Marick
Motorola Inc.[1]
1101 E. University
Urbana, Illinois 61801
Internet: Marick@urbana.mcd.mot.com
UUCP: uunet!uiucuxc!mcdurb!marick
(217) 384-8521
In much of industry, testing is an entry-level job filled
with uncreative drudgework. The testing profession is caught in
a vicious cycle - many of the talented, aggressive people escape
as quickly as possible to product development, instead of staying
where their talents could have the most leverage. They learn to
ignore testing, instead of using it. They work apart from the
testers that could help them, so they produce untestable software
whose bugs are discovered by customers.
This will change; economics dictates it. Manufacturing com-
panies are learning that quality is a prerequisite to profitabil-
ity. Software companies will follow. As they do, testing will
receive more attention. In this position paper, I discuss how
this change might happen, what we might do to hasten it, and what
research directions might produce the results industry will need
in the coming years.
I think it likely that the evolution of software quality
will mirror that of manufacturing quality. In manufacturing,
testing is no longer just statistical analysis of products at the
end of the assembly line. Instead, it is integral to the careful
design, monitoring, and continual refinement of the development
and manufacturing processes. Errors must be understood, not just
detected. That understanding is used to reduce the number of
errors, but also to detect them earlier, more cheaply, and more
reliably. Such understanding must be based on teamwork; it
appears only when designers, process engineers, and quality con-
trol people work together throughout the life of the product.
Software has not reached this level of maturity, partly
because the need is not understood, but also partly because
organizations haven't been growing their testers. What can be
done to increase the chance of testers growing into full contri-
butors, into people who are sought out both to plan and perform
testing of programs, designs, specifications, requirements, docu-
mentation, and the software development process itself?
A bad beginning makes a bad ending.
- Euripides
Your first job has a profound effect on the rest of your
career. First positions in testing often teach only that testing
____________________
[1] The opinions in this paper should be considered my own.
is miserable and to be avoided. My position is that we must pro-
duce more college graduates who know better: who are well versed
in general software engineering, in sophisticated testing tech-
niques and tools, and - most importantly - who have the testing
attitude, who understand the role of "designated pessimist",
someone who always asks the question, "how can this not work?"
My analogy is with UNIX[2]. UNIX enjoys its current popu-
larity for many reasons. One is that so many graduates knew it.
They entered industry agitating for its use. (UNIX is hardly an
isolated case, of course - why else do companies donate hardware
and software?) We need to produce bright-eyed and enthusiastic
students who will enter industry expecting to do testing right.
These students must come from universities that teach them test-
ing right.
Right now, few universities have the experience to do that.
Testing is best learned by apprenticeship and iteration. Because
universities have not concentrated on testing, they don't have
enough mentors to teach the apprentices. Practitioners must be
imported from industry.
I envision a university "center of testing excellence" with
faculty, industry practitioners on sabbatical, and both undergra-
duate and graduate students. The first role of the industry tes-
ters will be technology transfer into the university. In partic-
ular, they will transfer the techniques and perspectives that are
used in large-scale testing, and they will transfer an under-
standing of some of the important problems. They will help
develop the coursework and research program.
Once the center has a clear direction and is beginning to
create new technology, the practitioners will take on a new role.
Having facilitated the creation of appropriate technology, they
will begin facilitating its transfer. I will discuss that
transfer after discussing what the center will produce.
The Program
The center of excellence will produce people, techniques,
and tools.
People: It will produce good testers by apprenticeship and
classwork. Testing classes will heavily emphasize projects. The
ideal project is testing another student's class project - the
tester and developer work in a team throughout the entire
development process. Not only does the tester learn through
experience, the developer (one hopes) will learn the value of
having a tester. More advanced students will test other univer-
sity research projects, often those funded by industry - which
will certainly be interested in seeing well-tested research.
Techniques: The center will produce techniques as the result
of research conducted by faculty and by Master's and PhD candi-
dates. A very important area of research is the relationship
between testing and the early phases of software development. A
good question to answer is, "What does design for testability
____________________
[2] UNIX is a trademark of Bell Laboratories.
mean?" In the case of structural testing, it means that the
tests are not only based on the structure of the system, but that
they also influence that structure. In the case of functional
testing, it means exposing more function that can be used to
detect errors more readily. But these are too simple answers -
design for testability is a subset of "good design", and we're
far from understanding that. However, existing research areas
have not yet had their testing potential mined:
(1) Highly integrated environments encourage the creation
of reusable, sensibly partitioned components that are
more likely testable. Some of those components are
debugging tools that are readily adaptable into test-
ing tools.
(2) Object-oriented design and programming is a way of
decomposing a system into metaphors of concreteness
(objects). We're used to manipulating "things" in
various ways, so we're more likely to design an object
such that testing manipulations can take place. In-
heritance makes the modification of systems for test-
ing purposes simpler. Active values allow another
simple way to insert testing probes into a system.
(3) Formal methods of specification and verification in-
teract with testing. How does one test the adequacy
of a large specification? How can specifications be
used to drive automated test generation? How can for-
mal methods be merged into the tester's repertoire?
(4) Also important are time-motion studies. What do pro-
grammers really do? If there's a bug, what caused it?
- and why wasn't the test to catch it written? What
went wrong?
Further, there are hard kinds of systems whose testing is
not well understood. The center should concentrate on these.
Optimizing compilers, for example, are quite difficult to test,
largely because the set of classes of inputs is so large.
Operating systems and embedded systems are also hard, for an
additional reason - the inputs (including clock interrupts, power
failures, and the like) are very hard to control. With graphical
user interfaces, the output is the problem: it's often difficult
to express the expected results of an operation in a way that
allows automatic comparison to the actual results (which are,
themselves, often difficult to capture).
Tools: The center should create a freely-available tester's
toolchest, so that its graduates can quickly demonstrate their
skills with proper tools. Like the easy availability of DoD pro-
tocols in Berkeley UNIX, this could dramatically improve the
state of the practice. One of the most important tools would be
a general-purpose test harness for running tests and gathering
them into test suites. This might be built on a flexible hyper-
text system. My experience is that hypertext links work well for
integrating tests with production code. They allow developers to
easily examine tests, run them, and perhaps write new ones.
Technology Transfer
The center will produce the techniques and tools incremen-
tally, by applying them to projects. Each project will begin by
discarding the failures and incorporating the successes of previ-
ous projects. Because universities are free to fail, to try
promising approaches that don't work out, they are more efficient
at this cycle than industry can be. But what's the point of
pushing the state of the art if the state of the practice remains
constant?
Developing technology is hard, but effective technology
transfer is harder yet. Most typically, techniques are
transferred indirectly. Companies hire graduates who know the
techniques; these graduates apply them, often in a quite limited
way. They're limited because the techniques are based on tools
that are not transferred.
The center is designed to be more efficient at technology
transfer of both techniques and tools:
(1) The center will produce expert graduates. Experts are
more likely to be allowed to use ambitious new tech-
niques and tools. Some of the "graduates" will be
practitioners, people who already have strong track
records.
(2) The practitioners will guide the center toward the
creation of appropriate and adoptable technology,
based on iterated practical projects and the idea of a
portable tester's toolkit.
(3) Part of the technology transfer problem is that
researchers and development organizations are usually
separated - just as old-style product development
separates design and manufacturing engineers. So,
just as old-style designs are difficult to manufac-
ture, research results are difficult to apply. The
solution is the same: closer teamwork. The testing
center implements this solution by putting practition-
ers into both worlds: they work at the center, but for
their company.
A major problem will be finding exceptional practitioners:
good at testing and teaching, firmly practical while understand-
ing academic and theoretical issues, and with good visibility
into their parent corporation. If we can find these people, per-
suade them to join the center, and persuade their companies to
loan them, we stand a good chance of transferring technology,
both directly and indirectly.dave@bigsky.uucp (Dave Hughes) (07/21/89)
Test worked