RISKS@CSL.SRI.COM (RISKS FORUM, Peter G. Neumann -- Coordinator) (10/03/86)
RISKS-LIST: RISKS-FORUM Digest, Thursday, 2 October 1986 Volume 3 : Issue 73
FORUM ON RISKS TO THE PUBLIC IN COMPUTER SYSTEMS
ACM Committee on Computers and Public Policy, Peter G. Neumann, moderator
Contents:
Lessons from Viking Lander software (Bob Estell)
Software wears out? (Rob Austein)
Wrongful eviction through computer error (Bill Janssen)
Deliberate override (Herb Lin, Ray Chen)
Re: Piper Arrow Gear Override (Douglas Adams)
Undesirable breakins and causes (Ian Davis)
The RISKS Forum is moderated. Contributions should be relevant, sound, in good
taste, objective, coherent, concise, nonrepetitious. Diversity is welcome.
(Contributions to RISKS@CSL.SRI.COM, Requests to RISKS-Request@CSL.SRI.COM)
(Back issues Vol i Issue j available in CSL.SRI.COM:<RISKS>RISKS-i.j.
Summary Contents in MAXj for each i; Vol 1: RISKS-1.46; Vol 2: RISKS-2.57.)
----------------------------------------------------------------------
Date: 2 Oct 86 12:18:00 PST
From: "ESTELL ROBERT G" <estell@nwc-143b.ARPA>
Subject: Lessons from Viking Lander software
To: "risks" <risks@csl.sri.com>
Angry replies? Never! To the contrary, Thank you, Nancy.
In the last few weeks in the "pages" of this journal, we have established:
1. Software cannot be perfect, because it is designed by fallible humans.
2. Hardware cannot be absolutely predictable, adding to the software woes.
3. Very concise programs, though not "perfect" nevertheless have sometimes
run well enough to be considered "successful" - even on a "first chance."
[Other than during simulated tests.]
It's time to move forward. I concur heartily with Nancy's point that what
applies to small programs may not scale up to large programs. It's worth
noting that in her last message, Nancy alluded to the Viking Lander as a
"small" program, with "only" 18,000 words in assembly! How many of you
agree that we might use orders of magnitude to distinguish sizes? e.g.,
IF xx,000 = small, then x,000 = tiny, and x00 = toy; and xxx,000 = large,
and x,000,000 = very large; etc.
Maybe it then follows that xx,000,000 is "possible?" [SDI size?]
Maybe not. Depends a LOT on HOW it's designed, coded, and tested.
I'd like those more knowledgable than I to address some of the following;
maybe the software engineering journal is a better forum; maybe conclusions
could be reprinted in RISKS.
a. What are the measures of "acceptability?"
[Analogy: In baseball, the 1.000 batting average [perfection] is never
possible - except in trivial circumstances, then only with luck.
A century of experience says that .3xx is outstanding. Likewise, the
1.000 fielding average is "impossible." But anything less than .9xx
is terrible.
What are similar failure rates for software? Where should we set our
expectations? Where is the "point of diminishing returns?"
I'm not at all clear on just HOW to set these expectations.
[On a saturated UNIVAC 1110 a few years ago, we were suffering up to
3 crashes a day; we "engineered" that down to less than 3 per month.
Did that improvement make us just average, or much better?]
b. *IF* it's true that "small" programs can run "acceptably" albeit NOT
perfectly, and that "very large" programs probably cannot (?), then
why don't we get serious about building very large programs as orderly
structures of small modules? (Where "small" may mean xx,000 lines.)
At the moment, it seems to me that we're caught on the horns of a
dilemma of our own making; the "idealists" among us are saying that
very large systems cannot be perfect, hence should not be pursued;
the "realists" among us are saying that the present status of large,
real-time systems is a disaster; the "analysts" among us are saying
that there seems to be no good formula for success, yet; and the
"pragmatists" among us are saying that we can make SOME worthwhile
improvements in the status quo, and thus we should.
ALL FOUR VIEWPOINTS ARE "CORRECT."
Isn't it time now for all of us to start "groping" forward together?
Bob
------------------------------
Date: Thu, 2 Oct 1986 00:47 EDT
From: Rob Austein <SRA@XX.LCS.MIT.EDU>
To: risks@CSL.SRI.COM
Subject: Software wears out?
The other sense in which software "wears out" is that people lose
their ability to maintain it. I recently had to work on a mailer
daemon that is about 15 years old. Fine code, possibly one of the
best mailers ever written (certainly for its day), coded according to
good programming practices -- for the early 1970s. I almost went nuts
trying to modify it, people just don't think that way anymore (you
know, labels every ten instructions, GOTOs everywhere...).
I was the person modifying the code because everybody else had better sense
than to even try. At this point I can say with a fair amount of certainty
that -nobody- really understands that program anymore, although the people
who installed the various features (if still alive) usually remember having
done so within a month or so of being asked. And this is a fairly small
program compared to stuff done out in the Real World.
--Rob <BUG-COMSAT@MC.LCS.MIT.EDU>
------------------------------
Date: Thu, 2 Oct 86 19:10:10 CDT
From: Bill Janssen <janssen@mcc.com>
Subject: Wrongful eviction through computer error
To: risks@sri-csl.arpa
An interesting thing happened to me last month. I got home on the 5th of
September to find an eviction notice on my living room floor. Something
about not paying my rent. Well, I gathered up the checks and went over to
the office. Turns out the problem was that I had already paid for October,
as well as September, and the apartment management folks had just switched
to a new computer system! There must have been a line in it something like
if (last_month_paid_for != this_month
AND day == trigger_day_for_eviction)
issue_eviction_notice();
According to some of the office staff, 11 other people had already
been in with similar complaints.
Bill Janssen, MCC Software Technology, 9430 Research Blvd, Austin, Texas 78759
UUCP: {ihnp4,seismo,harvard,gatech,pyramid}!ut-sally!im4u!milano!janssen
------------------------------
Date: Thu, 2 Oct 1986 08:26 EDT
From: LIN@XX.LCS.MIT.EDU
To: George Adams <gba@ICARUS.RIACS.EDU>
Cc: Risks@CSL.SRI.COM
Subject: Deliberate override
From: George Adams <gba at riacs.edu>
Even if an incompetent driver forgot to enable the system on hard
pavement, performance would be no worse than now common. Without the
switch a competent driver might hit that cow instead of stopping in time.
But you are assuming that the average level of competence remains the
same in the presence of the automatic system. For tasks on which you
can enforce some high level of performance (such as flying), this may
be valid. But you can't do it for drivers. It is entirely possible
that drivers will come to RELY on the automated system, perhaps even
without knowing it; their abilities will decline, but they will still
feel capable of overriding the system. That's a recipe for disaster.
I'm not arguing that overrides should be forbidden. I'm saying that
there's a trade-off that has to be addressed before they are installed.
------------------------------
Date: Wed, 1 Oct 86 02:37:57 EDT
From: Ray Chen <chen%gt-stratus%gatech.csnet@CSNET-RELAY.ARPA>
To: RISKS@CSL.SRI.COM
Subject: Deliberate Overrides
Organization: The Clouds Project, School of ICS, Georgia Tech
Manual overrides are a nice idea, but chances are they'll be needed
most during a sudden emergency when there isn't time to think about,
much less trigger any kind of safety override.
How would you like to have to trigger a safety override while powering
into a corner trying to avoid an accident? Ugh.
Personally, I don't see any way around it. Total control after
all is just that -- the ability to specify exactly what the machine is
going to do, even if it's beyond the normal performance envelope.
Saftey restrictions on the other hand are designed to keep you
from exceeding the performance envelope.
There's an inherent contradiction in the two objectives which can't be
neatly resolved unless the safety system and the user/operator are
always in perfect agreement on the limits of the performance envelope
in every possible situation.
I think we have a trade-off here.
Ray Chen
uucp: ...!{akgua,decvax,hplabs,ihnp4,linus,seismo}!gatech!chen
CSNet: chen@GATech ARPA: chen%GATech.CSNET@CSNet-Relay.ARPA
------------------------------
Date: Thu, 2 Oct 86 08:59:20 PDT
From: crash!pnet01!adamsd@nosc.ARPA (Adams Douglas)
To: risks@sri-csl
Subject: Re: Piper Arrow Gear Override
Piper could also have installed the override system because some old lawsuit or
other related to a gear-up landing would have caused their insurance rates to
go through the roof if they didn't implement some kind of 'fix' that they
could point to.
Incidentally, I don't advocate it, the problem of leaving the override on could
be solved by having a flashing panel light next to the other two gear-status
lights on the glareshield such as OVERRIDE ENGAGED. But that would simply add
to the pilot's cockpit stimuli--which is never a good idea during takeoff.
------------------------------
Date: Thu, 2 Oct 86 17:32:33 edt
From: Ian Davis <ijdavis%watdaisy.waterloo.edu@CSNET-RELAY.ARPA>
To: RISKS@CSL.SRI.COM
Subject: Undesirable breakins and causes
Organization: U. of Waterloo, Ontario
Has anyone suggested that hardware can also seriously undermine security? [YES]
I tend to work from home and thus communicate via a modem, and have always
logged off by merely switching from data to voice on my modem, which both
drops the line, and hangs up the phone for me... unfortunately (and initially
unbeknown to me) at least one of the modems that answers incoming calls from
me (at a deliberately unspecified site) was hit by a current surge during a
recent lightning storm, and now no longer drops the communication line to
the CPU when I drop my line to it. This has disasterous consequences since
the next caller to use this recieving modem finds themselves logged into my
account with totally unrestricted access to the system. Fortunately most
users are honest and promptly sign off, but the risks are very real.
The moral for those of you who are concerned, is that one should always
log off from an operating system before dropping communication lines,
and that one should log back on as soon as possible if the line is dropped
accidentally.
Ian Davis
[We have been around that one several times now, although
lightning hitting the modem is a new wrinkle! PGN]
------------------------------
End of RISKS-FORUM Digest
************************
-------