RISKS@CSL.SRI.COM (RISKS FORUM, Peter G. Neumann -- Coordinator) (08/30/86)
RISKS-LIST: RISKS-FORUM Digest, Saturday, 30 August 1986 Volume 3 : Issue 46
FORUM ON RISKS TO THE PUBLIC IN COMPUTER SYSTEMS
ACM Committee on Computers and Public Policy, Peter G. Neumann, moderator
Contents:
Human error (Nancy Leveson, Lindsay F. Marshall)
Re: F-16 Tales (Earl Boebert, Phil Ngai)
Correction to note about flight simulators (Martin Minow)
Supermarket grinds to a halt (David Sherman)
Video processing (Guy Schafer)
ATMs (Jacob Palme)
The RISKS Forum is moderated. Contributions should be relevant, sound, in good
taste, objective, coherent, concise, nonrepetitious. Diversity is welcome.
(Contributions to RISKS@CSL.SRI.COM, Requests to RISKS-Request@CSL.SRI.COM)
(Back issues Vol i Issue j available in CSL.SRI.COM:<RISKS>RISKS-i.j.
Summary Contents in MAXj for each i; Vol 1: RISKS-1.46; Vol 2: RISKS-2.57.)
----------------------------------------------------------------------
Date: Fri, 29 Aug 1986 18:48 EDT
From: LIN@XX.LCS.MIT.EDU
To: Nancy Leveson <nancy@ICSD.UCI.EDU>
Cc: risks@CSL.SRI.COM, lin@XX.LCS.MIT.EDU
Subject: Human error
From: Nancy Leveson <nancy at ICSD.UCI.EDU>
... My real quibble is with the term "computer errors". Since I do
not believe that computers can perform acts of volition (they tend to
slavishly and often frustratingly follow directions to my frequent
chagrin),..
While I agree with the above sentiment for the most part, it strikes me that
there is a sense in which it is not true. Consider an situation in which a
very large computer program is used to control a very complex real-time
process. Imagine that the designers are generally competent, but do not
understand very well certain aspects of the process or the environment in
which the process operates. Something that they did not anticipate happens,
and the software performs an action that results in a disaster. How are we
to describe this occurrence?
We can describe this as a design (and thus human) error. But given
the circumstances, it is not implausible to argue that an "Act of God"
occurred. It was a rare happening; we didn't understand it very well;
we could not predict that it would happen. Is there a better
definition for "Act of God"? While computers are in principle
deterministic, a sufficiently complex program operating in a highly
varied environment is for all practical purposes non-deterministic, in
that its behavior is unpredictable under some circumstances.
The problem with calling such an event an "Act of God" is that it
leaves open the door to using that kind of defense when it is entirely
unwarranted; I personally do not want to get away from the idea that
human beings are (or should be) in control of their computers. I'm
only pointing out that sometimes they may not be, through no "fault"
of their own. Maybe computers should be used only in situations that
we understand very well, and that are predictable. My safety
instincts say that this should be true, but it's not clear that it is
practical.
... the term "computer error" is
also misleading since to me ... it seems to imply
some sort of volition on the part of the computer as if it were acting on
its own, without any human influence, to do these terrible things.
While I agree it is misleading for the reasons I gave above, from the
standpoint of being a user trying to behave defensively (as a driver
drives defensively) it could be useful -- no user should approach a
computer system as though its behavior is predictable and/or sensible
under all circumstances, and in the absence of the system designers
and programmers, that is probably an adaptive strategy from an
evolutionary perspective.
------------------------------
From: "Lindsay F. Marshall" <lindsay%kelpie.newcastle.ac.uk@Cs.Ucl.AC.UK>
Date: Fri, 29 Aug 86 12:08:04 bst
To: risks@csl.sri.com
Subject: Re: Human Error
Someone who has looked at the changing attitudes to "human error" as
against "mechanical failure" is Michael Lesk. He has been studying
reports of railway accidents in the UK to extract from them information
about the attitudes of the reporters and investigators towards the
causes of the accidents. I don't know if he has written this up anywhere
or not, nor do I know if he reads RISKS. He is well worth talking to
about the subject however and has uncovered some exceedingly
interesting points.
Lindsay F. Marshall
[Will someone at Bell Labs who reads this please give Mike a nudge? PGN]
------------------------------
Date: Fri, 29 Aug 86 10:51 CDT
From: Boebert@HI-MULTICS.ARPA
Subject: Re: F-16 Tales
To: Neumann@CSL.SRI.COM
ReSent-To: RISKS@CSL.SRI.COM
Weight on wheels is a basic sensor input that tells the flight program
whether or not the aircraft is airborne. In advanced systems like the
F-16 its is probably confirmed by air data computer and inertial
platform inputs; in older systems, where the computer does just nav and
weapons delivery, it is the prime indicator. It is therefore unlikely
in the extreme that this would be overlooked in a design or an ordnance
safety analysis ((weight_on_wheels = TRUE) & (master_arm = TRUE) &
(weapon_release = TRUE) is clearly an undesired state). I am also
skeptical that the gear would be controlled by the flight computer, but
I am not familiar with the F-16 so cannot comment further.
------------------------------
Date: Fri, 29 Aug 86 19:57:30 pdt
From: amdcad!phil@decwrl.DEC.COM (Phil Ngai)
To: risks@csl.sri.com
Subject: F-16 software
It sounds very funny that the software would let you drop a bomb on the wing
while in inverted flight but is it really important to prevent this? Is it
worth the chance of introducing a new bug to fix this very minor problem? Is
it worth the chance of making the code too big to fit in memory? What is the
chance that a pilot would really make this mistake?
[The probability is clearly NONZERO. It is very dangerous to start
making assumptions in programming about being able to leave out an
exception condition simply because you think it cannot arise. Such
assumptions have a nasty habit of interacting with other assumptions
or propagating. PGN]
------------------------------
Date: 29-Aug-1986 1406
From: minow%regent.DEC@decwrl.DEC.COM (Martin Minow, DECtalk Engineering ML3-1/U47 223-9922)
To: risks@csl.sri.com
Subject: Correction to note about flight simulators
In a private mail exchange, Danny Cohen ("COHEN@B.ISI.EDU") was
kind enough to point out that I had mis-remembered my article
from Smithsonian where I claimed the article stated that a flight
instructor flew as a flight engineer on a commercial flight.
> The plane encountered a wind-shear situation on take off. The
> instructor, from his flight engineer's position, reminded the pilot
> that the correct recovery for wind-shear is opposite to the correct
> recovery for a stall (which has a similar appearance to the pilot)."
According to Danny (I can't find my copy of this issue), the article does
not talk about anything being "opposite to the correct recovery for the stall."
I'm sorry for the confusion this might have caused anyone. At least, I did
learn a lot about flying and recovery from dangerous conditions. Danny did
ask me to clarify my purpose in submitting the article to RISKS -- whether
it was to show that computer-based simulators contribute to airline safety,
or to "highlight the risks in using computers for whatever purposes." To
set the matter straight, it was to show that computer-based simulation is a
factor in increased airline safety, as it lets pilots learn about situations
that are either dangerous or unusual (or both) in real life.
Danny is still looking for pointers to accidents caused by computer-based
simulators.
Martin
[I don't mean to take a potshot at Martin, who has been a delightful
contributor. But PLEASE, all of you, if you see something that you
think is appropriate for RISKS, make a note of it at the time rather
than subsequently half-remember it. I keep a huge stack of old items
next to my terminal just in case I have to dig back... PGN]
------------------------------
From: mnetor!lsuc!dave@seismo.CSS.GOV
Date: Fri, 29 Aug 86 17:04:53 edt
To: mnetor!seismo!CSL.SRI.COM!RISKS@seismo.CSS.GOV
Subject: Supermarket grinds to a halt
Last week I went to our local Miracle Food Mart supermarket (in
northern Toronto) at 9 a.m. on a Sunday, when they were just opening.
They discovered that they couldn't get any of the cash registers
to work; something was down in the central system. So they had the
cashiers writing each number down on a pad of paper and totalling
them up by hand, which slowed checkout down to a crawl. After a
while, someone found a desk calculator with a paper tape, which made
things a bit faster. When I left they had someone at the door warning
customers not to bother coming in because the terminals weren't working.
Obviously, this kind of thing can happen only where cash registers
are no longer cash registers but terminals connected to a central
system, which is becoming more and more the case. I can't believe
MFM doesn't have some type of backup system, since they're a large
chain. My speculation is that someone wasn't prepared for the system
to be running on Sunday morning; supermarkets must be closed in
Ontario on Sundays, and the ones near us started opening only about
a month ago...
David Sherman, The Law Society of Upper Canada, Toronto
{ ihnp4!utzoo seismo!mnetor utzoo hcr decvax!utcsri } !lsuc!dave
------------------------------
Date: Thu, 28 Aug 86 15:38:31 edt
From: decwrl!amdcad!amdimage!prls!philabs!linus!axiom!gts@ucbvax.Berkeley.EDU (Guy Schafer)
To: CSL.SRI.COM!RISKS
Subject: Video processing
Now that sophisticated hardware for capturing and altering video images
exists for even the modest IBM-PC (AT&T's Truevision products), several
concerns arise:
Because images can be captured in real time (for less than $5000), and
it has been proven that at least one method exists for over-powering
('hi-jacking') a cable video broadcast, some program can be altered and
re-broadcast (with a delay equal to the video processing time). This
could be especially dangerous if it is done to, say, 2 minutes of a
news broadcast or televised political proceedings.
Video post-processing can also be an effective means to control the behavior
of an individual by tapping directly into her cable coming into her house.
An appropriate stock tip given by a seemingly authentic Ruekeiser (sp?) might
cause a major stockholder to get on her phone to a broker with predictable
(and thus profitable) results. We always knew a hacker with a PC had quite
a bit of power; if this hacker can alter someone's main source of
information (television broadcasts) he suddenly has quite a bit more.
Also, video tapes which are used as evidence in court can be changed without
simple means of detection.
Actors can be cheated out of royalties--especially in commercials where
post-processing 30 seconds of video could cost less than the royalties of
an often-repeated performance. The features of the face can be "airbrushed"
or distinguishing marks can be added or removed (by software--e.g. TIPS by
AT&T) and the actor told that someone else got the part.
Any comments?
>< ...{ decvax!linus | seismo!harvard }!axiom!gts
------------------------------
Date: 30 Aug 86 16:57 +0200
From: Jacob_Palme_QZ%QZCOM.MAILNET@MIT-MULTICS.ARPA
To: "RISKS FORUM" <RISKS@CSL.SRI.COM>
Subject: ATMs (RISKS-3.45)
>>..Their dispensing machines cannot be cheated in this way, because they have
>>a steel door in front of the machine which does not open until you insert a
>>valid plastic card.
>
>People who swindle ATM's don't have cash cards?????
If you have a legally obtained cash card, and insert it into the
machine, this act is immediately recorded, so that if the swindle
is detected, they can find out who did it.
If you have an illegally obtained cash card, you probably do not
know the password you have to input on the keyboard. When you
input the wrong password (or do not input any password at all),
the machine swallows the card, and you never get it back again.
At least that is the way the Swedish ATM's work.
------------------------------
End of RISKS-FORUM Digest
************************
-------