risks@CSL.SRI.COM (RISKS Forum) (03/20/91)
RISKS-LIST: RISKS-FORUM Digest Tuesday 19 March 1991 Volume 11 : Issue 31
FORUM ON RISKS TO THE PUBLIC IN COMPUTERS AND RELATED SYSTEMS
ACM Committee on Computers and Public Policy, Peter G. Neumann, moderator
Contents:
Untested mods to amusement rides... (Peter da Silva)
The true risks of computerized voting (Hank Nussbacher)
California, driving, and privacy, again (Chris Hibbert)
Can't stop auto-withdrawal (Rick Simkin)
Re: Let your fingers do the walking... (Chris Thomas)
Re: About Risks in Believing AI Gurus (M.Minsky) (Bob Frankston)
Re: Telephone risks revisited (Kian-Tat Lim)
Re: Pilot Error - an impartial assessment? (Jerry Hollombe)
Re: Drawing correct conclusions from Miami NTSB report (Jerry Hollombe)
Re: "What the Laws Enforce" (Nancy Leveson)
Telecommunications Risks (Nigel Allen)
The RISKS Forum is moderated. Contributions should be relevant, sound, in
good taste, objective, coherent, concise, and nonrepetitious. Diversity is
welcome. CONTRIBUTIONS to RISKS@CSL.SRI.COM, with relevant, substantive
"Subject:" line. Others ignored! REQUESTS to RISKS-Request@CSL.SRI.COM. For
vol i issue j, type "FTP CRVAX.SRI.COM<CR>login anonymous<CR>AnyNonNullPW<CR>
CD RISKS:<CR>GET RISKS-i.j<CR>" (where i=1 to 11, j always TWO digits). Vol i
summaries in j=00; "dir risks-*.*<CR>" gives directory; "bye<CR>" logs out.
ALL CONTRIBUTIONS CONSIDERED AS PERSONAL COMMENTS; USUAL DISCLAIMERS APPLY.
Relevant contributions may appear in the RISKS section of regular issues
of ACM SIGSOFT's SOFTWARE ENGINEERING NOTES, unless you state otherwise.
----------------------------------------------------------------------
Date: 16 Mar 91 23:41:35 CST (Sat)
From: peter@taronga.hackercorp.com (Peter da Silva)
Subject: Untested mods to amusement rides...
Recently at Astroworld they decided to do a mod to their new ride, the
"Condor", for its initial public performance. They were giving a bunch of
reporters the first shot, so they decided to make it stop at the top to
give them plenty of time to take pictures.
Apparently they didn't test the change before the big day, because it went up
then got stuck halfway down, and it took 20 minutes of coaxing before they got
the press off.
Since this ride involves swinging people around at 40 MPH, it could have been a
*lot* worse! Astroworld officials insist that thanks to daily safety checks
there was never any danger, but I think we've all heard that before.
------------------------------
Date: Sun, 17 Mar 91 12:34:42 IST
From: Hank Nussbacher <HANK@VM.BIU.AC.IL>
Subject: The true risks of computerized voting
The following is a true story that happened on Wednesday March 13. Tel Aviv
University decided to hold this year's student elections via computer. 23
voting stations were set up, a VAX/VMS system was dedicated to the one day
project and a special voting program was developed.
Since every faculty had different students running for office and since
different students would arrive at the polling booth, the menu screen was
designed with a dynamically built vote selection. The menu would list 20 or so
candidates and the student would select their choices at the bottom. The
computer would then display the following menu with the selected choices in
positions 21, 22, 23 ... of the initial menu. The user could either accept
their choices by pressing enter or go back to the main menu.
When the user hit enter, the number of choices selected were to be extracted
from positions 21, 22, 23, etc and placed into the vote file. By mistake, the
program always took positions 1, 2, 3 from the original menu. When the voting
was done and the tallies were counted it was spotted that in every faculty the
people listed in the 1st, 2nd and 3rd positions were the ones that always won.
A quick check of the program showed the flaw. A single line of code with an
incorrect initial pointer destroyed 3,000 votes.
The test data that was run through the program did not catch this flaw since
the checking was done by name and not by number and since there were dozens of
names involved, no one took the time to double check that indeed the names
voted for the most were indeed the ones who won.
This week, Tel Aviv University will be doing a manual re-vote.
Hank Nussbacher, Israel
------------------------------
Date: Tue, 19 Mar 91 11:19:48 PST
From: hibbert@xanadu.UUCP (Chris Hibbert)
Subject: California, driving, and privacy, again
Reply-To: hibbert@xanadu.com
The California Department of Transportation (CalTrans) has been studying
Automatic Vehicle Identification (AVI) systems, which would be used to automate
the charging of tolls on bridges and highways throughout California. They've
asked for comments on a proposal titled "Compatibility Specification for
Automatic Vehicle Identification Equipment". The specification calls for an
active transponder that could be installed securely on vehicles. When it is
queried the transponder responds with a radio burst that would contain, among
other things, the VIN issued by the vehicle's manufacturer. This is a
first-rate RISK to privacy.
The system makes it possible for highway authorities (and anyone else who can
build or buy a reader that meets the spec.) to track the movements of anyone
who signs up in order to get faster service at tollbooths. The most plausible
charging system using this kind of reader requires that records be kept long
enough for disputes to be resolved. As we have seen here before, those records
(or their backups) can be examined with a subpoena or with the cooperation of
the vendors. (CalTrans would probably pay the vender of the system to maintain
the records, though that's not discussed in the spec.)
Other RISKS evident in this preliminary spec include the RISK of over-
specification. The stated accuracy requirement is "incorrect transaction
decodings and encodings shall not exceed 1 in 20,000". The box is required to
work without change in accuracy over a temperature range of -40 to +180 degrees
Fahrenheit; with vehicle spacing as close as 15 feet; at all speeds up to 100
mph; and with up to "6 inches of dry uncompacted snow," "2 inches of salt water
ice or slush," or "1 inch of dirt sand or salt" obstructing the path between a
transponder and reader.
There are quite a few alternative designs that don't require an identifier that
is traceable to an individual. At the low end there are low-tech
low-discrimination systems like color-coded or bar-coded monthly window
stickers (with many people getting the same sticker each month.) These allow
charging a monthly fee and charging a price based on number of occupants and
time of day (require paying manually in a separate lane on days when the car is
less full or traveling at rush hour.) They don't support charging based on the
amount of use within a month.
A more sophisticated system would use unique (but not traceable to individuals)
identifiers. The user has an anonymous account to deposit money into and money
would get deducted each time past a toll collector. No one has to maintain an
association between account numbers and people's names. New Orleans apparently
uses this kind of a system. Drivers get individual boxes, and pay into
anonymous accounts. If anyone can give me more details about this system, I'd
appreciate it.
The most sophisticated system might have active transponders in the car that
themselves maintained records about how much money had been deposited. The
toll collector would send a packet asking the box whether it had enough money
and then to deduct the current charge. When the box indicated it was running
low, it could be recharged like a Pitney-Bowes postage meter. This scheme
would obviously require much more security in the box and reasonably
sophisticated encryption mechanisms in order to make sure that the same packet
couldn't be used more than once to pay for trips, etc.
Any AVI system design that requires that records be kept to do billing or
resolve disputes allows tracking of individual movements. This would lessen
our individual privacy, which in California at least is protected by the state
Constitution.
Comments on the proposed specification can be sent to Les Kubel, Office of
Electrical and Electronics Engineering, California Department of
Transportation, 5900 Folsom Boulevard, PO Box 19128, Sacramento, CA 95819. I
have a scanned copy of the proposed specification. You can get a copy from me
or from that same address. You can also call and ask for a copy at
(916)739-2245. I wasn't able to find out what deadlines have been set when I
talked to Les.
Chris
------------------------------
Date: Mon, 18 Mar 91 8:20:39 CST
From: rsimkin@dlogics.dlogics.com (Rick Simkin)
Subject: Can't stop auto-withdrawal
Some banks offer a service which pays bills directly out of your checking
account. This service is started with a single authorization for each company
which will have this privilege. This could be considered a risk all by
itself--NO customer action is required for the bank to pay out whatever the
creditor bills. If there's an error in billing, the customer finds out only
when the bill comes, with the annotation "Paid by bank draft." At this point,
the money is already gone--there's no way to withold payment in the case of a
dispute.
But I discovered a more alarming risk this month: at the bank I have in mind,
this service, once started, continues forever. Although direct drafts can be
authorized by a signature given to the bank, they can't be stopped that way.
Instead, the customer has to tell the billing party, "Stop removing money from
my account," and keep doing so until the billing party stops. The bank
employee blamed this setup on the computer.
Maybe the bank's programmers studied under the Sorcerer's Apprentice?
Rick Simkin, Datalogics, Inc., 441 W. Huron St., Chicago, Illinois 60610-3498
uunet!dlogics!rsimkin rsimkin@dlogics.com +1 312 2664437
------------------------------
Date: Mon, 18 Mar 91 09:12:50 PST
From: thomas@sono.UUCP (Chris Thomas)
Subject: Re: Let your fingers do the walking...
John McMahon mentions the California Department of Transportation (Caltrans)
road-information service, and how it failed to have any information on the
quake-damaged Embarcadero freeway.
Some experimentation with the system reveals two facts:
- the default condition is that "the route you have selected
is reported open with no driving restrictions"; and
- there is no check to verify that the input is an actual
highway number in California. (Unless someone has created
a highway 9999 without telling anyone!)
Since Caltrans filed paperwork to have the Embarcadero officially removed
from the state system (sometime in 1990, I believe), highway 480 "no longer
exists", and Caltrans thus has no information about it.
Compare this with the information provided for another quake-damaged
freeway, I-280. The message correctly identifies the mile-long closure.
Chris Thomas thomas@sono.uucp (415) 969-9112 x2994
------------------------------
Date: Wed, 13 Mar 91 15:19 GMT
From: Bob Frankston <Bob_Frankston%Slate_Corporation@mcimail.com>
Subject: Re: About Risks in Believing AI Gurus (M.Minsky) (RISKS-11.19,21)
There are risks of unthinkingly believing various philosophies be they AI,
religion or other maintstream philosophies. It is not clear what the
particular risk is in the case of Minsky. I view him as a source of
questions not final answers.
Our understanding of what "intelligence" is minimal. I wouldn't expect
computers to be the same as humans short of synthesizing people from DNA.
But the idea of a symbiotic relationship between computers and people both
with highs levels of intelligence (whatever that is) is not that far fetched.
Klaus Brunnstein's claim: "In my analysis, it is this kind of misconceptions
that are an esssential reason for contemporary computer accidents -
misconceptions of scientists (uncritically following paradigms and unverified
assumptions), top- and medium-level-misconceptions, misimplementations,
misunderstandings on the users' side, conscious use of side effects and other
misuse..." seems to trivialize real issues in systems design and engineering.
------------------------------
Date: Thu, 14 Mar 91 19:44:35 GMT
From: ktl@wag.caltech.edu (Kian-Tat Lim)
Subject: Re: Telephone risks revisited (RISKS-11.27)
I'm sure many people will point out the similarity of this scenario to that in
John Brunner's 'The Shockwave Rider'.
Kian-Tat Lim (ktl@wag.caltech.edu, KTL @ CITCHEM.BITNET, GEnie: K.LIM1)
[Actually noone else did, but I am always amused at how many of
our current risks were in some way anticipated by Brunner. PGN]
------------------------------
Date: 12 Mar 91 21:12:58 GMT
From: hollombe@ttidca.tti.com (The Polymath)
Subject: Re: Pilot Error - an impartial assessment? (RISKS-11.)
Twenty-some years ago, when I worked for the L.A. County Engineer,
Aviation Division, I got FAA accident reports across my desk almost daily.
Without doing any statistical analysis, one relationship seemed to stand
out: If the pilot's dead, it's his fault.
Plus ca change ...
Jerry Hollombe, Citicorp, 3100 Ocean Park Blvd., Santa Monica, CA 90405
(213) 450-9111, x2483 {rutgers|pyramid|philabs|psivax}!ttidca!hollombe
------------------------------
Date: 19 Mar 91 02:30:35 GMT
From: hollombe@ttidca.tti.com (The Polymath)
Subject: Re: Drawing correct conclusions from Miami NTSB report (RISKS-11.29)
While the analysis of danorman@UCSD.EDU (Donald A Norman-UCSD Cog Sci Dept) is
probably true in the broadest sense, I can give a single "reason" for the
accident: The need to have an o-ring in the first place. Had the plugs been
designed to not require an o-ring, the accident wouldn't have happened.
Is such a design reasonable? Very much so. The Boeing 727 is designed to
operate without gaskets or o-rings. All joints where you would expect such are
simple metal to metal (or so I was taught in mechanic's school, some 22 years
ago. I admit I've never actually been that intimate with a 727).
The unfortunate fact is, it's cheaper to build and maintain a design requiring
gaskets and o-rings. The acceptable tolerances are much looser.
Possible moral: You get what you pay for.
Jerry Hollombe, Citicorp, 3100 Ocean Park Blvd., Santa Monica, CA 90405
(213) 450-9111, x2483 {rutgers|pyramid|philabs|psivax}!ttidca!hollombe
------------------------------
Date: Tue, 19 Mar 91 07:37:30 -0800
From: Nancy Leveson <nancy@ICS.UCI.EDU>
Subject: Re: "What the Laws Enforce"
The analogy here is not a good one. A more reasonable analogy is someone
breaking into your home or business and perusing your personal files.
"Breaking and Entering" IS a felony. After the Morris attack that precipitated
this discussion, I lost three days of work because the computers had to be
turned off and "cleansed." I don't consider that the equivalent of picking
flowers on my lawn but of someone breaking into my home, trashing the place,
and making it unlivable until I spend three days away from work cleaning it up.
Considering the number of lives and careers nationwide that were disrupted, it
is not unreasonable that this was highly publicized or that our society would
want to strongly discourage such behavior by making it a crime and not a
misdemeanor. Although such laws may not discourage true criminal behavior,
they do discourage potentially destructive "play" by essentially law-abiding
people.
In fact, personal and business privacy and property is extremely important in a
complex, crowded society such as ours. Many science fiction horror stories
about future societies involve the invasion of personal privacy -- whether it
is by the government or my fellow citizens matters little to me. A society
that values my privacy less than the curiousity of others about my personal
property is not a pleasant one to consider. And yes, there are serious and
important societal costs involved. Right now, I am being hampered in my
attempts to work on an important system that may save lives (or cost them if
done wrong) because a company with which I need to deal has had to severely
restrict outside computer access because of security fears. Draconian security
measures to prevent frivolous access and pranks (in situations where it would
not otherwise be necessary because there is nothing of value to steal) will
hurt us all and cost our society untold dollars and perhaps worse. What about
the kids who broke into the Sloan-Kettering Cancer Center and fooled around
with the patient records, changing some as a result? It would be surprising to
me that readers of Risks that value privacy highly would consider invasion of
privacy a misdemeanor on the same level as walking on the grass.
------------------------------
Date: Sun, 17 Mar 91 19:50:21 EST
From: ndallen@contact.UUCP (Nigel Allen)
Subject: Telecommunications Risks
Many telecommunications-related risks discussed here might also be of interest
to readers of the Telecom Digest, which is distributed within Usenet as the
comp.dcom.telecom newsgroup. If you would like to submit a
telecommunications-related article to the Telecom Digest, send it to
telecom@eecs.nwu.edu. If you have access to Usenet newsgroups, you can read
the Telecom Digest as the comp.dcom.telecom newsgroup; if not, you can be added
to the mailing list by sending a message to telecom-request@eecs.nwu.edu.
I'm not the moderator of the Telecom Digest; Patrick Townson is.
[Occasionally someone kindly forwards RISKS-relevant TELECOM stuff to me,
and Patrick sometimes picks up RISKS stuff. I'm not sure he is prepared
to handle the enormous volume of TELECOM RISKS that I am currently
experiencing, but for those of you who also are reading TELECOM, that
might be a better place for some of the more detailed stuff. However,
RISKS cuts across all disciplines and technologies, and I think it is
important that its perspective be as inclusive as possible -- precisely
because the risks themselves transcend artificial boundaries and because
the lessons to be learned do also. Thanks, Nigel. Peter]
------------------------------
End of RISKS-FORUM Digest 11.31
************************