[news.sysadmin] Getting bugs fixed

dplatt@coherent.com (Dave Platt) (11/24/88)

In article <2518@aplcomm.jhuapl.edu> trn@aplcomm.jhuapl.edu (Tony Nardo) writes:

> No, we sometimes need to fix a problem as soon as it is discovered.

> We don't want to wait *years* waiting for a vendor to patch a problem,
> and only have the problem addressed when an Internet
> worm/virus/demon-from-hell hits. 

> In my experience working for "FUBAR" (I prefer the proper initial
> slang), if you told that company about a bug *ONLY THROUGH PROPER
> CHANNELS*, maybe six or seven people would hear about it.  The problem
> would then be effectively buried until a sufficent number of customers
> (and potential customers) started talking about the bug amongst
> themselves.

> Worse, even when knowledge of a bug did reach the general engineering
> community in "FUBAR", management clones discouraged work on fixing the
> problem unless (a) some customer was footing the bill, (b) a chemical
> plant blew up (or something equally embarassing) and legal action was
> threatened, or (c) ex-customers started warning potential customers
> away.

The more things change, the more they stay the same...

Back in the mid-1970s, several of the system support staff at Motorola
(I believe it was) discovered a relatively simple way to crack system
security on the Xerox CP-V timesharing system (or it may have been
CP-V's predecessor UTS).  Through a simple programming strategy, it was
possible for a user program to trick the system into running a portion
of the program in "master mode" (supervisor state), in which memory
protection does not apply.  The program could then poke a large value
into its "privilege level" byte (normally write-protected) and could
then proceed to bypass all levels of security within the file-management
system, patch the system monitor, and do numerous other interesting
things.  In short, the barn door was wide open.

Motorola quite properly reported this problem to XEROX via an official
"level 1 SIDR" (a bug report with a perceived urgency of "needs to be
fixed yesterday").  Because the text of each SIDR was entered into a
database that could be viewed by quite a number of people, Motorola
followed the approved procedure: they simply reported the problem as
"Security SIDR", and attached all of the necessary documentation,
ways-to-reproduce, etc. separately.

Xerox apparently sat on the problem... they either didn't acknowledge
the severity of the problem, or didn't assign the necessary
operating-system-staff resources to develop and distribute an official
patch.

Time passed (months, as I recall).  The Motorola guys pestered their
Xerox field-support rep, to no avail.  Finally they decided to take
Direct Action, to demonstrate to Xerox management just how easily the
system could be cracked, and just how thoroughly the system security
systems could be subverted.

They dug around through the operating-system listings, and devised a
thoroughly devilish set of patches.  These patches were then
incorporated into a pair of programs called Robin Hood and Friar Tuck.
Robin Hood and Friar Tuck were designed to run as "ghost jobs" (daemons,
in Unix terminology);  they would use the existing loophole to subvert
system security, install the necessary patches, and then keep an eye on
one another's statuses in order to keep the system operator (in effect,
the superuser) from aborting them.

So... one day, the system operator on the main CP-V software-development 
system in El Segundo was surprised by a number of unusual phenomena.
These included the following (as I recall... it's been a while since I
heard the story):

-  Tape drives would rewind and dismount their tapes in the middle of a
   job.

-  Disk drives would seek back&forth so rapidly that they'd attempt to
   walk across the floor.

-  The card-punch output device would occasionally start up of itself
   and punch a "lace card" (every hole punched).  These would usually
   jam in the punch.

-  The console would print snide and insulting messages from Robin Hood
   to Friar Tuck, or vice versa.

-  The Xerox card reader had two output stackers;  it could be
   instructed to stack into A, stack into B, or stack into A unless a
   card was unreadable, in which case the bad card was placed into
   stacker B.  One of the patches installed by the ghosts added some
   code to the card-reader driver... after reading a card, it would flip
   over to the opposite stacker.  As a result, card decks would divide
   themselves in half when they were read, leaving the operator to
   recollate them manually.

I believe that there were some other effects produced, as well.

Naturally, the operator called in the operating-system developers.  They
found the bandit ghost jobs running, and X'ed them... and were once
again surprised.  When Robin Hood was X'ed, the following sequence of
events took place:

  !X id1

  id1:   Friar Tuck... I am under attack!  Pray save me!  (Robin Hood)
  id1: Off (aborted)

  id2: Fear not, friend Robin!  I shall rout the Sheriff of Nottingham's men!

  id3: Thank you, my good fellow! (Robin)

Each ghost-job would detect the fact that the other had been killed, and
would start a new copy of the recently-slain program within a few
milliseconds.  The only way to kill both ghosts was to kill them
simultaneously (very difficult) or to deliberately crash the system.

Finally, the system programmers did the latter... only to find that the
bandits appeared once again when the system rebooted!  It turned out
that these two programs had patched the boot-time image (the /vmunix
file, in Unix terms) and had added themselves to the list of programs
that were to be started at boot time...

The Robin Hood and Friar Tuck ghosts were finally eradicated when the
system staff rebooted the system from a clean boot-tape and reinstalled
the monitor.  Not long thereafter, Xerox released a patch for this
problem.

I believe that Xerox filed a complaint with Motorola's management about
the merry-prankster actions of the two employees in question.  To the
best of my knowledge, no serious disciplinary action was taken against
either of these guys.

Several years later, both of the perpetrators were hired by Honeywell,
which had purchased the rights to CP-V after Xerox pulled out of the
mainframe business.  Both of them made serious and substantial
contributions to the Honeywell CP-6 operating system development effort.
Robin Hood (Dan Holle) did much of the development of the PL-6
system-programming language compiler; Friar Tuck (John Gabler) was one
of the chief communications-software gurus for several years.  They're
both alive and well, and living in LA (Dan) and Orange County (John).
Both are among the more brilliant people I've had the pleasure of
working with.

Disclaimers: it has been quite a while since I heard the details of how
this all went down, so some of the details above are almost certainly
wrong.  I shared an apartment with John Gabler for several years, and he
was my Best Man when I married back in '86... so I'm somewhat
predisposed to believe his version of the events that occurred.

-- 
Dave Platt    FIDONET:  Dave Platt on 1:204/444        VOICE: (415) 493-8805
  UUCP: ...!{ames,sun,uunet}!coherent!dplatt     DOMAIN: dplatt@coherent.com
  INTERNET:   coherent!dplatt@ames.arpa,    ...@sun.com,    ...@uunet.uu.net 
  USNAIL: Coherent Thought Inc.  3350 West Bayshore #205  Palo Alto CA 94303