[net.rumor] apocryphal story about ESS crashes - can someone confirm it?

mogul@Shasta.ARPA (03/23/85)

Many years ago, I heard this story, and I was wondering if anybody
could confirm it:

    Back when this was still an unusual event, a new ESS exchange was
    installed and turned on.  Ma Bell, proud of her new child, invited the
    press in to be dazzled by modern, reliable, computer technology.

    After the obligatory tours and speeches, the photographers wanted to
    take some pictures.  Someone must have thought that a magtape drive
    would make a good picture, since it looked like part of a real
    computer.  A photographer pushed the shutter, the flash went off, and
    the ESS promptly crashed.

    Why did the ESS crash when the flash went off, you wonder?  So did Ma.
    It turns out that the tape drives had optical sensors for the foil
    end-of-tape markers.  Apparently, the photo flash set off the sensors
    when the ESS program wasn't expecting end-of-tape, and the program
    jumped off into hyperspace.  (One version of the story is that the
    drives had two sensors, one for detecting each end of the tape.  The
    programmer who coded this part of the program never expected the tape
    to run off both ends at the same time.)

    In spite of the vaunted reprogrammability of the ESS, it turned out to
    be too dangerous to fix the software bug immediately (it might break
    something else).  So, all the tape drives in all the ESS exchanges were
    fitted with little opaque hoods over their end-of-tape sensors.

It's a good story, even if it isn't true, but did this really happen?
If it did, do I have the details right?

-Jeff

minow@decvax.UUCP (Martin Minow) (03/26/85)

The version I heard -- many, many years ago -- was that, when the
first electronic office was installed outside Chicago (Morton Grove),
the engineers were so happy with the fancy flashing lights (gas tube
switchers) that switch panels were visible from the reception area.
When the press came to see it, their flashbulbs triggered all the
tubes at once.  And every phone in the exchange rang at once.

They had to turn power off to clear things up.

A curtain was installed the next day.

Martin Minow
decvax!minow

farber@rochester.UUCP (Dave Farber) (03/27/85)

That may have happened at the Morris Ill field trial of the Pre production
ESS (with a Gas tube network, flying spot store and Williams Tube
memory. But if it did I did not hear of it and I was in the Systems Eng
group responsible for the Morris ESS. 
-- 
Dave Farber

_______________________________________________________________________________
University of Rochester
Department of Electrical Engineering
Rochester, NY 14627

Arpa:farber@rochester.arpa
uucp: ..!{allegra,decvax,seismo}!rochester!farber
uucp (home system): ..{ihnp4!cfg,rochester,sco}!pcpond!farber
Compuserve: 76010,104                    Telex: 6501066405
MciMail: dfarber                         Source: bbk190
Telephones: Office(UD): 302-451-1163     Home: 215-274-8292
	    Office(UR): 716-275-4054
_______________________________________________________________________________

res@ihuxn.UUCP (Rich Strebendt) (03/27/85)

In response to:

> Many years ago, I heard this story, and I was wondering if anybody
> could confirm it:

The entire story as you have related it is pure bullsh*t.
 
>     Back when this was still an unusual event, a new ESS exchange was
>     installed and turned on.  

A switching office is not "installed and turned on" as a computer is --
the process is called "cut-over" and is done (or was done in the era
described in the posting) with axes and strong men yanking on ropes
at something like 2am.  Perhaps someone who has participated in one 
could relate the process in more detail than I am qualified to.

>     Ma Bell, proud of her new child, invited the
>     press in to be dazzled by modern, reliable, computer technology.

Certainly, a telephone company would be proud to have the latest in
technology (as long as it was cost-effective), but after the consent
decree of 1956 it could NOT be COMPUTER technology.

>     After the obligatory tours and speeches, the photographers wanted to
>     take some pictures.  Someone must have thought that a magtape drive
>     would make a good picture, since it looked like part of a real
>     computer.  

Here is where the BS starts.  The No. 1 ESS was designed and installed
WITHOUT TAPE DRIVES.  The nearest thing to a tape drive was the AMA
(Automatic Message Accounting) machine -- many tracks of PUNCHED PAPER
TAPE.

>     A photographer pushed the shutter, the flash went off, and
>     the ESS promptly crashed.

More BS -- ESS machines do not "crash".   Under failure
conditions they will first "phase" as they attempt to isolate the
failure and reconfigure without losing calls in progress.  If this does
not correct the problem with minimal corrective actions, then more
severe actions are taken progressively.  In the extreme, and very rare,
case that the problem is not recovered from in this way, then Emergency
Action (EA) is entered.  During EA the machine assumes that it is
insane.  Piece by piece it tries to find a working configuration --
sanity checking each configuration until it finds one whose sanity can
be proven, then it returns itself to service.  From the onset of EA
until operation is restored the machine's outage time is measured in
seconds.  The requirement on the design of the machine was that its
TOTAL down time (out of service) during its design lifetime (40 years)
was not to exceed 2 hours.  I repeat -- ESS machines do not "crash".

>     Why did the ESS crash when the flash went off, you wonder?  So did Ma.
>     It turns out that the tape drives had optical sensors for the foil
>     end-of-tape markers.  

Guess what -- most tape drives STILL use optical sensors to see the
reflection of light off of the foil "tape marks".  Indeed, I can 
think of no tape drive I have worked with recently that used any
OTHER way to detect the beginning or the end of tape.

>     In spite of the vaunted reprogrammability of the ESS, it turned out to
>     be too dangerous to fix the software bug immediately (it might break
>     something else).  So, all the tape drives in all the ESS exchanges were
>     fitted with little opaque hoods over their end-of-tape sensors.

Double bull roar.

The idea of "little opaque hoods" over the BOT/EOT sensors on ANY tape
drives is ludicrous.  As to the "vaunted reprogrammability of the ESS",
while the program storage medium for the No. 1 ESS was different from
core or semiconductor memory (magnetic twistor memory cards), it was as
reprogrammable as modern EPROMs.  However, it is true that Bell Labs
has always subjected program patches and new features to extensive
testing (though bugs do slip through the finest of testing nets), so
that a fix for this kind of problem would not be installed
"immediately" -- indeed, the conditions needed to cause this problem to
show up and the low frequency of occurrance of those conditions would
probably lead such a fix to be of very low priority and might make it
into the next generic release if someone had no more important problems
to fix.

> It's a good story, even if it isn't true, but did this really happen?
> If it did, do I have the details right?

Obviously, you do NOT have the details (or even the grossest
information) correct.

Lest I appear a know-it-all spoilsport, I have also heard a similar
story, but (more reasonably to my mind) it was supposed to have
happened in the Comp Center of an insurance company which wanted a
picture of the new computer for the Shareholders' Annual Report to show
what a modern company they were.  The flash was set off during a seven
tape merge sort causing the tapes to simultaneously go into rewind and
abruptly terminate several hours of work.  The story was newsworthy
because the operator tried to kill the photographer.  (Justifiable
homocide?)

Now to await the next cheap shot at AT&T.

					Rich Strebendt
					...!ihnp4!ihuxn!res

rfg@hound.UUCP (R.GRANTGES) (03/27/85)

[]
Not Morton Grove, Morris, Illinois.

-- 

"It's the thought, if any, that counts!"  Dick Grantges  hound!rfg

mfs@mhuxr.UUCP (SIMON) (03/27/85)

> In response to:
> 
> > Many years ago, I heard this story, and I was wondering if anybody
> > could confirm it:
> 
> The entire story as you have related it is pure bullsh*t.

Not quite, although it *is* apocryphal. The real story was considerably
less interesting.

Stebendt is correct that the only tape drives in a 1 ESS offices are the
Automatic Message Accounting (AMA) tapes, used for long distance billing.
There is one AMA drive per 1E processor (most central office equipment
is duplicated). WHat happened was that under the flash, both AMA units
went out of service. Since a condition for sanity is that the AMA unit
be working, and since the response to insanity is to switch to the other
processor, both 1E processors started switching control back and
forth. They did not "crash" or go out of service, although the condition
does raise a major alarm, which is characterized as alternating
"dongs", something to alarm (no pun intended) an unknowing press.
The condition was fixed by forcing one processor to become active, and to
ignore the major alarm. The the AMA unit could be restored to service
unconditionally, then the other processor. End of story. Less interesting,
but true. The tape drive *doors* were later modified so the tape
sensors would be shielded by them. There were no "little opaque hoods".

Marcel Simon

laura@utzoo.UUCP (Laura Creighton) (03/28/85)

The apocryphal story I heard had to do with the light from the sun shining
through a glass window and hitting all the end of tape sensors in a rack
of magtapes at once, thereby killing a supposed-to-be long running
program every Friday night after the staff had gone home. There was
much scratching of heads and looking at code until the switch from 
Standard to Daylight Savings time (or the reverse) was made, at which
point the problem became obvious.

Laura Creighton
utzoo!laura

chrise@ihlpa.UUCP (Chris Edmonds) (03/29/85)

> The apocryphal story I heard had to do with the light from the sun shining
> through a glass window and hitting all the end of tape sensors in a rack
> etc., etc.,

Yes this really happened (sort-of) with all the caveats previously 
mentioned in other responses (i.e., No crash, just moderate confusion
of the system which was easily resotered, no loss of service). I was
personally involved in this one and hand built the "little hood" which
was installed on ->ONE<- machine.

The situation was like this... In the late 70's WE was installed 2BESS
machines all over creation and its processor was known as the 3A Processor.
This processor was used in a number of other installations. I owned the
current engineering responsibility for the peripheral frame for this machine.
The system boot device was (and still is) a 3M like tape cartridge drive
physically mounted such that the optical sensors looked straight out the
front of the machine.  Normally these were not affected by visible light
as they are tuned to look for infared (LEDs with filters).

One of the installations was on the zillionth floor of the NW Bell
building in Seattle. It just so happened that this building had lotsa
windows (unusual for a Bell structure) and the processor was installed 
facing one on the west side of the building.  I'm sure you can guess the
rest.

It was summer so it took a little longer to figure it out.  Some evenings 
after the installers went home the tape would run off reel.  It finally
dawned :-) on us that it happened at sundown on sunny evenings.  The sun
was shining directly into the housing and swamping the sensor.  Since it
was edge sensitive the sensor never saw the BOT hole go away and never ramped
the motor down.  The result was a specially fabricated opaque cover which
had to be manually lifted to install the tape cartridge.  As far as I know
its still there.

Chris Edmonds @ Bell Labs, Naperville IL

nather@utastro.UUCP (Ed Nather) (03/30/85)

> It finally
> dawned :-) on us that it happened at sundown on sunny evenings.  The sun
> was shining directly into the housing and swamping the sensor.  Since it
> was edge sensitive the sensor never saw the BOT hole go away and never ramped
> the motor down.  
> 
> Chris Edmonds @ Bell Labs, Naperville IL

Don't be embarrassed.  During WWII the British radar watch of the French coast
was suddenly swamped (at dawn) with noise, and they alerted everybody in sight
because obviously the Germans were jamming their radar.  As the sun rose the
noise lessened, then went away ... and they could see normal costal activity
and de-activated the alert.  Then, the very next morning ...

Sunspots, as it finally turned out.

"Those who don't study astronomy are condemned to repeat it."

-- 
Ed Nather
Astronony Dept, U of Texas @ Austin
{allegra,ihnp4}!{noao,ut-sally}!utastro!nather

dee@cca.UUCP (Donald Eastlake) (04/02/85)

The ESS story I heard had to do with a high level panic error caused by
the ESS thinking it had no where to write billing information.  This
was because of a photographer's strobe setting off end of tape indications
simultaneously on its pair of mag tape drives.

A non-apocryphal story I observed was the case of the PDP-6 at the MIT
AI Lab which, for some time, seemed to have about a 1 in 4 or 5 chance
of crashing in the morning.  There were various jokes about morning
sickness.  None of the usual PM stuff, including running voltage margins,
etc., found anything wrong.  Then it was noticed that the exact times
of the crashes seemed to consistently drift a few minutes each day.
It turns outs that there were a bunch of windows behind the machine
(which was on the 9th floor) and sometimes the thermal transient inside
the machine caused by the sun rising and sunlight striking the back of
the machine caused it to crash.
-- 
	+1 617-492-8860		Donald E. Eastlake, III
	ARPA:  dee@CCA-UNIX	usenet:	{decvax,linus}!cca!dee