brian@sdcsvax.UCSD.EDU (Brian Kantor) (08/04/87)
It's been a few years since this was posted, and the discussion has
arisen again, so I'm going to repost the following horror story:
From: bob@islenet.UUCP (Bob Cunningham)
Subject: lightning, PACXs and computers
Date: 23 Jul 85 21:30:14 GMT
Organization: Hawaii Institute of Geophysics
Had serious problems from a lightning strike last week. Now that most
everything is working again, though I'd pass on some details. Those of you
in some parts of the U.S. might not find this sort of thing so unusual, but
if you think it Couldn't Happen Here, read on ...
Thunderstorms occur only a few times a year out here in the Hawaiian
Islands, and lightning tends to strike in relatively harmless places. Last
Wednesday at about 1500 a small but intense little storm was accompanied by
a relatively intense lightning strike on the the eastern end of the
University of Hawaii campus. Eyewitness reports of exactly where the
lightning struck are numerous and contradictory, observers up to five miles
away were mightily impressed by the huge bolt of lightning and very loud
thunder.
There was no apparent effect on the power lines, outside of the possible
"jiggling" of a few cycles. The affect on data communications lines was
more impressive.
Briefly, all of the on-campus computer facilities with data communications
lines extending beyond their immediate buildings (typically RS232
3-or-4-wire leased phone circuits or similar) suffered burned out
ports and burned out terminals, notably:
several VAX780s at the Center for Cultural Interchange between
East & West (an on-campus "think tank" type organization) lost
multiplexer (port) boards and distribution panels.
a VAX785 in the Information & Computing Sciences Dept. lost
multiplexer boards & distribution, and appears to have suffered
memory and/or cpu damage.
several VAX780s in the Management Systems Office (administrative
computing) lost multiplexer boards & distribution, and may have
more extensive damage.
the High Energy Physics group lost multiplexer boards & panels
and a tape drive on their VAX780.
a VAX750 in the Marine Sciences Building kept on operating,
oblivious to the fact that more than 8 of its ports had just burned
out.
an H800 at the Hawaii Institute of Geophysics lost virtually all of
its DMACP (port) daughter and mother boards.
All three of the larger port selectors on campus (1 large Gandalf
at the UH Computing Center, 1 smaller Gandalf at the East/West
Center, and 1 Micom 600/2 at the HIG) suffered a large number of
burned out line boards, along with some port boards.
None of the computers which crashed (all of the above, except the VAX750),
suffered disc crashes.
For the most part, most of the damage appears in obviously-burned-out RS232
driver circuitry, and continues back into logic on the various port
interface (multiplex) boards.
An as-yet-uncounted number of terminals (minimum estimate: several hundred)
were damaged. Typically, the RS232 driver chips (most often 1488) were
burned out. In about 30% of the terminals I've personally checked so far,
damage spreads further into the terminal logic board, and sometimes all the
way to the power supply.
Most of the affected terminals were on relatively long lines (more than
200' or so), however many terminals with much shorter lines -- even some
in the same room as their computer -- suffered damage. Usually, these were
attached to port/multiplexer cards along with one or more long-line
terminals.
Microcomputers (such as IBM PCs) chugged right along without noticing the
lightning strike -- unless they had an out-of-building asynch connection,
in which case they suffered damage like terminals.
In many cases (notably the IBM3081D, DEC20 and HP3000/64 at the UH
Computing Center), port selectors protected individual machines almost
completely.
I don't have an accurate monetary estimate of the damage, but
it will certainly run over $100,000.
Field engineering response from Digital was quite good, but handicapped by
the fact that DEC only carries one "kit" of spares in the islands for each
model of VAX. Assessing the damage took a day or so (UH has almost
entirely, basic service), some replacement boards arrived quickly, others
not until after the weekend. Response from other vendors was
similar; Harris pulled parts off its production line
in Florida and burned-in over the weekend for us. Micom responded
quickly to ERE requests, shipping within 24 hours via Federal
Express. I don't have details on Gandalf's response yet.
A wide variety of different maintenance agreements were in effect with the
various manufacturers. Many of the contracts turned out to have "acts of
god" exclusion clauses. Fortunately, all of the vendors involved took the
attitude that they'd fix now, and worry about who would pay later.
--
Bob Cunningham {dual|vortex|ihnp4}!islenet!bob
Honolulu, Hawaii
> ... In particular I wonder
> if your communications lines are above or below ground? What about
> your power lines? Are they above or below ground? Secondly, do your
> communications lines have any type lightning or surge protection?
> Were any of the affected lines shielded? Do you have surge protection
> on your power lines?
All of the comm lines are underground, in conduits (not steam tunnels,
which we don't have out here). In some cases these were leased "4-wire
control circuits" from the local phone company (Hawaiian Telephone Co.,
a GTE subsidiary); in other cases, self-installed. All power lines are
also underground (separate conduit, usually completely different trenches).
Note that power lines were not affected, nor were regular phone lines (not
a single modem was zapped, and no phone problems of any kind were
reported). None of the comm lines had lightning or surge protection. Some
(a few) were shielded -- though I'm not sure how well the shielding was
grounded. Some power lines have surge protection, some not; in any case
the path was NOT thru the power lines.
Interestingly, some of the affected lines were entirely within a single
building, typically of reinforced concrete (and for ordinary r.f. signals,
the rebar cage usually provides some shielding). My theory is that -- at
some point -- those comm lines were bundled with other lines passing from
building to building.
> We are in a lightning prone area, and I have been attempting to
> understand the intricacies of lightning protection recently. I
> confess that the more I hear the more confused I get. I have
> concentrated on power protection, but your problems seem to have resulted
> from a communications line hit.
That's what the evidence definitely indicates. Surprised us.
> I wouldn't have thought that such a
> hit would spread to so many machines, unless you had a tremendous
> earth current transient that affected buried lines in a wide area,
> or were unlucky enough to get a hit on some central distribution
> point.
There was no central distribution point -- several completely separate
computer "centers" were involved, none tied together in any way. I'd say
the ground current transient (or, perhaps several as a charge briefly
"bounded" back and forth from the clouds to the earth and back again until
settling) seems likely.
> Am I wrong in this evaluation? Is there some other way such
> a strike could get into a widespread communications network? How
> does one go about protecting such a network from lightning?
I'd like some of those answers myself. Anyone else with experience care to
comment?
--
Bob Cunningham {dual|vortex|ihnp4}!islenet!bob
Hawaii Institute of Geophysics Computing Facilities
Honolulu, Hawaii
From: hull@hao.UUCP (Howard Hull)
Subject: Lightning Protection (long)
Date: 31 Jul 85 23:22:25 GMT
Organization: High Altitude Obs./NCAR, Boulder CO
Protecting one building or tower from lightning is fairly straightforward:
1. First order protection -
On the tallest object associated with your structure, mount an
extended umbrella-like fixture a few meters in diameter,
with numerous sharp points along the periphery and across the
crown, spaced about 1 meter apart. (You can make the thing
from re-bar and heavy duty chicken wire unless you have high
winds like we have around here.) Use a large diameter conductor
(1 to 2 cm.) to connect the umbrella points together at the
center and thence down to a suitable ground stake located at
a place where soil moisture is prevalent, but more importantly,
try to make the conductor run in a straight line with *no* sharp
corners; use a minimum radius of 1.5 meters on any bends in the
ground wire. Keep this wire at least 2 meters from any power or
communications conduit at all places along its route.
Theory:
The multitude of points will emit a trickle corona continuously,
resulting in a space charge of ionized air within 20 meters of
the umbrella. The space charge will terminate the cloud-to-ground
electric field across a broad hemisphere and will reduce the local
field gradient to a value below that needed to form "leaders".
The umbrella will likely not ever be hit by lightning; however,
the conductor gauge is set to minimize the damage inherent in
such a strike. (A strike, if it occurs, will likely be a
secondary, (resulting from the shift in electrostatic field just
after a strike) to another object within a fraction of a km.)
This approach, you should note, puts additional stress on your
neighbors (they will see a slight rise in their hit statistics)
as it only postpones the discharge until the cloud has moved
past your installation. The ground conductor is spaced from
other conduits so that the Electromagnetic Pulse (EMP) associated
with the 10000 Ampere surge will not be able to develop equivalent
currents in parallel conductors adjacent to the ground wire.
Using a large diameter and avoiding bends reduces the per length
inductance discontinuities. This discourages the abandonment of
your ground conductor in favor of nearby metal objects such as
power conduits (resulting in hazardous elevation of the system
ground potential to thousands of volts above the mains).
2. Second order protection -
Protect your primary power entry by use of a surge protector
having four main elements a.) Line fuses for each hot main NO
FUSE FOR THE WHITE NEUTRAL. No circuit breakers (too slow).
b.) Self extinguishing gas discharge tubes or arc chutes routed
to a primary ground stake *separated* by 3 or more meters from the
umbrella ground mentioned above, *not* using the same stake, even,
and using the same linear routing algorithm mentioned above.
c.) Heavy gauge inductors, 1 microhenry or thereabouts for typical
30 to 50 Ampere per phase service levels, to choke the surge out
of the consumer side of the system. NONE IN THE WHITE NEUTRAL.
d.) Post choke line clamping to WHITE NEUTRAL. This is where the
witchcraft comes in. One candidate is the Metal Oxide Varistor
(MOV). They have two disadvantages: They age, gradually reducing
their threshold over time until one day they evaporate in a ball
of fire during a line surge. They have a rather remote threshold
characteristic compared to, say, a Silicon TransZorb. They have
several advantages: They are cheap. They come in packaging that
is familiar to professional electricians. They are generally
more robust than Selenium or Silicon protectors. They have a
smaller geometry than a Selenium protector. Another candidate
is a combination protector made up from a ground referenced
50 Ampere triac in series with either a lower rated voltage
MOV or TransZorb element, with the triac gate wired back to (an
artfully positioned) tap on the gas tube/arc chute ground. From
here (this stuff belongs in a fire-rated NEMA box) the WHITE
NEUTRAL and GREEN NEUTRAL are tied together at this one point
only, and passed through a medium size conductor to the primary
ground stake by a route that is separated by 1.5 meters from
the gas tube/arc chute ground.
Theory:
If your power line gets hit, the gas tube fires and conducts the
surge current to ground. The 20 kilovolts experienced by your
service entry (for about 10 microseconds) will go through the
chokes and will cause the MOV or complex protector shunt to break
down and draw a steadily rising current (to many tens of Amperes),
but immediately choked to a reduced voltage. The fuses will, after
a while, be blown away. Until then, the MOVs will clamp the
WHITE NEUTRAL to the mains (perhaps resulting in noticeable
rise of the common-mode voltage). It is this common-mode elevation
which destroys your out-of-building communications interfaces.
With everything in the building coming to 2000 volts above your
neighbors (including your local telephone operating company),
any common-mode paths will be severely stressed. However,
especially withing the building, they will be less stressed than
they would have been if the mains were allowed to diverge from
the WHITE NEUTRAL.
3. Third level protection
The most effective common-mode protection is an Ultra-Isolator
Transformer. It is also rather expensive compared to differential
line protectors and secondary Silicon TransZorb protectors.
Although many Ultra-Isolator Transformers were utilized during
the 1970's by sensitive computer installations, it was realized
eventually that the most damage to main-frame equipment was done
by differential surges (main to main on three-phase systems).
The common-mode threat was seen as too little to justify the
cost and complexity of installation of an ultra-isolator, which,
by the way, can also be done ineffectively, resulting in no net
improvement in the level of protection. The companies that
make ultra-isolators issue complete and effective instructions
concerning their installation. The difficulty is in getting
industrial electricians to follow the directions.
Thus for the benefit of the main-frame and peripheral power
supplies, for cost effective purposes, a good differential
surge eliminator inside the enclosure of each system power
supply is recommended. However, remember that the common mode
is the most destructive to your distributed data communications
peripherals; unfortunately, to protect them you must provide
the entire computer room and distributed CRT terminal load with
an ultra-isolator transformer, or see that each unit is designed
to withstand momentary local and global differences of thousands
of volts on the signal returns. Even then, on occasion, only one
violator located in a critical location and tied to a non-isolated
power system elsewhere in the building can blow the whole scheme.
Theory:
Not much theory here. The entire primary winding of the transformer
may get lifted to 2000 volts, but the secondary remains referenced
to the computer room ground stake. The box shields around the the
windings are tied to the stake, and short out the electric field
that might otherwise couple to the secondary. Saturation of the
transformer core protects the differential mode. The differential
protectors installed in each power supply dissipate the surges
locally and since each takes a small part of the surge energy, no
concentration of damage will likely occur.
4. Fourth order protection
You may get surge protectors for all communication lines leaving
the building. Each will need a reliable path to a stout ground.
(DEC usually specifies that the computer frame GREEN WIRE ground
be done with a heavy gauge wire, and all surge protector grounds
be separately returned to the distribution transformer secondary
neutral grounding point.) You may add Silicon TransZorbs to power
supply rails in data communications equipment.
Theory:
If one of your comm lines gets hit, or gets involved in an induced
surge, the elevation in voltage not dissipated by the protector is
conducted through the internal diode clamps included in most IC
line drivers and receivers to a ground or supply rail, and thence
to a TransZorb (a back-to-back zener with a heavy silver anode and
thermally conductive silver leads). If enough protectors are in
place, the common-mode surge is clubbed to death by the collective
capability of all peripheral surge protectors operating together.
And that about does it. Needless to say, if you do a good job of protecting
your site, and one of your neighbors gets hit, you may be damaged anyhow by
currents resulting from the elevation of your neighbor's electrical ground.
This is especially true in Hawaii (and even more so on their mountain tops)
where the ground is made of lava rock. If you get hit by lightning, your
entire site goes to 25000 volts with respect to the surrounding neighborhood.
This bleeds down to appx 2000 volts over the next 100 microseconds or so.
If you have several buildings to worry about, such as may be the case for a
university campus, putting an umbrella protector on every building will only
cause the cloud to ground potential to develop to the point that when you
finally do get a strike, it will be a *real killer*. It has been pointed
out elsewhere that most lightning strikes are from the ground up to the cloud.
Thus, More Theory (speculation):
I suspect that the mechanism is something like this: Collisions of air
molecules with each other and the things that make up the surface tend to
knock electrons off the air molecules. There are other charge pair generation
mechanisms as well, such as natural radioactive decay of Radon 222 and its
decay products. (This specific mechanism is not my theory - see JGR Vol 90
No D4 Pgs 5909-5916 June 30, 1985, Edward A Martell, NCAR.) The electrons,
because of their charge, are sticky. They cling to the surfaces of various
semi-insulators (rocks and dry dirt) and near the surface of conductors until
enough of them are implanted to provide a counter electrical field gradient to
repel later arrivals. The positive air ions are separated by thermal energy,
and molecular screening prevents the immediate recombination. The charge
separation is effected by the rising of the warmed positively ionized air.
Once the charge is separated, mutual repulsion drives the electrons into the
conductive ground layers. Later, as the air rises and water condenses,
positively charged droplets accumulate in descending air columns at the front
of the storm just ahead of the rising column. A field gradient is thus
established with respect to the ground, where all the electrons are. As the
ground is conductive, the electrons follow the cloud until, with the aid of
conductive moisture and the turbulence of the rising and descending air column
interface, leaders are established and a strike path is ionized and carried
into the descending air. The electrons travel up the path in a flash (parts
of which will have oscillations at radio frequency) and then distribute
themselves (at a more leisurely pace, accompanied with local flashes and
secondary flashes) in accordance with upper level gradients until there is
nolonger sufficient gradient to ionize the cloud-to-cloud paths. Time scales:
Main strike and individual secondary strikes each about 10 microseconds.
Duration of ionized path, reversals and secondaries about 100 microseconds.
Duration of high altitude electrical coronae readjustment about 1 millisecond.
Localized differences in the final potential may result in some reverse
strikes from a few overcharged negative clouds to the ground, or subsequently
more numerously (after air motion), cloud to cloud "readjustments".
Well, I've done it again. Darn. If this is too long, I suppose you should
flame me for it, or if I am guilty of mis-representing known (un)truths, that
would qualify as well. But I wanted to at least try to clear up the nature
of lightning and its hazards a little.
Howard Hull
[If yet unproven concepts are outlawed in the range of discussion...
...Then only the deranged will discuss yet unproven concepts]
{ucbvax!hplabs | allegra!nbires | harpo!seismo } !hao!hull
From: bob@islenet.UUCP (Bob Cunningham)
Subject: Re: lightning, PACXs and computers (followup)
Date: 11 Aug 85 00:14:22 GMT
Organization: Hawaii Institute of Geophysics
At last count, over 40% of the zapped terminals we had were repairable only by
replacing the 1488 and/or 1489 chips (usually the 1488). An
industrial-quality solder remover (heater + vacuum pump) is highly
recommended. The percentage seems to be slightly higher (better) for port
selector boards (Gandalf & Micom). Needless to say, we've been installing
sockets for those chips where practical.
On the computer side, all the port/multiplexer (DZ or whatever) boards
were just swapped out. Not sure what percentage just had driver chips
burned out or not. Subsidiary damage (one tape unit and some memory
boards) was limited to just 2 systems, and my opinion is that not all
the equipment at those locations was solidly tied to a single ground.
Besides the followup articles in this newsgroup, I've received a
considerable number of mail messages (all appreciated, though
I've not had time to send individual replies), falling into
two categories:
1) similar horror stories
2) thoughtful advice on lightning protection
Apparently, similar incidents (lightning damage via local data comm lines)
are much more common & widespread than I'd have thought. Unless you're in
a very unusual location, if you've got comm lines going between buildings,
be warned: something similar might just happen to you.
While I've received a number of very sensible suggestions on lightning
protection, there doesn't seem to be one single solution we could adopt
in all cases. Suggestions have ranged from using opto-isolators,
diodes (of various sorts, including MOVs), to using telco-type
spark gap devices, to fiber optics. Each approach has some good points.
However, the thought of having to install any particular suggestion
on the 800 or so data comm lines around campus which probably should have
protection is a sobering thought. Schemes cheap in material (e.g. diodes)
look to be rather labor-intensive (if both terminal and computer/port
selector ends both need protection -- which seems optimal). Schemes
cheap in time (e.g. fiber optics) look a bit expensive in parts
(though the thought of pulling a lot of fiber optic cables to replace
twisted-pair lines is also rather sobering).
So far, I think our best approach here is to stick to the basics.
For starters, going over all of the central grounds. All Computer
equipment and auxiliary racks within a room should be securely grounded
to a single point with generous-size braided ground straps.
The objective is to minimize any possible ground differences between
computers and their peripherals (including port selectors).
It seems to be a good idea to tie down all incoming terminal
grounds (RS232 pin 7) to that same point -- as the lines come in
(typically on punch blocks). This should localize damage (typically to the
port selectors). It also seems reasonable to dedicate whole computer-side
port/multiplexer boards to PACX lines ... no more mixing direct-connect
and port selector terminals. Typically, the only terminals I want
to leave directly connected will be those in the same building
(preferrably the same wing) as the computer site. This should also
minimize computer-side damage.
Fortunately, port selector boards are MUCH cheaper to repair/replace
than computer port/multiplexer boards.
Relatively expensive terminals (graphics types, for the most part) should
get some special form of isolation at their end from the RS232 lines
(preferrably something simple that plugs in between the RS232 line
and the terminal).
Cheaper terminals (< $1,000) I think are best left as-is for now.
All NEW inter-building trunks (multiple data comm lines) that I have
any control over will be fiber optic lines (stat muxes at each end,
of course).
We do have some plans for a couple of real LANs around campus. I
do believe that specs for those will now definitely include some
form of lightning protection.
--
Bob Cunningham {dual|vortex|ihnp4}!islenet!bob
Hawaii Institute of Geophysics Computing Facilities
Honolulu, Hawaiibob@uhccux.UUCP (Bob Cunningham) (08/19/87)
Some postscript notes on the 1985 lightning strike here at the U. of Hawaii. No one ever figured out the total extent of damage, but it certainly was in excess of $100,000 across the campus. The Hawaii Institute of Geophysics alone---where I'm responsible for various machines---documented $43,000 in damages for the claim on our multi-risk insurance policy (which, very fortunately specifically included fire & lightning...your typical insurance policy often specifically doesn't cover those). That doesn't include any of our DEC equipment which was replaced by DEC with no questions asked (they were kind to us, if they went by the fine print on their maintenance contracts, legally speaking they could have stuck us with some hefty bills). My guess is that HIG sustained about 20-25% of the damage on campus (the lightning striking just about on top of us); if so, then the total campus damage ran around $200,000. Again for HIG alone, over 40 terminals were damaged. Almost half of those we were able to repair by replacing 1488/1489 chips and such. The remaining terminals were much more thoroughly fried and weren't economically replaceable. I don't know what the campus total of damaged terminals was, but it was almost certainly over 100. There were some indications that one or two computer sites may indeed have suffered some sort of power surge, but the main damage was to data communications equipment and devices attached to that equipment. With several thousand datacom lines running around campus there were lots of "antennas" that picked up induced currents. We thought up all sorts of schemes to protect ourselves afterwards, but really didn't implement any protection measures systematically. The Meteorology department did a risk study which showed that the probability of a similarly damaging lightning strike on campus within the next 20 years was very, very small. I hope they're right. More (probably most now) campus terminals go through various port selectors now, which means the port selectors protect most of the larger computer systems from this sort of thing (our experience showed that replacing parts of the port selectors was substantially cheaper than replacing computer boards). On the other hand, we're much more thoroughly ethernetted on campus now and I don't have any experience with lightning zapping ethernets. One hopes that the transceivers are the weak links. Also on the other hand, we have many, many more PCs and fewer computer terminals. The PCs are more expensive to fix. If we were in Florida or the midwest where lightning strikes such as this were more common, we'd certainly take more systematic protection. 1985 was not a fun year. Several months afterwards we were hit by a hurricane that knocked out the power grid on Oahu (and Kauai), leaving Honolulu (the 11th most populous U.S. city) without power for several days. [during that time, hardly anybody was concerned about their computers, being preoccupied by more immediate problems like how to deal with the immediate shortages of water, gasoline and refrigerated food---all very dependent upon electrical power]. Outlying areas were without power for several weeks while the rest of us had to deal with rolling blackouts before power generating capacity and major electrical trunk capacity was restored. We'd start up our computer systems when we had power, and bring them down when the radio announcement came that the power company was about to "roll" the power over to another locality. Utility electric power was generally "flaky" for perhaps a year afterwards, with brief outages or major phase imbalances occuring more than once a week. HIG invested in an UPS (uninterruptible power supply) for our main computer systems. Other University sites weren't so fortunate and for various reasons wouldn't or couldn't obtain UPS's. During that time an extraordinary number of power supplies and various other computer components rolled over and died. It was, of course, virtually impossible to tie any specific failure to the power problems we know we had during that time, but my personal guess is that HIG alone probably saved the $40,000 or so we spent for our 50kva UPS during the following year in equipment that didn't fail because of power problems. All of that is is the past now, and we generally don't worry too much about these sorts of problems. Although...once every few months when we get an occasion power problem (particularly if it's a surge of some kind) I can count on 1-3% of our various models of PCs rolling over and dying (though it's usually just the power supplies). Bob Cunningham Hawaii Institute of Geophysics, University of Hawaii bob@loihi.hig.hawaii.edu