bob@islenet.UUCP (Bob Cunningham) (07/24/85)
Had serious problems from a lightning strike last week. Now that most everything is working again, though I'd pass on some details. Those of you in some parts of the U.S. might not find this sort of thing so unusual, but if you think it Couldn't Happen Here, read on ... Thunderstorms occur only a few times a year out here in the Hawaiian Islands, and lightning tends to strike in relatively harmless places. Last Wednesday at about 1500 a small but intense little storm was accompanied by a relatively intense lightning strike on the the eastern end of the University of Hawaii campus. Eyewitness reports of exactly where the lightning struck are numerous and contradictory, observers up to five miles away were mightily impressed by the huge bolt of lightning and very loud thunder. There was no apparent effect on the power lines, outside of the possible "jiggling" of a few cycles. The affect on data communications lines was more impressive. Briefly, all of the on-campus computer facilities with data communications lines extending beyond their immediate buildings (typically RS232 3-or-4-wire leased phone circuits or similar) suffered burned out ports and burned out terminals, notably: several VAX780s at the Center for Cultural Interchange between East & West (an on-campus "think tank" type organization) lost multiplexer (port) boards and distribution panels. a VAX785 in the Information & Computing Sciences Dept. lost multiplexer boards & distribution, and appears to have suffered memory and/or cpu damage. several VAX780s in the Management Systems Office (administrative computing) lost multiplexer boards & distribution, and may have more extensive damage. the High Energy Physics group lost multiplexer boards & panels and a tape drive on their VAX780. a VAX750 in the Marine Sciences Building kept on operating, oblivious to the fact that more than 8 of its ports had just burned out. an H800 at the Hawaii Institute of Geophysics lost virtually all of its DMACP (port) daughter and mother boards. All three of the larger port selectors on campus (1 large Gandalf at the UH Computing Center, 1 smaller Gandalf at the East/West Center, and 1 Micom 600/2 at the HIG) suffered a large number of burned out line boards, along with some port boards. None of the computers which crashed (all of the above, except the VAX750), suffered disc crashes. For the most part, most of the damage appears in obviously-burned-out RS232 driver circuitry, and continues back into logic on the various port interface (multiplex) boards. An as-yet-uncounted number of terminals (minimum estimate: several hundred) were damaged. Typically, the RS232 driver chips (most often 1488) were burned out. In about 30% of the terminals I've personally checked so far, damage spreads further into the terminal logic board, and sometimes all the way to the power supply. Most of the affected terminals were on relatively long lines (more than 200' or so), however many terminals with much shorter lines -- even some in the same room as their computer -- suffered damage. Usually, these were attached to port/multiplexer cards along with one or more long-line terminals. Microcomputers (such as IBM PCs) chugged right along without noticing the lightning strike -- unless they had an out-of-building asynch connection, in which case they suffered damage like terminals. In many cases (notably the IBM3081D, DEC20 and HP3000/64 at the UH Computing Center), port selectors protected individual machines almost completely. I don't have an accurate monetary estimate of the damage, but it will certainly run over $100,000. Field engineering response from Digital was quite good, but handicapped by the fact that DEC only carries one "kit" of spares in the islands for each model of VAX. Assessing the damage took a day or so (UH has almost entirely, basic service), some replacement boards arrived quickly, others not until after the weekend. Response from other vendors was similar; Harris pulled parts off its production line in Florida and burned-in over the weekend for us. Micom responded quickly to ERE requests, shipping within 24 hours via Federal Express. I don't have details on Gandalf's response yet. A wide variety of different maintenance agreements were in effect with the various manufacturers. Many of the contracts turned out to have "acts of god" exclusion clauses. Fortunately, all of the vendors involved took the attitude that they'd fix now, and worry about who would pay later. -- Bob Cunningham {dual|vortex|ihnp4}!islenet!bob Honolulu, Hawaii
fuller@mcnc.UUCP (Bill Fuller) (07/26/85)
This was certainly one of the most devastating lightning strikes I've heard of, and I was wondering if you could perhaps fill us in on a few details that weren't mentioned in your note. In particular I wonder if your communications lines are above or below ground? What about your power lines? Are they above or below ground? Secondly, do your communications lines have any type lightning or surge protection? Were any of the affected lines shielded? Do you have surge protection on your power lines? We are in a lightning prone area, and I have been attempting to understand the intricacies of lightning protection recently. I confess that the more I hear the more confused I get. I have concentrated on power protection, but your problems seem to have resulted from a communications line hit. I wouldn't have thought that such a hit would spread to so many machines, unless you had a tremendous earth current transient that affected buried lines in a wide area, or were unlucky enough to get a hit on some central distribution point. Am I wrong in this evaluation? Is there some other way such a strike could get into a widespread communications network? How does one go about protecting such a network from lightning? Bill Fuller {decvax,akgua}!mcnc!fuller
plw@mgwess.UUCP (Pete Wilson) (07/28/85)
My horror story doesn't involve such an impressive array of computational power, but an Onyx C8002 I have has suffered similar damage twice in the last two years. Last year, (while I was vacationing in Hawaii, by the way) lightning took out the I/O board and one terminal. This year, (while I was vacationing in Wyoming!) lightning took out the I/O board, three terminals, and the modem. Both times it also wiped out the phones in building (multikey + intercom). The lightning did not hit the building, but somewhere nearby. The communications lines for two of the terminals are strung across an 80 foot hangar on the metal support beams. The cable is sheilded, but I'm not sure if frame ground and signal ground are separated in the computer or terminals. Yes, I do intend to break out my trusty screwdriver, pop the covers off, and find out for sure (and make sure they are!). In the meantime, any advice from some wizard EEs about any protection methods would be greatly appreciated. Pete Wilson AT&T IS CBSG Montgomery Works ..!ihnp4!mgnetp!mgwess!plw
pjg@unrvax.UUCP (Paul Graham) (07/29/85)
[] I've experienced two lighting related problems that had nothing to do with power lines. Both cases involved nearby strikes that got onto the phone lines and crept into computers via serial ports. In one case the modem damage was obvious but it took awhile to conclude that a computer to computer line that ran parallel to a phone line for about 15 feet (under the same cable cover) had a current induced in which took out a serial port on one of the machines. Just about a month ago the same thing happened but two IBM PCs with internal modems had everthing inside fried. Interestingly an XT with the same modem just lost the modem. In response to this some users are getting phone line protectors. These differ from surge/transient suppressors by attempting to short the over voltage to ground. However they demand a fairly good ground (like a cold water pipe). One of PC owners wanted to know why the phone company didn't worry about stuff hanging on the other end of their lines, I asked him if his phone still worked (naturally it did). -- Thanks for your time. Paul Graham 702/784-6007 ucbvax!decvax!seismo!unr70!unrvax!pjg unr70!unvax!pjg@seismo.CSS.GOV
bob@islenet.UUCP (Bob Cunningham) (07/30/85)
> ... In particular I wonder > if your communications lines are above or below ground? What about > your power lines? Are they above or below ground? Secondly, do your > communications lines have any type lightning or surge protection? > Were any of the affected lines shielded? Do you have surge protection > on your power lines? All of the comm lines are underground, in conduits (not steam tunnels, which we don't have out here). In some cases these were leased "4-wire control circuits" from the local phone company (Hawaiian Telephone Co., a GTE subsidiary); in other cases, self-installed. All power lines are also underground (separate conduit, usually completely different trenches). Note that power lines were not affected, nor were regular phone lines (not a single modem was zapped, and no phone problems of any kind were reported). None of the comm lines had lightning or surge protection. Some (a few) were shielded -- though I'm not sure how well the shielding was grounded. Some power lines have surge protection, some not; in any case the path was NOT thru the power lines. Interestingly, some of the affected lines were entirely within a single building, typically of reinforced concrete (and for ordinary r.f. signals, the rebar cage usually provides some shielding). My theory is that -- at some point -- those comm lines were bundled with other lines passing from building to building. > We are in a lightning prone area, and I have been attempting to > understand the intricacies of lightning protection recently. I > confess that the more I hear the more confused I get. I have > concentrated on power protection, but your problems seem to have resulted > from a communications line hit. That's what the evidence definitely indicates. Surprised us. > I wouldn't have thought that such a > hit would spread to so many machines, unless you had a tremendous > earth current transient that affected buried lines in a wide area, > or were unlucky enough to get a hit on some central distribution > point. There was no central distribution point -- several completely separate computer "centers" were involved, none tied together in any way. I'd say the ground current transient (or, perhaps several as a charge briefly "bounded" back and forth from the clouds to the earth and back again until settling) seems likely. > Am I wrong in this evaluation? Is there some other way such > a strike could get into a widespread communications network? How > does one go about protecting such a network from lightning? I'd like some of those answers myself. Anyone else with experience care to comment? -- Bob Cunningham {dual|vortex|ihnp4}!islenet!bob Hawaii Institute of Geophysics Computing Facilities Honolulu, Hawaii
jxyp@lanl.ARPA (08/02/85)
[Apologies if this reaches you twice... problems with an upstream feed.] > > Briefly, all of the on-campus computer facilities with data communications > > lines extending beyond their immediate buildings (typically RS232 > > 3-or-4-wire leased phone circuits or similar) suffered burned out > > ports and burned out terminals ... > details that weren't mentioned in your note. In particular I wonder > if your communications lines are above or below ground? We are in the midst of our second storm season at our new campus. It's a small campus, but we do have several terminals in buildings separate from the computing center (where the cpu is located). Except for scale, we are experiencing (repeatedly!) exactly what happened in Hawaii. Our remote terminals are connected with rs232 cables, underground, in conduit. Our power supply to the VAX is protected. We have an Emulex CS11 (DH11 emulator) with 4 distribution panels. During most electrical storms we have lost one of the panels and various other peripherals including terminals and our line printer (which is in the computing center). Our first attempt at defense was to install individual surge protectors at the power connection of each remote terminal. Next we added surge protectors to our dialup phone lines. I assumed that the data lines to the remote terminals *couldn't* be the culprits, since they are underground and in conduit. The latest advice (from one of DEC's field people) is that this is a typical problem with sites where equipment is in several buildings; that, during an electrical storm, each building will have a different potential to ground! Thus, the ground wire in the rs232 cables suddenly carries the difference. I have disconnected (at both ends) all of the terminals outside the computing center until we can get protection on the data lines. We have had several lightning storms since, and (knock on wood) no more damage. Please, can someone with experience and *knowledge* join this discussion? -- Jay Plett {cmcl2,ihnp4}!lanl!unm-la!jay {gatech,ucbvax}!unmvax!unm-la!jay jxyp@lanl.ARPA
chris@umcp-cs.UUCP (Chris Torek) (08/02/85)
>The latest advice (from one of DEC's field people) is that this is a >typical problem with sites where equipment is in several buildings; >that, during an electrical storm, each building will have a different >potential to ground! That's why Ethernet cables are supposed to be grounded in one and only one place. No? -- In-Real-Life: Chris Torek, Univ of MD Comp Sci Dept (+1 301 454 4251) UUCP: seismo!umcp-cs!chris CSNet: chris@umcp-cs ARPA: chris@maryland
smb@ulysses.UUCP (Steven Bellovin) (08/02/85)
> The latest advice (from one of DEC's field people) is that this is a > typical problem with sites where equipment is in several buildings; > that, during an electrical storm, each building will have a different > potential to ground! Thus, the ground wire in the rs232 cables suddenly > carries the difference. I have disconnected (at both ends) all of the > terminals outside the computing center until we can get protection on > the data lines. We have had several lightning storms since, and (knock > on wood) no more damage. > I'd say that DEC is right. RS-232 pin 7 ties together the electrical grounds on both ends; any difference in ground potential -- whether from lightning or simply a different electrical feed -- can cause trouble. A solution -- damnifIknow; we tried all sorts of things when I was at UNC Chapel Hill without notable success. Optoisolators tend to protect the equipment but get fried themselves, but they're often better than nothing.
larry@kitty.UUCP (Larry Lippman) (08/05/85)
> > > Briefly, all of the on-campus computer facilities with data communications > > > lines extending beyond their immediate buildings (typically RS232 > > > 3-or-4-wire leased phone circuits or similar) suffered burned out > > > ports and burned out terminals ... > > We are in the midst of our second storm season at our new campus. It's > a small campus, but we do have several terminals in buildings separate > from the computing center (where the cpu is located). Except for scale, > we are experiencing (repeatedly!) exactly what happened in Hawaii. > > Our remote terminals are connected with rs232 cables, underground, in > conduit. Our power supply to the VAX is protected. We have an Emulex > CS11 (DH11 emulator) with 4 distribution panels. During most electrical > storms we have lost one of the panels and various other peripherals > including terminals and our line printer (which is in the computing > center). Our first attempt at defense was to install individual surge > protectors at the power connection of each remote terminal. Next we > added surge protectors to our dialup phone lines. I assumed that the > data lines to the remote terminals *couldn't* be the culprits, since > they are underground and in conduit. > > The latest advice (from one of DEC's field people) is that this is a > typical problem with sites where equipment is in several buildings; > that, during an electrical storm, each building will have a different > potential to ground! Thus, the ground wire in the rs232 cables suddenly > carries the difference. I have disconnected (at both ends) all of the > terminals outside the computing center until we can get protection on > the data lines. We have had several lightning storms since, and (knock > on wood) no more damage. [Sorry about leaving so much of the original article, but I felt it important for anyone just joining in...] I am going to give you a simple and inexpensive suggestion which is predicated on the following assumptions: (1) You are running async at 9,600 baud or less. (2) You do not require any RS-232 handshaking leads (RTS, CTS, DSR, DTR, etc.) and that any flow control is character oriented. The suggestion: Use optically isolated (at both ends) 30 mA current loop, which requires two pairs - one for RxD and one for TxD. You can get fairly inexpensive current loop-to-RS-232 converters with *indivually* isolated power supplies for well under $ 200.00 per pair. If you or any of your people are electronically handy, you can build what you need for less than $ 60.00 in parts per data line. If your DTE and/or DTE already has a current loop option, then you will not require RS-232 conversion. You should make certain that any current loop option is *passive* and optically isolated. In that case, all you need do is provide a current limited, isolated source of excitation current for the RxD and TxD pairs. Using current loop as above, there is *NO* DC path from the DTE apparatus to the DCE apparatus; the current loop which connects the two ends is optically isolated and floats between them. As a conservative measure, you should use shielded pairs, but UNDER NO CIRCUMSTANCES should you connect the shields to either DTE or DCE apparatus; just connect the shields to good earth ground at both ends (i.e., water pipe). Larry Lippman Recognition Research Corp. Clarence, New York UUCP {decvax,dual,rocksanne,rocksvax,watmath}!sunybcs!kitty!larry {rice,shell}!baylor!kitty!larry syr!buf!kitty!larry VOICE 716/741-9185 TELEX {via WUI} 69-71461 answerback: ELGECOMCLR "Have you hugged your cat today?"
olson@fortune.UUCP (Dave Olson) (08/06/85)
I used to work at a company in Mt. View, CA, that regularly had problems with terminals and DZ-11 interface boards (on 11/60's and 11/70's) being destroyed by lightning. Because of the product we manufactured, we had lots of UARTs, and just kept replacing the terminals' UARTs, and getting the occasional DZ-11 replaced. Finally, in one particularly bad storm, we lost 12 of 15 DZ boards, and ~40 terminals. About 80% of the terminal damage was restricted to the UARTs, the rest had more extensive damage. All the systems went down about 2-3 minutes later, due to a related power outage, however there was no other damage to the computers.) After that, we built a number of optical isolators. We never lost any boards or terminals after that. Some of our RS232 lines ran 900 feet, most were 80-200 ft. All connections were unshielded 4-wire to a plug-board, and std 25-pair telephone wire fanning out to various locations from there. Most of the destroyed terminals were in the same ferro-cement building as the computers. (Side note: all of them were used at 9600 baud, even those 900 feet away. Just goes to show how far you can push the standard...) We also had related problems because some of the terminals were in a building with a different power supply; we measured differences of 5-8 volts (DC) in the ground potential... The isolators (and making sure signal and frame grounds were NOT connected) solved that problem too. Dave Olson, Fortune Systems
olson@fortune.UUCP (Dave Olson) (08/06/85)
I used to work at a company in Mt. View, CA, that regularly had problems with terminals and DZ-11 interface boards (on 11/60's and 11/70's) being destroyed by lightning. Because of the product we manufactured, we had lots of UARTs, and just kept replacing the terminals' UARTs, and getting the occasional DZ-11 replaced. Finally, in one particularly bad storm, we lost 12 of 15 DZ boards, and ~40 terminals. About 80% of the terminal damage was restricted to the UARTs, the rest had more extensive damage. All the systems went down about 2-3 minutes later, due to a related power outage, however there was no other damage to the computers.) After that, we built a number of optical isolators. We never lost any boards or terminals after that. Some of our RS232 lines ran 900 feet, most were 80-200 ft. All connections were unshielded 4-wire to a plug-board, and std 25-pair telephone wire fanning out to various locations from there. Most of the destroyed terminals were in the same ferro-cement building as the computers. (Side note: all of them were used at 9600 baud, even those 900 feet away. Just goes to show how far you can push the standard...) We also had related problems because some of the terminals were in a building with a different power supply; we measured substantial differences in the ground potential (which no doubt fluctuated radically with time). The isolators (and making sure signal and frame grounds were NOT connected) solved that problem too. Dave Olson, Fortune Systems
w8sdz@brl-tgr.ARPA (Keith B. Petersen ) (08/06/85)
The most effective lightning protection for RS-232 is no hardwire connection. This can be accomplished by using fibre optics. Several manufacturers offer RS-232<-->fibre optic<-->RS-232 at reasonable prices. Belden (the wire manufacturer) is one. -- Keith Petersen
bob@islenet.UUCP (Bob Cunningham) (08/11/85)
At last count, over 40% of the zapped terminals we had were repairable only by replacing the 1488 and/or 1489 chips (usually the 1488). An industrial-quality solder remover (heater + vacuum pump) is highly recommended. The percentage seems to be slightly higher (better) for port selector boards (Gandalf & Micom). Needless to say, we've been installing sockets for those chips where practical. On the computer side, all the port/multiplexer (DZ or whatever) boards were just swapped out. Not sure what percentage just had driver chips burned out or not. Subsidiary damage (one tape unit and some memory boards) was limited to just 2 systems, and my opinion is that not all the equipment at those locations was solidly tied to a single ground. Besides the followup articles in this newsgroup, I've received a considerable number of mail messages (all appreciated, though I've not had time to send individual replies), falling into two categories: 1) similar horror stories 2) thoughtful advice on lightning protection Apparently, similar incidents (lightning damage via local data comm lines) are much more common & widespread than I'd have thought. Unless you're in a very unusual location, if you've got comm lines going between buildings, be warned: something similar might just happen to you. While I've received a number of very sensible suggestions on lightning protection, there doesn't seem to be one single solution we could adopt in all cases. Suggestions have ranged from using opto-isolators, diodes (of various sorts, including MOVs), to using telco-type spark gap devices, to fiber optics. Each approach has some good points. However, the thought of having to install any particular suggestion on the 800 or so data comm lines around campus which probably should have protection is a sobering thought. Schemes cheap in material (e.g. diodes) look to be rather labor-intensive (if both terminal and computer/port selector ends both need protection -- which seems optimal). Schemes cheap in time (e.g. fiber optics) look a bit expensive in parts (though the thought of pulling a lot of fiber optic cables to replace twisted-pair lines is also rather sobering). So far, I think our best approach here is to stick to the basics. For starters, going over all of the central grounds. All Computer equipment and auxiliary racks within a room should be securely grounded to a single point with generous-size braided ground straps. The objective is to minimize any possible ground differences between computers and their peripherals (including port selectors). It seems to be a good idea to tie down all incoming terminal grounds (RS232 pin 7) to that same point -- as the lines come in (typically on punch blocks). This should localize damage (typically to the port selectors). It also seems reasonable to dedicate whole computer-side port/multiplexer boards to PACX lines ... no more mixing direct-connect and port selector terminals. Typically, the only terminals I want to leave directly connected will be those in the same building (preferrably the same wing) as the computer site. This should also minimize computer-side damage. Fortunately, port selector boards are MUCH cheaper to repair/replace than computer port/multiplexer boards. Relatively expensive terminals (graphics types, for the most part) should get some special form of isolation at their end from the RS232 lines (preferrably something simple that plugs in between the RS232 line and the terminal). Cheaper terminals (< $1,000) I think are best left as-is for now. All NEW inter-building trunks (multiple data comm lines) that I have any control over will be fiber optic lines (stat muxes at each end, of course). We do have some plans for a couple of real LANs around campus. I do believe that specs for those will now definitely include some form of lightning protection. -- Bob Cunningham {dual|vortex|ihnp4}!islenet!bob Hawaii Institute of Geophysics Computing Facilities Honolulu, Hawaii
chris@umcp-cs.UUCP (Chris Torek) (08/15/85)
>... All Computer equipment and auxiliary racks within a room should >be securely grounded to a single point with generous-size braided >ground straps. It seems to be a good idea to tie down all incoming >terminal grounds (RS232 pin 7) to that same point. . . . Everything I thought I knew about grounding says you never tie signal ground (pin 7) to frame ground. Am I wrong, or is this a Big Mistake? -- In-Real-Life: Chris Torek, Univ of MD Comp Sci Dept (+1 301 454 4251) UUCP: seismo!umcp-cs!chris CSNet: chris@umcp-cs ARPA: chris@maryland
jeq@laidbak.UUCP (Jonathan E. Quist) (08/16/85)
In article <1242@umcp-cs.UUCP> chris@umcp-cs.UUCP (Chris Torek) writes: >>... All Computer equipment and auxiliary racks within a room should >>be securely grounded to a single point with generous-size braided >>ground straps. It seems to be a good idea to tie down all incoming >>terminal grounds (RS232 pin 7) to that same point. . . . > >Everything I thought I knew about grounding says you never tie signal >ground (pin 7) to frame ground. Am I wrong, or is this a Big Mistake? >-- According to the standard (Yes, it is a published standard, though I haven't actually seen a copy in years), pin 7 is signal ground, pin 1 is "safety" ground. While some equipment ties the two together internally, most _well designed_ distribution panels seperate the two. In my experience, several terminals (on long cable runs) which had noise problems improved when the I cut the strapping between pins 1 and 7 at the terminal end. (A colleague and I had routinely tied the two together until we noticed this.) If you are running shielded cable (what a concept!) to your terminals, it may help to tie the shield to ground at _only_ one end of the cable run. Since the distribution panel is likely the only sure ground in the system, all shields should be tied at that point. (If I'm beginning to confuse audio practices with digital practices, light your torches and let me know!) Jonathan E. Quist Lachman Associates, Inc. ihnp4!laidbak!jeq ``I deny this is a disclaimer.''
bob@islenet.UUCP (Bob Cunningham) (08/17/85)
> >... All Computer equipment and auxiliary racks within a room should > >be securely grounded to a single point with generous-size braided > >ground straps. It seems to be a good idea to tie down all incoming > >terminal grounds (RS232 pin 7) to that same point. . . . > > Everything I thought I knew about grounding says you never tie signal > ground (pin 7) to frame ground. Am I wrong, or is this a Big Mistake? This is an almost-counter-intuitive trick I picked up at another site where we were having a few seemingly-weird problems with RS232 signals that seemed to "float" far too much. The point is to make sure that -- at the computer port side -- all the signal grounds sit at the same potential. Easiest way to accomplish this is to tie them all to the reference ground. We don't carry pin 1 (frame ground) from the terminals to the computer. From my experience, it works, even though there is potential (no pun intended) ground loop problem with remote terminals that also tie pin 7 to their frame ground, and have a different reference ground (say, in another building). -- Bob Cunningham {dual|vortex|ihnp4}!islenet!bob Hawaii Institute of Geophysics Computing Facilities Honolulu, Hawaii
dgary@ecsvax.UUCP (D Gary Grady) (08/20/85)
> According to the standard (Yes, it is a published standard, though > I haven't actually seen a copy in years), pin 7 is signal ground, > pin 1 is "safety" ground. For what it's worth, IBM's asynchronous communications adapter makes no use of pin 1. I'm not surprised, are you? :=) <- smiley pig -- D Gary Grady Duke U Comp Center, Durham, NC 27706 (919) 684-3695 USENET: {seismo,decvax,ihnp4,akgua,etc.}!mcnc!ecsvax!dgary
jbn@wdl1.UUCP (08/23/85)
The idea is that your grounding system must be loop-free. This is often difficult to achieve without such annoyances as breaking the ground line at one end of a cable. Another reason that differential signalling (a la RS-449) is a Good Thing. John Nagle