[clari.nb.telecom] Inside Story of the Great AT&T Outage

newsbytes@clarinet.com (01/18/90)

NEW YORK, NEW YORK, U.S.A., 1990 JAN 16 (NB) -- During a 
teleconference carried nationwide over a free 900 number, AT&T 
Chairman Robert Allen blamed the great phone brown-out of Jan. 
15 on software designed to make the system even more redundant. 
 
American Telephone & Telegraph Co.'s long-distance service was 
disrupted Monday. Service was crippled about 2:50 p.m. EST, when 
callers began receiving recorded messages saying the company's 
network, the largest in the nation, was overloaded. The company 
announced around 11:45 PM that a software "bandage" had returned 
the network to full service.                        

Chairman Robert Allen took personal blame for the problem. "Even 
though it was a one-time hit, this was the most far-reaching 
service problem we ever experienced," he said. "We responded with 
every resource at our disposal. We worked throughout the night, 
nonstop and flat out. And we'll do whatever it takes to make sure 
this doesn't recur." 

"What we believe triggered the trouble was a software problem in 
a new signaling feature we were introducing, which would add 
redundancy. I wish I could tell you exactly what went wrong. I 
can't. But here's the scenario. 

"At about 2:25 Eastern, a software problem occurred in the 
network" [apparently when AT&T engineers tried to add additional 
circuits to a New York City switch.] "Preliminary indications of 
problems in signaling spread rapidly"  [as the switch overloaded, 
passing its problems along.] "That spread to the switch network, 
including toll-free 800 lines. AT&T private line service and 
other special networks, including those of the government, were 
not affected" because they're not part of the normal switched 
network.  

Normally, Allen added, AT&T handles about 110 million calls. Due 
to the Martin Luther King holiday, only about 100 million calls 
were attempted Jan. 15. At the problem's height, call 
completions dropped 50%.               

Ken Garrett, senior vice president of network services, said the 
company believes there was no sabotage and no computer virus 
involved. "That's always been a suspicion, but we have a growing 
level of confidence that's not the case. We won't rule it out 
until we're absolutely sure," he added.  The switch that failed 
was an AT&T 4ESS, which is programmed in the C language. 

Allen declined continued requests to discuss the company's 
marketing strategy in the wake of the disaster, saying: "Our first 
problem is to make sure the network is working... [and] secondly, 
we're going to be talking to customers today, reassuring them of 
our actions." He hinted, however, that the company might take out 
full-page newspaper ads as early as Jan. 17. "It's the first 
time in 32 1/2 years I've been in the business we've had this 
kind of failure."

Allen said the company would also try to compensate customers, 
while denying that AT&T had any legal obligation to do that. "We 
have an 800-number assurance policy, which we'll honor, even 
though this condition doesn't fit. We're also looking at our 
other options on a customer-by-customer basis. We do intend to 
file an emergency petition with the FCC today, calling for a 
special day of discounted calling, which will be some small 
compensation to all customers." 

The 800-number assurance policy says that if a customer has a 
problem AT&T will switch them over within an hour to alternate 
facilities. Compensation follows ifthat does not occur. Only one 
customer requested that AT&T move its 800 services to another carrier, 
and Allen said the company's attempts to do that were rebuffed. 

Allen also discussed a "New York Times" report that, early on, 
operators were unwilling to give callers information on how to 
make their calls with other carriers. "That was accurate early in 
the day. We asked our operators later to offer those numbers to 
our customers in the event they asked for them." 
                                                 
Allen said the company has not calculated how much revenue was 
lost directly, nor have they estimated how much market share AT&T 
could lose as a result of its problems. 

(Dana Blankenhorn/19900116/Press Contact: Daisy Ottman, AT&T, 
201-221-6227)