On January 15th, 1990, AT&T's New Jersey operations center identified a widespread system malfunction, marked by numerous red warnings on their network display.
Despite efforts to resolve the issue, the network remained compromised for 9 hours, resulting in a 50% failure rate in call connections.
The fallout was significant, with AT&T experiencing a loss exceeding $60 million, leaving over 60,000 Americans without functional phones. Additionally, 500 airline flights were delayed, affecting 85,000 people.
AT&T's long-distance network, typically praised for its efficiency, handling a substantial portion of the nation's calls, faced a critical failure on this day. A software bug in a recent update caused a fault to cascade through the network, impacting its 114 switches. The bug was a result of a small, untested software patch implemented at management's request.
The root cause was traced back to a coding error within a C program involved in a software update across the network's switches. The misplaced break statement within nested conditional statements led to data overwrites and system resets.
Pseudocode:
The erroneous break statement caused data overwrites instead of proper processing, leading to system resets across all switches.
Engineers worked for 9 hours to restore AT&T's system by rolling back switches to a previous, functional version of the code. It took an additional two weeks of thorough code reading, testing, and replication to identify and comprehend the bug.
Despite the rigorous testing and resilient network design, a single line of code brought down half the country's main communication network.
In conclusion, AT&T faced numerous system crashes in the '90s, and while today's companies have improved processes, occasional bugs still slip through. The incident serves as a reminder that outages often result from human error and process gaps, emphasizing the importance of continuous learning from such events.
Comments