Partners Pick Up Pieces After Massive CenturyLink Outage Hits Business Users, Consumers

CenturyLink closed out its year on a low note after a nationwide outage took down phone and internet services for consumers and business users, including 911 services in some states.

Residential and business customers from Massachusetts to California began reporting issues with their phone and internet services on Dec. 27. The widespread outage knocked out services completely for some users, while other users experienced limited functionality. CenturyLink said two days later that it had resolved the outage, which it blamed on a single malfunctioning network card.

One telecom solution provider that partners with the country's largest carriers, including CenturyLink, said that many of its clients experienced service interruptions late last week because of the outage, with some being offline for two days.

"It was a little scary because you wouldn't think a network of that size with one bad card could have this much of an impact," the solution provider, who requested anonymity, said.

[Related: CenturyLink 'Not Satisfied' With Low-Margin Products; Plans To Double Down On Strategic IT Solutions]

The solution provider was notified of the outage via CenturyLink's online partner portal. "I could see panic from CenturyLink -- it was really all hands on deck, but they basically sent themselves on a goose chase while also trying to be proactive by taking its equipment offline," the partner said.

The outage isn't as much of a problem as the time it took for CenturyLink to resolve the issue, the partner added.

"It seemed like they should have been able to pinpoint something this big more quickly," the partner said.

Federal Communications Commission Chairman Ajit Pai on Dec. 28 called the outage “completely unacceptable” because of its impact on emergency services, and said that the FCC would investigate.

“When an emergency strikes, it’s critical that Americans are able to use 911 to reach those who can help,” Pai said. “The breadth and duration [of the outage is] particularly troubling.”

Monroe, Louisiana-based CenturyLink tweeted on Dec. 29 that all services for businesses, residents and emergency call systems were restored. However, some business customers are still reporting limited functionality.

"CenturyLink knows how important connectivity is to our customers, so we view any disruption as a serious matter and sincerely apologize for any inconvenience that resulted. We now return to normal business operations, and customers who have a service issue should contact CenturyLink’s repair department," the carrier said in Facebook post.

CenturyLink said the event impacted visibility into its network management system, which impairing the carrier's ability to troubleshoot and prolonged the duration of the outage.

"The outage was caused by a faulty network management card from a third-party equipment vendor that caused invalid traffic replication … Steps are being taken to help prevent the issue from reoccurring,” said the company in a statement.

An executive with Carrier Access, a solution provider based in Clive, Iowa, that partners with CenturyLink, said that his company builds redundancy into its voice and connectivity solutions in the form of other routes or secondary providers for its global customers, so that if disaster strikes, these businesses won't be completely down.

"Outages like this typically help the partner community more than they hurt us," said Shane Stark, director of vendor and channel relations for Carrier Access. "It truly shows the value of the partner community, and it’s the best commercial you'll ever have for SD-WAN."