IBM Blames Massive Cloud Outage On Third-Party Network Provider

Big Blue has yet to provide a detailed post-mortem of a multi-hour, multi-region cloud services disruption. ‘It affected everything,’ says one IBM partner. ‘The whole environment was down.’

IBM said Wednesday a networking failure involving a third party caused a serious cloud outage the day before, the impact of which IBM partners said was widely felt by their customers.

The CEO of one IBM Business Partner that builds cutting-edge solutions on Big Blue’s platform told CRN that customers across the U.S. lost access to their environments, their status screens and consoles, and they had “no sense of what was happening.”

“It affected everything,” he said. “The whole environment was down.”

[Related: The 10 Biggest Cloud Outages Of 2019 (So Far)]

The IBM Cloud status page, which also was briefly down during Tuesday’s disruption, reports a slew of issues that were resolved after 6:30 p.m. ET.

“The network operations team adjusted routing policies to fix an issue introduced by a 3rd party provider and this resolved the incident,” the status page explained.

Among the hodgepodge of reported problems was the inability to load Virtual Private Clouds, provision infrastructure, and simply access IBM Cloud.

Complaints of a lack of transparency have streamed across Twitter, as many customers said IBM Cloud was slow to acknowledge problems or respond to inquiries. It took roughly two hours after problems that crippled enterprise environments and some popular websites arose before IBM chimed in.

At 8:26 p.m. ET, IBM Cloud’s Twitter account first said it was aware of the outage. Slightly more than an hour later, the Armonk, N.Y.-based tech company tweeted it was working to restore services, and by 9:54 pm, it reported all IBM Cloud services had been restored.

IBM tweeted at 1:56 p.m. ET Wednesday that it has “focused on external network provider issues as cause of the disruption of IBM Cloud services.”

“Was a huge network issue is what we know so far,” said the IBM partner.