Technolgy Rules: Here’s what caused yesterday’s O2 and SoftBank outages

Friday, December 7, 2018

Here’s what caused yesterday’s O2 and SoftBank outages

It appears that most mobile carriers, including O2 and SoftBank, have recovered from yesterday’s cell phone network outage that was triggered by a shutdown of Ericsson equipment running on their networks. That shutdown appears to have been triggered by expired software certificates on the equipment itself.

While Ericsson acknowledged in their press release yesterday that expired certificates were at the root of the problem, you may be wondering why this would cause a shutdown. It turns out that it’s likely due to a fail-safe system in place, says Tim Callan, senior fellow at Sectigo (formerly Comodo CA), a U.S. certificate-issuing authority. Callan has 15 years of experience in the industry.

He indicated that while he didn’t have specific information on this outage, it would be consistent with industry best practices to shut down the system when encountering expired certificates “We don’t have specific visibility into the Ericsson systems in question, but a typical application would require valid certificates to be in place in order to keep operating. That is to protect against breach by some kind of agent that is maliciously inserted into the network,” Callan told TechCrunch.

In fact, Callan said that in 2009 a breach at Heartland Payments was directly related to such a problem. “2009’s massive data breach of Heartland Payment Systems occurred because the network in question did NOT have such a requirement. Today it’s common practice to use certificates to avoid that same vulnerability,” he explained.

Ericsson would not get into specifics about what caused the problem.”Ericsson takes full responsibility for this technical failure. The problem has been identified and resolved. After a complete analysis Ericsson will take measures to prevent such a failure from happening again.”

Among those affected yesterday were millions of O2 customers in Great Britain and SoftBank customers in Japan. SoftBank issued an apology in the form of a press release on the company website. “We deeply apologize to our customers for all inconveniences it caused. We will strive to take all measures to prevent the same network outage.”

As for O2, they also apologized this morning after restoring service, tweeting:

Our 4G network was restored earlier this morning. Our technical teams will continue to monitor service performance closely and we’re starting the full review to understand what happened. We are really sorry for the issues yesterday.

— O2 in the UK (@O2) December 7, 2018