The IT Nerd

I’ll put the news out there first before I tell you what I think of it. Rogers CEO Tony Staffieri has put out another statement that speaks to their intentions to make their network better. Here’s what you need to know:

First, emergency calls to 911 simply have to work. Every time. We have made meaningful progress on a formal agreement between carriers to switch 911 calls to each other’s networks automatically – even in the event of an outage on any carrier’s network. I believe this is the only responsible way forward and I am personally committed to making it possible for all Canadians.

That’s a good first step seeing that Rogers failed miserably in that department. There’s also the fact that telcos like Rogers are legally required to have 911 calls work 100% of the time without fail.

Second, Rogers will set a higher standard for reliability by physically separating our wireless and internet services to create an ‘always on’ network – to help make sure our customers don’t experience an outage with both cellular and internet services again.

I will admit that splitting the networks will help to keep Canadians online if something like this happens again. Though the cynic in me says that if there’s a way for Rogers to have both networks fail, they will find it and it will happen.

Third, we will continue to focus on reliability, investing $10 billion over the next three years. This includes more oversight, more testing and greater use of Artificial Intelligence to ensure we’re able to deliver the reliable service you deserve.

That implies to me that they weren’t spending either enough money or any money in any of those areas. So I guess you can call this an improvement. But you have to ask yourself why haven’t they done any or all of this prior to the network crashing down around them a couple of weeks ago.

Finally, we are partnering with leading technology firms to do a full review of our network to help us learn from the outage. We will share lessons with our industry for the benefit of every Canadian.

Alternate plan: How about sharing those lessons with every Canadian so that Rogers can be held accountable for addressing their shortcomings? Right. I keep forgetting. That’s not going to happen. Rogers isn’t that sort of company to be completely transparent as proven by their CRTC filing.

Frankly, I am not impressed.

What I read from this is that Rogers is in full panic mode as I am guessing that businesses and individuals such as myself are rapidly fleeing to other telcos such as Bell and Telus. Thus they have to say something to keep their churn rates from skyrocketing massively, which in turn would cause their stock value to drop. Not to mention that they really need to do something to keep their attempt to merge with Shaw Communications from becoming a train wreck next to a dumpster fire. What I also read from this is that Rogers was likely underfunding their network, which is something that always comes back to bite you. I’ll give Staffieri points for coming out with statements like these, but I don’t think it’s going to be enough to save Rogers.

You might recall that the CRTC gave Rogers 10 days to explain what happened during its network outage from earlier this month. Now according to CBC News, they’ve done that:

The outage, which started early on July 8 and for some customers lingered for days, left millions without cellphone and internet service — prompting questions from the federal government and the Canadian Radio-television and Telecommunications Commission (CRTC).

“An update in our core IP [internet protocol] network … caused our IP routing network to malfunction,” the letter read.

And:

In its letter, Rogers said coding from the update deleted a routing filter that “allowed for all possible routes to the Internet to pass through the routers,” which flooded and overwhelmed the core network, causing it to stop processing internet traffic altogether.

“As a result, the Rogers network lost connectivity to the Internet.”

This seems to dovetail with this observation by Cloudflare where Rogers effectively deleted itself off the Internet. Here’s where it gets interesting:

The letter, posted on the CRTC website, met the regulator’s deadline for Rogers to answer questions about the outage. However it has many redactions where Rogers is believed to have offered more specific details about the problem and its plans to prevent something similar from occurring again. 

The CRTC said Rogers submitted two versions of the letter, one unabridged and the other with redactions, and that it released the latter to protect “highly sensitive information” about Rogers’ operations. 

Hmmm… I would like to believe that Rogers is protecting “highly sensitive information”. But something about this doesn’t sit right with me. Thus I have to wonder what they really have to hide. Is it perhaps that their network wasn’t really in a good place when the outage happened? If Rogers really wants to meet the mark on this:

“In order to regain the trust of Canadians, it is important that we provide open answers to the questions that they have about the outage,” the Rogers letter read. “That is why when answering the CRTC … Rogers is being as transparent as possible.”

Then their need to be transparent needs to be 100%. The fact that they have a redacted document that is available here doesn’t inspire confidence. They’ll need to do better if they want to regain the trust of Canadians.

Though to be completely fair to Rogers, they are doing some things that I believe that they need to do to regain the trust of Canadians:

It is clear that what matters most is that Rogers ensures this does not happen again. We are conducting a full review of the outage. Our engineers and technical experts have been and are continuing to work alongside our global equipment vendors to fully explore the root cause and its effects. We will also increase resiliency in our networks and systems which will include fully segregating our wireless and wireline core networks. Lastly, we have additionally hired an external review team to further assess and provide insights into the outage. This will involve a complete evaluation of all our processes, including the performance of network upgrades, disaster recovery procedures, and communication with the public. 

Additionally, Rogers will work with governmental agencies and our industry peers to further strengthen the resiliency of our network and improve communication and co-operation during events like this. Most importantly, we will explore additional measures to maintain or transfer to other networks 9-1-1 and other essential services during events like these.

All of this is good because as I said here, their communication with the public sucked during this outage. And the other items that they mentioned like having a third party come in and review how they work, and working with industry peers which they were ordered to do are all good as well. But the only thing that matters is results and transparency. Let’s see if Rogers has it within them to do both so that they regain the trust of Canadians.