The IT Nerd

You might recall that the CRTC gave Rogers 10 days to explain what happened during its network outage from earlier this month. Now according to CBC News, they’ve done that:

The outage, which started early on July 8 and for some customers lingered for days, left millions without cellphone and internet service — prompting questions from the federal government and the Canadian Radio-television and Telecommunications Commission (CRTC).

“An update in our core IP [internet protocol] network … caused our IP routing network to malfunction,” the letter read.

And:

In its letter, Rogers said coding from the update deleted a routing filter that “allowed for all possible routes to the Internet to pass through the routers,” which flooded and overwhelmed the core network, causing it to stop processing internet traffic altogether.

“As a result, the Rogers network lost connectivity to the Internet.”

This seems to dovetail with this observation by Cloudflare where Rogers effectively deleted itself off the Internet. Here’s where it gets interesting:

The letter, posted on the CRTC website, met the regulator’s deadline for Rogers to answer questions about the outage. However it has many redactions where Rogers is believed to have offered more specific details about the problem and its plans to prevent something similar from occurring again. 

The CRTC said Rogers submitted two versions of the letter, one unabridged and the other with redactions, and that it released the latter to protect “highly sensitive information” about Rogers’ operations. 

Hmmm… I would like to believe that Rogers is protecting “highly sensitive information”. But something about this doesn’t sit right with me. Thus I have to wonder what they really have to hide. Is it perhaps that their network wasn’t really in a good place when the outage happened? If Rogers really wants to meet the mark on this:

“In order to regain the trust of Canadians, it is important that we provide open answers to the questions that they have about the outage,” the Rogers letter read. “That is why when answering the CRTC … Rogers is being as transparent as possible.”

Then their need to be transparent needs to be 100%. The fact that they have a redacted document that is available here doesn’t inspire confidence. They’ll need to do better if they want to regain the trust of Canadians.

Though to be completely fair to Rogers, they are doing some things that I believe that they need to do to regain the trust of Canadians:

It is clear that what matters most is that Rogers ensures this does not happen again. We are conducting a full review of the outage. Our engineers and technical experts have been and are continuing to work alongside our global equipment vendors to fully explore the root cause and its effects. We will also increase resiliency in our networks and systems which will include fully segregating our wireless and wireline core networks. Lastly, we have additionally hired an external review team to further assess and provide insights into the outage. This will involve a complete evaluation of all our processes, including the performance of network upgrades, disaster recovery procedures, and communication with the public. 

Additionally, Rogers will work with governmental agencies and our industry peers to further strengthen the resiliency of our network and improve communication and co-operation during events like this. Most importantly, we will explore additional measures to maintain or transfer to other networks 9-1-1 and other essential services during events like these.

All of this is good because as I said here, their communication with the public sucked during this outage. And the other items that they mentioned like having a third party come in and review how they work, and working with industry peers which they were ordered to do are all good as well. But the only thing that matters is results and transparency. Let’s see if Rogers has it within them to do both so that they regain the trust of Canadians.

This entry was posted on July 23, 2022 at 3:40 pm and is filed under Commentary with tags Rogers. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.