The IT Nerd

Now that it looks like the multi-day Rogers outage is on it’s way to being resolved, I figured it would be worth doing a post mortem of this incident. The reason being is that I believe there are things that can be taken away from this incident that can either help to avoid another one like it, or to improve the response to a similar situation.

The first thing that jumps out at me is the system that failed and caused this outage. Whatever systems that Rogers had that authorizes its hardware when you connect it to their network clearly failed. And it took days for Rogers to get things fixed to a point that customers could come back online. This is the sort of system that sounds business critical to me, which in my world is something that usually has a backup system. But based on what seems to have gone down from my perspective, it appears on the surface that this may not be the case. If that is accurate, then that is reason to worry if you’re a Rogers customer as this could happen again. Now if Rogers was a small ISP, I could see them not having a backup system of some sort. But this is Canada’s largest telco. That should not be the case. The other possibility is that if they did have a backup system, it clearly didn’t work. If that’s what happened, then Rogers has a much bigger issue that they need to deal with. In either case, one hopes that they address this going forward. On top of that, one has to wonder if there are other business critical systems within Rogers that lack any sort of backup at all or one that they know is in a working state.

That brings me to point number two, it took Rogers days to resolve this. During that time the following happened:

• Retail stores allegedly had no idea that this was going on. But they still handed out hardware that had zero chance of working, which in turn compounded the amount of angry customers the telco had to deal with.
• Tech support was somewhat aware of this issue, but there was nothing they could say or do other than offer a service credit if you pushed hard enough.
• The Rogers social media team didn’t start to admit that there was a widespread problem until this past weekend. During that time their responses weren’t consistent. Then when they did admit to it, they sent out at least a couple of messages saying that things were resolved when in reality they were not even close to being resolved. Not to mention a secondary issue of some sort was uncovered. And late on in this crisis, customers were unable to get responses for them for extended periods of time which didn’t help their cause.

The net result is that Rogers customers who were affected by this outage were not happy campers. If you look through any of my stories covering this outage that’s clear as I posted Tweets in each story from customers who took to Twitter to express their displeasure. These are people who counted on Rogers to provide them with quality Internet service, and were willing to give Rogers some slack when this incident started. But when the above happened, and the days dragged on, any goodwill those customers had went out the window. I’ve talked to exactly 103 people via e-mail, phone and Twitter about this issue. None of them are happy. If these 103 people represent how some, most or all of Rogers customers who were affected by this feel, then Rogers has a very big problem on their hands. These are people who will at the very least bad mouth Rogers to anyone who will listen. At worst, people will do that and leave Rogers for other telcos such as Bell Canada. Either way, it’s not good for Canada’s largest telco.

Now this does sound dire. But strange as it may seem, Rogers can rescue this. If I ran Rogers, here’s what I would do:

1. Tell Rogers customers what happened, why it happened, and why it will never happen again in a robust manner and give details on how things are going to change. For example, in this case Rogers let people get new hardware that had zero chance of working and allegedly their retail staff had no idea that an critical event was going on. The fix for that is to ensure that all parts of the organization know when a critical event happens and take the best action in the interest of their customer base. As in not handing out hardware to customers that had zero chance of working. The bottom line is that customers have to know that you recognize that you’ve made a mistake and you’re willing to invest to fix it because they want to trust you. If customers can’t trust you, they’ll go elsewhere to someone who they can trust.
2. Rogers needs to apologize. They need to do so in a robust and fulsome way that shows that they regret the inconvenience that they’ve caused their customers. Sending a direct email to all affected customers would work best in this situation. Because when one does that, most people will respond positively. Conversely, when when that doesn’t happen, customers go elsewhere because they feel that the organization doesn’t care.
3. Make amends by giving a credit of some sort to customers. And by that, I don’t mean a credit that amounts to a week or two of service. I mean a credit that takes into account that some Rogers customers had difficulty working from home, making money if they were self employed, and had to use expensive cellular data to get online. A credit that is worth a month of service should cut it based on what I saw online. By offering a credit that is meaningful, they recognize that they seriously inconvenienced their customers. And customers will respond positively to to that.

By doing all of the above, Rogers stands a healthy chance of retaining the customers that they upset during this outage. But will they do it? At best I think the odds are 50/50 that they will. But I do know that if they don’t do anything, their odds of retaining customers who were affected by this outage plummet dramatically. How will you know for sure which way this goes? Watch next quarter’s results for increased churn in their cable, TV, and home phone operations relative to last quarter. If it goes up, then you know that the people who were ranting about Rogers on social media were deadly serious about leaving Canada’s largest telco.

One other thing. I want to circle back to the 83 year old client of mine who I had to set up my spare cell phone in hotspot mode (which ended up consuming 5GB of data and blowing by my data cap with Fido by 1GB by the way) so that she could get onto the Internet as having Internet access allows her to order medication and food. That’s important because she is housebound. She’s up and running as of yesterday and I will monitor her situation. But what was interesting was that Rogers reached out to me directly on Monday to see if they could assist. I told this to my wife and her response in a very cynical tone was “they likely want to get her story off the field because it looks really bad to take a old woman offline.” That statement further highlights the challenge that Rogers has which is how the public now perceives them. But the reality is that Rogers didn’t have to try and help her. Thus I applaud them for wanting to assist. The thing is, the fact that Rogers wanted to try and help this woman illustrates that they aren’t bad guys. I think that something happened here that went sideways on multiple levels. That along with some less than optimal decision making on the part of Rogers conspired to make them look worse than the really are. I believe that they can remedy that if they choose to do the right thing. The question I have is if they have the desire to do so.

I guess we’re about to find out.

UPDATE: Rogers is apparently still working towards bringing people back online. Details here.