The IT Nerd

For the last 28 hours, I have been testing the firmware update that Linksys has released for the WRT32X router that according to them solves the connectivity issues that were introduced a month or so ago with another firmware update. When the firmware was released to the public, I wanted to test the firmware over an extended period time because some of the issues manifested themselves over a 24 hour period. Other issues manifested themselves when Android devices were present. Thus I threw a couple of the Android phones I keep around for testing with car infotainment systems at it as well. The results were positive:

1. I noted that the WRT32X was faster than before as it gained just under 100 Mbps in raw speed getting up to a speed of 823 Mbps. That put it in the range of the ASUS and Netgear gaming routers which had beaten it rather handily when I tested each of those products.
2. All the stability and connectivity issues that I noted were no longer present.

As far as I am concerned, this firmware is a winner and if you have this router (or the Linksys WRT3200ACM which had similar issues), you need to install it NOW.

So now that this story which I have been covering from a few angles since it started last month seems to be over, I want to take a moment to do a post mortem of the events of the last month as there were positive and negative points in terms of how this was handled:

1. Seeing as a very crippling bug slipped out the door, I would hope that Linksys is reviewing their development and QA practices. Now I get that bugs get out the door from time to time. But this one was pretty bad and you have to wonder how it slipped through whatever processes they had in place. Thus a full review of how they do things from a development and QA perspective is in order. For bonus points, they should make the results public so that they can reassure their users that this was a one time event.
2. At the start of this, my support experience when this issue first popped up was a fail. And it was apparently not isolated as Linksys customers emailed and Tweeted me saying that they had experiences that were similar to mine. Now the good news on that front is that Linksys is looking how they deliver support to their customers, and they are introducing a level 3 support group that is based in the USA (as their level 1 and 2 support is overseas) for those situations where they have a baffling issue that needs advanced levels of support. That’s a good move. But one thing that I tell my clients is that when you support customers, you have to have your best equipped and best prepared people on the front lines because they will solve issues faster and generate higher levels of customer satisfaction. I would suggest to Linksys that they look at that to head off something like this in the future.
3. Also from the negative camp comes the fact that Linksys was slow to react to this. You would think that an influx of cases coming into their support channels, be it by phone, email, or social media would have set off alarm bells in the company. But it didn’t. From what I understand it was media coverage such as my stories on this issue that set off alarm bells inside Linksys. What I tell my clients who ask me how to head off something like this is that you have to have your ear to the ground at all times and be ready to jump into action instantly. After all, you should never find out about your bad news on CNN.
4. When Linksys did finally react to this, they did right by their customers by offering up replacement products such as Velop mesh routers. That gave their customers the ability to get back online and it gave them breathing space to try and fix this issue. That was good and I applaud them for that. But the flipside to that was that even though they stopped selling these routers directly, they were very easy to find on the streets and online. Thus they were in effect adding to their problems every time one of these routers were sold. I would say that Linksys needs look at how to stop that from happening in the future because if I were the person who bought a router and then have it go wonky, and then I find out that this was a known and widespread issue, I would not be happy and I would not buy Linksys product again.
5. Linksys going forward has to come up with the means to fix this stuff quickly. I say that because during the month that it took them to fix this, the following happened:
   1. Customers raged about this on social media, which is not the sort of press that I am sure Linksys wants.
   2. Customers who were fed up of waiting went to other router brands and are likely not to return to Linksys again. Not to mention that they won’t recommend the brand to their friends.
   3. Both Netgear and ASUS served up routers to me in hopes that I would review them, which I did. The reason being is that they sensed an opportunity to steal marketshare from Linksys. And from what they tell me, they succeeded on that front. And it’s a safe bet that if they did that with me, they did that with other people who review products. That’s not good if you’re Linksys.

Linksys should take a page out of the Apple playbook when it came to that vulnerability that allowed anyone to log into a Mac with root level access. It was fixed inside of 24 hours. Sure people said that that Apple dropped the ball when it came to letting that bug slip out the door. I know that I did. But Apple dropped everything and fixed it quickly, which meant that once people installed the fix, it became a non issue and nobody discussed it. Having this firmware issue sit out there for a month doesn’t do Linksys any favors. Thus they need to have the ability to react to these sorts of situations.

Is there anything that I am missing from this port mortem? If you think there is, or you have some wisdom to share, please leave a comment and share your thoughts.

UPDATE: Kieran Hannon who is the CMO of Belkin which owns Linksys reached out to me:

Understand and includes valid points. Unfortunately the root cause was a chipset issue which this thread might help shed a light: https://t.co/lOdv9FsDyC

— Kieran Hannon (@kieranhannon) January 13, 2018

The tweet that he referenced says this:

https://twitter.com/principian/status/952260343656366080

CB – it was a chipset issue. Took a while for Marvell to troubleshoot this particular issue caused by a recent Android software update. This particular fix was offered once a patch was received from Marvell. It underwent a rigorous testing but still beta.

— Kieran Hannon (@kieranhannon) January 13, 2018

What the customer (CB) is referring to is the fact that the firmware that Linksys has put out there for the WRT32X and WRT3200ACS is essentially still going through their QA process and is in effect a beta. I can see on one hand that one might be reluctant to install a firmware update that is still a beta. But consider this. The firmware works, Linksys is telling you up front that it’s a beta, and it’s a whole lot better than having a router with firmware on it that doesn’t work. Having said that, the above illustrates in a nutshell the challenge that Linksys has going forward.

This entry was posted on January 13, 2018 at 2:32 pm and is filed under Commentary with tags Linksys. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.