The IT Nerd

A reader pointed me towards a Bleeping Computer article which then led me to this advisory from ASUS which basically says this. There’s a list of ASUS routers that ASUS is urging customers to upgrade to the latest firmware ASAP because of a security threat that seems to be out there. The company is recommending the following:

We strongly encourage you to periodically audit both your equipment and your security procedures, as this will ensure that you will be better protected. As a user of an ASUS router, we advise taking the following actions:

1. Update your router to the latest firmware. We strongly recommend that you do so as soon as new firmware is released. You will find the latest firmware available for download from the ASUS support page at https://www.asus.com/support/or the appropriate product page at https://www.asus.com/Networking/. ASUS has provided a link to new firmware for selected routers at the end of this notice.
2. Set up separate passwords for your wireless network and router-administration page. Use passwords with a length of at least eight characters, including a mix of capital letters, numbers and symbols. Do not use the same password for multiple devices or services.
3. Enable ASUS AiProtection, if your router supports this feature. Instructions on how to do this can be found in your router’s manual, or on the relevant ASUS support page, at https://www.asus.com/Networking/.

Please note, if you choose not to install this new firmware version, we strongly recommend disabling services accessible from the WAN side to avoid potential unwanted intrusions. These services include remote access from WAN, port forwarding, DDNS, VPN server, DMZ, port trigger.

The advisory also lists the following routers as being affected by whatever threat is out there:

• GT6
• GT-AXE16000
• GT-AX11000 PRO
• GT-AX6000
• GT-AX11000
• GS-AX5400
• GS-AX3000
• XT9
• XT8
• XT8 Version 2
• RT-AX86U PRO
• RT-AX86U
• RT-AX86S
• RT-AX82U
• RT-AX58U
• RT-AX3000
• TUF-AX6000
• TUF-AX5400

Now what strikes me as interesting is that a cursory look at the affected routers shows that a fair number of them have had firmware updates in the last few days. While others have not. Take the XT8 that I own for example. The last firmware update was in May which I wrote about my experiences with that. Others like the RT-AX86U PRO has have firmware updates in the last day or two. That might imply that all of these routers are about to get firmware updates shortly. But I am not 100% sure about that. It might be worth your while to keep an eye on the firmware page for your ASUS router to see if there’s an update for your router. As for if you should update to this new firmware, I would do so as there’s a clear benefit to doing so which is is security. But I would do it with caution and be prepared to roll back to an earlier firmware version as ASUS firmware can either be fine or buggy and there seems to be no middle ground in terms of that.

So I have been spending the last week working on trying to understand why the latest ASUS firmware for the ZenWifi XT8 is such a dumpster fire of a firmware. When I initially updated to it, it was so bad that I put out a PSA telling people not to upgrade to it. And to frank, even though I have things working now, that PSA will still stand because of a couple of reasons.

The first reason is that this firmware, which is version 3.0.0.4.388.23285, seems not to play nice with the advanced DMZ functionality of the Bell HH4000. Specifically the ASUS ZenWifi XT8 would get an external IP address from the HH4000, but the HH4000 would give the ZenWifi XT8 a self assigned address which looks like 169.xxx.xxx.xxx. If you see an address like this, it means that whatever device has this address will not be able to communicate with other devices on your local network or to the Internet properly, if at all. However on the previous version of the firmware which is 3.0.0.4.388.23012, I could not reproduce this issue. How I ultimately fixed this with the new firmware installed was to completely undo the advanced DMZ setup. Then reboot the HH4000. Followed by removing all the entries from the DHCP table in the HH4000. Rebooting again. Then redoing the advanced DMZ setup. Since no other ASUS firmware has required me to go through these gymnastics to make everything work after the firmware update, logic suggests that this is clearly a bug in the 3.0.0.4.388.23285 version of the ASUS firmware. Though to be fair to ASUS, this is likely a use case that they don’t test.

The second reason is that DNS resolution for reasons that I do not understand seems to be problematic. Before I get into why I say that, let me give you a quick lesson on DNS. When you go to a website, say http://www.itnerd.blog, your computer or smart phone has to ask a server called a Domain Name Server for the address of that website. And that server will respond with the exact address. Now that process should be milliseconds to do. But I was finding that it was taking 5 to 10 seconds for that to happen. Now I never use the DNS servers for any ISP that I do business with as ISPs have been known to track your activities and sell that information to third parties. Instead I normally use the Canadian Shield DNS service. But with this firmware, I could not do that. Instead, I had to use Quad 9 which is one of the list of pre-defined DNS services that are baked into ASUS routers. While I do get the same result which is to keep my browsing history private from my ISP, I question why I am no longer able to use Canadian Shield on my ZenWiFi XT8. I say that because as part of my troubleshooting I went back to the previous firmware which is 3.0.0.4.388.23012 and I was not able to reproduce this issue. That again implies that this firmware has an issue of some sort of issue with entering your own DNS settings.

One other thing that I should mention is that as another troubleshooting step, I factory reset the router and set it up from scratch rather than restoring the configuration from a backup. I did that because I thought that the backup might have been the issue. But as you can see above, that wasn’t the case.

The bottom line is this: While I do have things working at present with version 3.0.0.4.388.23285, I cannot recommend that any ASUS ZenWiFi XT8 user upgrade to this version. And this experience underlines the fact that ASUS really has issues with their firmware and their upgrade process that they really need to get sorted sooner rather than later. On top of issues like the ones above that managed to get out of their QA lab into the public domain, the fact I have to recommend that ASUS users reset their router config and redo it either via a backup file or from scratch when said user does a firmware upgrade makes zero sense. No other consumer or prosumer router works that way. And that reflects poorly on ASUS. This is why I have as of late been recommending Netgear and Linksys routers. With both of those brands, I am able to install them and forget about them so to speak. And ASUS needs to be in that camp or they will lose customers due to the fact that ease of setup and ease of use are thing that really matter to customers.

It seems that firmware version 3.0.0.4.388.23285 for the ASUS ZenWiFi XT8 is a must pass. I say that because when I tried to update to it the following happened:

• My Wife’s work supplied SIP phone stopped working.
• HomeKit broke in spectacular fashion
• VPNs would not work
• Surfing the web was difficult do to the slow speed.
• It seem to have issue with Bell and the Advanced DMZ method of bypassing their hardware.

Clearly there’s an issue with this firmware as rolling back to 3.0.0.4.388.23012 allowed me to get things back online. Though I will admit that I had to do two factory resets to make that happen. This reminds me of the last time that I had to put out a PSA about ASUS firmware which was this instance a few months ago.

Until ASUS formally addresses this, your best bet is to stay on firmware 3.0.0.4.388.23012. If you have already upgraded to 3.0.0.4.388.23285, then your best bet is to do the following: 

• Backup the configuration using these instructions. 
• Download and install firmware 3.0.0.4.388.23012 using method 2 from these instructions. 
• After updating do a factory reset of the router using these instructions. 
• Using a computer and a web browser, connect to the router and using the advanced options, upload the backup of the configuration that you saved in the first step. 

The reason why I recommend going this route is that for whatever reason, ASUS routers do not cleanly update the firmware. And that leads to all sorts of weird issues that are hard to track down.

I truly wish ASUS could consistently come out with firmware updates that improve the user experience rather than frustrate users. But until that day comes, if it ever does, your best bet might be to pick a firmware that you know to be stable, and stick to it until there is clear evidence that any newer firmware won’t break your setup.

Earlier this week there were reports of ASUS routers being knocked offline. The reports started to come in on May 16, 2023 and if you were affected by this (I own an ASUS router and I wasn’t affected, though I should have been. More on that shortly), your issues might have lasted a couple of days. Unless you factory reset your router to get back online.

Now the company was pretty silent about what was going on. In fact people complained about the lack of communication from ASUS. But the mystery was solved via this story from ARS Technica which a reader pointed me to:

Two days later, the Taiwan-based hardware maker has finally answered the calls for help. The mass outage, the company said, was the result of “an error in the configuration of our server settings file.” After fixing the glitch, most users needed to only reboot their devices. In the event that didn’t fix the problem, the company’s support team advised users to save their current configuration settings and perform a factory reset. The company also apologized.

Here’s the root cause in detail:

Asus still hasn’t provided details about the configuration error. Various users have offered explanations online that appear to be correct.

“On the 16th, Asus pushed a corrupted definition file for ASD, a built-in security daemon present in a wide range of their routers,” one person wrote. “As routers automatically updated and fetched the corrupted definition file, they started running out of filesystem space and memory and crashing.”

The explanation answered the question of what was causing routers to crash, but it raised a new one: Why were routers affected even when they had been configured to not automatically update and no manual update had been performed? Asus has yet to address this, but the likely answer is that the definitions file for ASD, which resides in memory and scans devices for security threats, gets updated whether or not automatic updates are enabled.

I might be able to answer some of this.

ASD is the AIProtection functionality that is built into many ASUS routers which is made by anti-virus vendor Trend Micro. What this does is block access to questionable websites, protecting users from spyware, malware, and other unwanted applications while preventing potential distributed denial of service (DDOS) attacks and other security incidents. It relies on definition files to update itself. And the downloading of those definition files are completely independent of the downloading of firmware updates for the routers. That’s why users got this update even if firmware updates were turned off. In fact doing some experiments on my ZenWiFi XT8 indicate that the only way to turn off the updates for AIProtection is to turn off AIProtection. The other thing that AIProtection does is send your browsing history to Trend Micro. Presumably to help to improve AIProtection. But I can see that some people would be bothered by this as it makes you the product seeing as AIProtection is free.

Now this incident highlights the risks of having this sort of functionality built into your router. And if you’re someone who is concerned about this and want another option, I’ll give you two. There’s CIRA Canadian Shield which is a DNS service that offers a lot of this sort of functionality. Along with that is HYAS Protect At Home which is the same sort of product, but it’s a lot more advanced as it is based on their corporate security tech.

A final word about this. ASUS has a bit of a history of finding themselves in bad situations, and screwing the attempts to diffuse the situation in question. This incident is an example of that. It should not have taken ASUS two days to say anything about this issue. And their apology is pretty lame. ASUS really need to learn how to do a better job of managing a crisis situation that affected a large number of their customers. Otherwise, they will not have any customers.

Over the last few months, I’ve been telling out to either avoid or be cautious about firmware updates for the ASUS ZenWiFi XT8. And in the latter case, I said this:

ASUS really needs to get a firmware release out that stabilizes things for the vast majority of their users. And unfortunately, this specific firmware doesn’t seem to be it. Based on what I am reading in the Reddit threads that I linked to above, some people are getting fed up with being treated as “beta testers”. That in the long term will affect the probability that these users will buy another ASUS product in a negative way. Thus ASUS would be well advised to get on getting a firmware out that is stable for all.

Well we might, key word MIGHT have that firmware. Last week ASUS rolled out version 3.0.0.4.388.22525 of their firmware and it from all reports has been stable for most. Specifically, the connection between the nodes which has been a source of grief for many. I’ve been testing this for the last few days and have found zero issues with it myself. But I should note that I found zero issues with the last firmware that ASUS put out before Christmas, while many other had issues. But what gives me hope that this is stable is that looking at places like SNB Forums, the majority of users seem to be having a good experience with this firmware.

My firmware upgrade process for ASUS routers is as follows: 

• Log into the router using a computer and a web browser
• Backup the configuration using these instructions. 
• Update the firmware.
• After updating I do a factory reset of the router using these instructions. 
• Using a computer and a web browser, connect to the router and using the advanced options in the setup wizard, upload the backup of the configuration that I saved in the first step.

I do this because I have found that simply upgrading to the latest ASUS firmware can create problems. And doing this while taking up to 30 – 40 minutes to perform results in zero issues.

I would be very interested to hear the experience of other XT8 owners with this firmware. Is it better? Is it worse? Please leave a comment and share your thoughts.

I promised a follow up to the odd decision by ASUS to release a new firmware for their ZenWiFi XT8 mesh router just before Christmas. My experience has been positive so far. I have had zero issues with it. I had a brief look at my logs as since I factory reset my XT8 nodes after upgrading to this firmware. I didn’t see anything out of the ordinary. I also note that memory usage on both nodes was around 72% of the 512MB of RAM. Previous firmwares had the RAM usage about 10% higher. That implies that the mesh router is operating more efficiently.

But to make sure that I wasn’t simply missing out on anything, I’ve been following a pair of threads on Reddit along with a thread on SNBForums. And in those places, the results are more mixed. The most common issue that has been reported is that there have been issues with the nodes disconnecting. Once that happens, it may be a challenge to get them to reconnect. Or they may never reconnect at all. Meanwhile others report no issues.

Based on that, this is clearly a your mileage may vary situation. If you are on a ASUS firmware for the XT8 that is stable, you may be better off staying on that version until whatever stability issues that I am not seeing are sorted out. However, if you do decide to upgrade, you should note the version of firmware that you are currently running so that you can downgrade back to it if problems present themselves. Specifically, losing connection to one of the nodes.

ASUS really needs to get a firmware release out that stabilizes things for the vast majority of their users. And unfortunately, this specific firmware doesn’t seem to be it. Based on what I am reading in the Reddit threads that I linked to above, some people are getting fed up with being treated as “beta testers”. That in the long term will affect the probability that these users will buy another ASUS product in a negative way. Thus ASUS would be well advised to get on getting a firmware out that is stable for all.

I got pinged by a reader while I was dealing with a client this morning. I couldn’t look into it then, but when I got home I did. When I logged into my ZenWifi XT8 mesh WiFi system I say that an update was available. I clicked on it and saw this:

I also checked the ASUS website and didn’t see anything on this firmware. That was odd because ASUS usually updates the website with new firmware releases pretty quickly. Unless this slipped out by accident which I have seen a couple of times. In that case, ASUS usually releases a newer firmware to replace it. The other theory that I have is that ASUS felt that they had to roll out this firmware four days before Christmas because this firmware fix couldn’t wait until after the holidays. I point that out because companies don’t typically roll out stuff like this a week or two before Christmas due to not having staff on hand to answer phone calls or fix something if it goes off the rails. Making this an unusual release.

Since I have told you not to upgrade in the past due to serious issues with their firmware, as well as telling you when it was safe to upgrade, I decided to install it and report back to you my early impressions. And I also will follow up with you in a week’s time with longer term impressions. My upgrade process for ASUS routers is as follows:

• Log into the router using a computer and a web browser
• Backup the configuration using these instructions. 
• Update the firmware.
• After updating I do a factory reset of the router using these instructions. 
• Using a computer and a web browser, connect to the router and using the advanced options in the setup wizard, upload the backup of the configuration that I saved in the first step.

I do this because I have found that simply upgrading to the latest ASUS firmware can create problems. For example, one firmware upgrade broke HomeKit for almost a day until I figured out that doing factory reset it followed by setting it up from scratch was the fastest and best way to resolve the issue. Since then, this has been my upgrade process and it has never failed me.

After walking through those steps I was back online and I started doing some testing. Right away I noticed that TimeMachine backups were way faster from all the Macs in the home. I also noted that when I had to do a VPN connection to fix something for a client, that was way faster as well. The reason why I put the words “way faster” in bold is because it was truly much faster than what I had been used to. I tried to run Speed Tests from my iPhone 14 Pro and didn’t find a difference in terms of WiFi speed. So my best guess is that besides what little is in the release notes, ASUS must have done something to make device to router connections faster. Other than that, I have noted no stability issues like I have seen previously. Nor have I noted any other improvements.

I’ll be running this for about a week and I will report back as it takes about that long before any serious issues become apparent. Also, if release notes do appear, I’ll be sure to link them here.

UPDATE 12/22/2022: ASUS just posted release notes for this firmware update on their website. It has more details than the screenshot above:

So these release notes explained why I am seeing the better VPN performance. But it doesn’t explain the improved WiFi performance that I am seeing. In terms of my observations, it’s the same as what I reported above. It seem stable and reliable thus far.

UPDATE #2: I have an update on this firmware here.

You might remember that a few weeks ago I put out a public service announcement for owners of The ASUS ZenWiFi XT8 to not install firmware 3.0.0.4.388.21099 as it was incredibly unstable. In fact, it was so unstable that I recommended rolling back to the previous firmware and provided instructions on how to do so. Earlier this week, ASUS released an updated firmware which is listed as version 3.0.0.4.388.21617 that claims to optimize memory usage and improve stability among other things. You can read the release notes here. Since I suggested to people that they shouldn’t try the previous firmware, I decided to try this firmware. Now first, let me describe how to update the firmware as for whatever reason, if you don’t follow this method you will have issues. More on the issues that you will have in a minute. But here’s my process:

• Log into the router using a computer and a web browser
• Backup the configuration using these instructions. 
• Update the firmware using method 1 from these instructions.
• After updating do a factory reset of the router using these instructions. 
• Using a computer and a web browser, connect to the router and using the advanced options in the setup wizard, upload the backup of the configuration that you saved in the first step.

The reason why I recommend going this route is that for whatever reason, ASUS routers do not cleanly update the firmware. And that leads to all sorts of weird issues that are hard to track down. In my case, it breaks HomeKit unless I go through the steps above. Going this route avoids all of that.

Now I have been testing this firmware for a few days, and so far it has been stable for me. And it has been stable for others based on reports from places like SNB Forums. Thus I am cautiously optimistic that you should be okay to update your ZenWiFi XT8 to this firmware. Though I would wait another week or maybe two to see if any issues appear by monitoring the thread in SNB Forums just to be safe.

If you have any feedback about this firmware, please share them in the comments below. And if I notice anything that is a deal breaker, I will will post an update.

I’ve only done this once before with Linksys routers, but I am being forced to do this again as ASUS has put out a firmware version for their ZenWiFi XT8 router that will cause you problems. Specifically the firmware version is 3.0.0.4.388.21099 which was released on October 3rd. This firmware appears to have some sort memory leak issue. Meaning that it consumes all available RAM memory on the router until it runs out and crashes. When this happens, you will see the following:

• The router will work fine for four or five days.
• The child node will suddenly disconnect and you will see a blue flashing light.
• The primary node will look fine with a white light, but there is no WiFi available.

A reboot will bring everything back online. But only for four or five days where the above will repeat. This has been reported in a couple of places like Reddit and SNBForums. What’s interesting is that UKTechHub has posted that ASUS has released a new firmware to a user on that forum that seems to address this issue. That implies that ASUS knows that this issue exists. Why ASUS hasn’t widely released a newer firmware that addresses this problem that they seem to know about remains a bit of an open question.

Until ASUS formally addresses this, your best bet is to stay on firmware 3.0.0.4.386.49873. If you have already upgraded to 3.0.0.4.388.21099, then your best bet is to do the following:

• Backup the configuration using these instructions.
• Download and install firmware 3.0.0.4.388.21099 using method 2 from these instructions.
• After updating do a factory reset of the router using these instructions.
• Using a computer and a web browser, connect to the router and using the advanced options, upload the backup of the configuration that you saved in the first step.

The reason why I recommend going this route is that for whatever reason, ASUS routers do not cleanly update the firmware. And that leads to all sorts of weird issues that are hard to track down. In my case, it breaks HomeKit unless I go through the steps above.

Now you could just stay on 3.0.0.4.388.21099 until ASUS decides to fix this. But you’ll have to reboot your router every four or five days which is a pain. Thus I would hope that ASUS decides to step up to the plate and addresses this with a firmware fix. But I am not holding my breath as based on my recent experience with them and how they support their customers, ASUS doesn’t seem to be that sort of company. Though they are free to prove me wrong.

UPDATE: ASUS has released a newer firmware to address these issues. More info here.