The IT Nerd

If you use anything from Snapchat to Reddit to Lloyds, you were likely affected by an outage over at Amazon Web Services or AWS. The fact is that two companies largely provide the computing resources that apps and websites need. The other one is Microsoft with their Azure platform. So when one of those companies has an outage, we are all going to have a bad day.

The cause of today’s outage according to Amazon was as follows:

We have identified a potential root cause for error rates for the DynamoDB APIs in the US-EAST-1 Region. Based on our investigation, the issue appears to be related to DNS resolution of the DynamoDB API endpoint in US-EAST-1. 

It’s always DNS at the end of the day when a network or an Amazon has an outage. As it stands, Amazon has largely gotten things back online. But the company is still dealing with issues as recently as a few minutes ago. So don’t be surprised if your favorite app or website might still have problems.

Aras Nazarovas, senior security researcher at Cybernews has this commentary: 

“Today’s outages for multiple services was the result internal DNS failures at Amazon Web Services in their US-EAST-1 region of AWS Cloud, similar failures have been common causes for major outages in the past, and usually stem from incorrect, updated configurations, or due to poor monitoring of expiration timelines for configurations, certificates, etc. 

From initial reporting there are no indications of any security breach, however failing to keep information or resources available for clients can be classified as a cyber incident, even if there was no malicious outsider or malicious intent. 

Similar outages occur almost every year, and they can be a reminder of how extensive software supply chains have become, showing how a simple issue on a handful of Amazon Data Centers caused thousands of issues to their clients.

Clients of affected services were impacted by failing to access their resources and data hosted by AWS for ~4hours impact of such a failure to ensure availability can vary greatly depending on the specific business and industry that used impacted AWS services, in worst case scenarios such an outage could have had serious consequences in critical infrastructure sectors.

In the event of such disruptions users should immediately seek alternative solutions for communication (different app, phone calls, SMS, radio) to be able to coordinate next steps towards recovering from such a disruption. It is a good practice to have a “Disaster Recovery Plan” where alternative communication channels and other critical steps have been planned in advance.”

I’ll be keeping an eye on this to see if there are any knock on effects. But it looks like things are trending towards normalcy…. Hopefully.

UPDATE: Sergiy Balynsky, VP of Engineering at cybersecurity company Spin.AI, provided the following comment:

“The AWS outage is a reminder that business continuity planning isn’t optional. Organizations should maintain independent backups and diversify across multiple cloud providers – so a disruption in one platform doesn’t bring operations to a halt.

Even the most reliable clouds can fail. A strong business continuity plan should include not only reliable backups, but also cross-platform and multi-cloud redundancy to minimize business disruption and maintain access to critical data when one provider experiences downtime.”

The Amazon AWS status page is listing a major outage. This was the first posting that the cloud provider posted at 9:37 AM today:

We are seeing impact to multiple AWS APIs in the US-EAST-1 Region. This issue is also affecting some of our monitoring and incident response tooling, which is delaying our ability to provide updates. We have identified the root cause and are actively working towards recovery. 

It’s actually worse than that. AWS powers a huge part of the Internet. So if you have a Ring smart home system, cameras, and Alexa speakers, you have a problem as they likely don’t work. On top of that, if you want to watch Hawkeye or some other show on Disney+, you will likely have issues doing that as well as Disney+ is hosted on AWS. In fact, games like PUBG, League of Legends, and Valorant don’t work either. Not to mention Tinder, Roku, Coinbase, both Cash App and Venmo are down too. And finally, Amazon is having issues delivering stuff as they themselves are powered by AWS. And the list goes on from there.

Expect this outage to last a very long time given that it has gone on for most of the day.

UPDATE: As of 50 minutes ago, this was the posting on the AWS Status Page:

[3:03 PM PST] Many services have already recovered, however we are working towards full recovery across services. Services like SSO, Connect, API Gateway, ECS/Fargate, and EventBridge are still experiencing impact. Engineers are actively working on resolving impact to these services.

UPDATE #2: The outage appears to be largely over as per this:

[4:35 PM PST] With the network device issues resolved, we are now working towards recovery of any impaired services. We will provide additional updates for impaired services within the appropriate entry in the Service Health Dashboard.