The IT Nerd

If you use anything from Snapchat to Reddit to Lloyds, you were likely affected by an outage over at Amazon Web Services or AWS. The fact is that two companies largely provide the computing resources that apps and websites need. The other one is Microsoft with their Azure platform. So when one of those companies has an outage, we are all going to have a bad day.

The cause of today’s outage according to Amazon was as follows:

We have identified a potential root cause for error rates for the DynamoDB APIs in the US-EAST-1 Region. Based on our investigation, the issue appears to be related to DNS resolution of the DynamoDB API endpoint in US-EAST-1. 

It’s always DNS at the end of the day when a network or an Amazon has an outage. As it stands, Amazon has largely gotten things back online. But the company is still dealing with issues as recently as a few minutes ago. So don’t be surprised if your favorite app or website might still have problems.

Aras Nazarovas, senior security researcher at Cybernews has this commentary: 

“Today’s outages for multiple services was the result internal DNS failures at Amazon Web Services in their US-EAST-1 region of AWS Cloud, similar failures have been common causes for major outages in the past, and usually stem from incorrect, updated configurations, or due to poor monitoring of expiration timelines for configurations, certificates, etc. 

From initial reporting there are no indications of any security breach, however failing to keep information or resources available for clients can be classified as a cyber incident, even if there was no malicious outsider or malicious intent. 

Similar outages occur almost every year, and they can be a reminder of how extensive software supply chains have become, showing how a simple issue on a handful of Amazon Data Centers caused thousands of issues to their clients.

Clients of affected services were impacted by failing to access their resources and data hosted by AWS for ~4hours impact of such a failure to ensure availability can vary greatly depending on the specific business and industry that used impacted AWS services, in worst case scenarios such an outage could have had serious consequences in critical infrastructure sectors.

In the event of such disruptions users should immediately seek alternative solutions for communication (different app, phone calls, SMS, radio) to be able to coordinate next steps towards recovering from such a disruption. It is a good practice to have a “Disaster Recovery Plan” where alternative communication channels and other critical steps have been planned in advance.”

I’ll be keeping an eye on this to see if there are any knock on effects. But it looks like things are trending towards normalcy…. Hopefully.

UPDATE: Sergiy Balynsky, VP of Engineering at cybersecurity company Spin.AI, provided the following comment:

“The AWS outage is a reminder that business continuity planning isn’t optional. Organizations should maintain independent backups and diversify across multiple cloud providers – so a disruption in one platform doesn’t bring operations to a halt.

Even the most reliable clouds can fail. A strong business continuity plan should include not only reliable backups, but also cross-platform and multi-cloud redundancy to minimize business disruption and maintain access to critical data when one provider experiences downtime.”