The IT Nerd

After pretty much bricking every Windows 10/11 computer that ran CrowdStrike Falcon, CrowdStrike put out a post that details the technical ins and outs of what led up to what happened on Friday. You can read it here. In it they seem very open. And the company has committed to providing additional details and a root cause analysis.

The thing is that what happened on Friday is a warning to the planet, and to the IT industry. CrowdStrike really screwed up here and disrupted the planet in the process. The mitigation for this is relatively easy to apply as I did that a whole bunch of times on Friday and Saturday. But because of the scale of this event, we’re talking about days before this problem is fully dealt with. In other words, this was bad. But it could have been worse. We need to learn from that and be prepared for the next event like this. Because there will be a next event. That starts with CrowdStrike being completely open to laying bare what happened and what they will do to ensure that it never happens again. And that’s followed up by other companies learning from this event and ensuring that they don’t become the next Crowdstrike.

UPDATE: After I posted this, I got this commentary from John Gunn, CEO, Token:

If anyone wants to know what a full-blown cyberattack from China or other enemy nations might look like, this event just gave us a small preview of the interruptions and havoc that could be inflicted. Every day we hear about new ransomware attacks, but these are revealed because of the immediate financial payoff the attackers seek. There are undoubtedly countless significant network intrusions throughout our infrastructure and essential services that are undetected which are like sleeper cells waiting to be activated if we enter a major conflict with these nations.

This CrowdStrike created nightmare via a bad antivirus update is a massive problem. How massive? Let me give you some perspective:

• Plus or minus a billion computers are basically bricked worldwide.
• These are mostly corporate ones as corporate computers are most likely to use the CrowdStrike AV software.
• Every affected computer needs to be rebooted in Safe Mode and have a driver manually removed. That should take 4 to 5 minutes a computer. I know that because I’ve done that about 50 times today.
• Smart companies take away the rights for common employees to do this.
• Even if they had the rights to this, imagine the average end user trying to handle a moderately complex task like this.

This is most non trivial event that could possibly exist. But there’s more. I sourced comments from a number of industry experts on this:

Evan Dornbush, former NSA cybersecurity expert:

   “This is of course  a phishing attack opportunity. Don’t make a bad situation worse. Only follow recommended instructions direct from your CrowdStrike rep. There will be a lot of misinformation about how to reconfigure your computers or which critical system files to delete.  Don’t fall victim to downloading phony solutions.

   “Similarly, this is a great time to reflect on password management, since the fix may eventually require administrative access to systems that have not rebooted in quite some time.”

Omdia Senior Director, Cybersecurity Maxine Holt

The global IT outage crisis is escalating, and organizations everywhere are in full scramble mode, desperately implementing workarounds to keep their businesses afloat. Microsoft has pointed fingers at a third-party software update, while CrowdStrike admits to a “defect found in a single content update for Windows hosts” and is working feverishly with affected customers. Omdia analysts connect the dots: this isn’t a cyberattack, but it’s unquestionably a cybersecurity disaster.

Cybersecurity’s role is to protect and ensure uninterrupted business operations. Today, on 19 July 2024, many organizations are failing to operate, proving that even non-malicious cybersecurity failures can bring businesses to their knees. The workaround, involving booting into safe mode, is a nightmare for cloud customers. Cloud-dependent businesses are facing severe disruptions.

Omdia’s Cloud and Data Center analysts have long warned about over-reliance on cloud services. Today’s outages will make enterprises rethink moving mission-critical applications off-premises. The ripple effect is massive, hitting CrowdStrike, Microsoft, AWS, Azure, Google, and beyond. CrowdStrike’s shares have plummeted by more than 20% in unofficial pre-market trading in the US, translating to a staggering $16 billion loss in value.

Looking forward, there’s a shift towards consolidating security tools into integrated platforms. However, as one CISO starkly put it, “Consolidating with fewer vendors means that any issue has a huge operational impact. Businesses must demand rigorous testing and transparency from their vendors.”

CrowdStrike’s testing procedures will undoubtedly be scrutinized in the aftermath. For now, the outages continue to rise, and the tech world watches as the fallout unfolds.

Steve Hahn, Executive VP, BullWall:

   “This event, more than any other, is precisely why companies need a defense in depth strategy. One issue on your endpoint security and not only can your infrastructure go down, but you can be left wide open for a myriad of attacks. Ransomware uses endpoints, and other attack vectors, as their launch mechanism for their attack and you need layers of security over your critical data and fileshares.

   “It will be interesting to see if we have a ripple of downstream consequences. Right now we are dealing with outages at airlines and other critical businesses but will we also see a wave of Ransomware attacks that follow? Time will tell.”

I wish every help desk globally well in dealing with this as this is going to be days if not a week or two of remediation. I also hope that CrowdStrike gets hauled in front of the relevant authorities globally to explain why this happened, and why corporate users should trust them again.

UPDATE: Madison Horn for Congress (OK-5) adds this comment:

With 15 years of experience in both the private and public sectors, I bring a deep insight into complex technological issues. If elected, I will be the most credentialed cybersecurity lawmaker in U.S. history. My leadership transcends partisan divides, focusing on practical solutions. By bridging the gap between technology and policy, I will address workforce development, AI regulation, and trust in government. My candidacy represents a path toward bipartisan cooperation to confront our nation’s complex challenges.” 

“Today, we face the largest IT blackout in history, caused not by a cyber attack or malicious actor, but by human error. This outage has impacted communities and 911 operators, and what we can assume at this time, caused billion dollar losses across the global economy – starkly highlighting the fragility of our interconnected world. 

While today’s events could not have been prevented with a single solution, any set of systems that have the potential to cause massive societal impact in the event of failure—such as the 9/11 communication outages for first responders—must have right-sized regulations that protect human life and ensure economic stability. 

Presently, the critical infrastructure and financial sectors have requirements that ensure the classification of systems that could be single points of failure, yet misclassification and outdated regulations persist. In many cases, existing regulations are not properly tailored to specific industries. This issue is compounded by the fact that governing bodies struggle to keep pace with rapid technological change — leading to a disconnect in understanding the underlying technology, its dependencies, capabilities, cost of implementation, and workforce limitations. 

This gap between our regulatory landscape and the demands of the rapid advancement of technology impacting society are widening. To address today’s critical challenges, we need leaders who have expertise in technology, enabling Congress to effectively collaborate with the private sector to drive solutions. The technology we use today, which fits in a device smaller than a deck of cards, has the potential to disrupt critical infrastructure like our electric grid. To safeguard our future, we need elected leaders who not only grasp the gravity of this technological reality but also have the expertise to address and mitigate these risks effectively. 

UPDATE #2:  Tom Marsland, VP of Technology, Cloud Range adds this comment:

Recovery is going to be painful, to put it lightly. The recovery steps outlined by CrowdStrike involve manually booting the affected PC into a recovery mode, deleting a file, and restarting.  This is not something that can be done remotely, and in many organizations, will require an administrator. This means someone from IT Support going computer to computer and doing this manually. This was most certainly preventable. This sort of release goes to the importance of change / configuration management.  This update should’ve been tested internally by CrowdStrike, then released to a small subset of users, then to their broader ecosystem. That is done specifically to catch problems with updates before they affect the entire ecosystem. Either that didn’t happen here at all, or that process failed to catch this bug, which is a  problem in and of itself.

This will take days, probably weeks for larger organizations. Unfortunately, as is the case in many cyber breaches as well, this is nothing new. Organizations failure to follow best practices with testing and deploying patches (both from a CrowdStrike side and from an organization receiving updates side) is the root cause of this.  When major patches roll out or become available, putting on auto-updates is one way to make sure your organization gets patched, but if there’s any concern about the operability or function of that update, organizations generally roll those out within their own businesses to a small set at first, and then to everyone else. The organizations affected today seem to be the ones that turned on automatic updates and that was it.

UPDATE #3: Tom Siu, CISO, Inversion6

This case with CrowdStrike Falcon and Microsoft Windows highlights one of the low-frequency and high-impact risks that don’t often rise to the top of your Risk Index. I call it the “auto-immune response risk” situation where your security tools and services misidentify normative files and services, and automated corrective actions lead to system outage. In the military, we called these “blue on blue” engagements.

The lesson that cybersecurity professionals need to know is that in the real world, errors can happen and propagate throughout our environments. This is why cybersecurity and IT teams need to have clear shared objectives and cogent leadership to first recover the IT systems, avoid lowering the security posture, and then plan/execute a path forward.  Uptime may be important, and CISOs will have to justify arguments for extended outages caused by security tooling. Often the toughest call in an incident response scenario is to take systems offline due to a vulnerability; here we have them offline already. The planning and execution we see going on currently is basically a disaster recovery scenario.

One risk mitigation for this scenario is to use mildly diverse portfolio of endpoint security solutions. For example, one product on your endpoints, and a different product on your infrastructure. I know vendors, and CISOs, often desire to unify these applications under “one pane of glass” (licensing simplicity is a big factor) but this type of low-frequency risk is going to a harder argument for a multiple solutions.  

This doesn’t sound like a patch that went awry, but more of a more complex systems interaction that hasn’t been fully evaluated; I suggest we make our judgements about the vendors by the quality and transparency of their communications and assistance. Additionally, we as security professionals need to incorporate public and internal communications for this type of event into our Incident Response Plans.

In conclusion, one question I’ve seen today is, “Is this an IT outage or a security incident?”

My answer is, “Yes.”