The IT Nerd

This CrowdStrike created nightmare via a bad antivirus update is a massive problem. How massive? Let me give you some perspective:

• Plus or minus a billion computers are basically bricked worldwide.
• These are mostly corporate ones as corporate computers are most likely to use the CrowdStrike AV software.
• Every affected computer needs to be rebooted in Safe Mode and have a driver manually removed. That should take 4 to 5 minutes a computer. I know that because I’ve done that about 50 times today.
• Smart companies take away the rights for common employees to do this.
• Even if they had the rights to this, imagine the average end user trying to handle a moderately complex task like this.

This is most non trivial event that could possibly exist. But there’s more. I sourced comments from a number of industry experts on this:

Evan Dornbush, former NSA cybersecurity expert:

   “This is of course  a phishing attack opportunity. Don’t make a bad situation worse. Only follow recommended instructions direct from your CrowdStrike rep. There will be a lot of misinformation about how to reconfigure your computers or which critical system files to delete.  Don’t fall victim to downloading phony solutions.

   “Similarly, this is a great time to reflect on password management, since the fix may eventually require administrative access to systems that have not rebooted in quite some time.”

Omdia Senior Director, Cybersecurity Maxine Holt

The global IT outage crisis is escalating, and organizations everywhere are in full scramble mode, desperately implementing workarounds to keep their businesses afloat. Microsoft has pointed fingers at a third-party software update, while CrowdStrike admits to a “defect found in a single content update for Windows hosts” and is working feverishly with affected customers. Omdia analysts connect the dots: this isn’t a cyberattack, but it’s unquestionably a cybersecurity disaster.

Cybersecurity’s role is to protect and ensure uninterrupted business operations. Today, on 19 July 2024, many organizations are failing to operate, proving that even non-malicious cybersecurity failures can bring businesses to their knees. The workaround, involving booting into safe mode, is a nightmare for cloud customers. Cloud-dependent businesses are facing severe disruptions.

Omdia’s Cloud and Data Center analysts have long warned about over-reliance on cloud services. Today’s outages will make enterprises rethink moving mission-critical applications off-premises. The ripple effect is massive, hitting CrowdStrike, Microsoft, AWS, Azure, Google, and beyond. CrowdStrike’s shares have plummeted by more than 20% in unofficial pre-market trading in the US, translating to a staggering $16 billion loss in value.

Looking forward, there’s a shift towards consolidating security tools into integrated platforms. However, as one CISO starkly put it, “Consolidating with fewer vendors means that any issue has a huge operational impact. Businesses must demand rigorous testing and transparency from their vendors.”

CrowdStrike’s testing procedures will undoubtedly be scrutinized in the aftermath. For now, the outages continue to rise, and the tech world watches as the fallout unfolds.

Steve Hahn, Executive VP, BullWall:

   “This event, more than any other, is precisely why companies need a defense in depth strategy. One issue on your endpoint security and not only can your infrastructure go down, but you can be left wide open for a myriad of attacks. Ransomware uses endpoints, and other attack vectors, as their launch mechanism for their attack and you need layers of security over your critical data and fileshares.

   “It will be interesting to see if we have a ripple of downstream consequences. Right now we are dealing with outages at airlines and other critical businesses but will we also see a wave of Ransomware attacks that follow? Time will tell.”

I wish every help desk globally well in dealing with this as this is going to be days if not a week or two of remediation. I also hope that CrowdStrike gets hauled in front of the relevant authorities globally to explain why this happened, and why corporate users should trust them again.

UPDATE: Madison Horn for Congress (OK-5) adds this comment:

With 15 years of experience in both the private and public sectors, I bring a deep insight into complex technological issues. If elected, I will be the most credentialed cybersecurity lawmaker in U.S. history. My leadership transcends partisan divides, focusing on practical solutions. By bridging the gap between technology and policy, I will address workforce development, AI regulation, and trust in government. My candidacy represents a path toward bipartisan cooperation to confront our nation’s complex challenges.” 

“Today, we face the largest IT blackout in history, caused not by a cyber attack or malicious actor, but by human error. This outage has impacted communities and 911 operators, and what we can assume at this time, caused billion dollar losses across the global economy – starkly highlighting the fragility of our interconnected world. 

While today’s events could not have been prevented with a single solution, any set of systems that have the potential to cause massive societal impact in the event of failure—such as the 9/11 communication outages for first responders—must have right-sized regulations that protect human life and ensure economic stability. 

Presently, the critical infrastructure and financial sectors have requirements that ensure the classification of systems that could be single points of failure, yet misclassification and outdated regulations persist. In many cases, existing regulations are not properly tailored to specific industries. This issue is compounded by the fact that governing bodies struggle to keep pace with rapid technological change — leading to a disconnect in understanding the underlying technology, its dependencies, capabilities, cost of implementation, and workforce limitations. 

This gap between our regulatory landscape and the demands of the rapid advancement of technology impacting society are widening. To address today’s critical challenges, we need leaders who have expertise in technology, enabling Congress to effectively collaborate with the private sector to drive solutions. The technology we use today, which fits in a device smaller than a deck of cards, has the potential to disrupt critical infrastructure like our electric grid. To safeguard our future, we need elected leaders who not only grasp the gravity of this technological reality but also have the expertise to address and mitigate these risks effectively. 

UPDATE #2:  Tom Marsland, VP of Technology, Cloud Range adds this comment:

Recovery is going to be painful, to put it lightly. The recovery steps outlined by CrowdStrike involve manually booting the affected PC into a recovery mode, deleting a file, and restarting.  This is not something that can be done remotely, and in many organizations, will require an administrator. This means someone from IT Support going computer to computer and doing this manually. This was most certainly preventable. This sort of release goes to the importance of change / configuration management.  This update should’ve been tested internally by CrowdStrike, then released to a small subset of users, then to their broader ecosystem. That is done specifically to catch problems with updates before they affect the entire ecosystem. Either that didn’t happen here at all, or that process failed to catch this bug, which is a  problem in and of itself.

This will take days, probably weeks for larger organizations. Unfortunately, as is the case in many cyber breaches as well, this is nothing new. Organizations failure to follow best practices with testing and deploying patches (both from a CrowdStrike side and from an organization receiving updates side) is the root cause of this.  When major patches roll out or become available, putting on auto-updates is one way to make sure your organization gets patched, but if there’s any concern about the operability or function of that update, organizations generally roll those out within their own businesses to a small set at first, and then to everyone else. The organizations affected today seem to be the ones that turned on automatic updates and that was it.

UPDATE #3: Tom Siu, CISO, Inversion6

This case with CrowdStrike Falcon and Microsoft Windows highlights one of the low-frequency and high-impact risks that don’t often rise to the top of your Risk Index. I call it the “auto-immune response risk” situation where your security tools and services misidentify normative files and services, and automated corrective actions lead to system outage. In the military, we called these “blue on blue” engagements.

The lesson that cybersecurity professionals need to know is that in the real world, errors can happen and propagate throughout our environments. This is why cybersecurity and IT teams need to have clear shared objectives and cogent leadership to first recover the IT systems, avoid lowering the security posture, and then plan/execute a path forward.  Uptime may be important, and CISOs will have to justify arguments for extended outages caused by security tooling. Often the toughest call in an incident response scenario is to take systems offline due to a vulnerability; here we have them offline already. The planning and execution we see going on currently is basically a disaster recovery scenario.

One risk mitigation for this scenario is to use mildly diverse portfolio of endpoint security solutions. For example, one product on your endpoints, and a different product on your infrastructure. I know vendors, and CISOs, often desire to unify these applications under “one pane of glass” (licensing simplicity is a big factor) but this type of low-frequency risk is going to a harder argument for a multiple solutions.  

This doesn’t sound like a patch that went awry, but more of a more complex systems interaction that hasn’t been fully evaluated; I suggest we make our judgements about the vendors by the quality and transparency of their communications and assistance. Additionally, we as security professionals need to incorporate public and internal communications for this type of event into our Incident Response Plans.

In conclusion, one question I’ve seen today is, “Is this an IT outage or a security incident?”

My answer is, “Yes.”

Earlier today I posted a story about security company CrowdStrike taking down the entire planet with a bad antivirus update. In that story I pointed out that the CEO of CrowdStrike George Kurtz posted a Tweet where he completely failed to apologize for bringing the entire planet to a standstill. At the time I said this:

The problem with this Tweet is that he completely failed to apologize for basically taking down the entire planet because of a screw up with his product. If I used his product, I’d be looking to move to some other antivirus product. Because this Tweet to be frank, sucks.

Well, I guess someone must have told him that the Tweet in question didn’t go over well because I just found this on the YouTube channel of NBC:

You tell me about the quality of his response. I’m really not impressed by this as this kind of looks like a hostage video. But to be fair, he’s likely been up all night and he’s likely reconsidering his life choices. At least he apologized for taking down the entire planet, but maybe you have a different view. If so, post a comment and share it.

This is not a good look for cybersecurity company Crowdstrike.

A bad update for the CrowdStrike Falcon antivirus product installed on Windows PCs has taken down millions of PC globally. I am hearing stories of flights being grounded, Sky News being taken off air for a few hours, and retailers being unable to sell anything.

In short, this is extraordinary bad.

This is made worse by the CEO of CrowStrike who decided to Tweet. This Tweet was not a good idea:

CrowdStrike is actively working with customers impacted by a defect found in a single content update for Windows hosts. Mac and Linux hosts are not impacted. This is not a security incident or cyberattack. The issue has been identified, isolated and a fix has been deployed. We…

— George Kurtz (@George_Kurtz) July 19, 2024

The problem with this Tweet is that he completely failed to apologize for basically taking down the entire planet because of a screw up with his product. If I used his product, I’d be looking to move to some other antivirus product. Because this Tweet to be frank, sucks.

There is a workaround that was posted to Reddit. But it will have to be done on a PC by PC basis. Which means that companies could be down for days. Which is of course bad.

I’ll be watching this situation and posting updates when it warrants an update. But this situation is bad and likely won’t improve for a while. And when this is resolved, CrowdStrike will have a whole lot of explaining to do.

At this year’s RSA Conference, CrowdStrike’s Joshua Shaprio said this:

Are adversaries still relying on malware? Joshua weighs in. pic.twitter.com/SofmFgkJki

— CrowdStrike (@CrowdStrike) April 26, 2023

In short, Crowstrike has been dealing with about one malwareless cyber issue a week during the last couple quarters reaffirming data reported earlier this year that 71% of cyberattacks were carried out without malware and highlighting the challenges cybersecurity teams face trying to combat such compromises.
Using a case study, the two illustrated the “layer A problem” involving the bad actor’s in-depth reconnaissance and use of dedicated machines to hide identities and avoid detection resulting in the threat actors set up with their own users on the network, free to exfiltrate data, compromise the cloud, and add themselves as a SQL server admin.

During their RSA keynote, both CrowdStrike CEO George Kurtz and President Michael Sentonas used a case study to illustrate the “layer A problem” involving the bad actor’s in-depth reconnaissance and use of dedicated machines to hide identities and avoid detection resulting in the threat actors set up with their own users on the network, free to exfiltrate data, compromise the cloud, and add themselves as a SQL server admin. More on that in a moment.

From an Akamai report on that attack:
 
    “The attack starts with a password brute-force on the MySQL service. Once successful, the attacker runs a sequence of queries in the database, gathering data on existing tables and users. By the end of execution, the victim’s data is gone – it’s archived in a zipped file which is sent to the attackers’ servers and then deleted from the database. A ransom note is left in a table named WARNING, demanding a ransom payment of up to 0.08 BTC.”

During their RSA keynote both Kurtz and Sentonas highlighted that without the standard malicious code to detect, companies need to consider strategies with robust telemetry gathering activities from the endpoint to the cloud, and to manage identity data with greater granularity, and, with the use of AI and machine learning, find anomalous activity among that data.

CrowdStrike CEO George Kurtz spoke about this to Bloomberg:

CrowdStrike CEO @George_Kurtz joined @Bloomberg live from the #RSAC floor to discuss today's cyber threat environment and the role of AI in cybersecurity.

Watch now. ⬇️ https://t.co/KEKlJw1jJx

— CrowdStrike (@CrowdStrike) April 26, 2023

Dave Ratner, CEO, HYAS:

   “Increasing an organization’s visibility into the real-time activities inside the network is quickly becoming critical for business resiliency against modern attacks. The ability to identify anomalous outbound communications from both the IT and OT networks can dramatically reduce the elapsed time from infection to detection and remediation and may be the only signal that allows organizations to get ahead of an attack before data exfiltration, encryption, and other actions that impact business continuity.”

Clearly the use of AI by those who defend against attacks is growing. Just look at Google and the announcement that they made at RSA. This is something that defenders need to consider in order to keep our digital assets safe.

I have some bad news if you’re responsible for defending your organization against threat actors.

Attackers have reduced the average time required to move laterally through systems by 14% last year, down to just 84 minutes, according to a new report out by Crowdstrike, giving defenders even less time to contain breaches after the initial breakout. 

Increasing the difficulty for defenders, a full 71% of the attacks used valid credentials for access, as opposed to malware, making detection by automated systems extremely difficult, up from 62% in 2021. Using “hands on keyboard” techniques make it harder for traditional anti-malware tools to detect activity according to CrowdStrike.

Like I said, this is bad news.

Ted Miracco, CEO of Approov Mobile Security:

   “It’s important to note that no single security measure can completely prevent all types of attacks, especially social engineering attacks. That said, mobile app attestation, runtime secrets protection, and RASP can all be highly effective measures in preventing credential access, SIM swapping, and MFA fatigue in mobile applications. 

   “Attestation techniques can help not only ensure that only genuine apps, and not tampered or cloned versions, are accessing APIs, it also uses the authorized application seamlessly as the second factor before accessing sensitive data. By verifying the integrity of the app at runtime, it both prevents attackers from injecting malicious code or accessing sensitive data. Runtime secrets protection can ensure that only valid app instances running in un-compromised environments can access the API keys and secrets stored in the cloud. This can prevent attackers from accessing these secrets even if they manage to gain control of the device. RASP can monitor, detect, and instantly block computer attacks, including new threats that were unforeseen during development. By continuously analyzing app behavior and detecting anomalies, RASP can prevent “interactive intrusions” and other types of attacks.

   “It’s best to use a combination of security measures, including those mentioned above, along with other security best practices such as proper authentication and authorization, encryption, and regular security testing and updates.”

The fact that attacks are getting faster and faster to execute means that we all have to work much harder to stop organizations from being victims. And the approach outlined above can certainly help with that… If everyone adopts that approach.