The IT Nerd

Here’s a new twist in the Delta Airlines gets taken down by the CrowdStrike snafu. Passengers who were affected by Delta getting taken down by the CrowdStrike snafu are now suing Delta. The lawsuit, filed by Sauder Schelkopf and Webb, Klase & Lemond on behalf of Delta passengers whose flights were canceled, alleges the following:

“[b]y the end of the weekend, nearly every airline had managed to recover and resume normal operations. Delta, however, did not resume normal operations. By the start of the workweek, Delta continued to cancel a staggering number of flights. On Monday, July 22, it was reported that Delta canceled more than 1,250 flights. These cancellations accounted for nearly 70% of all flights within, to, or from the United States that had been canceled on Monday. No other US airline had canceled one-tenth as many flights.”  It is further alleged that Delta failed to give some affected passengers automatic refunds for canceled flights and often times conditioned its offer of partial reimbursements to passengers on a waiver releasing Delta of all legal claims passengers have against Delta.

And:

“While nearly every other airline recovered quickly from the July 19th ‘Tech Outage,’ Delta’s passengers remained stranded, waiting in lines for days trying to get to their destinations. When our clients sought refunds, Delta again failed to deliver. We look forward to litigating the case on their behalf,” said Joe Sauder of Sauder Schelkopf, an attorney for the passengers.

You can bet that Delta saw this coming. Which is likely why they are going after CrowdStrike for compensation. It will be interesting to see two things. One, if this gets certified as a class action lawsuit. And two, If Delta somehow drags CrowdStrike into this on top of their other pending legal action against them.

It’s taken far longer than I anticipated, but CrowdStrike has finally responded to news that Delta Airlines has retained legal counsel to get compensation from them when it comes to their faulty software patch taking down Delta and a whole lot of other people:

CrowdStrike reiterated its apology to Delta in a letter responding to public comments about the airline pursuing legal claims, but said it “strongly rejects any allegation that it was grossly negligent or committed willful misconduct.” CrowdStrike says the litigation threat “has contributed to a misleading narrative that CrowdStrike is responsible for Delta’s IT decisions and response to the outage,” noting that competing airlines restored their operations much more swiftly.

“CrowdStrike’s CEO personally reached out to Delta’s CEO to offer onsite assistance, but received no response,” CrowdStrike lawyer Michael Carlinsky said in the letter. Carlinsky said CrowdStrike had made several other attempts to provide assistance, including an offer for onsite support, but was told that resources for the latter were not required.

I’m going to go out on a limb and say that CrowdStrike didn’t get a response because Delta was too busy trying to get their systems back online because of CrowdStrike’s screw up. And by the time they did respond, Delta was so mad at CrowdStrike that Delta flipped them off. If there’s an alternate view to this that I should be aware of, leave that view in the comments below.

Anyway…..

“Should Delta pursue this path, Delta will have to explain to the public, its shareholders, and ultimately a jury why CrowdStrike took responsibility for its actions — swiftly, transparently, and constructively — while Delta did not,” said Carlinsky. The letter also notes that CrowdStrike’s contractual liability is capped “in the single-digit millions,” and that the company will “respond aggressively” to litigation “if forced to do so.” We have reached out to Delta for comment and will update this story if we hear back.

This sounds like a threat to me. And I can see why CrowdStrike would fire threats in Delta’s direction. CrowdStrike doesn’t want a mountain of lawsuits filed against it because it’s pretty safe to say that any one of these lawsuits would “end” CrowdStrike, never mind a whole bunch of them. Thus they’re trying to use Delta to deter others from doing what Delta has done. The thing is that I am not sure that this is a viable strategy. On top of that, it doesn’t paint CrowdStrike in the best light. Not that CrowdStrike is going to listen to me, but maybe they should rethink how they respond to this before their problems multiply. Just a thought.

TechCrunch is reporting that they have in their possession an email where CrowdStrike who took down millions of PCs worldwide is offering up a $10 gift card as their way of saying sorry:

CrowdStrike, the cybersecurity firm that crashed millions of computers with a botched update all over the world last week, is offering its partners a $10 Uber Eats gift card as an apology, according to several people who say theyreceived the gift card, as well as a source who also received one.

On Tuesday, a source told TechCrunch that they received an email from CrowdStrike offering them the gift card because the company recognizes “the additional work that the July 19 incident has caused.” 

“And for that, we send our heartfelt thanks and apologies for the inconvenience,” the email read, according to a screenshot shared by the source. The same email was also posted on X by someone else. “To express our gratitude, your next cup of coffee or late night snack is on us!”

Now this email was sent to their partners as opposed to their end customers. Likely because their partners are taking the brunt of the anger over this epic fail by CrowdStrike. Still is $10 worth it for partners who have had to do heroics to get customers back online, and if they’re like me are likely still doing heroics to get their customers online? Personally, I don’t think so. But you tell me by leaving a comment below and sharing your thoughts.

Crowdstrike has posted a root cause analysis in regards to them taking down a whole lot of PCs last Friday. Some of which are still down because of how huge their screw up was. In any case, this global IT nightmare was caused by an “undetected error” in the content configuration update for its Falcon platform affecting Windows machines. And that their fix for this is that the company will do more internal testing as well as putting in place “a new check” to stop “this type of problematic content” from being deployed again.

In short, something slipped through their QA process or perhaps lack of one as either is plausible, that caused millions of PCs to blue screen. That’s a total fail.

There’s something else that should be pointed out. CrowdStrike CEO George Kurtz has lived this nightmare before:

On April 21, 2010, the antivirus company McAfee released an update to its software used by its corporate customers. The update deleted a key Windows file, causing millions of computers around the world to crash and repeatedly reboot. Much like the CrowdStrike mistake, the McAfee problem required a manual fix.

Kurtz was McAfee’s chief technology officer at the time. Months later, Intel acquired McAfee. And several months after that Kurtz left the company. He founded CrowdStrike in 2012 and has been its CEO ever since.

Clearly he’s learned nothing from that experience. And I am sure that someone will be asking him about that real soon as he’s been requested to answer questions about this epic fail in Washington.

When the CrowdStrike snafu happened on Friday, I said this:

I’ll be watching this situation and posting updates when it warrants an update. But this situation is bad and likely won’t improve for a while. And when this is resolved, CrowdStrike will have a whole lot of explaining to do.

And later that day, I said this:

I wish every help desk globally well in dealing with this as this is going to be days if not a week or two of remediation. I also hope that CrowdStrike gets hauled in front of the relevant authorities globally to explain why this happened, and why corporate users should trust them again.

Well, it looks like I will get my wish based on this:

Today, House Committee on Homeland Security Chairman Mark E. Green, MD (R-TN) and Subcommittee on Cybersecurity and Infrastructure Protection Andrew Garbarino (R-NY) sent a letter to CrowdStrike Chief Executive Officer George Kurtz, requesting his public testimony before the Committee regarding the global information technology (IT) outage that occurred last Friday. The outage was attributed to a “defect” in a CrowdStrike software update. The cascading effects impacted key functions of the global economy including aviation, healthcare, banking, media, and emergency services. 

It will be interesting to see if he actually show’s up, and what he says if he does. This company pretty much executed the most successful cyberattack in the history of the planet….. By accident. So I am not surprised that Congress wants to ask some questions about this. And what he’s going to ensure that this won’t happen again.

After pretty much bricking every Windows 10/11 computer that ran CrowdStrike Falcon, CrowdStrike put out a post that details the technical ins and outs of what led up to what happened on Friday. You can read it here. In it they seem very open. And the company has committed to providing additional details and a root cause analysis.

The thing is that what happened on Friday is a warning to the planet, and to the IT industry. CrowdStrike really screwed up here and disrupted the planet in the process. The mitigation for this is relatively easy to apply as I did that a whole bunch of times on Friday and Saturday. But because of the scale of this event, we’re talking about days before this problem is fully dealt with. In other words, this was bad. But it could have been worse. We need to learn from that and be prepared for the next event like this. Because there will be a next event. That starts with CrowdStrike being completely open to laying bare what happened and what they will do to ensure that it never happens again. And that’s followed up by other companies learning from this event and ensuring that they don’t become the next Crowdstrike.

UPDATE: After I posted this, I got this commentary from John Gunn, CEO, Token:

If anyone wants to know what a full-blown cyberattack from China or other enemy nations might look like, this event just gave us a small preview of the interruptions and havoc that could be inflicted. Every day we hear about new ransomware attacks, but these are revealed because of the immediate financial payoff the attackers seek. There are undoubtedly countless significant network intrusions throughout our infrastructure and essential services that are undetected which are like sleeper cells waiting to be activated if we enter a major conflict with these nations.

This CrowdStrike created nightmare via a bad antivirus update is a massive problem. How massive? Let me give you some perspective:

• Plus or minus a billion computers are basically bricked worldwide.
• These are mostly corporate ones as corporate computers are most likely to use the CrowdStrike AV software.
• Every affected computer needs to be rebooted in Safe Mode and have a driver manually removed. That should take 4 to 5 minutes a computer. I know that because I’ve done that about 50 times today.
• Smart companies take away the rights for common employees to do this.
• Even if they had the rights to this, imagine the average end user trying to handle a moderately complex task like this.

This is most non trivial event that could possibly exist. But there’s more. I sourced comments from a number of industry experts on this:

Evan Dornbush, former NSA cybersecurity expert:

   “This is of course  a phishing attack opportunity. Don’t make a bad situation worse. Only follow recommended instructions direct from your CrowdStrike rep. There will be a lot of misinformation about how to reconfigure your computers or which critical system files to delete.  Don’t fall victim to downloading phony solutions.

   “Similarly, this is a great time to reflect on password management, since the fix may eventually require administrative access to systems that have not rebooted in quite some time.”

Omdia Senior Director, Cybersecurity Maxine Holt

The global IT outage crisis is escalating, and organizations everywhere are in full scramble mode, desperately implementing workarounds to keep their businesses afloat. Microsoft has pointed fingers at a third-party software update, while CrowdStrike admits to a “defect found in a single content update for Windows hosts” and is working feverishly with affected customers. Omdia analysts connect the dots: this isn’t a cyberattack, but it’s unquestionably a cybersecurity disaster.

Cybersecurity’s role is to protect and ensure uninterrupted business operations. Today, on 19 July 2024, many organizations are failing to operate, proving that even non-malicious cybersecurity failures can bring businesses to their knees. The workaround, involving booting into safe mode, is a nightmare for cloud customers. Cloud-dependent businesses are facing severe disruptions.

Omdia’s Cloud and Data Center analysts have long warned about over-reliance on cloud services. Today’s outages will make enterprises rethink moving mission-critical applications off-premises. The ripple effect is massive, hitting CrowdStrike, Microsoft, AWS, Azure, Google, and beyond. CrowdStrike’s shares have plummeted by more than 20% in unofficial pre-market trading in the US, translating to a staggering $16 billion loss in value.

Looking forward, there’s a shift towards consolidating security tools into integrated platforms. However, as one CISO starkly put it, “Consolidating with fewer vendors means that any issue has a huge operational impact. Businesses must demand rigorous testing and transparency from their vendors.”

CrowdStrike’s testing procedures will undoubtedly be scrutinized in the aftermath. For now, the outages continue to rise, and the tech world watches as the fallout unfolds.

Steve Hahn, Executive VP, BullWall:

   “This event, more than any other, is precisely why companies need a defense in depth strategy. One issue on your endpoint security and not only can your infrastructure go down, but you can be left wide open for a myriad of attacks. Ransomware uses endpoints, and other attack vectors, as their launch mechanism for their attack and you need layers of security over your critical data and fileshares.

   “It will be interesting to see if we have a ripple of downstream consequences. Right now we are dealing with outages at airlines and other critical businesses but will we also see a wave of Ransomware attacks that follow? Time will tell.”

I wish every help desk globally well in dealing with this as this is going to be days if not a week or two of remediation. I also hope that CrowdStrike gets hauled in front of the relevant authorities globally to explain why this happened, and why corporate users should trust them again.

UPDATE: Madison Horn for Congress (OK-5) adds this comment:

With 15 years of experience in both the private and public sectors, I bring a deep insight into complex technological issues. If elected, I will be the most credentialed cybersecurity lawmaker in U.S. history. My leadership transcends partisan divides, focusing on practical solutions. By bridging the gap between technology and policy, I will address workforce development, AI regulation, and trust in government. My candidacy represents a path toward bipartisan cooperation to confront our nation’s complex challenges.” 

“Today, we face the largest IT blackout in history, caused not by a cyber attack or malicious actor, but by human error. This outage has impacted communities and 911 operators, and what we can assume at this time, caused billion dollar losses across the global economy – starkly highlighting the fragility of our interconnected world. 

While today’s events could not have been prevented with a single solution, any set of systems that have the potential to cause massive societal impact in the event of failure—such as the 9/11 communication outages for first responders—must have right-sized regulations that protect human life and ensure economic stability. 

Presently, the critical infrastructure and financial sectors have requirements that ensure the classification of systems that could be single points of failure, yet misclassification and outdated regulations persist. In many cases, existing regulations are not properly tailored to specific industries. This issue is compounded by the fact that governing bodies struggle to keep pace with rapid technological change — leading to a disconnect in understanding the underlying technology, its dependencies, capabilities, cost of implementation, and workforce limitations. 

This gap between our regulatory landscape and the demands of the rapid advancement of technology impacting society are widening. To address today’s critical challenges, we need leaders who have expertise in technology, enabling Congress to effectively collaborate with the private sector to drive solutions. The technology we use today, which fits in a device smaller than a deck of cards, has the potential to disrupt critical infrastructure like our electric grid. To safeguard our future, we need elected leaders who not only grasp the gravity of this technological reality but also have the expertise to address and mitigate these risks effectively. 

UPDATE #2:  Tom Marsland, VP of Technology, Cloud Range adds this comment:

Recovery is going to be painful, to put it lightly. The recovery steps outlined by CrowdStrike involve manually booting the affected PC into a recovery mode, deleting a file, and restarting.  This is not something that can be done remotely, and in many organizations, will require an administrator. This means someone from IT Support going computer to computer and doing this manually. This was most certainly preventable. This sort of release goes to the importance of change / configuration management.  This update should’ve been tested internally by CrowdStrike, then released to a small subset of users, then to their broader ecosystem. That is done specifically to catch problems with updates before they affect the entire ecosystem. Either that didn’t happen here at all, or that process failed to catch this bug, which is a  problem in and of itself.

This will take days, probably weeks for larger organizations. Unfortunately, as is the case in many cyber breaches as well, this is nothing new. Organizations failure to follow best practices with testing and deploying patches (both from a CrowdStrike side and from an organization receiving updates side) is the root cause of this.  When major patches roll out or become available, putting on auto-updates is one way to make sure your organization gets patched, but if there’s any concern about the operability or function of that update, organizations generally roll those out within their own businesses to a small set at first, and then to everyone else. The organizations affected today seem to be the ones that turned on automatic updates and that was it.

UPDATE #3: Tom Siu, CISO, Inversion6

This case with CrowdStrike Falcon and Microsoft Windows highlights one of the low-frequency and high-impact risks that don’t often rise to the top of your Risk Index. I call it the “auto-immune response risk” situation where your security tools and services misidentify normative files and services, and automated corrective actions lead to system outage. In the military, we called these “blue on blue” engagements.

The lesson that cybersecurity professionals need to know is that in the real world, errors can happen and propagate throughout our environments. This is why cybersecurity and IT teams need to have clear shared objectives and cogent leadership to first recover the IT systems, avoid lowering the security posture, and then plan/execute a path forward.  Uptime may be important, and CISOs will have to justify arguments for extended outages caused by security tooling. Often the toughest call in an incident response scenario is to take systems offline due to a vulnerability; here we have them offline already. The planning and execution we see going on currently is basically a disaster recovery scenario.

One risk mitigation for this scenario is to use mildly diverse portfolio of endpoint security solutions. For example, one product on your endpoints, and a different product on your infrastructure. I know vendors, and CISOs, often desire to unify these applications under “one pane of glass” (licensing simplicity is a big factor) but this type of low-frequency risk is going to a harder argument for a multiple solutions.  

This doesn’t sound like a patch that went awry, but more of a more complex systems interaction that hasn’t been fully evaluated; I suggest we make our judgements about the vendors by the quality and transparency of their communications and assistance. Additionally, we as security professionals need to incorporate public and internal communications for this type of event into our Incident Response Plans.

In conclusion, one question I’ve seen today is, “Is this an IT outage or a security incident?”

My answer is, “Yes.”