The Great CrowdStrike Outage of 2024

In the world of cybersecurity, even the mightiest can stumble. On July 19th, 2024, the cybersecurity titan, CrowdStrike, found itself at the center of an unexpected drama: a global outage that left IT teams across the globe in a synchronized state of panic. Here’s the rundown of the incident with a bit of humor because if you’ve ever been involved in an incident even half this scale, you know it’s needed.

What is a CrowdStrike?

CrowdStrike is a leading cybersecurity technology company. Founded in 2011, CrowdStrike specializes in endpoint protection, threat intelligence, and incident response services, leveraging cloud-native architecture and artificial intelligence. CrowdStrike’s flagship product and the focus on the recent global outage, Falcon, utilizes machine learning to detect threats in real-time and provide comprehensive security coverage across various devices and platforms.

Initial Symptoms: Houston, we have a problem

Around 1000 hrs UTC (that’s 5 a.m. Eastern for you non-UTC-using folks!), users began experiencing unusual latency issues. Thirty minutes later, the latency had escalated into a full-blown connectivity problem. Across the globe, security operations centers (SOCs) went into overdrive, with IT staff frantically refreshing dashboards and questioning their life choices.

The root cause? A misconfiguration in the update deployment pipeline. This update, intended to optimize performance, inadvertently triggered a cascading failure. CrowdStrike’s incident response team was alerted and responded to the incident. They focused their initial investigations on identifying any recent changes to the system, as these are often the culprits behind sudden outages. Indeed, a software update had been deployed earlier that morning, which appeared to be the starting point of the issues.

Misconfiguration Details: pt 1

Gosh…where to start. Ok, so the faulty software driver was analyzed by loads of brilliant C++ experts and so we know that the driver had a null pointer that was pointing towards an invalid region of memory for a program. Initial belief by many was that this driver was included in the update, was pushed to production on a Friday morning, and caused the systems to crash with a BSOD (blue screen of death). A fix was deployed, but the affected systems had to be rebooted. The issue here is that it needs access to safe boot, and if it’s encrypted with something like BitLocker, you need that ungodly long key with you to boot into safe mode. Also not to mention that if you have 1000 systems, you have to do that 1000 times. Scale that up to global proportions and you can see why air travel was halted for some time.

Here’s a little stack trace for the nerds among us who wanna see it! This screenshot is thanks to @Perpetualmaniac on twitter.

Misconfiguration Details: pt 2 (uhhh…what now?)

So as it turns out, the faulty driver wasn’t released during the update. The driver was actually already present on MAc, Linux, and Windows systems. It was essentially a logic bomb that just needed the right data to activate it, which is exactly what happened here. Typically, deploying new data files is considered safe since the data isn’t executable and doesn’t undergo the same review as new code. So, with the bug already being pre-installed, the update was rolled out to the Windows systems running Crodstrike causing this insane situation.

This same vulnerability exists in Linus and macOS, but because they weren’t targeting in the data update, they didn’t crash. However, if a similar update had been pushed to Linus, it could’ve caused an even more catastrophic incident.

Adding to the severity, Microsoft granted CrowdStrike's driver "boot-start" privileges, typically reserved for Microsoft drivers. A boot-start driver is essential for the Windows operating system to start. To fix the system, one must boot into safe mode, bypass the boot-start drivers, and delete the faulty data files. This removes part B of the detonator, but CrowdStrike's boot-start drivers, which have special privileges, must remain for the system to boot properly. Below are the faulty data files.

faulty data files to remove in safe mode

The theories that follow…

Anytime something major happens, there will always be those with theories of information that are still not shared, and this incident has been no different. There are several conspiracy theories surrounding the CrowdStrike event. Some of the prominent ones include:

Deliberate Sabotage: Some theorists claim that the inclusion of the faulty driver was a deliberate act of sabotage, potentially orchestrated by a state actor or a competing cybersecurity firm to damage CrowdStrike's reputation and compromise their clients' security.

Cover-Up by Tech Giants: Another theory suggests that tech giants, including Microsoft and CrowdStrike, knew about the vulnerability for a long time but chose to cover it up to avoid public backlash and potential financial losses. This includes the idea that these companies worked together to downplay the severity of the issue.

Insider Threat: Some believe that an insider within CrowdStrike or Microsoft planted the bug, either as a disgruntled employee or as part of an espionage operation. This insider would have the knowledge and access needed to introduce such a vulnerability.

Government Surveillance: A popular conspiracy theory posits that the bug was intentionally placed by government agencies to create a backdoor for surveillance purposes. This would allow them to monitor activities on a vast number of computers worldwide without the users' knowledge.

Global Cyberwarfare: Some theorists argue that this incident is part of a larger scheme of global cyberwarfare, where various countries are continuously trying to infiltrate and disrupt each other's critical infrastructure. The CrowdStrike bug is seen as one of many tools used in these ongoing covert operations.

Financial Gain: There is speculation that financial entities or stock market manipulators might have had a hand in this event, aiming to profit from the resulting fluctuations in stock prices of the companies involved.

These conspiracy theories often lack substantial evidence and are fueled by speculation and mistrust of large corporations and government entities. But it certainly doesn’t bode well for the companies overall image that two of it’s executives recently sold large amounts of CrowdStrike stock. According to investing.com, Crowdstrikes Chief Security Officer, Shawn Henry sold 4000 shares of the company’s Class A common stock on July 15th, just days before the global outage, for a totaling sum of approximately $1.49 million. Additionally, the company’s Chief Accounting Officer, Saha Anurag sold 1,662 shares (a total value of $630,000) on June 21st, 2024. Both transactions were disclosed in filings with the SEC. While these two suspiciously timed transactions could absolutely be benign, there certainly is a lot of speculation happening. Only time will tell and we’ll update you with additional information as we obtain it.