CrowdStrike Struck The World

Faulty CrowdStrike update puts Windows PCs in a boot loop

Numerous Windows machines are experiencing a Blue Screen of Death (BSOD) issue at boot time, impacting banks, airlines, hospitals, TV broadcasters, stock exchanges, supermarkets and many more businesses around the world. The well known security provider CrowdStrike, issued a buggy update for their EDR that is preventing Windows PCs and Server from booting properly. The solution is widely used, hence the huge number of reports on social media of systems being down, with lots of them affecting critical sectors such as aviation and hospitals.

Examples of impacted services and companies

• Ryanair was “experiencing disruption across the network due to a Global 3rd party IT outage”.
• Berlin Airport issued a statements stating that operational procedures are affected by IT problems “at an external provider with worldwide consequences”.
• Delta Airlines impacted.
• Dubai Airport had their check-in servers down.
• Hong King airport switched to manual check-in due to IT outages.
• KLM forced to suspend most of operations due to outage.
• A few Hospitals in Germany canceled surgery.
• Public transport in Sydney.
• Sky News, a UK broadcaster was unable to broadcast its morning news bulletins.
• London Stock Exchange wasn’t showing prices.
• Las Vegas casino computers are crashing.
• More reports are coming from the UK, Netherlands, India, Japan, Germany, US, Philippines, Israel, Spain and other countries. DownDetector shows nearly every big company being impacted by the issue.
• Some Reddit posts say they’re entire company is down.

Is there a fix for the CrowdStrike update?

CrowdStrike is treating this as a priority zero incident and a fix was already made public.

If you don’t have Bitlocker enabled or you do have the encryption key:

1. Boot Windows into Safe Mode or the Windows Recovery Environment
2. Navigate to the C:\Windows\System32\drivers\CrowdStrike directory
3. Locate the file matching "C-00000291*.sys" file, right click and rename it to "C-00000291* renamed"
4. Boot the host normally.

Things might not be so easy for non technical people. Also the above solution probably won’t work for Bitlocker enabled drives, unless you have the key (probably all the work from home people) but it is what it is. Also, it might be hard to delete files on a machine you can’t connect to since it doesn’t boot.

A solution was posted on X for Bitlocker enabled drives (haven’t tested it, so beware):

1. Cycle through BSODs until you get the recovery screen.
2. Navigate to Troubleshoot>Advanced Options> Startup Settings
3. Press “Restart”
4. Skip the first Bitlocker recovery key prompt by pressing Esc
5. Skip the second Bitlocker recovery key prompt by selecting “Skip This Drive” in the bottom right
6. Navigate to Troubleshoot> Advanced Options> Command Prompt
7. Type “bcdedit /set {default) safeboot minimal”, then press enter.
8. Go back to the WinRE main menu and select Continue.
9. It may cycle 2-3 times.
10. If you booted into safe mode, log in per normal.
11. Open Windows Explorer, navigate to C:\Windows\System32\drivers\Crowdstrike
12. Delete the offending file (starts with C-00000291*, sys file extension)
13. Open command prompt (as administrator)
14. Type “bcdedit /deletevalue {default) safeboot”, then press enter.
15. Restart as normal, confirm normal behavior.

Why did this happen?

To work properly, EDRs/XDRs require low level access to the operating system. This usually means installing some kind of drivers that work in the Ring 0 protection level. User applications run in Ring 3. Ring 0 is basically kernel access and has visibility into everything that Ring 3 does. This also means that Ring 3 can’t influence things running in Ring 0. Security solutions have components running in Ring 0 so that malware (which runs in Ring 3, at least at the beginning) can’t influence or shutdown those components. If security solutions and malware would only run in Ring 3, there wouldn’t be anything stopping malware from interfering with the AV/EDR, hence bypassing protections.

Kernel code is very hard to write properly. A small coding error or just not considering all possible behaviours can cause the whole system to crash. Software errors are common and hard to avoid, but when it happens in a driver that’s installed on thousands of computers, the consequences are not easy to deal with as we can see in this particular example.

Common security best practices these days as well as compliance checkboxes require running some kind of AV/EDR solution. And it’s common to have these auto-update mechanisms in place as the security cat-and-mouse game moves extremely quickly. Perhaps we need to be more careful when checking a box to avoid similar situations from happening again? Perhaps use two different solutions and split your computer fleet in two? If something happened, at least you would only loose half your systems. But the overhead to do that in a big organization is pretty huge. Or maybe take the Apple route and disable kernel drivers and think of another way of doing things? We’ll see where the industry moves from this lesson.

Not an easy weekend for IT folks, so hang in there!