Widespread IT Outage Due to CrowdStrike Update
Posted: Sun Jul 21, 2024 11:23 am
Yesterday, Microsoft servers across the world displayed the dreaded "blue screen of death," leading to mass IT outages that disrupted business, airlines and flights, healthcare providers, banks, and more. The cause: A defective update to CrowdStrike Falcon Sensor, a widely used cloud-based endpoint detection and prevention (EDR) software program.
CrowdStrike said its engineering team has identified the issue that caused the massive disruption to Windows-based systems: A bug in the Memory Scanning prevention policy, which was not identified during their testing stages, Callie Guenther, senior manager at Critical Start, noted in an emailed statement.
"While CrowdStrike likely performed standard regression and functionality tests, these were insufficient because they did not simulate the real-world deployment environment where the bug caused the Falcon sensor to consume 100% of a CPU core," she wrote. This ultimately led to system performance issues.
CrowdStrike has since reverted the flawed Falcon software update. Even so, some users are still experiencing system crashes or are unable to stay online to receive the new and fixed version. The cybersecurity vendor has provided workaround steps for this issue.
What we Know?
endpoint security company Crowdstrike released an update that is causing widespread "blue screens of death" (BSOD) on Windows systems. Crowdstrike released an advisory, which is only available after logging into the Crowdstrike support platform. A brief public statement can be found here.
Crowdstrike now also published a detailed public document with tips to recover:
https://www.crowdstrike.com/blog/statem ... ows-hosts/
---
Update: Some reports we have seen indicate that there may be phishing emails circulating claiming to come from "Crowdstrike Support" or "Crowdstrike Security". I do not have any samples at this point, but attackers are likely leveraging the heavy media attention. Please be careful with any "patches" that may be delivered this way.
One domain possibly associated with these phishing attacks is : crowdfalcon-immed-update [ .] com
---
Linux and MacOS systems are not affected by this issue.
The quickest fix appears to boot the system into "Windows Safemode with Network". This way, Crowdstrike will not start, but the current version may be downloaded and applied, which will fix the issue. This "quick version" of the fix is not part of Crowdstrike's recommendations but may be worth a try if you have many systems to apply the fix to or if you need to talk a non-computer-savvy person through the procedure. Some users have reported that this will succeed.
Casimir Pulaski (@cybermactex) mentioned on X that a simple reboot sometimes works if the latest update was downloaded before the system crashed.
The support portal statement offers the following steps to get affected systems back into business:
CrowdStrike Engineering has identified a content deployment related to this issue and reverted those changes.
Workaround Steps:
1 - Boot Windows into Safe Mode or the Windows Recovery Environment
2 - Navigate to the C:\Windows\System32\drivers\CrowdStrike directory
3 - Locate the file matching “C-00000291*.sys”, and delete it.
4 - Boot the host normally.
For a Bitlocker-protected system, you will have to provide the recovery key to delete the file.
Virtual systems are easier to fix as you should be able to just shut them down, mount the virtual disk to the host or a different virtual system (Linux?
), and remove the file.
CrowdStrike said its engineering team has identified the issue that caused the massive disruption to Windows-based systems: A bug in the Memory Scanning prevention policy, which was not identified during their testing stages, Callie Guenther, senior manager at Critical Start, noted in an emailed statement.
"While CrowdStrike likely performed standard regression and functionality tests, these were insufficient because they did not simulate the real-world deployment environment where the bug caused the Falcon sensor to consume 100% of a CPU core," she wrote. This ultimately led to system performance issues.
CrowdStrike has since reverted the flawed Falcon software update. Even so, some users are still experiencing system crashes or are unable to stay online to receive the new and fixed version. The cybersecurity vendor has provided workaround steps for this issue.
What we Know?
endpoint security company Crowdstrike released an update that is causing widespread "blue screens of death" (BSOD) on Windows systems. Crowdstrike released an advisory, which is only available after logging into the Crowdstrike support platform. A brief public statement can be found here.
Crowdstrike now also published a detailed public document with tips to recover:
https://www.crowdstrike.com/blog/statem ... ows-hosts/
---
Update: Some reports we have seen indicate that there may be phishing emails circulating claiming to come from "Crowdstrike Support" or "Crowdstrike Security". I do not have any samples at this point, but attackers are likely leveraging the heavy media attention. Please be careful with any "patches" that may be delivered this way.
One domain possibly associated with these phishing attacks is : crowdfalcon-immed-update [ .] com
---
Linux and MacOS systems are not affected by this issue.
The quickest fix appears to boot the system into "Windows Safemode with Network". This way, Crowdstrike will not start, but the current version may be downloaded and applied, which will fix the issue. This "quick version" of the fix is not part of Crowdstrike's recommendations but may be worth a try if you have many systems to apply the fix to or if you need to talk a non-computer-savvy person through the procedure. Some users have reported that this will succeed.
Casimir Pulaski (@cybermactex) mentioned on X that a simple reboot sometimes works if the latest update was downloaded before the system crashed.
The support portal statement offers the following steps to get affected systems back into business:
CrowdStrike Engineering has identified a content deployment related to this issue and reverted those changes.
Workaround Steps:
1 - Boot Windows into Safe Mode or the Windows Recovery Environment
2 - Navigate to the C:\Windows\System32\drivers\CrowdStrike directory
3 - Locate the file matching “C-00000291*.sys”, and delete it.
4 - Boot the host normally.
For a Bitlocker-protected system, you will have to provide the recovery key to delete the file.
Virtual systems are easier to fix as you should be able to just shut them down, mount the virtual disk to the host or a different virtual system (Linux?
), and remove the file.