In a significant disruption, a recent update from CrowdStrike has led to widespread crashes in Windows systems. The bug in their Endpoint Detection and Response (EDR) agent, which was not thoroughly tested, resulted in a global outage affecting numerous installations. This incident has sparked a discussion about the importance of rigorous testing, the risks associated with automatic updates, and the need for robust backup solutions.
The issue began with a routine security update from CrowdStrike, intended to enhance security but instead causing widespread system crashes. Kevin Reed, Chief Information Security Officer at Acronis, explained, “The recent CrowdStrike outage appears to stem from a bug in their EDR agent, which was unfortunately not thoroughly tested. This resulted in widespread disruption as many installations were affected globally. The flawed update necessitates manual intervention to resolve, specifically rebooting systems in ‘safe mode’ and deleting the faulty driver file. This process is cumbersome and leaves systems vulnerable in the interim, potentially inviting opportunistic attacks.”
Reed’s comments underscore a critical point: the necessity of rigorous testing for updates, especially those related to security. “This incident highlights the importance of rigorous testing and staged updates for EDR agents. Normally, testing is done with every release and can take days to weeks, depending on the size of the update or changes. The ease with which their driver files can be deleted also raises questions about the self-protection mechanisms of CrowdStrike’s software,” Reed added.
Andreas Hassellöf, CEO at Ombori, echoed this sentiment, emphasizing the delicate balance between cybersecurity and operational stability. “This massive global IT outage, reportedly caused by a faulty security update from CrowdStrike affecting Microsoft Windows systems, highlights the delicate balance between maintaining cybersecurity and ensuring operational stability.”
Hassellöf pointed out a significant risk: companies might become hesitant to apply crucial updates, fearing similar outages. “There’s now a risk that companies might become hesitant to apply crucial updates, fearing similar outages. However, this approach would leave them more susceptible to cyberattacks. It’s absolutely vital that organizations don’t overreact by avoiding updates altogether.”
He recommended a more controlled, methodical approach to managing updates. “Companies should implement robust testing procedures, including staging updates in isolated environments that mirror their production systems before rolling them out widely. This approach allows for the identification and mitigation of potential issues before they can impact critical operations. While no update process is entirely risk-free, a careful, staged approach to updates can significantly reduce the likelihood of such widespread disruptions while maintaining strong cybersecurity defenses.”
Alois Reitbauer, Chief AI Strategist at Dynatrace, highlighted the role of AI in managing complex IT operations. “Given the increasing complexity of software, all software developers and organizations are susceptible to outages. When outages do occur, organizations need the capability to pinpoint root cause and remediate immediately. AI-driven approaches have become essential for complex IT operations to deploy as manual processes cannot keep up. A power of three approach to AI leveraging predictive, causal, and generative AI is increasingly critical to help organizations deliver the highest availability and performance of software as well as minimize disruption to end-user experience.”
In light of the incident, Reed advised businesses to ensure robust backup solutions. “Those with recent backups can restore their systems to a stable state, minimizing downtime and exposure. Moving forward, we recommend all businesses ensure robust backup solutions and advocate for better testing protocols from their security vendors.”
Mark Jow, Security Evangelist EMEA at Gigamon, also emphasized the need for preparedness. “This Microsoft IT outage demonstrates the need for more robust and resilient solutions so that when these issues do arise, they can be resolved quickly without causing such widespread customer chaos and security risk. Preparedness is key – every IT and security vendor must have a robust system in place across its software development lifecycle to test upgrades before they are rolled out to ensure that there are no security flaws within the updates.”
Alexey Lukatsky, Managing Director and Cybersecurity Business Consultant at Positive Technologies, drew parallels with past incidents such as the SolarWinds hack. “This case reminds us of the importance of secure development, since in this case it was most likely the lack of update checking both on the side of the manufacturer – CrowdStrike – and on the side of consumers who automatically installed all the updates that reached them, and led to a massive global outage around the globe. With the exception of those countries that are not using infosec products from this American corporation.”
Lukatsky noted that while this incident does not appear to be a malicious attack, it still highlights vulnerabilities in the development and update processes.
Darren Anstee, Chief Technology Officer for Security at NETSCOUT, emphasized the need for better balance in the update process. “The worldwide IT outage currently affecting airlines, media, banks and much more appears to have been caused by a faulty software update which was automatically applied, and not a cyberattack. This is another demonstration of how dependent we are on both our IT infrastructure, and the supply chains that deliver tightly integrated capabilities within it.”
Anstee highlighted the necessity of testing and controlled roll-outs. “Most enterprise software goes through testing and controlled roll-out before it is pushed to a whole population, but this doesn’t seem to be the case in this instance.”
Jake Moore, Global Security Advisor at ESET, discussed the broader implications of such outages. “These outages are increasing in volume due to the sheer increase in numbers of online users and traffic. After witnessing the blue screen of death (BSOD), many people are quick to suspect a cyberattack or find similarities to Netflix’s ‘Leave The World Behind’ but this can often add to the confusion. It highlights the importance of these services and the millions of people they serve.”
Moore stressed the importance of cyber-resilience plans and the challenges of simulating large-scale issues. “Businesses must test their infrastructure and have multiple fail safes in place, however large the company is, this is typically referred to as a cyber-resilience plan. But as often it is with the case, it is simply impossible to simulate the size and magnitude of the issue in a safe environment without testing the actual network.”
The CrowdStrike update incident serves as a stark reminder of the complexities and risks associated with software updates. It underscores the necessity of rigorous testing, robust backup solutions, and methodical update processes. As businesses continue to navigate the challenges of cybersecurity, this incident provides valuable lessons for improving IT infrastructure and resilience against future disruptions.