The outage was caused by a faulty update sent to CrowdStrike software running on Microsoft’s Windows operating system. Devices went spiraling, unable to properly restart unless someone removed the flawed file from their systems.
Adam Meyers, CrowdStrike’s senior vice president of counter adversary operations, told members of a House Homeland Security subcommittee that the company had instituted new safeguards to ensure that such a failure couldn’t happen again.
Lawmakers pushed Meyers to explain why the error had happened in the first place and how the company planned to answer for the outage’s harm to consumers.
“I want to make sure that you all know what happened, can explain it, and then how you’re making sure it’s not going to happen again,” said Rep. Andrew Garbarino, R-N.Y.
The July incident underscored how dependent modern commerce and communications have become on just a handful of large technology companies. Travelers were stranded as airlines canceled flights. Emergency services were disrupted as 911 operators found that their systems had failed. Hospitals paused some of their services.
Discover the stories of your interest
Tuesday’s hearing pointed to the persistent questions that governments have about the power and influence of tech companies that dominate the modern internet era. Lawmakers around the world have created recent laws to regulate how companies such as Microsoft, Amazon, Apple, Google and Meta, the owner of Facebook and Instagram, do business. They have accused the companies of entrenching themselves by shutting out smaller competitors, and have set new rules for how social media platforms handle content. “This heavy dependence on the tech sector is new, and it does increase the vulnerability to large shocks,” said Jonathan Welburn, a senior researcher at the RAND Corp. who studies and models the supplier connections and dependencies among companies, in a recent interview.
The July outage swept across the globe as computers received the faulty update. Its immediate impact was largely confined to computers used by businesses, rather than individual consumers, because CrowdStrike works with big corporate clients.
The infamous Windows “blue screen of death” appeared on disabled machines. Inside CrowdStrike, engineers were told to focus on fixing the problem rather than on tracing its cause. The company eventually posted instructions to tell its customers how to fix the problem and issued a software patch designed to stop devices from rebooting continuously.
George Kurtz, the company’s CEO, was absent from the hearing even though the committee had initially demanded his testimony. In his place was Meyers, who said in his opening statement that CrowdStrike had “let our customers down.”
“We are deeply sorry and are determined to prevent this from ever happening again,” he said.
Many lawmakers praised CrowdStrike’s overall response. But they pushed Meyers to account for how a routine update — the kind the company sends 10 to 12 times a day — could have gone so wrong.
He said the company’s process for screening updates had failed to catch the issue. “It tested as clean or good, and that’s why it was allowed to roll out,” he said.
The company has since updated its internal processes to incorporate more rigorous testing to protect against a similar situation, Meyers said. CrowdStrike customers can now opt to wait to receive updates, too.
While CrowdStrike has made clear that a cyberattack did not cause the outage, lawmakers signaled that they were still concerned about the way the public had suffered.
Rep. William R. Timmons IV, R-S.C., asked Meyers how the company planned to hold itself accountable. Timmons said his “constituents that missed flights and were stuck in airports for weeks” probably didn’t care about the company’s distinction between a security breach and the flawed update.
On the company’s most recent earnings call, CrowdStrike executives said it was dedicating $60 million to “customer commitment packages,” paid out in the form of credits to clients that were affected.
That is a much smaller figure than the $500 million that Delta Air Lines says it lost to the outage. The airline said in an August securities filing that it was “pursuing legal claims against CrowdStrike and Microsoft to recover damages caused by the outage.” CrowdStrike executives previously said insurance would limit the company’s losses.
Rep. Mark E. Green, R-Tenn., asked Meyers whether artificial intelligence had been involved in sending out the faulty update.
“AI was not responsible for making any decision in that process,” Meyers said.
Garbarino suggested that he was worried that if this failure was a “perfect storm,” it could happen again without changes from CrowdStrike.
“Because a lot of perfect storms or hundred-year floods are happening now every other year,” he said.
