![]() | A request that this article title be changed is
under discussion. Please do not move this article until the discussion is closed. |
![]() Multiple
blue screens of death caused by a faulty software update on airport luggage conveyor belts at
LaGuardia Airport, New York City | |
Date | 19 July 2024 |
---|---|
Location | Worldwide |
Type | IT outage, computer crash |
Cause | Faulty CrowdStrike software update |
Outcome | ~8.5 million Microsoft Windows operating systems crash worldwide, causing global disruption of critical services |
On 19 July 2024, American cybersecurity company CrowdStrike distributed a faulty update to its security software that caused widespread problems with computers running Microsoft Windows. As a result, roughly 8.5 million systems crashed and were unable to properly restart [1] in what has been called the largest outage in the history of information technology [2] and "historic in scale". [3]
The outage disrupted daily life, businesses, and governments around the world. Many industries were affected—airlines, airports, banks, hotels, hospitals, manufacturing, stock markets, broadcasting, gas stations, retail stores, and more—as were governmental services, such as emergency services and websites. [4] [5] The worldwide financial damage has been estimated to be at least US$10 billion. [6] [7]
Within hours, the error was discovered and a fix was released, [8] but because affected computers had to be fixed manually, [9] outages continued to linger on many services. [10] [11]
CrowdStrike produces a suite of security software products for businesses, designed to protect computers from cyberattacks. The Falcon Sensor product, CrowdStrike's vulnerability scanner, installs an endpoint sensor at the operating system kernel level on individual computers to detect and prevent threats. [12] Patches are routinely distributed by CrowdStrike to its clients to enable their computers to address new threats. [13]
On 19 July at 04:09 UTC, CrowdStrike distributed a faulty configuration update for its Falcon sensor software running on Windows PCs and servers. The update caused machines to go into either a boot loop or boot recovery mode. [14] [15]
Almost immediately, Windows virtual machines on the Microsoft Azure cloud platform began rebooting and crashing, [16] and at 06:48 UTC, Google Compute Engine also reported the problem. The problem affected systems running Windows 10 and Windows 11 running the CrowdStrike Falcon software. [17] [14] Most personal Windows PCs were unaffected, as CrowdStrike's software is primarily used by organizations. [17] Computers running macOS and Linux were unaffected, as the problematic content file was only for Windows. [18] Because it was a content file, the update could not be delayed. [19] However, similar problems affected Linux distributions in April 2024. [20] [21]
CrowdStrike reverted the content update at 05:27 UTC, [22] and devices that booted after the revert were not affected. [23]
At 07:15 UTC, Google said that the CrowdStrike update was at fault. [24] Within hours, CrowdStrike CEO George Kurtz confirmed that CrowdStrike's faulty kernel configuration file update had caused the problem. [9] [8] At 09:45 UTC, Kurtz confirmed that the fix was deployed [25] [26] and that the problem was not the result of a cyberattack. [9] [27]
The day before the faulty update, the Azure platform had an outage that blocked some companies' access to their storage and to Microsoft 365 applications in Azure's Central United States region. [28] Microsoft said the 18 July incident was unrelated to the CrowdStrike problem, but that the two incidents compounded problems for these customers. [29]
An update to a configuration file which was responsible for screening named pipes, Channel File 291, caused an out-of-bounds memory read [30] in the Windows sensor client that resulted in an invalid page fault.
Affected machines could be restored by booting into
safe mode or the
Windows Recovery Environment and deleting any .sys
file beginning with C-00000291-
and with timestamp 0409 UTC in the
%windir%\
System32\
drivers\CrowdStrike\
directory.
[31] As this process must be done locally on each individual machine, it was "expected to take days" for affected businesses to restore all systems.
[32] Further, technical staff will have to reboot the affected computers individually with manual intervention on each system, which could be a "monumental task".
[33]
[34]
[35]
On devices with Windows' BitLocker disk encryption enabled, as corporations often do to increase security, fixing the problem was exacerbated because a recovery key could be required that was stored on a server that itself had crashed. [36] [37]
Some Microsoft Azure customers discovered that they could fix the problem by rebooting affected virtual machines up to 15 times, ideally while connected to Ethernet. [38] [22] Microsoft has also recommended restoring a backup from before 18 July to fix the issue. [39]
Outages were experienced worldwide, [40] [41] [42] reflecting the wide use of Microsoft Windows and CrowdStrike software by global corporations in numerous business sectors. [43] At the time of the incident, CrowdStrike said it had more than 24,000 customers, [44] including nearly 60% of Fortune 500 companies and more than half of the Fortune 1000. [45] [46] Microsoft estimates that 8.5 million devices were affected by the update. [47]
Widespread outages were immediately reported across multiple countries, with major global disturbances experienced by the general public sweeping from east to west from time zone to time zone. At 04:09 UTC on 19 July, the time when the faulty update was issued, it was the middle of the business day of Oceania and Asia, the early morning hours in Europe, and midnight in much of the Americas.
Some countries were less affected. China, which has striven toward self-sufficiency in IT, saw little impact to key services such as airlines and banks, although foreign businesses and luxury hotels in the country were affected. [48] Russia and Iran—both restricted by international sanctions from using the services of American high-tech companies—reported no disruptions. [49] [50]
Despite the losses companies have suffered, CrowdStrike was said to be only minimally liable for the damage or lost revenue caused. [51] The terms for CrowdStrike's Falcon software limits liability to 'fees paid', [52] so the maximum compensation an affected company could recover if this provision is enforceable are the fees that the company has paid to CrowdStrike. [53]
In the EU, it is possible that CrowdStrike will be held liable under a GDPR regulation related to the impact of security incidents on user data. The regulation is best known in relation to data leaks but also applies to data destruction. It is unclear whether temporary loss of access to data is enough to trigger liability, or whether GDPR applies to all incidents related to security or only unauthorised access. [54]
Globally 5,078 air flights, 4.6% of those scheduled that day, were cancelled. [55] [56]
Australian airlines Qantas, Virgin Australia, and Jetstar were affected. [57] [58] A Sydney Airport spokesperson said that the outage had affected some operations and that "there may be some delays throughout the evening". [59] Melbourne Airport saw check-in procedures disrupted; officials advised passengers to consult with their airlines. Canberra Airport, Darwin Airport, Adelaide Airport, Perth Airport, Hobart Airport, Launceston Airport, and Brisbane Airport were also affected. [41] [60] [61] [62] In New Zealand, Christchurch Airport was having problems. [63]
Hong Kong International Airport experienced delays during check-in, primarily for passengers of the local budget carrier Hong Kong Express, whose staff members used handwritten signs to direct passengers to check-in counters. [65] The Hong Kong Airport Authority activated the emergency response after airline websites and automatic check-in malfunctioned. Local airlines Cathay Pacific, Hong Kong Express, and Hong Kong Airlines's booking systems were unavailable. [66] HKExpress cancelled some flights on 20 July. [67] Jeju Air [68] and Spring Japan [68] experienced problems.[ clarification needed] Jetstar Japan had to cancel many flights (mostly domestic flights). [69] [70] Some of the self-check-in kiosks in Singapore Changi Airport were affected, delaying and forcing airlines to switch to manual check-in, and Singapore Airlines and Scoot reported various[ weasel words] levels of service difficulties throughout 19 July. [71] [72] Cebu Pacific and Philippines AirAsia flights were delayed. [73] [74] [75] Long queues formed at Ninoy Aquino International Airport. [76] In Taiwan, airline system disruptions were reported at Taoyuan International Airport. [77] [78] In Indonesia, disruptions were reported for the check-in systems of AirAsia and Citilink. [79] In Thailand, Thai AirAsia's reservation and check-in systems were affected. [80]
In India, the outage affected Indigo Airlines, Akasa Air, SpiceJet, and Vistara. Handwritten boarding passes were being issued during the outage. The Ministry of Civil Aviation asked and ordered the airlines as well as the airports to be compassionate while providing food as well as seats if need be. [81] [82] [83] [84] As of 18:14 IST (12:44 UTC), over 200 Indian flights were cancelled; IndiGo alone cancelled 192. [85] Only those airlines that had a heavy reliance on Microsoft Azure for all their services, from hosting websites and booking engines to revenue management systems and Departure Control Systems faced the impact. Air India said that none of its flights were cancelled or delayed due to the outage, attributing it to their robust cyber system. [86]
Prague Airport [60] [87] in Czechia, Budapest Airport [88] in Hungary, Bratislava Airport [89] in Slovakia, and Schiphol Airport [90] in the Netherlands experienced problems. Planes were not allowed to land at Zurich Airport. [68] Near Brussels, Charleroi Airport employees manually checked passengers in, but other software alleviated problems by 10:00 (UTC+2) and there were minimal delays. [91] ENAIRE's Aena, the Spanish national airport traffic control manager, mentioned an IT outage in their website and social media. [92] All Spanish airports reported disruptions. [93] Charles de Gaulle Airport and Orly Airport experienced check-in problems and suspended flights. [94] Poznań–Ławica Airport and Warsaw Chopin Airport experienced check-in disruptions. An emergency system was activated, but check-in processes were slower. [95] Berlin Brandenburg Airport announced that since around 07:00 (UTC+2), operational processes were affected by "IT problems at an external provider", and that they planned to stop flights until 8 UTC. [84] [96] While passenger handling continued with some restrictions, there were delays and airlines cancelled some flights. [97] Several airlines ( Eurowings, Ryanair, Vueling, and Turkish Airlines) in Hamburg Airport had to issue tickets by hand. [98] Croatian and Swedish air traffic control were also disrupted. [99] [100]
Swiss International Air Lines had 30% of flights grounded. [101] Lufthansa in Germany experienced problems with the "profile and booking retrieval" features of their website. [102] Ryanair's booking and check-in services were unavailable and the airline was "forced to cancel a small number of flights", advising passengers to arrive at airports at least three hours before departure. [103] [104] Wizz Air said the outage put its online services offline. [105] [106] Dutch airline KLM suspended most operations, announcing that flight handling is impossible with the issue, [68] and Transavia Airlines experienced problems. [90] [107] Finnair reported that they were having trouble sending emails and SMS messages to customers. [108] In Greece, citizens and tourists saw delays at major airports, notably at Athens International Airport and at Heraklion International Airport. This disruption, occurring at the peak of the tourist season, resulted in chaotic scenes as passengers were forced to wait for hours for their flights. Contributing factors included severe staff shortages and new schedules. [109] [110] In Heraklion, eight flights were problematic. The airport's chief, George Pliakas, indicated that flights were being manually arranged to manage the disruption, but the influx of arriving flights strained the system. [111] [112]
Several UK airports had problems, including Edinburgh Airport, whose departure boards froze, [113] and Gatwick Airport, where automatic barcode scanning stopped working and had to be checked manually. [68] Amadeus, which manages UK baggage at Heathrow, said they were affected by the IT outage. [114] Disruption to flights was anticipated in the Isle of Man, particularly to and from the UK, but ultimately minimal. [115]
Tunisia experienced temporary airport disruptions. [116] Turkish Airlines cancelled some of its flights due to the outage. [117]
A ground stop was issued by United, Delta, and American Airlines. Flights in the air continued flying, but no new flights were to take off. [15] [118] Allegiant Air were also grounded by the outage according to the Federal Aviation Administration (FAA). [5] [119] About 1,500 flights were cancelled in the United States on 19 July due to the outage. [69] While American Airlines, United and Allegiant recovered relatively quickly after Friday, Delta Air Lines experienced an operational meltdown which continued through the weekend. [120] [121]
Delta Air Lines was by far the hardest hit of the US major airlines, with over 1,200 flights cancelled on Friday, 19 July, and thousands more cancelled on Saturday and Sunday. [121] [122] Thousands of stranded travelers were forced to spend the night sleeping on the floor in the terminals at Delta's largest hub, Hartsfield–Jackson Atlanta International Airport (which is also the busiest airport in the world by passenger traffic). [122] Metro Atlanta hotels and rental car companies were overwhelmed by the scale of the crisis, leaving travelers no other option but to stay in the airport. [123] One traveler attempting to return home to Tampa (after giving up on reaching California) reported that Amtrak was charging $1,000 for a one-way train ticket from Atlanta to Tampa. [124] Visibly distraught passengers with nowhere to go were seen trying to sleep in the airport on hard linoleum floors without blankets or food. [122] Without warning, Delta immediately banned unaccompanied minors on its flights through the end of 23 July. [125] This imposed considerable hardship on parents who had been counting on that service to avoid having to fly with their children to escort them to summer camp or a relative's house, along with minors who had expected to use that service to fly home. [125]
On 20 July, Delta cancelled over 1,400 flights. [126]
On 21 July, Delta cancelled over 1,300 flights. [127] With so many passengers still stuck in Hartsfield–Jackson after two consecutive nights, the airport implemented traveler assistance measures such as a "concessions crisis plan" and a plan to reunite passengers with their checked baggage. [126] However, passengers in Atlanta on that date continued to report "jam-packed" conditions and witnessing "heartbreaking" scenes in the terminals. [128]
On 21 July, Delta CEO Ed Bastian apologized to customers in a statement and revealed that the outage had affected one of Delta's crew-tracking software programs. [127] According to Bastian, the software was "unable to effectively process the unprecedented number of changes triggered by the system shutdown". [129] According to Delta CIO Rahul Samant, the program was restored and brought back online at around 11 a.m. on 19 July, but was overwhelmed by the backlog of updates awaiting processing and had been trying to catch up ever since. [128] After the ground stop left too many crew members in the wrong places, Delta was struggling to assemble enough pilots and flight attendants at airport gates to operate scheduled flights. [129] Many flights were repeatedly delayed and finally cancelled because the one or two crew members who made it to the gate for a particular flight kept hitting their legal flight time limit before the airline could finish fully staffing the flight, and this caused the crisis to snowball as those crew and their aircraft were now in the wrong place for the following day's flights. [129] A similar issue had been blamed for the 2022 Southwest Airlines scheduling crisis. [127] That same day, US Secretary of Transportation Pete Buttigieg said on social media that the US Department of Transportation had received hundreds of complaints about Delta, and reminded the airline of its legal obligations to affected passengers. [127]
On 22 July, Delta cancelled over 1,200 flights. [128] On 23 July, the Department of Transportation announced the launch of a formal investigation into Delta's treatment of passengers. [128] Delta promised to cooperate but stated that it was focused on its recovery. [128] Senator Maria Cantwell in her capacity as chair of the Senate Committee on Commerce, Science, and Transportation wrote to Bastian to express her concern for Delta passengers. [128]
United Airlines' smaller number of cancellations had a significant impact on its hubs. For example, San Mateo County hotels around San Francisco International Airport rapidly filled up with travelers on 19 July. [130] Guests reported difficulty with checking into the local Marriott hotel because Marriott International was also recovering from the outage. [130]
Southwest Airlines (the third largest US major airline by domestic passengers) was entirely unaffected. [131] A Southwest spokesperson confirmed that the airline had seen no impact from the CrowdStrike outage but refused to confirm speculation among aviation industry analysts that it had been shielded by its notoriously outdated software.
The flight delays meant that many people who had traveled to the 2024 Republican National Convention—which concluded the day the outages started—were stuck in the convention's host city of Milwaukee, Wisconsin. [132] [133]
Montréal–Trudeau International Airport and Toronto Pearson International Airport were affected in Canada, and Porter Airlines cancelled all flights. [134] Vancouver International Airport was also reportedly affected in Canada, although it was unclear whether this was directly related to the global outages. [135]
Microsoft and CrowdStrike stocks fell as a result of the outage. CrowdStrike's stock fell over 11% on 19 July though Microsoft was down less than 1%. [136] [5]
Banks that were affected included Chase, Bank of America, Wells Fargo, U.S. Bank, Capital One and Charles Schwab in the US, [137] RBC and TD Bank in Canada, [138] [139] Capitec Bank and other South African banks, [88] several Israeli banks, [140] and several banks in the Philippines, such as RCBC, Metrobank, LandBank, BDO, UnionBank, BPI, and PNB. [141] [142] E-wallets such as Maya and GCash also experienced problems in the Philippines. [143] The website and mobile banking application of DenizBank in Turkey could not be accessed. [144] Visa was affected. [145] Numerous Singaporean companies, including Singapore Exchange (SGX) and DBS Bank, reported various levels of service difficulties throughout 19 July. [71] [72]
In India, the Reserve Bank of India said that only 10 banks and NBFCs were affected by the outage; [146] [147] few banks use CrowdStrike tools and many banks' critical systems do not run on the cloud. NSE, BSE, [147] and India's largest bank, State Bank of India, said they were unaffected. [148]
In Brazil, Bradesco Bank confirmed it was affected. During the morning customers were able to login, but at 12:00 UTC the bank disabled the login button. [149]
New Zealand banks ASB and Kiwibank, Australian banks Westpac [150] and ANZ had problems. [63] Apps of Australian banks NAB, Westpac, ANZ, Commonwealth Bank, Bendigo Bank, and Suncorp were affected. [60]
The London Stock Exchange, while operating normally, was unable to push news updates to its website. [68] English gambling company Ladbrokes Coral and English supermarket chain Morrisons also reported problems. [68] Polish banks, including Santander Bank Polska, ING Bank Śląski and mBank, encountered issues related to the outage. Santander BP's helpline, video, and chat services were affected. PKO Bank Polski clarified that its iPKO and IKO services were stable, but other banks faced difficulties. [151] In Finland, OP Financial Group reported minor disruptions on investment partner and stock savings accounts. [152] Sense Bank in Ukraine experienced outages due to the update. [153]
Paraguayan banks Ueno and Banco Continental were affected; their customers were unable to log in. [154]
The United States Department of Homeland Security, NASA, Federal Trade Commission, National Nuclear Security Administration, Department of Justice, and Department of Education were affected, and the Department of the Treasury and Department of State reported minor disruptions. The Department of Veterans Affairs and Department of Energy experienced disruptions, but it is not currently known if they are related to the incident. [155] DMV agencies for the states of Georgia, Kansas, Missouri, North Carolina, Tennessee, and the District of Columbia were affected. [156] [157] [158] Ted Wheeler, the mayor of Portland, Oregon, declared the outages to be a city emergency. [159] [160] Election and voting registration databases in Arizona, South Dakota, Texas and the state of Washington were affected. [161] The website for the city of Sioux Falls, South Dakota, went down. [162]
In the United States, there were outages in 911 service or disruptions in 911 call centres' operation in some parts of Alaska, [163] Arizona, [164] Florida, [165] Iowa, [166] Indiana, [167] Kansas, [168] Michigan, [169] Minnesota, [170] New York, [171] Ohio, [172] Oregon, [173] Pennsylvania, [174] and Virginia. [166] 911 was down for all of New Hampshire. [175] [176] In addition, Alaska was experiencing issues with non-emergency call centers. [176] Many call centers switched to working backup systems. [161]
The CM/ECF and PACER computer systems used by the US federal courts were unaffected. [177] However, several state courts reported problems with their computer systems, including courts in Alaska, California, Delaware, Idaho, Kansas, Maryland, Massachusetts, Michigan, Nevada, New York, and Pennsylvania. [177] In New York City, courts and correctional facilities were disrupted, delaying a hearing in the trial of Harvey Weinstein for sex offenses. [178]
Government websites in the Philippines, such as the website of the House of Representatives of the Philippines, were down due to the outage. [179]
In Canada, services in Toronto were affected, [180] and Canada Child Benefit payments were delayed. [181] [182] New Zealand Parliament had problems. [84] Sunshine Coast Council was one of several councils affected in Australia. [60] The National Security Authority spokesman confirmed several institutions in Slovakia were affected. [183]
The fire department in Copenhagen, Denmark, was unable to receive automatic fire alerts from buildings. [184]
Traffic disruptions were reported at the US–Canada border, [185] [181] [139] including long delays at the Ambassador Bridge and Detroit–Windsor tunnel between Ontario and Michigan. [180] The Canada Border Services Agency blamed a partial outage of its telephone reporting system which was later resolved. [134] [185] There were long delays and police advised motorists to avoid the area. [186] The Washington Metro Area Transit Authority suffered minor service delays in the early morning in America; their website/live tracking was unavailable until around 9:30 am on 19 July. [187] The Massachusetts Bay Transportation Authority in Boston, as well as the Metropolitan Transportation Authority in New York, lost vehicle tracking and arrival notices for passengers. [188] Most of North American freight and passenger train operators went largely unaffected aside from some technical issues within Union Pacific and Canadian Pacific Kansas City, Amtrak was mostly unaffected aside from issues with credit card processing during the morning. [189]
Malaysia's railway operator, Keretapi Tanah Melayu, confirmed that its KITS ticketing system was experiencing technical issues. [190] Transport for Ireland said its apps were down due to the outage. [191] Ireland's Road Safety Authority said it was experiencing "significant disruption" to its National Car Test (NCT) centres. [192] In Singapore, the entrance and exit gantries of over 185 car parks managed by the Housing and Development Board (HDB) were affected. [193]
Fuel stations have also been affected in Australia, with people stuck at fuel pumps unable to pay for petrol because payment systems were not working. [59] Auckland Transport's HOP card in New Zealand had problems. [63] Australian freight train operator Aurizon was affected. [60] Regional trains in New South Wales, Australia on the Hunter Line and the Southern Highlands Line were cancelled or delayed with the Regional Bus and Train network in Victoria operated by V/Line having all lines suspended. [60] [194]
UK rail companies were also affected. Suburban rail services in the United Kingdom were heavily affected. [113] [195] Cab riders in London could not pay with credit or with debit cards and thus required cash. [68] [196] In Sweden [197] and Belgium, [198] tickets for public transport could not be sold, and Keolis Nederland experienced issues. [90] [107]
Many hospitals across North America paused non-urgent surgeries and visits. [199] Some affected hospitals, while remaining open, had limited, if any, access to patient records. [200] In the United States, Memorial Sloan Kettering Cancer Center postponed all procedures that required anaesthesia, the Mass General Brigham hospital system cancelled all non-emergency procedures and medical visits, [201] [202] and the Cincinnati Children's Hospital Medical Center was also affected. [203] University Health Network experienced technical issues in Canada, saying hospitals' clinical activity would continue but warning that appointments may be delayed. [185] A number of other Canadian hospitals faced difficulties, with Newfoundland and Labrador Health Services activating contingency plans as patient record systems were affected. [186] LabCorp [204] and Quest Diagnostics [205] were impacted by the outage.
Britain's National Health Service (NHS) said that the issues are "causing disruption in the majority of [English] GP practices", [88] with some of its services, such as GP surgeries, which rely on a software product called EMIS Web, unable to view and manage medical records, issue and manage prescriptions, or make appointments. [68] Manx Radio reported that GP surgeries were affected in the Isle of Man. [206] The London Ambulance Service experienced an unprecedented surge [55] in 999 and 111 calls following the outage, responding to 4,500 emergency calls by 17:00 ( BST). [207]
Two-thirds of Northern Ireland's general practices (GPs) were affected. At hospitals radiation therapy, bookings for operating theatres, and staff rosters are also affected. [196]
In Belgium, FPS Public Health said the outage disrupted new-patient admissions in two hospitals, which activated their emergency IT plans. [208] Two hospitals in Lübeck and Kiel, Germany, cancelled non-emergency operations. [68] The Spanish regional governments of Aragon, Basque Country, Castilla-La Mancha, Catalonia, and Galicia reported problems with their healthcare services. [209] Hospital Fernando Fonseca in Portugal reported problems, [210] while the Catholic Health system in New York experienced outages that caused delays in services. [203]
In the Netherlands, the outages affected two hospitals—the Scheperziekenhuis in Emmen and the Slingeland Ziekenhuis in de Achterhoek—and numerous emergency aid stations were also affected, including those in Emmen, Hoogeveen, and Stadskanaal. [211]
Systems in Wesley Hospital and St Andrews Hospital in Brisbane, Australia, were affected. [59]
The Central Health information system in Croatia was affected, although it was clarified that it was due to a concurrent issue tied with moving their servers to a new location. [212]
In Israel, Magen David Adom and its emergency service hotline was affected. Hospitals including Sheba Medical Center, Rambam Hospital and Laniado Hospital were experiencing problems that led to longer waiting times and delayed surgeries. [140]
The pharmaceutical company Krka in Slovenia suffered a full production outage and sent its workforce home. [213]
Numerous American TV stations were unable to broadcast because of the global outage. KSHB-TV, one of the affected stations, had to resort to airing national news via Scripps News. [69] ESPN was unable to air the morning editions of SportsCenter on the morning of the outage in America, instead airing ESPN Radio's Unsportsmanlike, simulcasting with ESPN2. [214] ESPN and ESPN2 later simulcasted Get Up! and First Take in place of SportsCenter, albeit without on-air graphics or b-roll. [215] [216] Various Paramount channels were also affected including Nicktoons (with its West Coast feed switching to an old emergency feed), TeenNick, BET Her, and most channels on the Pluto TV service. MeTV Toons was sent off the air for five and a half hours. Mercedes AMG PETRONAS F1 Team also suffered issues on the Friday of the Hungarian Grand Prix, with a Mercedes spokesperson confirming that the team had to manually address the problem on every computer it used. The issue also affected their engine customers, McLaren, Aston Martin and Williams. [217] Many video screens in New York City's Times Square turned off. [218]
When some companies let their employees go home early as a result of the incident, [219] the topic "Thank you Microsoft for an early vacation" momentarily became Weibo's most popular term. [220] Universal Studios Japan announced that they would not be selling tickets via ticket booths over the weekend due to the outage; however, tickets would still be sold online or via designated ticket sales sites. [69]
Vodafone experienced outages. [153] The issue affected the office laptops of DPG Media Belgium – which impacts JOE and QMusic Radio, banks, post services, and government agencies. [208] Telephone communication with the urban services in Antwerp were also affected. [221] The Centre for Cybersecurity Belgium stated that the impact in Belgium was limited. [208] Sky News was unable to broadcast live in the UK, [222] as was the BBC's CBBC, a free-to-air children's television channel. [68] Irish national broadcaster RTÉ said its newsroom was hit by "intermittent internet outages" with minimal impact to output. [223] [224] The Canadian Broadcasting Corporation was also impacted. [134]
Several French TV channels affected by the issues include TF1, TFX, LCI and Canal+ Group networks. [225] Phone and internet service provider Bouygues Telecom has also announced the unavailability of its customer service as a result of the outage. [226] The operations of the 2024 Summer Olympics, scheduled to start the following week in Paris, France, were also affected. The outage occurred a day after the Olympic Village opened and organisers were processing the arrivals of athletes and delegates. The organising committee said that a contingency plan was activated and that only the delivery of uniforms and accreditations were affected. [227] The incident slowed down the operations, with the accreditation desk at the press centre closed and security checks done manually using a list of names. [68] [228]
IT workers and the BPO industry were affected in the Philippines. [229] Numerous Singaporean companies, including SPH Media, Singtel, and M1, reported various levels of service difficulties throughout the day on 19 July. [71] [72]
Australian media firms affected by the issues include the ABC, SBS, Seven Network and Nine Network. [40] Ticketing at Docklands Stadium for Friday night's Australian Football League match between the Essendon Bombers and the Adelaide Crows was affected. [194]
Israel Post was affected [140] and Ukrainian Nova Poshta experienced outages. [153] In the US, UPS and FedEx were affected. [230]
Sim racing service iRacing was also affected by the outage in America. [231] Various Korean online games, like Black Desert Online, Ragnarok Online, and Ragnarok Origin shut down. [232]
Amazon Web Services, eBay, Google Cloud, Instagram, and Plenty of Fish were also affected. [145] [203]
Supermarkets in the Philippines were affected due to crashed POS systems. [233][ dubious – discuss] German supermarket chain Tegut closed some of its stores. [234] Customers have also experienced payment issues at Foodstuffs and Woolworths supermarkets in New Zealand. [63] British grocery chain Waitrose could only accept cash from customers. [235] Australian retailers and fast food chains are also hit by the outage, causing self-checkout and online order systems to be out of service. [236] Supermarkets affected included Woolworths and Coles. [60]
The international chain Starbucks's mobile application was limited to basic viewing of accounts made before the update, alongside ordering being limited to in-store purchases only, and cash register software crashes. [237]
In the United States, system issues caused by the incident forced sporting goods retailer Dick's Sporting Goods to close some of its stores and caused temporary outages to its website. [238] [239]
Convenience store chain 7-Eleven experienced issues at its Speedway branded locations still running on Speedway's legacy software using BlueCube and Radiant Site Manager dating from the days Speedway was owned by Marathon Petroleum Corporation, with some stores unable to accept credit or debit transactions while others were closed outright. [240] 7-Eleven, which was in the process of converting Speedway locations to its proprietary RIS software, [241] did not experience issues at Speedway locations that already switched to RIS, nor did it experience issues at standard 7-Eleven or Stripes Convenience Stores locations.[ citation needed]
In Norway, the pharmacy chain Apotek1 and the insurance company Tryg were compelled to suspend services; Vitusapotek and Boots were also affected. [242] [243] Beyond these disruptions, Norway experienced minimal impact owing to CrowdStrike's limited market share in the country. [244]
Amazon suffered disruption to its warehouse operations and internal software. An app used in Amazon warehouses to manage schedules and time-off requests called 'A to Z' was taken down by the outage and an internal service called 'Anytime Pay' became unavailable to employees. Operations were briefly halted at some sites, while Amazon's trucking operations were disrupted, with drivers saying a platform they use called 'Relay' suffered issues meaning they were briefly unable to pick up loads at warehouses. [245]
The outage affected terminal operations at DCT Gdańsk, a major container hub in the Baltic port of Gdańsk in Poland. [246] Shipping ports in the US were unaffected for the most part, although the Port of Houston (which handles the most foreign tonnage) closed briefly. [247]
In Sweden, the Malmberget mine was evacuated as a precaution. [248] Tickets for soccer games could not be sold. [249]
In the United States, security provider ADT was affected. [145]
In Germany, Tesla halted production at its Gigafactory Berlin-Brandenburg for about four hours. [250]
In a live interview on NBC's Today, CrowdStrike CEO George Kurtz apologised to the public. He said company leaders were "deeply sorry for the impact that we've caused to customers, to travelers, to anyone affected by this, including our companies". [251] CrowdStrike warned that malicious actors might try to pose as its staff or independent researchers claiming to help fix the problem. [252]
The Australian government held a national emergency meeting to address the outage. The National Coordination Mechanism was activated; Prime Minister Anthony Albanese said, "I understand Australians are concerned about the outage that is unfolding globally and affecting a wide range of services. My Government is working closely with the National Cyber Security Coordinator". [63] [253] He later said, "There is no impact to critical infrastructure, government services, or Triple-0 services at this stage". [254] Victorians were advised to call Triple-0 if a fire alarm sounds or smoke is detected, as some automatic alarms in buildings may not automatically call fire services due to the outage. [60]
United States president Joe Biden's administration was in touch with CrowdStrike to offer assistance, and on 23 July, Kurz was invited to appear before Congress to explain how the outage occurred and what Crowdstrike is doing to prevent future incidents. [69]
The UK government's COBR committee met to discuss the incident. [255]
India's Minister of Information and Technology Ashwini Vaishnav said that the government was in touch with Microsoft. The government's CERT-IN cybersecurity agency classified the incident as "critical". [256]
In Russia, the government noted that the sanctions and boycotts placed on Russia as a result of its invasion of Ukraine in 2022 had unintentionally shielded it from the outage. [257] Russia's Digital Communications Ministry said, "At the moment, the ministry has not received reports of system failures at Russian airports," and "The situation with Microsoft once again shows the importance of import substitution of foreign software, primarily at critical information infrastructure facilities." The Russian Federal Air Transport Agency confirmed that no domestic airlines were affected. [257] The Kremlin stated that its systems were working as normal. [257]
Cybersecurity consultant Troy Hunt called the incident the "largest IT outage in history" and adding: "This is basically what we were all worried about with Y2K, except it's actually happened this time". [258] [259] Slate described it as "Y2K Lite". [260]
Elon Musk—CEO of Tesla, X Corp, Neuralink, and SpaceX—posted on X that CrowdStrike has been "deleted from all our systems". [261]
AirAsia CEO Tony Fernandes demanded answers and compensation for millions of dollars in revenue he said the company had lost in the incident. [262]
Chinese cybersecurity companies such as 360 Security, QAX and Tencent took advantage of the CrowdStrike incident to promote their own software. [263]
News reporters have used the term "digital pandemic" to describe the outage. [264] [265] [266] [267]
Governments worldwide and cybersecurity agencies warned of digital phishing scams following the incident. Cyber criminals started sending phishing emails purporting to be CrowdStrike support and impersonating CrowdStrike staff in phone calls shortly afterward. [268] [269]
The outage raised questions about oligopoly and centralisation in the information technology sector. [270] [271] The majority of the world's computers use Microsoft Windows, creating a monoculture that reduces resiliency. [134] [271] Ciaran Martin, a cybersecurity expert, said, "This is a very, very uncomfortable illustration of the fragility of the world's core internet infrastructure". [134] Critical infrastructure expert Gregory Falco said, "Cybersecurity providers are part of this homogenous backbone of modern systems and are so core to how we operate that a glitch in their operations will have similar impacts to failures in systems that are household names". [272] Security experts suggested more redundancy to avoid single points of failure, [270] wider use of decentralised and heterogeneous federated systems, [271] and public anger at the failure of political leaders to regulate for diversity and competition. [134]
Experts speculate that the update was not put through routine patch management procedures (testing the update in a sandbox) to verify there were no problems. [273]
Mandating disclosure of breaches and vulnerabilities has also been suggested. [274] In an interview with Wired, cybersecurity consultant Jake Williams said that this outage has "shown why pushing updates without IT intervention is unsustainable," and that "people may now demand changes in this operating model." [275]
Microsoft blamed a 2009 antitrust agreement with the European Union that they said forced them to sustain low-level kernel access to third-party developers. [276] [277] [278] The document does not actually state Microsoft has to provide kernel-level access, just the same APIs used by its own security products. [277] The EU rejected the allegations. [279]
In Linux, it is possible to use eBPF instead of kernel modules to program this type of software. [280]
Since macOS Catalina (2019), this type of software can use the Endpoint Security Framework instead of kext, which has been gradually enforced. [281]
Technical staff will need to go and reboot each and every computer affected, which could be a monumental task.
The fixes we've seen so far mean that you have to physically go to every machine, which will take days.
It's going to need manual intervention on each system.
{{
cite news}}
: CS1 maint: url-status (
link)
In its last earnings report, Crowdstrike declared a total of nearly 24,000 customers.
CrowdStrike doesn't have to shell out anything more than a simple refund.
limit liability to "fees paid."
That means that if a company had a claim against CrowdStrike for the damage or lost revenue to its business, the most it could recover is just what it paid to CrowdStrike
It goes on to say that HK Express regrets to announce the cancellation of some 24 of its flights originally scheduled for 20 July.
Part of RTÉ's operation was affected by the issue with minimal impact to output.
The scale of the impact of the "blue screen" outage is still being uncovered, but airlines, offices and more are affected
{{
cite web}}
: CS1 maint: url-status (
link)
{{
cite web}}
: CS1 maint: url-status (
link)
'Digital pandemic', 'havoc' and 'meltdown' were some of the most common phrases in UK headlines after botched CrowdStrike software update
![]() | A request that this article title be changed is
under discussion. Please do not move this article until the discussion is closed. |
![]() Multiple
blue screens of death caused by a faulty software update on airport luggage conveyor belts at
LaGuardia Airport, New York City | |
Date | 19 July 2024 |
---|---|
Location | Worldwide |
Type | IT outage, computer crash |
Cause | Faulty CrowdStrike software update |
Outcome | ~8.5 million Microsoft Windows operating systems crash worldwide, causing global disruption of critical services |
On 19 July 2024, American cybersecurity company CrowdStrike distributed a faulty update to its security software that caused widespread problems with computers running Microsoft Windows. As a result, roughly 8.5 million systems crashed and were unable to properly restart [1] in what has been called the largest outage in the history of information technology [2] and "historic in scale". [3]
The outage disrupted daily life, businesses, and governments around the world. Many industries were affected—airlines, airports, banks, hotels, hospitals, manufacturing, stock markets, broadcasting, gas stations, retail stores, and more—as were governmental services, such as emergency services and websites. [4] [5] The worldwide financial damage has been estimated to be at least US$10 billion. [6] [7]
Within hours, the error was discovered and a fix was released, [8] but because affected computers had to be fixed manually, [9] outages continued to linger on many services. [10] [11]
CrowdStrike produces a suite of security software products for businesses, designed to protect computers from cyberattacks. The Falcon Sensor product, CrowdStrike's vulnerability scanner, installs an endpoint sensor at the operating system kernel level on individual computers to detect and prevent threats. [12] Patches are routinely distributed by CrowdStrike to its clients to enable their computers to address new threats. [13]
On 19 July at 04:09 UTC, CrowdStrike distributed a faulty configuration update for its Falcon sensor software running on Windows PCs and servers. The update caused machines to go into either a boot loop or boot recovery mode. [14] [15]
Almost immediately, Windows virtual machines on the Microsoft Azure cloud platform began rebooting and crashing, [16] and at 06:48 UTC, Google Compute Engine also reported the problem. The problem affected systems running Windows 10 and Windows 11 running the CrowdStrike Falcon software. [17] [14] Most personal Windows PCs were unaffected, as CrowdStrike's software is primarily used by organizations. [17] Computers running macOS and Linux were unaffected, as the problematic content file was only for Windows. [18] Because it was a content file, the update could not be delayed. [19] However, similar problems affected Linux distributions in April 2024. [20] [21]
CrowdStrike reverted the content update at 05:27 UTC, [22] and devices that booted after the revert were not affected. [23]
At 07:15 UTC, Google said that the CrowdStrike update was at fault. [24] Within hours, CrowdStrike CEO George Kurtz confirmed that CrowdStrike's faulty kernel configuration file update had caused the problem. [9] [8] At 09:45 UTC, Kurtz confirmed that the fix was deployed [25] [26] and that the problem was not the result of a cyberattack. [9] [27]
The day before the faulty update, the Azure platform had an outage that blocked some companies' access to their storage and to Microsoft 365 applications in Azure's Central United States region. [28] Microsoft said the 18 July incident was unrelated to the CrowdStrike problem, but that the two incidents compounded problems for these customers. [29]
An update to a configuration file which was responsible for screening named pipes, Channel File 291, caused an out-of-bounds memory read [30] in the Windows sensor client that resulted in an invalid page fault.
Affected machines could be restored by booting into
safe mode or the
Windows Recovery Environment and deleting any .sys
file beginning with C-00000291-
and with timestamp 0409 UTC in the
%windir%\
System32\
drivers\CrowdStrike\
directory.
[31] As this process must be done locally on each individual machine, it was "expected to take days" for affected businesses to restore all systems.
[32] Further, technical staff will have to reboot the affected computers individually with manual intervention on each system, which could be a "monumental task".
[33]
[34]
[35]
On devices with Windows' BitLocker disk encryption enabled, as corporations often do to increase security, fixing the problem was exacerbated because a recovery key could be required that was stored on a server that itself had crashed. [36] [37]
Some Microsoft Azure customers discovered that they could fix the problem by rebooting affected virtual machines up to 15 times, ideally while connected to Ethernet. [38] [22] Microsoft has also recommended restoring a backup from before 18 July to fix the issue. [39]
Outages were experienced worldwide, [40] [41] [42] reflecting the wide use of Microsoft Windows and CrowdStrike software by global corporations in numerous business sectors. [43] At the time of the incident, CrowdStrike said it had more than 24,000 customers, [44] including nearly 60% of Fortune 500 companies and more than half of the Fortune 1000. [45] [46] Microsoft estimates that 8.5 million devices were affected by the update. [47]
Widespread outages were immediately reported across multiple countries, with major global disturbances experienced by the general public sweeping from east to west from time zone to time zone. At 04:09 UTC on 19 July, the time when the faulty update was issued, it was the middle of the business day of Oceania and Asia, the early morning hours in Europe, and midnight in much of the Americas.
Some countries were less affected. China, which has striven toward self-sufficiency in IT, saw little impact to key services such as airlines and banks, although foreign businesses and luxury hotels in the country were affected. [48] Russia and Iran—both restricted by international sanctions from using the services of American high-tech companies—reported no disruptions. [49] [50]
Despite the losses companies have suffered, CrowdStrike was said to be only minimally liable for the damage or lost revenue caused. [51] The terms for CrowdStrike's Falcon software limits liability to 'fees paid', [52] so the maximum compensation an affected company could recover if this provision is enforceable are the fees that the company has paid to CrowdStrike. [53]
In the EU, it is possible that CrowdStrike will be held liable under a GDPR regulation related to the impact of security incidents on user data. The regulation is best known in relation to data leaks but also applies to data destruction. It is unclear whether temporary loss of access to data is enough to trigger liability, or whether GDPR applies to all incidents related to security or only unauthorised access. [54]
Globally 5,078 air flights, 4.6% of those scheduled that day, were cancelled. [55] [56]
Australian airlines Qantas, Virgin Australia, and Jetstar were affected. [57] [58] A Sydney Airport spokesperson said that the outage had affected some operations and that "there may be some delays throughout the evening". [59] Melbourne Airport saw check-in procedures disrupted; officials advised passengers to consult with their airlines. Canberra Airport, Darwin Airport, Adelaide Airport, Perth Airport, Hobart Airport, Launceston Airport, and Brisbane Airport were also affected. [41] [60] [61] [62] In New Zealand, Christchurch Airport was having problems. [63]
Hong Kong International Airport experienced delays during check-in, primarily for passengers of the local budget carrier Hong Kong Express, whose staff members used handwritten signs to direct passengers to check-in counters. [65] The Hong Kong Airport Authority activated the emergency response after airline websites and automatic check-in malfunctioned. Local airlines Cathay Pacific, Hong Kong Express, and Hong Kong Airlines's booking systems were unavailable. [66] HKExpress cancelled some flights on 20 July. [67] Jeju Air [68] and Spring Japan [68] experienced problems.[ clarification needed] Jetstar Japan had to cancel many flights (mostly domestic flights). [69] [70] Some of the self-check-in kiosks in Singapore Changi Airport were affected, delaying and forcing airlines to switch to manual check-in, and Singapore Airlines and Scoot reported various[ weasel words] levels of service difficulties throughout 19 July. [71] [72] Cebu Pacific and Philippines AirAsia flights were delayed. [73] [74] [75] Long queues formed at Ninoy Aquino International Airport. [76] In Taiwan, airline system disruptions were reported at Taoyuan International Airport. [77] [78] In Indonesia, disruptions were reported for the check-in systems of AirAsia and Citilink. [79] In Thailand, Thai AirAsia's reservation and check-in systems were affected. [80]
In India, the outage affected Indigo Airlines, Akasa Air, SpiceJet, and Vistara. Handwritten boarding passes were being issued during the outage. The Ministry of Civil Aviation asked and ordered the airlines as well as the airports to be compassionate while providing food as well as seats if need be. [81] [82] [83] [84] As of 18:14 IST (12:44 UTC), over 200 Indian flights were cancelled; IndiGo alone cancelled 192. [85] Only those airlines that had a heavy reliance on Microsoft Azure for all their services, from hosting websites and booking engines to revenue management systems and Departure Control Systems faced the impact. Air India said that none of its flights were cancelled or delayed due to the outage, attributing it to their robust cyber system. [86]
Prague Airport [60] [87] in Czechia, Budapest Airport [88] in Hungary, Bratislava Airport [89] in Slovakia, and Schiphol Airport [90] in the Netherlands experienced problems. Planes were not allowed to land at Zurich Airport. [68] Near Brussels, Charleroi Airport employees manually checked passengers in, but other software alleviated problems by 10:00 (UTC+2) and there were minimal delays. [91] ENAIRE's Aena, the Spanish national airport traffic control manager, mentioned an IT outage in their website and social media. [92] All Spanish airports reported disruptions. [93] Charles de Gaulle Airport and Orly Airport experienced check-in problems and suspended flights. [94] Poznań–Ławica Airport and Warsaw Chopin Airport experienced check-in disruptions. An emergency system was activated, but check-in processes were slower. [95] Berlin Brandenburg Airport announced that since around 07:00 (UTC+2), operational processes were affected by "IT problems at an external provider", and that they planned to stop flights until 8 UTC. [84] [96] While passenger handling continued with some restrictions, there were delays and airlines cancelled some flights. [97] Several airlines ( Eurowings, Ryanair, Vueling, and Turkish Airlines) in Hamburg Airport had to issue tickets by hand. [98] Croatian and Swedish air traffic control were also disrupted. [99] [100]
Swiss International Air Lines had 30% of flights grounded. [101] Lufthansa in Germany experienced problems with the "profile and booking retrieval" features of their website. [102] Ryanair's booking and check-in services were unavailable and the airline was "forced to cancel a small number of flights", advising passengers to arrive at airports at least three hours before departure. [103] [104] Wizz Air said the outage put its online services offline. [105] [106] Dutch airline KLM suspended most operations, announcing that flight handling is impossible with the issue, [68] and Transavia Airlines experienced problems. [90] [107] Finnair reported that they were having trouble sending emails and SMS messages to customers. [108] In Greece, citizens and tourists saw delays at major airports, notably at Athens International Airport and at Heraklion International Airport. This disruption, occurring at the peak of the tourist season, resulted in chaotic scenes as passengers were forced to wait for hours for their flights. Contributing factors included severe staff shortages and new schedules. [109] [110] In Heraklion, eight flights were problematic. The airport's chief, George Pliakas, indicated that flights were being manually arranged to manage the disruption, but the influx of arriving flights strained the system. [111] [112]
Several UK airports had problems, including Edinburgh Airport, whose departure boards froze, [113] and Gatwick Airport, where automatic barcode scanning stopped working and had to be checked manually. [68] Amadeus, which manages UK baggage at Heathrow, said they were affected by the IT outage. [114] Disruption to flights was anticipated in the Isle of Man, particularly to and from the UK, but ultimately minimal. [115]
Tunisia experienced temporary airport disruptions. [116] Turkish Airlines cancelled some of its flights due to the outage. [117]
A ground stop was issued by United, Delta, and American Airlines. Flights in the air continued flying, but no new flights were to take off. [15] [118] Allegiant Air were also grounded by the outage according to the Federal Aviation Administration (FAA). [5] [119] About 1,500 flights were cancelled in the United States on 19 July due to the outage. [69] While American Airlines, United and Allegiant recovered relatively quickly after Friday, Delta Air Lines experienced an operational meltdown which continued through the weekend. [120] [121]
Delta Air Lines was by far the hardest hit of the US major airlines, with over 1,200 flights cancelled on Friday, 19 July, and thousands more cancelled on Saturday and Sunday. [121] [122] Thousands of stranded travelers were forced to spend the night sleeping on the floor in the terminals at Delta's largest hub, Hartsfield–Jackson Atlanta International Airport (which is also the busiest airport in the world by passenger traffic). [122] Metro Atlanta hotels and rental car companies were overwhelmed by the scale of the crisis, leaving travelers no other option but to stay in the airport. [123] One traveler attempting to return home to Tampa (after giving up on reaching California) reported that Amtrak was charging $1,000 for a one-way train ticket from Atlanta to Tampa. [124] Visibly distraught passengers with nowhere to go were seen trying to sleep in the airport on hard linoleum floors without blankets or food. [122] Without warning, Delta immediately banned unaccompanied minors on its flights through the end of 23 July. [125] This imposed considerable hardship on parents who had been counting on that service to avoid having to fly with their children to escort them to summer camp or a relative's house, along with minors who had expected to use that service to fly home. [125]
On 20 July, Delta cancelled over 1,400 flights. [126]
On 21 July, Delta cancelled over 1,300 flights. [127] With so many passengers still stuck in Hartsfield–Jackson after two consecutive nights, the airport implemented traveler assistance measures such as a "concessions crisis plan" and a plan to reunite passengers with their checked baggage. [126] However, passengers in Atlanta on that date continued to report "jam-packed" conditions and witnessing "heartbreaking" scenes in the terminals. [128]
On 21 July, Delta CEO Ed Bastian apologized to customers in a statement and revealed that the outage had affected one of Delta's crew-tracking software programs. [127] According to Bastian, the software was "unable to effectively process the unprecedented number of changes triggered by the system shutdown". [129] According to Delta CIO Rahul Samant, the program was restored and brought back online at around 11 a.m. on 19 July, but was overwhelmed by the backlog of updates awaiting processing and had been trying to catch up ever since. [128] After the ground stop left too many crew members in the wrong places, Delta was struggling to assemble enough pilots and flight attendants at airport gates to operate scheduled flights. [129] Many flights were repeatedly delayed and finally cancelled because the one or two crew members who made it to the gate for a particular flight kept hitting their legal flight time limit before the airline could finish fully staffing the flight, and this caused the crisis to snowball as those crew and their aircraft were now in the wrong place for the following day's flights. [129] A similar issue had been blamed for the 2022 Southwest Airlines scheduling crisis. [127] That same day, US Secretary of Transportation Pete Buttigieg said on social media that the US Department of Transportation had received hundreds of complaints about Delta, and reminded the airline of its legal obligations to affected passengers. [127]
On 22 July, Delta cancelled over 1,200 flights. [128] On 23 July, the Department of Transportation announced the launch of a formal investigation into Delta's treatment of passengers. [128] Delta promised to cooperate but stated that it was focused on its recovery. [128] Senator Maria Cantwell in her capacity as chair of the Senate Committee on Commerce, Science, and Transportation wrote to Bastian to express her concern for Delta passengers. [128]
United Airlines' smaller number of cancellations had a significant impact on its hubs. For example, San Mateo County hotels around San Francisco International Airport rapidly filled up with travelers on 19 July. [130] Guests reported difficulty with checking into the local Marriott hotel because Marriott International was also recovering from the outage. [130]
Southwest Airlines (the third largest US major airline by domestic passengers) was entirely unaffected. [131] A Southwest spokesperson confirmed that the airline had seen no impact from the CrowdStrike outage but refused to confirm speculation among aviation industry analysts that it had been shielded by its notoriously outdated software.
The flight delays meant that many people who had traveled to the 2024 Republican National Convention—which concluded the day the outages started—were stuck in the convention's host city of Milwaukee, Wisconsin. [132] [133]
Montréal–Trudeau International Airport and Toronto Pearson International Airport were affected in Canada, and Porter Airlines cancelled all flights. [134] Vancouver International Airport was also reportedly affected in Canada, although it was unclear whether this was directly related to the global outages. [135]
Microsoft and CrowdStrike stocks fell as a result of the outage. CrowdStrike's stock fell over 11% on 19 July though Microsoft was down less than 1%. [136] [5]
Banks that were affected included Chase, Bank of America, Wells Fargo, U.S. Bank, Capital One and Charles Schwab in the US, [137] RBC and TD Bank in Canada, [138] [139] Capitec Bank and other South African banks, [88] several Israeli banks, [140] and several banks in the Philippines, such as RCBC, Metrobank, LandBank, BDO, UnionBank, BPI, and PNB. [141] [142] E-wallets such as Maya and GCash also experienced problems in the Philippines. [143] The website and mobile banking application of DenizBank in Turkey could not be accessed. [144] Visa was affected. [145] Numerous Singaporean companies, including Singapore Exchange (SGX) and DBS Bank, reported various levels of service difficulties throughout 19 July. [71] [72]
In India, the Reserve Bank of India said that only 10 banks and NBFCs were affected by the outage; [146] [147] few banks use CrowdStrike tools and many banks' critical systems do not run on the cloud. NSE, BSE, [147] and India's largest bank, State Bank of India, said they were unaffected. [148]
In Brazil, Bradesco Bank confirmed it was affected. During the morning customers were able to login, but at 12:00 UTC the bank disabled the login button. [149]
New Zealand banks ASB and Kiwibank, Australian banks Westpac [150] and ANZ had problems. [63] Apps of Australian banks NAB, Westpac, ANZ, Commonwealth Bank, Bendigo Bank, and Suncorp were affected. [60]
The London Stock Exchange, while operating normally, was unable to push news updates to its website. [68] English gambling company Ladbrokes Coral and English supermarket chain Morrisons also reported problems. [68] Polish banks, including Santander Bank Polska, ING Bank Śląski and mBank, encountered issues related to the outage. Santander BP's helpline, video, and chat services were affected. PKO Bank Polski clarified that its iPKO and IKO services were stable, but other banks faced difficulties. [151] In Finland, OP Financial Group reported minor disruptions on investment partner and stock savings accounts. [152] Sense Bank in Ukraine experienced outages due to the update. [153]
Paraguayan banks Ueno and Banco Continental were affected; their customers were unable to log in. [154]
The United States Department of Homeland Security, NASA, Federal Trade Commission, National Nuclear Security Administration, Department of Justice, and Department of Education were affected, and the Department of the Treasury and Department of State reported minor disruptions. The Department of Veterans Affairs and Department of Energy experienced disruptions, but it is not currently known if they are related to the incident. [155] DMV agencies for the states of Georgia, Kansas, Missouri, North Carolina, Tennessee, and the District of Columbia were affected. [156] [157] [158] Ted Wheeler, the mayor of Portland, Oregon, declared the outages to be a city emergency. [159] [160] Election and voting registration databases in Arizona, South Dakota, Texas and the state of Washington were affected. [161] The website for the city of Sioux Falls, South Dakota, went down. [162]
In the United States, there were outages in 911 service or disruptions in 911 call centres' operation in some parts of Alaska, [163] Arizona, [164] Florida, [165] Iowa, [166] Indiana, [167] Kansas, [168] Michigan, [169] Minnesota, [170] New York, [171] Ohio, [172] Oregon, [173] Pennsylvania, [174] and Virginia. [166] 911 was down for all of New Hampshire. [175] [176] In addition, Alaska was experiencing issues with non-emergency call centers. [176] Many call centers switched to working backup systems. [161]
The CM/ECF and PACER computer systems used by the US federal courts were unaffected. [177] However, several state courts reported problems with their computer systems, including courts in Alaska, California, Delaware, Idaho, Kansas, Maryland, Massachusetts, Michigan, Nevada, New York, and Pennsylvania. [177] In New York City, courts and correctional facilities were disrupted, delaying a hearing in the trial of Harvey Weinstein for sex offenses. [178]
Government websites in the Philippines, such as the website of the House of Representatives of the Philippines, were down due to the outage. [179]
In Canada, services in Toronto were affected, [180] and Canada Child Benefit payments were delayed. [181] [182] New Zealand Parliament had problems. [84] Sunshine Coast Council was one of several councils affected in Australia. [60] The National Security Authority spokesman confirmed several institutions in Slovakia were affected. [183]
The fire department in Copenhagen, Denmark, was unable to receive automatic fire alerts from buildings. [184]
Traffic disruptions were reported at the US–Canada border, [185] [181] [139] including long delays at the Ambassador Bridge and Detroit–Windsor tunnel between Ontario and Michigan. [180] The Canada Border Services Agency blamed a partial outage of its telephone reporting system which was later resolved. [134] [185] There were long delays and police advised motorists to avoid the area. [186] The Washington Metro Area Transit Authority suffered minor service delays in the early morning in America; their website/live tracking was unavailable until around 9:30 am on 19 July. [187] The Massachusetts Bay Transportation Authority in Boston, as well as the Metropolitan Transportation Authority in New York, lost vehicle tracking and arrival notices for passengers. [188] Most of North American freight and passenger train operators went largely unaffected aside from some technical issues within Union Pacific and Canadian Pacific Kansas City, Amtrak was mostly unaffected aside from issues with credit card processing during the morning. [189]
Malaysia's railway operator, Keretapi Tanah Melayu, confirmed that its KITS ticketing system was experiencing technical issues. [190] Transport for Ireland said its apps were down due to the outage. [191] Ireland's Road Safety Authority said it was experiencing "significant disruption" to its National Car Test (NCT) centres. [192] In Singapore, the entrance and exit gantries of over 185 car parks managed by the Housing and Development Board (HDB) were affected. [193]
Fuel stations have also been affected in Australia, with people stuck at fuel pumps unable to pay for petrol because payment systems were not working. [59] Auckland Transport's HOP card in New Zealand had problems. [63] Australian freight train operator Aurizon was affected. [60] Regional trains in New South Wales, Australia on the Hunter Line and the Southern Highlands Line were cancelled or delayed with the Regional Bus and Train network in Victoria operated by V/Line having all lines suspended. [60] [194]
UK rail companies were also affected. Suburban rail services in the United Kingdom were heavily affected. [113] [195] Cab riders in London could not pay with credit or with debit cards and thus required cash. [68] [196] In Sweden [197] and Belgium, [198] tickets for public transport could not be sold, and Keolis Nederland experienced issues. [90] [107]
Many hospitals across North America paused non-urgent surgeries and visits. [199] Some affected hospitals, while remaining open, had limited, if any, access to patient records. [200] In the United States, Memorial Sloan Kettering Cancer Center postponed all procedures that required anaesthesia, the Mass General Brigham hospital system cancelled all non-emergency procedures and medical visits, [201] [202] and the Cincinnati Children's Hospital Medical Center was also affected. [203] University Health Network experienced technical issues in Canada, saying hospitals' clinical activity would continue but warning that appointments may be delayed. [185] A number of other Canadian hospitals faced difficulties, with Newfoundland and Labrador Health Services activating contingency plans as patient record systems were affected. [186] LabCorp [204] and Quest Diagnostics [205] were impacted by the outage.
Britain's National Health Service (NHS) said that the issues are "causing disruption in the majority of [English] GP practices", [88] with some of its services, such as GP surgeries, which rely on a software product called EMIS Web, unable to view and manage medical records, issue and manage prescriptions, or make appointments. [68] Manx Radio reported that GP surgeries were affected in the Isle of Man. [206] The London Ambulance Service experienced an unprecedented surge [55] in 999 and 111 calls following the outage, responding to 4,500 emergency calls by 17:00 ( BST). [207]
Two-thirds of Northern Ireland's general practices (GPs) were affected. At hospitals radiation therapy, bookings for operating theatres, and staff rosters are also affected. [196]
In Belgium, FPS Public Health said the outage disrupted new-patient admissions in two hospitals, which activated their emergency IT plans. [208] Two hospitals in Lübeck and Kiel, Germany, cancelled non-emergency operations. [68] The Spanish regional governments of Aragon, Basque Country, Castilla-La Mancha, Catalonia, and Galicia reported problems with their healthcare services. [209] Hospital Fernando Fonseca in Portugal reported problems, [210] while the Catholic Health system in New York experienced outages that caused delays in services. [203]
In the Netherlands, the outages affected two hospitals—the Scheperziekenhuis in Emmen and the Slingeland Ziekenhuis in de Achterhoek—and numerous emergency aid stations were also affected, including those in Emmen, Hoogeveen, and Stadskanaal. [211]
Systems in Wesley Hospital and St Andrews Hospital in Brisbane, Australia, were affected. [59]
The Central Health information system in Croatia was affected, although it was clarified that it was due to a concurrent issue tied with moving their servers to a new location. [212]
In Israel, Magen David Adom and its emergency service hotline was affected. Hospitals including Sheba Medical Center, Rambam Hospital and Laniado Hospital were experiencing problems that led to longer waiting times and delayed surgeries. [140]
The pharmaceutical company Krka in Slovenia suffered a full production outage and sent its workforce home. [213]
Numerous American TV stations were unable to broadcast because of the global outage. KSHB-TV, one of the affected stations, had to resort to airing national news via Scripps News. [69] ESPN was unable to air the morning editions of SportsCenter on the morning of the outage in America, instead airing ESPN Radio's Unsportsmanlike, simulcasting with ESPN2. [214] ESPN and ESPN2 later simulcasted Get Up! and First Take in place of SportsCenter, albeit without on-air graphics or b-roll. [215] [216] Various Paramount channels were also affected including Nicktoons (with its West Coast feed switching to an old emergency feed), TeenNick, BET Her, and most channels on the Pluto TV service. MeTV Toons was sent off the air for five and a half hours. Mercedes AMG PETRONAS F1 Team also suffered issues on the Friday of the Hungarian Grand Prix, with a Mercedes spokesperson confirming that the team had to manually address the problem on every computer it used. The issue also affected their engine customers, McLaren, Aston Martin and Williams. [217] Many video screens in New York City's Times Square turned off. [218]
When some companies let their employees go home early as a result of the incident, [219] the topic "Thank you Microsoft for an early vacation" momentarily became Weibo's most popular term. [220] Universal Studios Japan announced that they would not be selling tickets via ticket booths over the weekend due to the outage; however, tickets would still be sold online or via designated ticket sales sites. [69]
Vodafone experienced outages. [153] The issue affected the office laptops of DPG Media Belgium – which impacts JOE and QMusic Radio, banks, post services, and government agencies. [208] Telephone communication with the urban services in Antwerp were also affected. [221] The Centre for Cybersecurity Belgium stated that the impact in Belgium was limited. [208] Sky News was unable to broadcast live in the UK, [222] as was the BBC's CBBC, a free-to-air children's television channel. [68] Irish national broadcaster RTÉ said its newsroom was hit by "intermittent internet outages" with minimal impact to output. [223] [224] The Canadian Broadcasting Corporation was also impacted. [134]
Several French TV channels affected by the issues include TF1, TFX, LCI and Canal+ Group networks. [225] Phone and internet service provider Bouygues Telecom has also announced the unavailability of its customer service as a result of the outage. [226] The operations of the 2024 Summer Olympics, scheduled to start the following week in Paris, France, were also affected. The outage occurred a day after the Olympic Village opened and organisers were processing the arrivals of athletes and delegates. The organising committee said that a contingency plan was activated and that only the delivery of uniforms and accreditations were affected. [227] The incident slowed down the operations, with the accreditation desk at the press centre closed and security checks done manually using a list of names. [68] [228]
IT workers and the BPO industry were affected in the Philippines. [229] Numerous Singaporean companies, including SPH Media, Singtel, and M1, reported various levels of service difficulties throughout the day on 19 July. [71] [72]
Australian media firms affected by the issues include the ABC, SBS, Seven Network and Nine Network. [40] Ticketing at Docklands Stadium for Friday night's Australian Football League match between the Essendon Bombers and the Adelaide Crows was affected. [194]
Israel Post was affected [140] and Ukrainian Nova Poshta experienced outages. [153] In the US, UPS and FedEx were affected. [230]
Sim racing service iRacing was also affected by the outage in America. [231] Various Korean online games, like Black Desert Online, Ragnarok Online, and Ragnarok Origin shut down. [232]
Amazon Web Services, eBay, Google Cloud, Instagram, and Plenty of Fish were also affected. [145] [203]
Supermarkets in the Philippines were affected due to crashed POS systems. [233][ dubious – discuss] German supermarket chain Tegut closed some of its stores. [234] Customers have also experienced payment issues at Foodstuffs and Woolworths supermarkets in New Zealand. [63] British grocery chain Waitrose could only accept cash from customers. [235] Australian retailers and fast food chains are also hit by the outage, causing self-checkout and online order systems to be out of service. [236] Supermarkets affected included Woolworths and Coles. [60]
The international chain Starbucks's mobile application was limited to basic viewing of accounts made before the update, alongside ordering being limited to in-store purchases only, and cash register software crashes. [237]
In the United States, system issues caused by the incident forced sporting goods retailer Dick's Sporting Goods to close some of its stores and caused temporary outages to its website. [238] [239]
Convenience store chain 7-Eleven experienced issues at its Speedway branded locations still running on Speedway's legacy software using BlueCube and Radiant Site Manager dating from the days Speedway was owned by Marathon Petroleum Corporation, with some stores unable to accept credit or debit transactions while others were closed outright. [240] 7-Eleven, which was in the process of converting Speedway locations to its proprietary RIS software, [241] did not experience issues at Speedway locations that already switched to RIS, nor did it experience issues at standard 7-Eleven or Stripes Convenience Stores locations.[ citation needed]
In Norway, the pharmacy chain Apotek1 and the insurance company Tryg were compelled to suspend services; Vitusapotek and Boots were also affected. [242] [243] Beyond these disruptions, Norway experienced minimal impact owing to CrowdStrike's limited market share in the country. [244]
Amazon suffered disruption to its warehouse operations and internal software. An app used in Amazon warehouses to manage schedules and time-off requests called 'A to Z' was taken down by the outage and an internal service called 'Anytime Pay' became unavailable to employees. Operations were briefly halted at some sites, while Amazon's trucking operations were disrupted, with drivers saying a platform they use called 'Relay' suffered issues meaning they were briefly unable to pick up loads at warehouses. [245]
The outage affected terminal operations at DCT Gdańsk, a major container hub in the Baltic port of Gdańsk in Poland. [246] Shipping ports in the US were unaffected for the most part, although the Port of Houston (which handles the most foreign tonnage) closed briefly. [247]
In Sweden, the Malmberget mine was evacuated as a precaution. [248] Tickets for soccer games could not be sold. [249]
In the United States, security provider ADT was affected. [145]
In Germany, Tesla halted production at its Gigafactory Berlin-Brandenburg for about four hours. [250]
In a live interview on NBC's Today, CrowdStrike CEO George Kurtz apologised to the public. He said company leaders were "deeply sorry for the impact that we've caused to customers, to travelers, to anyone affected by this, including our companies". [251] CrowdStrike warned that malicious actors might try to pose as its staff or independent researchers claiming to help fix the problem. [252]
The Australian government held a national emergency meeting to address the outage. The National Coordination Mechanism was activated; Prime Minister Anthony Albanese said, "I understand Australians are concerned about the outage that is unfolding globally and affecting a wide range of services. My Government is working closely with the National Cyber Security Coordinator". [63] [253] He later said, "There is no impact to critical infrastructure, government services, or Triple-0 services at this stage". [254] Victorians were advised to call Triple-0 if a fire alarm sounds or smoke is detected, as some automatic alarms in buildings may not automatically call fire services due to the outage. [60]
United States president Joe Biden's administration was in touch with CrowdStrike to offer assistance, and on 23 July, Kurz was invited to appear before Congress to explain how the outage occurred and what Crowdstrike is doing to prevent future incidents. [69]
The UK government's COBR committee met to discuss the incident. [255]
India's Minister of Information and Technology Ashwini Vaishnav said that the government was in touch with Microsoft. The government's CERT-IN cybersecurity agency classified the incident as "critical". [256]
In Russia, the government noted that the sanctions and boycotts placed on Russia as a result of its invasion of Ukraine in 2022 had unintentionally shielded it from the outage. [257] Russia's Digital Communications Ministry said, "At the moment, the ministry has not received reports of system failures at Russian airports," and "The situation with Microsoft once again shows the importance of import substitution of foreign software, primarily at critical information infrastructure facilities." The Russian Federal Air Transport Agency confirmed that no domestic airlines were affected. [257] The Kremlin stated that its systems were working as normal. [257]
Cybersecurity consultant Troy Hunt called the incident the "largest IT outage in history" and adding: "This is basically what we were all worried about with Y2K, except it's actually happened this time". [258] [259] Slate described it as "Y2K Lite". [260]
Elon Musk—CEO of Tesla, X Corp, Neuralink, and SpaceX—posted on X that CrowdStrike has been "deleted from all our systems". [261]
AirAsia CEO Tony Fernandes demanded answers and compensation for millions of dollars in revenue he said the company had lost in the incident. [262]
Chinese cybersecurity companies such as 360 Security, QAX and Tencent took advantage of the CrowdStrike incident to promote their own software. [263]
News reporters have used the term "digital pandemic" to describe the outage. [264] [265] [266] [267]
Governments worldwide and cybersecurity agencies warned of digital phishing scams following the incident. Cyber criminals started sending phishing emails purporting to be CrowdStrike support and impersonating CrowdStrike staff in phone calls shortly afterward. [268] [269]
The outage raised questions about oligopoly and centralisation in the information technology sector. [270] [271] The majority of the world's computers use Microsoft Windows, creating a monoculture that reduces resiliency. [134] [271] Ciaran Martin, a cybersecurity expert, said, "This is a very, very uncomfortable illustration of the fragility of the world's core internet infrastructure". [134] Critical infrastructure expert Gregory Falco said, "Cybersecurity providers are part of this homogenous backbone of modern systems and are so core to how we operate that a glitch in their operations will have similar impacts to failures in systems that are household names". [272] Security experts suggested more redundancy to avoid single points of failure, [270] wider use of decentralised and heterogeneous federated systems, [271] and public anger at the failure of political leaders to regulate for diversity and competition. [134]
Experts speculate that the update was not put through routine patch management procedures (testing the update in a sandbox) to verify there were no problems. [273]
Mandating disclosure of breaches and vulnerabilities has also been suggested. [274] In an interview with Wired, cybersecurity consultant Jake Williams said that this outage has "shown why pushing updates without IT intervention is unsustainable," and that "people may now demand changes in this operating model." [275]
Microsoft blamed a 2009 antitrust agreement with the European Union that they said forced them to sustain low-level kernel access to third-party developers. [276] [277] [278] The document does not actually state Microsoft has to provide kernel-level access, just the same APIs used by its own security products. [277] The EU rejected the allegations. [279]
In Linux, it is possible to use eBPF instead of kernel modules to program this type of software. [280]
Since macOS Catalina (2019), this type of software can use the Endpoint Security Framework instead of kext, which has been gradually enforced. [281]
Technical staff will need to go and reboot each and every computer affected, which could be a monumental task.
The fixes we've seen so far mean that you have to physically go to every machine, which will take days.
It's going to need manual intervention on each system.
{{
cite news}}
: CS1 maint: url-status (
link)
In its last earnings report, Crowdstrike declared a total of nearly 24,000 customers.
CrowdStrike doesn't have to shell out anything more than a simple refund.
limit liability to "fees paid."
That means that if a company had a claim against CrowdStrike for the damage or lost revenue to its business, the most it could recover is just what it paid to CrowdStrike
It goes on to say that HK Express regrets to announce the cancellation of some 24 of its flights originally scheduled for 20 July.
Part of RTÉ's operation was affected by the issue with minimal impact to output.
The scale of the impact of the "blue screen" outage is still being uncovered, but airlines, offices and more are affected
{{
cite web}}
: CS1 maint: url-status (
link)
{{
cite web}}
: CS1 maint: url-status (
link)
'Digital pandemic', 'havoc' and 'meltdown' were some of the most common phrases in UK headlines after botched CrowdStrike software update