A fast-moving conveyor belt transmits a package through a scanning machine en route to a delivery truck during operations on Cyber Monday at the Amazon Fulfillment Center in Robbinsville, New Jersey, U.S., November 29, 2021.
Facebook Facebook logo Subscribe to Facebook to connect with Mike Segar Reuters
Robotic vacuum cleaners cannot be called. Whole Foods orders are suddenly canceled. Parts of Amazon’s massive retail operations have slowed to a standstill.
Amazon Web Services, the leading provider of cloud infrastructure technology for businesses large and small, hit a historic, hours-long shutdown on Tuesday. Popular websites and frequently used services have been disabled offline, angering users and emphasizing the seriousness of the problems that can arise from having too much technology-based economic activity by few vendors.
AWS controlled 33% of the global cloud infrastructure market in the second quarter, according to Synergy Research Group, followed by Microsoft with 20% and Google with 10%. Revenue at AWS jumped 39% in the prior year third quarter to $16.1 billion, outpacing the 15% growth across Amazon.
Tuesday’s blackout began around 11 a.m. ET and was mostly resolved by Tuesday night. Amazon confirmed that service issues in AWS’ main US-East-1 region, located in Northern Virginia, were causing problems for its warehouse and delivery network. The company did not mention the reason for the outage.
Amazon said its fulfillment center and deliveries have stopped in some US pockets. The power outage stopped internal apps used to check packages and load delivery routes, according to workers’ posts to Facebook groups and a notice sent to drivers seen by CNBC.
Workers were asked to stand alert in break rooms and loading areas, and an Amazon driver tweeted a video of his co-worker performing karaoke at a warehouse.
Whole Foods, which was acquired by Amazon for $13.7 billion in 2017, has canceled orders from some users in affected areas, offering refunds as consolation. Amazon Flex drivers, contractors who make deliveries using their own vehicles, were promised payment after they were sent home because shifts were not available, according to a notice from Amazon.
AWS snafu has paralyzed Amazon’s retail operations at a particularly inconvenient time. The company was in the middle of high season, when it was hit by a flurry of requests from holiday shoppers. Third-party merchants, who make up more than half of the retail volume sold on Amazon, depend on a few weeks at the end of the year for a large percentage of their annual sales.
Joe Stefani, an Amazon seller in Chicago, said his company, Desert Cactus, was unable to get stock in the company’s warehouses because of the outage. Stephanie said Amazon handles 90% of his company’s orders, and ships products to customers from its fulfillment centers.
Sellers like Stephanie haven’t been able to access Seller Central, an internal system Amazon uses to manage customer orders. This means that Stephanie was unable to print the required shipping labels for any shipments sent to Amazon warehouses.
“We couldn’t send at least 10,000 to 12,000 items,” Stephanie said, including NBA and National Hockey League merchandise. “It will end up costing us money in the long run.”
Web services and other major infrastructure companies have seen major outages this year. Fastly, whose technology helps companies speed up the delivery of digital content to consumers, faced an outage in June that disrupted major websites including Amazon, The New York Times and Hulu. In October, Facebook experienced its worst outage since 2008 due to a configuration issue.
Amazon has experienced its own turmoil in the recent past. AWS experienced an outage in November 2020, when issues with a service called Kinesis brought down a handful of websites. This time the damage was more widespread, affecting businesses of all shapes and sizes.
Roombas, smart cat litter boxes taken offline
Evan Coleman knew something was wrong when he couldn’t load an app connected to his self-cleaning cat litter box, which had just arrived in the mail.
Coleman, the lead software engineer for wedding planning service The Knot, logged on to Twitter and saw a series of posts about the AWS outage. Then note that the app-controlled ceiling fan and internet-connected cat feeder also don’t work.
Fortunately for Coleman and his two cats, Leo and Luna, the feeder can still dispense food manually.
“The cats didn’t go hungry,” he said.
Evan Coleman, a software engineer, said the online cat litter box has been shut down due to an AWS outage. Fortunately, his cats Leo (left) and Luna (right) weren’t starving.
Steve Peters, an experienced designer in California, had a different kind of mess on his hands. After breakfast on Tuesday morning, Peters summoned the iRobot Roomba broom to clean his girl.
Roomba is based on AWS.
“I was forced to dig into the closet and find a dust bowl because I was screaming so loud,” Peters said. “I’m glad I don’t have a fridge connected to the internet.”
The outage also disrupted Canvas, an online learning platform with more than 30 million users, as well as Respondus’s LockDown Browser, a test monitoring service that blocks certain web browsing functions while students take an exam.
The timing was as bad for educational institutions as it was for retail, with many college students across the country in the middle of finals week. Chantal Lamorell, an assistant professor at Santa Ana College in California, said she had to postpone an exam for a childhood development class that was supposed to be taken online Tuesday morning.
Ananai Arora, a computer science student at Arizona State University, said he does not have access to the materials needed to study for an upcoming exam. On Tuesday, he received a bunch of Discord and Snapchat messages from friends at other universities with similar issues.
While outages like Tuesday appear to be catastrophic right now and could have major repercussions for businesses and organizations that depend on the service, such disruptions are rare, said Karl Malamud, a Bay Area-based Internet Archive technologist and specialist.
It took Amazon nearly nine hours to solve the problems — an impressive feat given the size and scope of AWS, Malamud said, adding that AWS and other major cloud services are generally very reliable.
“I bet all hands are on board,” Malamud said.
He watches: How Amazon beats supply chain chaos with its own containers, charter ships and long-haul planes