It was Amazon Simple Storage Service, better known as S3, in the Northern Virginia region called US-EAST-1. At 9:37 a.m. Pacific Time on 28 February 2017, an authorized S3 team member entered one command input incorrectly while investigating a slow billing process, causing more servers to be removed than intended. According to Amazon’s official post-mortem, the GET, LIST and DELETE APIs were fully restored after three hours and 41 minutes, while complete S3 operations returned after four hours and 17 minutes.
Amazon described the initiating error plainly: “Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended.” The command came from an established operational playbook and was supposed to remove only a small number of servers supporting an S3 billing subsystem.
The unexpectedly large removal also took capacity away from two much more important systems. The index subsystem tracked the metadata and location of every S3 object in the region, while the placement subsystem decided where newly uploaded objects would be stored. Without enough capacity in those systems, S3 could not process its normal requests.
Why one region mattered so much
AWS divides its infrastructure into geographic regions, with each region containing multiple isolated Availability Zones. Those zones protect against many local failures, but services and customers can still have dependencies that exist at the regional level, as the AWS infrastructure documentation explains.
By 2017, US-EAST-1 was one of AWS’s oldest and most heavily used regions. Many businesses had placed applications, storage buckets and supporting services there, and Wired described the eastern region as Amazon’s most popular. A disruption confined to Northern Virginia could therefore become visible around the world.
S3 itself had been accumulating that importance for more than a decade. Amazon launched the service on 14 March 2006 as a way for developers to store and retrieve data without building their own storage systems. Four years before the outage, AWS reported that S3 already contained more than two trillion objects and regularly peaked above 1.1 million requests per second.
A company did not need to run its entire website from S3 to be affected. A page might obtain its main HTML from one system while loading images, scripts, attachments or configuration files from an S3 bucket. When those supporting files disappeared, the service could look broken even if some of its servers remained online.
The cascade took longer than expected
Removing so much capacity forced both the index and placement subsystems into full restarts. While those restarts were under way, S3 in US-EAST-1 could not service normal API requests.
The recovery revealed a second weakness that had been hidden by years of otherwise successful operation. Amazon said the two subsystems had not been completely restarted in its larger regions for many years, while S3 had grown enormously during the same period. Restarting the systems and validating the integrity of their metadata consequently took longer than the team expected.
The clock began at 9:37 a.m. Pacific Time, when the incorrect input was executed. By 12:26 p.m., the index subsystem had recovered enough capacity to begin handling GET, LIST and DELETE requests again. That was the first meaningful sign of recovery, but normal service had not yet returned.
At 1:18 p.m., the index subsystem was fully recovered and those three APIs were functioning normally. PUT requests still depended on the placement subsystem, which completed its own recovery at 1:54 p.m. Only then did Amazon describe S3 as operating normally again.
The damage also spread to AWS services that depended on S3 inside the same region. New EC2 instance launches, some EBS operations, Lambda and the S3 console were among the affected services, and several continued processing accumulated backlogs after S3 itself recovered.
What broke outside Amazon
The visible casualty list included Slack, Trello, Medium, Quora, Coursera and Docker, although the symptoms differed from service to service. Some applications failed completely, while others displayed missing images, broken attachments or screens that never finished loading. Contemporary reporting collected by Data Center Knowledge documented how widely the disruption travelled.
Amazon’s own communication system also became part of the story. Until 11:37 a.m. Pacific, AWS could not update individual service statuses because the administration console for its Service Health Dashboard depended on S3. The company used its AWSCloud social-media account and a plain-text dashboard banner until normal status updates became possible.
Some of the strangest failures appeared inside people’s homes. Because the automation platform IFTTT was affected, users reported that certain internet-connected light bulbs would no longer respond to cloud-based commands, according to The Guardian’s contemporary account. The bulbs had electricity and their local hardware had not necessarily failed, but the remote chain connecting an app to the device had broken.
Financial estimates were similarly striking. Cyber-risk modelling company Cyence estimated that the interruption cost S&P 500 companies approximately $150 million and US financial-services companies another $160 million. Those figures were estimates of lost business during the disruption, not payments made by Amazon.
The numbers should therefore be treated as modelled exposure rather than a precise audit of every lost transaction. They also excluded many smaller businesses and customers of third-party services that relied indirectly on AWS. Even with those qualifications, the estimate illustrated how quickly a regional infrastructure problem could become an economic event.
Why the guardrails failed
The initiating error was ordinary, but the operational tool gave it an extraordinary reach. Amazon acknowledged that the tool allowed too much capacity to be removed too quickly and could take a subsystem below the minimum capacity needed to remain operational. The problem was not simply that a person entered something incorrectly, but that one incorrect input was allowed to travel so far.
AWS changed the tool so that capacity would be removed more slowly. It also added safeguards preventing an operation from taking a subsystem below its required minimum level. Those controls were intended to stop a similar input error from recreating the same failure.
The company also began auditing its other operational tools for comparable weaknesses. It accelerated work to divide the index subsystem into smaller partitions known internally as cells, allowing recovery procedures to be tested on smaller units and limiting the area affected by a failure. Amazon had already used cells elsewhere in S3, but further partitioning of the index subsystem had still been scheduled for later in 2017.
The Service Health Dashboard received its own architectural change. After its S3 dependency prevented timely updates during the outage, Amazon moved the dashboard’s administration console across multiple AWS regions.
Human error was therefore the trigger, but it does not fully explain the scale or duration of the event. The available guardrails determined how much capacity could disappear, while the size and restart history of the subsystems determined how long recovery would take. A smaller removal or a faster-tested restart process might have produced a routine maintenance incident instead of an internet-wide spectacle.
What the outage actually proved
The incident exposed an important difference between durability and availability. Amazon currently describes S3 as designed for 99.999999999 percent data durability, meaning the system is engineered to avoid losing stored objects. That figure does not promise that every object will be reachable through every regional API at every moment.
The architectural conclusion is also narrower than simply declaring that every company must operate in several regions. AWS’s current multi-region guidance says many workloads can meet their resilience objectives within one region by using multiple Availability Zones. Multi-region designs become relevant when a business must survive regional service failures or meet especially strict recovery targets.
The pattern resembles other infrastructure stories in which a safe-looking assumption survives until scale finally exposes it. YouTube once had to move beyond the 32-bit view counter that Gangnam Style was approaching, while the domain name system depends on layers of caching and root-server infrastructure that most users never see. In each case, the visible product depends on machinery whose limits remain quiet until one of them is reached.
At 12:37 p.m. Eastern Time on that Tuesday in Virginia, one incorrect input began removing more capacity than its operator intended. By 4:54 p.m., S3 was operating normally again, but during the intervening four hours status updates stalled, attachments disappeared and some cloud-connected lights ignored their owners. A routine debugging command had briefly made the internet’s invisible storage layer impossible to ignore.
