The outage began just after 9:37 a.m. Pacific time, in a data centre region called US-EAST-1, in Northern Virginia. Within minutes, Slack workspaces stopped loading images, Trello boards went blank, Medium articles refused to render, smart lightbulbs from companies like LIFX became unreachable, and even Amazon’s own service health dashboard — the page customers were meant to check for updates — could not update, because it depended on the same system that had just broken. Downdetector logged spikes across hundreds of services simultaneously. For roughly four hours, a meaningful slice of what people call “the internet” was, in practical terms, gone.
The cause, as Amazon itself later explained in an unusually candid public post-mortem, was a single command, typed by a single engineer, with one field wrong.
What the engineer was actually doing
The service that failed was Amazon S3, short for Simple Storage Service. S3 is the quiet workhorse of the modern web: an object store where developers park files, images, video, backups, static websites, machine-learning training data, and the assets that other applications assume will always be there. By 2017 S3 already held trillions of objects and served over a million requests per second. If a website’s images live on S3 and S3 stops answering, the website looks broken even if its own servers are fine.
On that Tuesday morning, the S3 team in US-EAST-1 was debugging an issue with the S3 billing system, which had been running more slowly than expected. To diagnose it, an authorised engineer ran a command from an established operational playbook. The command was meant to take a small number of servers offline — specifically, servers belonging to one of the S3 billing subsystems. Removing a handful of machines from a subsystem that has thousands of them is a routine, low-risk operation.
The engineer entered the command. One of the inputs was wrong. Instead of removing a small set of billing servers, the command removed a much larger set — and, critically, the larger set included servers that belonged to two other S3 subsystems, not just the billing one.
The two subsystems that should never have gone down together
The first of the two subsystems accidentally caught in the command was the index subsystem, which stores the metadata and location information for every single object in the region — essentially the catalogue that tells S3 where any given file actually lives on disk. Without it, S3 knows a file’s name but has no idea where the bytes are.
The second was the placement subsystem, which decides where new objects should be written when a customer uploads something. Crucially, placement cannot work unless the index subsystem is already healthy. The two are not peers; one leans on the other. Removing capacity from both at once meant each needed a full restart before S3 could serve requests again.
With too many index-subsystem servers suddenly gone, S3 could no longer reliably answer the simplest question it exists to answer: given an object name, where is the data? Read requests (GET), write requests (PUT), list operations, and delete operations for the entire US-EAST-1 region began to fail, and because placement depended on index, it went down too.
And US-EAST-1 is Amazon’s oldest and largest region — the default choice for a huge swathe of AWS customers since 2006. A very large number of unrelated companies discovered, all at once, that their applications quietly assumed S3 was infallible.
The damage was not confined to S3 either. Other AWS services in the region that rely on S3 for storage stumbled with it: new Amazon EC2 instance launches, Amazon EBS volumes that needed data from an S3 snapshot, and AWS Lambda functions were all impacted while the S3 APIs were unavailable. A single storage layer had quietly become load-bearing for services that customers thought of as entirely separate.
Why recovery took nearly four hours
Restarting the missing servers sounds simple. It was not. Amazon’s own write-up admitted something slightly embarrassing: the index and placement subsystems in its larger regions had not been completely restarted for many years, even though S3 had grown enormously in that time. The routine of removing and replacing capacity was well practised; a full cold restart at this scale simply had not been exercised.
So the index subsystem had to walk through its full safety checks — validating that the metadata for a vastly larger dataset was consistent before it resumed answering requests — on a much larger volume of data than it had ever been booted with before. That process, not the mistyped command itself, is what stretched the outage.
According to Amazon’s own timeline, the index subsystem had activated enough capacity to begin servicing GET, LIST, and DELETE requests by 12:26 p.m. Pacific, and those APIs were functioning normally by 1:18 p.m. — roughly three hours and forty minutes after the incident began. PUT requests took longer, because they also depend on the placement subsystem, which only finished its own recovery at 1:54 p.m. Pacific. Only then was S3 operating normally again.
Even that was not quite the end. Downstream of Amazon, the ripple was almost comic in its breadth. eWeek’s tally of knocked-out or degraded services ran to dozens of names, including Quora, Business Insider, Giphy, IFTTT, Coursera, Docker, Slack, and Trello, alongside hundreds of smaller sites. Even connected home devices misbehaved: smart plugs stopped responding, because the phone app that controlled them tried to authenticate against a service that lived on S3. The status pages of many services showed all green, because those status pages were themselves hosted on S3.
The bill was real. By one estimate from the cyber-risk modelling firm Cyence, the disruption cost S&P 500 companies in the region of $150 million; Docker reportedly pushed back a major announcement by two days. Four hours of one storage service in one region translated into a very large number spread across a great many balance sheets.
What Amazon changed afterwards
Amazon’s public message ended with a list of specific fixes. The tool the engineer had used was modified so it could no longer remove capacity below a safe minimum threshold, and so it took servers offline more slowly, giving humans time to notice. Amazon also audited its other operational tools for the same class of safeguard.
The index and placement subsystems were being re-partitioned into smaller “cells” so that a failure in one cell could not take down the whole region, and the service health dashboard was moved so that it no longer depended on the region it was reporting on — an obvious fix in retrospect, and a reminder that even Amazon had allowed circular dependencies to sneak in.
The wider point is not that one engineer was careless. Amazon deliberately did not name the individual, and the post-mortem is pointed about the fact that a system in which one keystroke can break a continent’s worth of services is a system with a design problem, not a personnel problem. The same principle governs how aviation investigates crashes: individual error is treated as the trigger, not the cause. The cause is the environment that let the trigger matter.
Nor was it the first time AWS had learned this lesson in public. Amazon keeps an archive of post-event summaries stretching back years, including a 2015 DynamoDB disruption in the same US-East region. The recurring shape is familiar: one dependency, quietly relied upon by more systems than anyone had fully mapped, fails and takes the rest with it.
Why so much of the internet lived in one place
The outage also exposed something structural. US-EAST-1 exists in Northern Virginia because that corner of the United States, particularly Ashburn and Loudoun County, is where a very large fraction of global internet infrastructure has physically clustered since the 1990s — close to the original NSFNET peering points and the MAE-East exchange. When AWS launched S3 in 2006, US-EAST-1 was the only region. Customers built there by default, and many never moved. A decade later, “in the cloud” often quietly meant “in one specific set of buildings in Virginia”.
The plumbing that carries the modern web has more single points of failure than people assume — not because engineers are lazy, but because scale hides dependency. In the same way that Wi-Fi in a crowded café isn’t really slow, it’s queued, or that a deleted file isn’t really deleted, the internet’s outages tend to reveal how the thing actually works — usually at the least convenient moment.
The last time a comparable single-command mistake reshaped how people thought about internet fragility was in 1988, when a Cornell graduate student’s worm crashed a tenth of the machines then online. The 2017 S3 outage was, in a sense, its heir. One command, one wrong parameter, one region — and a reminder that the cloud is still made of buildings, and the buildings are still run by people.
