AWS S3 outage: a mistyped command removes too many servers in us-east-1
An engineer running a routine playbook entered one input incorrectly and removed far more S3 servers than intended, taking S3 in Northern Virginia down for about four hours along with services that depend on it.
- Security:
- No determination
- Started
- Feb 28, 2017, 17:37 UTC
- S3 operating normally
- Feb 28, 2017, 21:54 UTC
- Duration
- 4 h 17 min
- Full recovery
- Complete
- Geographic scope
- United States (us-east-1)
- Last verified
- 1 source
What happened
- Systems and services affected
- Amazon S3S3 consoleEC2 new instance launchesEBS volumes restoring from S3 snapshotsAWS LambdaAWS Service Health Dashboard
- Kind of failure
- Human error
- Sector
- Cloud infrastructure
- Provider report
- Published
- Status and recovery
- Resolved. AWS says S3 was operating normally at 1:54 PM PST; some services that depend on S3 took longer to work through backlogs.[1]
AWS gives times in Pacific time: 9:37 AM to 1:54 PM PST. Some dependent services needed longer to clear backlogs.
What caused it?
Root cause: ConfirmedWhile debugging the S3 billing system, an authorized team member ran a command from an established playbook to remove a small number of servers. One input was entered incorrectly, and a much larger set of servers was removed, including servers for the S3 index and placement subsystems.[1]
Provider report: Published. AWS published a summary of the event. The page does not show its publication date.[1]
Was it malicious?
No determinationAWS attributes the event to an incorrectly entered command. Its summary does not mention an attack.[1]
How the failure happened
- 1TriggerPlaybook command to remove a few servers
- 2FailureOne input entered incorrectly
- 3SpreadIndex and placement subsystems restarted
- Affected6 services and systems
Evidence for each step
- 1Playbook command to remove a few servers. Run while debugging the S3 billing system[1]Starting point.
- 2One input entered incorrectly. A much larger set of servers was removed[1]Led from step 1: Stated in a provider report or official record.
- 3Index and placement subsystems restarted. A full restart S3 had not done in its larger regions for many years[1]Led from step 2: Stated in a provider report or official record.
Only steps the sources state are drawn. Affected services are listed under What happened.
Explanation
Both versions summarize the sourced claims on this page. Our own interpretation appears only under Independent analysis.
Plain language: what happened and why people were affected.
On February 28, 2017, Amazon's S3 storage service stopped working in its Northern Virginia region for about four hours. Other AWS services that rely on S3, such as new server launches and Lambda, were affected too.
An engineer meant to take a few servers offline using a standard procedure, but typed one value wrong and removed far more. Two core S3 systems had to restart completely, which had not been done in years and took much longer than expected.
Even AWS's own status page could not be updated at first, because it relied on S3.
Confirmed and unknown
Confirmed
- ConfirmedOne input to a playbook command was entered incorrectly, removing a much larger set of servers than intended.[1]Said by: AWS · Evidence: Official
- ConfirmedThe index and placement subsystems had to fully restart, which S3 had not done in its larger regions for many years.[1]Said by: AWS · Evidence: Official
- ConfirmedAWS could not update individual service statuses on its Service Health Dashboard until 11:37 AM PST because the dashboard depended on S3.[1]Said by: AWS · Evidence: Official
- ConfirmedS3 was operating normally at 1:54 PM PST.[1]Said by: AWS · Evidence: Official
Still unknown
No open questions: the provider has published a full account.
Timeline
Times in UTC. Dates without a time mean the sources give no time. Only events a source documents are listed.
- First observedFeb 28, 2017, 17:37 UTC
- AcknowledgedFeb 28, 2017, 19:37 UTC
- MitigationFeb 28, 2017, 20:26 UTC
- RecoveryFeb 28, 2017, 21:18 UTC
- ResolvedFeb 28, 2017, 21:54 UTC
- Incident reportPublished, date not shown
- First observedThe command is run with an incorrect input; S3 servers are removed and errors begin.[1]
- AcknowledgedAWS can update individual service statuses on its Service Health Dashboard again.[1]
- MitigationIndex subsystem partly back; S3 can serve GET, LIST and DELETE requests.[1]
- RecoveryIndex subsystem fully recovered.[1]
- ResolvedPlacement subsystem recovered; S3 operating normally.[1]
- Start to: S3 operating normally4 h 17 min
Related incidents
Links stated by providers or officials carry their evidence label. “Similar failure” is a comparison of documented causes, not a causal link.
- Similar failure (comparison)
Both in us-east-1, and both made worse by recovery steps taking far longer than expected.[1]
Evidence and sources
Grouped by type of evidence. Numbers match the citations above.
Primary: provider and official sources
Published by the provider or agency itself: incident reports, status records, advisories, statements.
- 1.Summary of the Amazon S3 Service Disruption in the Northern Virginia (US-EAST-1) Region (opens in a new tab)Amazon Web Services · Provider report · Publication date not recorded · Accessed
Last verified against these sources on Nov 14, 2026.