Amazon Web Services·

AWS S3 outage: a mistyped command removes too many servers in us-east-1

An engineer running a routine playbook entered one input incorrectly and removed far more S3 servers than intended, taking S3 in Northern Virginia down for about four hours along with services that depend on it.

ResolvedEnded Feb 28, 2017
Security:
No determination
Started
Feb 28, 2017, 17:37 UTC
S3 operating normally
Feb 28, 2017, 21:54 UTC
Duration
4 h 17 min
Full recovery
Complete
Geographic scope
United States (us-east-1)
Last verified
1 source

What happened

Systems and services affected
Amazon S3S3 consoleEC2 new instance launchesEBS volumes restoring from S3 snapshotsAWS LambdaAWS Service Health Dashboard
Kind of failure
Human error
Sector
Cloud infrastructure
Provider report
Published
Status and recovery
Resolved. AWS says S3 was operating normally at 1:54 PM PST; some services that depend on S3 took longer to work through backlogs.[1]

AWS gives times in Pacific time: 9:37 AM to 1:54 PM PST. Some dependent services needed longer to clear backlogs.

What caused it?

Root cause: Confirmed

While debugging the S3 billing system, an authorized team member ran a command from an established playbook to remove a small number of servers. One input was entered incorrectly, and a much larger set of servers was removed, including servers for the S3 index and placement subsystems.[1]

Provider report: Published. AWS published a summary of the event. The page does not show its publication date.[1]

Was it malicious?

No determination

AWS attributes the event to an incorrectly entered command. Its summary does not mention an attack.[1]

How the failure happened

  1. 1TriggerPlaybook command to remove a few servers
  2. 2FailureOne input entered incorrectly
  3. 3SpreadIndex and placement subsystems restarted
  4. Affected6 services and systems

Evidence for each step

  1. 1
    Playbook command to remove a few servers. Run while debugging the S3 billing system[1]
    Starting point.
  2. 2
    One input entered incorrectly. A much larger set of servers was removed[1]
    Led from step 1: Stated in a provider report or official record.
  3. 3
    Index and placement subsystems restarted. A full restart S3 had not done in its larger regions for many years[1]
    Led from step 2: Stated in a provider report or official record.

Only steps the sources state are drawn. Affected services are listed under What happened.

Explanation

Both versions summarize the sourced claims on this page. Our own interpretation appears only under Independent analysis.

Plain language: what happened and why people were affected.

On February 28, 2017, Amazon's S3 storage service stopped working in its Northern Virginia region for about four hours. Other AWS services that rely on S3, such as new server launches and Lambda, were affected too.

An engineer meant to take a few servers offline using a standard procedure, but typed one value wrong and removed far more. Two core S3 systems had to restart completely, which had not been done in years and took much longer than expected.

Even AWS's own status page could not be updated at first, because it relied on S3.

Confirmed and unknown

Confirmed

  • Confirmed
    One input to a playbook command was entered incorrectly, removing a much larger set of servers than intended.[1]
    Said by: AWS · Evidence: Official
  • Confirmed
    The index and placement subsystems had to fully restart, which S3 had not done in its larger regions for many years.[1]
    Said by: AWS · Evidence: Official
  • Confirmed
    AWS could not update individual service statuses on its Service Health Dashboard until 11:37 AM PST because the dashboard depended on S3.[1]
    Said by: AWS · Evidence: Official
  • Confirmed
    S3 was operating normally at 1:54 PM PST.[1]
    Said by: AWS · Evidence: Official

Still unknown

No open questions: the provider has published a full account.

Timeline

Times in UTC. Dates without a time mean the sources give no time. Only events a source documents are listed.

  1. First observedFeb 28, 2017, 17:37 UTC
  2. AcknowledgedFeb 28, 2017, 19:37 UTC
  3. MitigationFeb 28, 2017, 20:26 UTC
  4. RecoveryFeb 28, 2017, 21:18 UTC
  5. ResolvedFeb 28, 2017, 21:54 UTC
  6. Incident reportPublished, date not shown
Feb 28, 2017, 17:37 UTCFeb 28, 2017, 21:54 UTC
  1. First observed
    The command is run with an incorrect input; S3 servers are removed and errors begin.[1]
  2. Acknowledged
    AWS can update individual service statuses on its Service Health Dashboard again.[1]
  3. Mitigation
    Index subsystem partly back; S3 can serve GET, LIST and DELETE requests.[1]
  4. Recovery
    Index subsystem fully recovered.[1]
  5. Resolved
    Placement subsystem recovered; S3 operating normally.[1]
  6. Start to: S3 operating normally4 h 17 min

Related incidents

Links stated by providers or officials carry their evidence label. “Similar failure” is a comparison of documented causes, not a causal link.

Evidence and sources

Grouped by type of evidence. Numbers match the citations above.

Primary: provider and official sources

Published by the provider or agency itself: incident reports, status records, advisories, statements.

  1. 1.
    Summary of the Amazon S3 Service Disruption in the Northern Virginia (US-EAST-1) Region (opens in a new tab)
    Amazon Web Services · Provider report · Publication date not recorded · Accessed

Last verified against these sources on Nov 14, 2026.