Amazon Web Services·

AWS us-east-1 outage: automated scaling floods AWS's internal network

An automated capacity-scaling job triggered a surge of connections that congested the devices linking AWS's internal and main networks, impairing APIs, the console and AWS's own monitoring for most of a day.

ResolvedEnded Dec 8, 2021
Security:
No determination
Started
Dec 7, 2021, 15:30 UTC
Last service recovered (EventBridge)
Dec 8, 2021, 02:40 UTC
Duration
11 h 10 min
Full recovery
Complete
Geographic scope
United States (us-east-1)
Last verified
1 source

What happened

Systems and services affected
EC2 APIsAWS Management ConsoleRoute 53 APIsAWS STSAPI GatewayAmazon EventBridgeAmazon ConnectAmazon CloudWatchAWS Service Health Dashboard
Kind of failure
Software defect
Sector
Cloud infrastructure
Provider report
Published
Status and recovery
Resolved. AWS reports network devices fully recovered at 2:22 PM PST and services recovering through the afternoon, with EventBridge delivery latency last to clear at 6:40 PM PST.[1]

AWS gives times in Pacific time. Network devices fully recovered at 2:22 PM PST; individual services recovered later, the last (EventBridge delivery latency) at 6:40 PM PST.

What caused it?

Root cause: Confirmed

An automated activity to scale capacity of an AWS service in the main network triggered unexpected behavior from many clients inside AWS's internal network. The surge overwhelmed the devices connecting the two networks, and a latent issue stopped clients from backing off as designed, so retries kept the network congested.[1]

Provider report: Published. AWS published its summary on December 10, 2021.[1]

Was it malicious?

No determination

AWS attributes the event to an internal automated scaling activity and network congestion. Its summary does not mention an attack.[1]

How the failure happened

  1. 1TriggerAutomated capacity scaling
  2. 2FailureSurge from internal clients
  3. 3SpreadClients did not back off
  4. Affected9 services and systems

Evidence for each step

  1. 1
    Automated capacity scaling. In AWS's main network[1]
    Starting point.
  2. 2
    Surge from internal clients. Overwhelmed the devices connecting the internal and main networks[1]
    Led from step 1: Stated in a provider report or official record.
  3. 3
    Clients did not back off. A latent issue kept retries going, sustaining the congestion[1]
    Led from step 2: Stated in a provider report or official record.

Only steps the sources state are drawn. Affected services are listed under What happened.

Explanation

Both versions summarize the sourced claims on this page. Our own interpretation appears only under Independent analysis.

Plain language: what happened and why people were affected.

On December 7, 2021, many AWS services in Northern Virginia stopped working properly for much of the day. Apps that depend on them, and AWS's own support and status tools, were affected.

AWS runs a separate internal network for its own systems. An automatic job that adds capacity set off a flood of connection attempts from that internal network, which jammed the equipment linking it to the main network.

Because AWS's monitoring also ran over that internal network, engineers had limited visibility while they fixed it.

Confirmed and unknown

Confirmed

  • Confirmed
    An automated scaling activity triggered a surge from clients in AWS's internal network that overwhelmed the devices between the internal and main networks.[1]
    Said by: AWS · Evidence: Official
  • Confirmed
    A latent issue prevented clients from backing off as designed, sustaining the congestion.[1]
    Said by: AWS · Evidence: Official
  • Confirmed
    Service Health Dashboard failover was impaired; updates resumed by 8:22 AM PST.[1]
    Said by: AWS · Evidence: Official
  • Confirmed
    Network devices fully recovered at 2:22 PM PST.[1]
    Said by: AWS · Evidence: Official

Still unknown

No open questions: the provider has published a full account.

Timeline

Times in UTC. Dates without a time mean the sources give no time. Only events a source documents are listed.

  1. First observedDec 7, 2021, 15:30 UTC
  2. AcknowledgedDec 7, 2021, 16:22 UTC
  3. MitigationDec 7, 2021, 17:28 UTC
  4. RecoveryDec 7, 2021, 22:22 UTC
  5. ResolvedDec 8, 2021, 02:40 UTC
  6. Incident reportDec 10, 2021
Dec 7, 2021, 15:30 UTCDec 8, 2021, 02:40 UTC
  1. First observed
    Automated scaling activity triggers the surge; congestion begins.[1]
  2. Acknowledged
    Service Health Dashboard updates resume.[1]
  3. Mitigation
    Internal DNS errors fully recovered.[1]
  4. Network congestion significantly improved.[1]
  5. Recovery
    All network devices fully recovered.[1]
  6. Resolved
    EventBridge delivery latency returns to normal; last service recovered.[1]
  7. After the incident ended
    Incident report
    AWS publishes its summary.[1]
  8. Start to: Last service recovered (EventBridge)11 h 10 min

Related incidents

Links stated by providers or officials carry their evidence label. “Similar failure” is a comparison of documented causes, not a causal link.

Evidence and sources

Grouped by type of evidence. Numbers match the citations above.

Primary: provider and official sources

Published by the provider or agency itself: incident reports, status records, advisories, statements.

  1. 1.
    Summary of the AWS Service Event in the Northern Virginia (US-EAST-1) Region (opens in a new tab)
    Amazon Web Services · Provider report · Published · Accessed

Last verified against these sources on Nov 14, 2026.