AWS us-east-1 outage: automated scaling floods AWS's internal network
An automated capacity-scaling job triggered a surge of connections that congested the devices linking AWS's internal and main networks, impairing APIs, the console and AWS's own monitoring for most of a day.
- Security:
- No determination
- Started
- Dec 7, 2021, 15:30 UTC
- Last service recovered (EventBridge)
- Dec 8, 2021, 02:40 UTC
- Duration
- 11 h 10 min
- Full recovery
- Complete
- Geographic scope
- United States (us-east-1)
- Last verified
- 1 source
What happened
- Systems and services affected
- EC2 APIsAWS Management ConsoleRoute 53 APIsAWS STSAPI GatewayAmazon EventBridgeAmazon ConnectAmazon CloudWatchAWS Service Health Dashboard
- Kind of failure
- Software defect
- Sector
- Cloud infrastructure
- Provider report
- Published
- Status and recovery
- Resolved. AWS reports network devices fully recovered at 2:22 PM PST and services recovering through the afternoon, with EventBridge delivery latency last to clear at 6:40 PM PST.[1]
AWS gives times in Pacific time. Network devices fully recovered at 2:22 PM PST; individual services recovered later, the last (EventBridge delivery latency) at 6:40 PM PST.
What caused it?
Root cause: ConfirmedAn automated activity to scale capacity of an AWS service in the main network triggered unexpected behavior from many clients inside AWS's internal network. The surge overwhelmed the devices connecting the two networks, and a latent issue stopped clients from backing off as designed, so retries kept the network congested.[1]
Provider report: Published. AWS published its summary on December 10, 2021.[1]
Was it malicious?
No determinationAWS attributes the event to an internal automated scaling activity and network congestion. Its summary does not mention an attack.[1]
How the failure happened
- 1TriggerAutomated capacity scaling
- 2FailureSurge from internal clients
- 3SpreadClients did not back off
- Affected9 services and systems
Evidence for each step
- 1
- 2Surge from internal clients. Overwhelmed the devices connecting the internal and main networks[1]Led from step 1: Stated in a provider report or official record.
- 3Clients did not back off. A latent issue kept retries going, sustaining the congestion[1]Led from step 2: Stated in a provider report or official record.
Only steps the sources state are drawn. Affected services are listed under What happened.
Explanation
Both versions summarize the sourced claims on this page. Our own interpretation appears only under Independent analysis.
Plain language: what happened and why people were affected.
On December 7, 2021, many AWS services in Northern Virginia stopped working properly for much of the day. Apps that depend on them, and AWS's own support and status tools, were affected.
AWS runs a separate internal network for its own systems. An automatic job that adds capacity set off a flood of connection attempts from that internal network, which jammed the equipment linking it to the main network.
Because AWS's monitoring also ran over that internal network, engineers had limited visibility while they fixed it.
Confirmed and unknown
Confirmed
- ConfirmedAn automated scaling activity triggered a surge from clients in AWS's internal network that overwhelmed the devices between the internal and main networks.[1]Said by: AWS · Evidence: Official
- ConfirmedA latent issue prevented clients from backing off as designed, sustaining the congestion.[1]Said by: AWS · Evidence: Official
- ConfirmedService Health Dashboard failover was impaired; updates resumed by 8:22 AM PST.[1]Said by: AWS · Evidence: Official
- ConfirmedNetwork devices fully recovered at 2:22 PM PST.[1]Said by: AWS · Evidence: Official
Still unknown
No open questions: the provider has published a full account.
Timeline
Times in UTC. Dates without a time mean the sources give no time. Only events a source documents are listed.
- First observedDec 7, 2021, 15:30 UTC
- AcknowledgedDec 7, 2021, 16:22 UTC
- MitigationDec 7, 2021, 17:28 UTC
- RecoveryDec 7, 2021, 22:22 UTC
- ResolvedDec 8, 2021, 02:40 UTC
- Incident reportDec 10, 2021
- First observedAutomated scaling activity triggers the surge; congestion begins.[1]
- AcknowledgedService Health Dashboard updates resume.[1]
- MitigationInternal DNS errors fully recovered.[1]
- Network congestion significantly improved.[1]
- RecoveryAll network devices fully recovered.[1]
- ResolvedEventBridge delivery latency returns to normal; last service recovered.[1]
- After the incident endedIncident reportAWS publishes its summary.[1]
- Start to: Last service recovered (EventBridge)11 h 10 min
Related incidents
Links stated by providers or officials carry their evidence label. “Similar failure” is a comparison of documented causes, not a causal link.
- Similar failure (comparison)
Both started in automation inside us-east-1 and spread through AWS's own control-plane services.[1]
Evidence and sources
Grouped by type of evidence. Numbers match the citations above.
Primary: provider and official sources
Published by the provider or agency itself: incident reports, status records, advisories, statements.
- 1.Summary of the AWS Service Event in the Northern Virginia (US-EAST-1) Region (opens in a new tab)Amazon Web Services · Provider report · Published · Accessed
Last verified against these sources on Nov 14, 2026.