OpenAI·

ChatGPT, API and Sora outage: a new telemetry service overwhelms Kubernetes

A newly deployed monitoring service made every node in OpenAI's clusters hammer the Kubernetes API, taking down cluster control planes and the DNS that services use to find each other, for over four hours.

ResolvedEnded Dec 12, 2024
Security:
Explicitly ruled out
Started
Dec 11, 2024, 23:16 UTC
All services fully recovered
Dec 12, 2024, 03:38 UTC
Duration
4 h 22 min
Full recovery
Complete
Geographic scope
Global
Last verified
1 source

What happened

Systems and services affected
ChatGPTOpenAI APISora
Kind of failure
Configuration change
Sector
AI services
Provider report
Published
Status and recovery
Resolved. OpenAI reports substantial recovery for the API at 5:36 PM PST and ChatGPT at 5:45 PM, ChatGPT and Sora fully recovered at 7:01 PM, and the API across all models at 7:38 PM.[1]

OpenAI gives times in PST: 3:16 PM to 7:38 PM on December 11.

What caused it?

Root cause: Confirmed

A new telemetry service deployed at 3:12 PM PST had a configuration that made every node in each cluster run resource-intensive Kubernetes API operations. The API servers were overwhelmed, taking down the control plane in most large clusters; DNS-based service discovery depends on the control plane, so services could no longer find each other.[1]

Provider report: Published. OpenAI published a write-up on its status page. The page does not show its publication date.[1]

Was it malicious?

Explicitly ruled out

OpenAI: the event was "not caused by a security incident or a recent launch."[1]

How the failure happened

  1. 1TriggerNew telemetry service deployed
  2. 2FailureKubernetes API servers overwhelmed
  3. 3SpreadControl plane down; service discovery failed
  4. Affected3 services and systems

Evidence for each step

  1. 1
    New telemetry service deployed[1]
    Starting point.
  2. 2
    Kubernetes API servers overwhelmed. Every node ran resource-intensive API operations[1]
    Led from step 1: Stated in a provider report or official record.
  3. 3
    Control plane down; service discovery failed. DNS-based service discovery depends on the control plane[1]
    Led from step 2: Stated in a provider report or official record.

Only steps the sources state are drawn. Affected services are listed under What happened.

Explanation

Both versions summarize the sourced claims on this page. Our own interpretation appears only under Independent analysis.

Plain language: what happened and why people were affected.

On December 11, 2024, ChatGPT, OpenAI's API and Sora stopped working for more than four hours.

OpenAI rolled out a new tool to monitor its computing clusters. Its settings made every machine send heavy requests to the system that manages each cluster, which buckled under the load.

The fix meant removing the new tool, but engineers needed that same overloaded management system to do it, which slowed recovery.

Confirmed and unknown

Confirmed

  • Confirmed
    A new telemetry service's configuration made every node run resource-intensive Kubernetes API operations, overwhelming the API servers.[1]
    Said by: OpenAI · Evidence: Official
  • Confirmed
    DNS caching made the issue less visible at first, so the rollout continued.[1]
    Said by: OpenAI · Evidence: Official
  • Confirmed
    Fixing it required control-plane access that the overload had blocked.[1]
    Said by: OpenAI · Evidence: Official

Denied

  • Denied
    It was caused by a security incident or a product launch.[1]
    Said by: OpenAI · Evidence: Official

Still unknown

No open questions: the provider has published a full account.

Timeline

Times in UTC. Dates without a time mean the sources give no time. Only events a source documents are listed.

  1. First observedDec 11, 2024, 23:16 UTC
  2. AcknowledgedNot documented
  3. MitigationDec 12, 2024, 01:36 UTC
  4. RecoveryDec 12, 2024, 01:45 UTC
  5. ResolvedDec 12, 2024, 03:38 UTC
  6. Incident reportPublished, date not shown
Dec 11, 2024, 23:12 UTCDec 12, 2024, 03:38 UTC
  1. New telemetry service deployed (3:12 PM PST).[1]
  2. First observed
    All products begin degrading.[1]
  3. Mitigation
    Substantial recovery for the API.[1]
  4. Recovery
    Substantial recovery for ChatGPT.[1]
  5. Resolved
    API fully recovered across all models.[1]
  6. Start to: All services fully recovered4 h 22 min

Evidence and sources

Grouped by type of evidence. Numbers match the citations above.

Primary: provider and official sources

Published by the provider or agency itself: incident reports, status records, advisories, statements.

  1. 1.
    API, ChatGPT & Sora Facing Issues (write-up) (opens in a new tab)
    OpenAI · Provider report · Publication date not recorded · Accessed

Last verified against these sources on Nov 14, 2026.