GitHub·

GitHub outage: a 43-second network break leaves the site degraded for 24 hours

A brief loss of connectivity during network maintenance caused GitHub's databases to fail over across the country, leaving data out of sync and the site degraded for 24 hours and 11 minutes.

ResolvedEnded Oct 22, 2018
Security:
No determination
Started
Oct 21, 2018, 22:52 UTC
Status returned to green
Oct 22, 2018, 23:03 UTC
Duration
24 h 11 min
Full recovery
Complete
Geographic scope
Global
Last verified
1 source

What happened

Systems and services affected
GitHub.comWebhooksGitHub Pages
Kind of failure
Network equipment
Sector
Software development
Provider report
Published
Status and recovery
Resolved. GitHub processed all pending webhooks and Pages builds and set its status to green at 23:03 UTC on October 22. Reconciling writes that had not replicated was still in progress when GitHub published its analysis.[1]

What caused it?

Root cause: Confirmed

Routine maintenance to replace failing 100G optical equipment caused a 43-second loss of connectivity between GitHub's US East Coast network hub and its primary US East Coast data center. Database failover automation (Orchestrator) promoted West Coast databases, and writes that had not replicated had to be reconciled.[1]

Provider report: Published. GitHub published its post-incident analysis on October 30, 2018.[1]

Was it malicious?

No determination

GitHub attributes the incident to maintenance work and a network partition. Its analysis does not mention an attack.[1]

How the failure happened

  1. 1TriggerMaintenance on failing optical equipment
  2. 2Failure43-second loss of connectivity
  3. 3ResponseDatabases failed over to the West Coast
  4. Affected3 services and systems

Evidence for each step

  1. 1
    Maintenance on failing optical equipment. Replacing 100G optical equipment[1]
    Starting point.
  2. 2
    43-second loss of connectivity. Between the US East Coast hub and the primary US East Coast data center[1]
    Led from step 1: Stated in a provider report or official record.
  3. 3
    Databases failed over to the West Coast. Writes that had not replicated had to be reconciled[1]
    Led from step 2: Stated in a provider report or official record.

Only steps the sources state are drawn. Affected services are listed under What happened.

Explanation

Both versions summarize the sourced claims on this page. Our own interpretation appears only under Independent analysis.

Plain language: what happened and why people were affected.

On October 21, 2018, GitHub stopped showing up-to-date information and paused some features for about a day.

During maintenance, the connection between GitHub's main East Coast data center and its network was cut for 43 seconds. In that time, automatic systems moved the main databases to the West Coast. When the connection came back, the two coasts had different data.

GitHub chose to keep the site degraded rather than risk losing data while it put everything back in order. It says no user data was lost.

Confirmed and unknown

Confirmed

  • Confirmed
    Connectivity between the East Coast hub and the primary data center was lost for 43 seconds during maintenance on failing 100G optical equipment.[1]
    Said by: GitHub · Evidence: Official
  • Confirmed
    GitHub had degraded service for 24 hours and 11 minutes.[1]
    Said by: GitHub · Evidence: Official
  • Confirmed
    No user data was lost.[1]
    Said by: GitHub · Evidence: Official
  • Confirmed
    About 200,000 webhook payloads that had outlived an internal TTL were dropped.[1]
    Said by: GitHub · Evidence: Official

Still unknown

No open questions: the provider has published a full account.

Timeline

Times in UTC. Dates without a time mean the sources give no time. Only events a source documents are listed.

  1. First observedOct 21, 2018, 22:52 UTC
  2. AcknowledgedOct 21, 2018, 23:09 UTC
  3. MitigationOct 21, 2018, 23:19 UTC
  4. RecoveryOct 22, 2018, 16:24 UTC
  5. ResolvedOct 22, 2018, 23:03 UTC
  6. Incident reportOct 30, 2018
Oct 21, 2018, 22:52 UTCOct 22, 2018, 23:03 UTC
  1. First observed
    43-second connectivity loss; database clusters fail over to the West Coast.[1]
  2. Acknowledged
    GitHub sets its status to yellow, then red at 23:11.[1]
  3. Mitigation
    Webhook delivery and Pages builds paused to protect data integrity.[1]
  4. All database primaries re-established on the East Coast.[1]
  5. Recovery
    Failover back to the original topology; backlog processing continues.[1]
  6. Resolved
    Pending webhooks and Pages builds processed; status set to green.[1]
  7. After the incident ended
    Incident report
    GitHub publishes its post-incident analysis.[1]
  8. Start to: Status returned to green24 h 11 min

Evidence and sources

Grouped by type of evidence. Numbers match the citations above.

Primary: provider and official sources

Published by the provider or agency itself: incident reports, status records, advisories, statements.

  1. 1.
    October 21 post-incident analysis (opens in a new tab)
    GitHub · Provider report · Published · Accessed

Last verified against these sources on Nov 14, 2026.