Roblox·

Roblox outage: a 73-hour shutdown traced to its Consul service-discovery cluster

Roblox was offline for most of three days after a new Consul feature, under very high load, combined with a database storage problem to cripple the system its services use to find each other.

ResolvedEnded Oct 31, 2021
Security:
No determination
Started
Oct 28, 2021
All players given access
Oct 31, 2021
Duration
Not computable from sources
Full recovery
Complete
Geographic scope
Global
Last verified
1 source

What happened

Systems and services affected
Roblox platformHashiCorp Consul (service discovery and key-value store)Nomad schedulingVaultCaching system
Kind of failure
Software defect
Sector
Gaming
Provider report
Published
Status and recovery
Resolved. Roblox says 100% of players were given access and the service was fully operational at 16:45 Pacific time on October 31.[1]

Roblox gives times in Pacific time (labelled PST): first detection at 13:37 on October 28, full operation at 16:45 on October 31. Roblox states the outage lasted 73 hours; the times above span about 75.

What caused it?

Root cause: Confirmed

Enabling a relatively new Consul streaming feature under unusually high read and write load caused contention. A pathological issue in BoltDB, the storage Consul uses, made freelist maintenance extremely expensive. A single Consul cluster supported many workloads, which widened the impact.[1]

Provider report: Published. Roblox published its report on January 20, 2022, written with HashiCorp.[1]

Was it malicious?

No determination

Roblox says "there was no user data loss or access by unauthorized parties of any information during the incident." Its report attributes the outage to Consul and BoltDB issues.[1]

How the failure happened

  1. 1TriggerConsul streaming feature enabled
  2. 2FailureContention and BoltDB slowdown
  3. 3SpreadImpact widened
  4. Affected5 services and systems

Evidence for each step

  1. 1
    Consul streaming feature enabled. Under unusually high read and write load[1]
    Starting point.
  2. 2
    Contention and BoltDB slowdown. Freelist maintenance became extremely expensive[1]
    Led from step 1: Stated in a provider report or official record.
  3. 3
    Impact widened. A single Consul cluster supported many workloads[1]
    Led from step 2: Stated in a provider report or official record.

Only steps the sources state are drawn. Affected services are listed under What happened.

Explanation

Both versions summarize the sourced claims on this page. Our own interpretation appears only under Independent analysis.

Plain language: what happened and why people were affected.

From October 28 to 31, 2021, Roblox was down for most of three days.

Roblox's many services use a system called Consul to find and talk to each other. A newly enabled Consul feature slowed down under very heavy use, and a hidden storage problem made it worse. Without Consul, almost nothing else could run.

Engineers from Roblox and HashiCorp, the company that makes Consul, worked together to bring it back, then let players in gradually.

Confirmed and unknown

Confirmed

  • Confirmed
    Enabling Consul's streaming feature under unusually high read and write load caused contention and poor performance.[1]
    Said by: Roblox · Evidence: Official
  • Confirmed
    A BoltDB issue made freelist maintenance extremely expensive; compacting BoltDB resolved it.[1]
    Said by: Roblox and HashiCorp · Evidence: Official
  • Confirmed
    The outage lasted 73 hours.[1]
    Said by: Roblox · Evidence: Official
  • Confirmed
    There was no user data loss or access by unauthorized parties.[1]
    Said by: Roblox · Evidence: Official

Still unknown

No open questions: the provider has published a full account.

Timeline

Times in UTC. Dates without a time mean the sources give no time. Only events a source documents are listed.

  1. First observedOct 28, 2021
  2. AcknowledgedNot documented
  3. MitigationOct 28, 2021
  4. RecoveryOct 31, 2021
  5. ResolvedOct 31, 2021
  6. Incident reportJan 20, 2022
  1. First observed
    Problems detected at 13:37 Pacific time; online players fall to half of normal by 16:35.[1]
  2. Mitigation
    Consul streaming disabled for all systems at 15:51 Pacific time.[1]
  3. Consul cluster reset from a snapshot; performance degrades again once load returns.[1]
  4. Recovery
    Consul and caching healthy by 05:00 Pacific time; players admitted in steps from 10:00.[1]
  5. Resolved
    All players given access; fully operational at 16:45 Pacific time.[1]
  6. After the incident ended
    Incident report
    Roblox publishes its report, written with HashiCorp.[1]

Evidence and sources

Grouped by type of evidence. Numbers match the citations above.

Primary: provider and official sources

Published by the provider or agency itself: incident reports, status records, advisories, statements.

  1. 1.
    Roblox Return to Service (opens in a new tab)
    Roblox · Provider report · Published · Accessed

Last verified against these sources on Nov 14, 2026.