Roblox outage: a 73-hour shutdown traced to its Consul service-discovery cluster
Roblox was offline for most of three days after a new Consul feature, under very high load, combined with a database storage problem to cripple the system its services use to find each other.
- Security:
- No determination
- Started
- Oct 28, 2021
- All players given access
- Oct 31, 2021
- Duration
- Not computable from sources
- Full recovery
- Complete
- Geographic scope
- Global
- Last verified
- 1 source
What happened
- Systems and services affected
- Roblox platformHashiCorp Consul (service discovery and key-value store)Nomad schedulingVaultCaching system
- Kind of failure
- Software defect
- Sector
- Gaming
- Provider report
- Published
- Status and recovery
- Resolved. Roblox says 100% of players were given access and the service was fully operational at 16:45 Pacific time on October 31.[1]
Roblox gives times in Pacific time (labelled PST): first detection at 13:37 on October 28, full operation at 16:45 on October 31. Roblox states the outage lasted 73 hours; the times above span about 75.
What caused it?
Root cause: ConfirmedEnabling a relatively new Consul streaming feature under unusually high read and write load caused contention. A pathological issue in BoltDB, the storage Consul uses, made freelist maintenance extremely expensive. A single Consul cluster supported many workloads, which widened the impact.[1]
Provider report: Published. Roblox published its report on January 20, 2022, written with HashiCorp.[1]
Was it malicious?
No determinationRoblox says "there was no user data loss or access by unauthorized parties of any information during the incident." Its report attributes the outage to Consul and BoltDB issues.[1]
How the failure happened
- 1TriggerConsul streaming feature enabled
- 2FailureContention and BoltDB slowdown
- 3SpreadImpact widened
- Affected5 services and systems
Evidence for each step
- 1
- 2Contention and BoltDB slowdown. Freelist maintenance became extremely expensive[1]Led from step 1: Stated in a provider report or official record.
- 3Impact widened. A single Consul cluster supported many workloads[1]Led from step 2: Stated in a provider report or official record.
Only steps the sources state are drawn. Affected services are listed under What happened.
Explanation
Both versions summarize the sourced claims on this page. Our own interpretation appears only under Independent analysis.
Plain language: what happened and why people were affected.
From October 28 to 31, 2021, Roblox was down for most of three days.
Roblox's many services use a system called Consul to find and talk to each other. A newly enabled Consul feature slowed down under very heavy use, and a hidden storage problem made it worse. Without Consul, almost nothing else could run.
Engineers from Roblox and HashiCorp, the company that makes Consul, worked together to bring it back, then let players in gradually.
Confirmed and unknown
Confirmed
- ConfirmedEnabling Consul's streaming feature under unusually high read and write load caused contention and poor performance.[1]Said by: Roblox · Evidence: Official
- ConfirmedA BoltDB issue made freelist maintenance extremely expensive; compacting BoltDB resolved it.[1]Said by: Roblox and HashiCorp · Evidence: Official
- ConfirmedThe outage lasted 73 hours.[1]Said by: Roblox · Evidence: Official
- ConfirmedThere was no user data loss or access by unauthorized parties.[1]Said by: Roblox · Evidence: Official
Still unknown
No open questions: the provider has published a full account.
Timeline
Times in UTC. Dates without a time mean the sources give no time. Only events a source documents are listed.
- First observedOct 28, 2021
- AcknowledgedNot documented
- MitigationOct 28, 2021
- RecoveryOct 31, 2021
- ResolvedOct 31, 2021
- Incident reportJan 20, 2022
- First observedProblems detected at 13:37 Pacific time; online players fall to half of normal by 16:35.[1]
- MitigationConsul streaming disabled for all systems at 15:51 Pacific time.[1]
- Consul cluster reset from a snapshot; performance degrades again once load returns.[1]
- RecoveryConsul and caching healthy by 05:00 Pacific time; players admitted in steps from 10:00.[1]
- ResolvedAll players given access; fully operational at 16:45 Pacific time.[1]
- After the incident endedIncident reportRoblox publishes its report, written with HashiCorp.[1]
Evidence and sources
Grouped by type of evidence. Numbers match the citations above.
Primary: provider and official sources
Published by the provider or agency itself: incident reports, status records, advisories, statements.
- 1.Roblox Return to Service (opens in a new tab)Roblox · Provider report · Published · Accessed
Last verified against these sources on Nov 14, 2026.