In September 2026, AWS said it could not restore access to its Bahrain region or to resources and data hosted exclusively in one Availability Zone in the United Arab Emirates after physical damage earlier in the year. In Bahrain, the damage crossed multiple Availability Zones and exceeded what AWS said its regional and multi-AZ services were designed to withstand. Most affected customers had moved or rebuilt elsewhere. For data that had not been migrated, AWS said it had exhausted its restoration options. (Reuters, WIRED)
This is not a story about war.
It is a reminder that every architecture is built on assumptions, and some of those assumptions will eventually change.
The next broken assumption may come from a fire, flood, heatwave, prolonged power failure, submarine cable cut, compromised administrator account, cloud control-plane outage, new data-transfer rule, or simply growth into a market the original system was never designed to serve.
We cannot predict every future event. We can, however, design the ability to continue, recover, move, and adapt when the conditions around a service change.
That is a much larger goal than uptime.
Resilience Has Three Different Jobs
High availability usually answers a narrow question: can the service absorb a failed process, server, or Availability Zone without a visible interruption?
A future-ready service must answer three broader questions.
Continuity: Which player and business capabilities must continue during disruption, even in a reduced mode?
Recoverability: Can the service and its data be restored to a known, valid point within an agreed time?
Adaptability: Can the system change region, provider, data boundary, capacity, or operating model without being rebuilt from zero?
These goals overlap, but they are not interchangeable. Replication can improve availability while copying corruption or deletion to every replica. A backup can preserve data while still taking days to restore. A multi-region deployment can survive a local failure while depending on one global identity provider, key service, DNS account, or release pipeline.
The first step is therefore not “add another region.” It is to identify what the game must protect, which assumptions it depends on, and how those assumptions can fail together.
1. A Different Server Is Not Always a Different Failure Domain
In March 2021, a fire at OVHcloud’s Strasbourg site destroyed the SBG2 building. Facepunch said 25 European Rust servers were affected, with a total loss of the affected servers and no way to restore the lost player data. Servers returned, but progress on those worlds was reset. (Game Developer)
For a game, this is more than infrastructure downtime. A persistent world is a record of player time, relationships, competition, construction, and ownership. The machines can be replaced; the missing history cannot.
The broken assumption was not simply that a server would remain online. It was that the operational copies and the recovery copies would not share the same physical fate.
Teams often count instances, database replicas, or zones and conclude that the service is redundant. The more useful question is whether those copies share a building, metropolitan area, power grid, network corridor, cloud account, administrator, encryption-key hierarchy, or deletion path.
A practical failure-domain map should cover more than compute:
- facility, power, cooling, and metropolitan area;
- Availability Zone, region, and legal jurisdiction;
- cloud provider, account hierarchy, IAM, and KMS;
- DNS, certificate issuance, identity, and deployment systems;
- backup credentials, storage location, and retention controls;
- people, runbooks, and authority to declare a failover.
The executive decision is not “How many copies can we afford?” It is “Which losses must the business be able to survive?”
For one game, losing a temporary match may be acceptable while losing inventory or purchases is not. For another, the persistent world itself is the product. Recovery investment should follow that difference.
2. A Game Can Be Online and Still Be Unreachable
On 14 March 2024, a suspected underwater rock slide off Côte d’Ivoire took four submarine cable systems offline: ACE, SAT-3, WACS, and MainOne. Thirteen African countries experienced degraded service or near-total internet outages. The four cables looked like separate connections, but they converged through the same coastal area. An unaffected cable, Equiano, and terrestrial routes became vital alternatives. (Internet Society)
This was not reported as the outage of one named game. It is still a direct game-service lesson. Login, matchmaking, voice, telemetry, commerce, and real-time play all depend on the path between the player and the service—not only on whether the destination server is healthy.
For a website, an extra 100 milliseconds may be an inconvenience. For a competitive game, path instability can change the outcome of play. Riot Games describes Riot Direct as a backbone and point-of-presence network created to reduce the uncertainty of the public internet and bring game traffic closer to players. Its architecture also reflects that different games have different protocol, latency, and server-location requirements. (Riot Games)
The design lesson is not that every studio should build a private global backbone. It is that network reachability is a product dependency and should be designed with the same care as databases or compute.
Useful questions include:
- Do two carriers use genuinely different physical routes and landing points?
- Can traffic enter through more than one edge or point of presence?
- Can matchmaking move players to another healthy location without creating an unacceptable latency or fairness problem?
- Can a regional cell keep essential account and session functions local when international connectivity is degraded?
- Are routing quality, packet loss, jitter, and failed connection attempts visible by ISP and geography?
- Is there a reduced mode for social, chat, store, leaderboard, or telemetry functions when the core game session remains playable?
Contract diversity is not necessarily route diversity. A diagram showing two providers is not proof that their cables, exchanges, or upstream dependencies do not meet at the same physical point.
3. A Security Incident Can Attack the Ability to Change the Game
In January 2023, Riot Games disclosed that a social-engineering attack had compromised its development environment. Attackers stole source code for League of Legends, Teamfight Tactics, and a legacy anti-cheat platform, then demanded a ransom. Riot refused to pay. The incident affected the company’s patch cadence and created a risk that exposure of source code could help new cheats emerge. (Axios)
The live game was not the only asset at risk. The systems used to build, inspect, patch, and defend it had become part of the incident.
This matters because many continuity plans focus on the production runtime while leaving the control path concentrated. A game may run in several regions but still depend on one identity system for administrators, one CI/CD organization, one artifact registry, one signing key, and one support workflow capable of changing player accounts or the game economy.
The important separation is between three planes:
The service plane runs the player experience: APIs, sessions, game servers, authoritative data, queues, and caches.
The control plane changes the service: cloud consoles, CI/CD, LiveOps tools, feature flags, account support, secret management, and privileged access.
The recovery plane rebuilds the other two: immutable backups, infrastructure definitions, deployment artifacts, independent credentials, recovery documentation, and a clean environment.
If the same compromised identity can modify production, delete backups, change DNS, and disable monitoring, these are not independent planes.
For game systems, security recovery should also cover game-specific integrity. Entitlement and economy changes should be traceable through an append-only or otherwise tamper-evident record. High-risk support actions should require stronger approval. Build artifacts should be signed and reproducible. Administrative access should use phishing-resistant authentication, short-lived privileges, and a tested break-glass path.
The operational question is not only “Can we block the attacker?” It is “Can we still ship a safe patch, protect the economy, communicate with players, and restore trusted operations?”
4. A Global Service Can Still Have One Shared Brain
From 28 to 31 October 2021, Roblox experienced a 73-hour outage. At the time, Roblox said around 50 million players used the platform each day. Its postmortem described two difficult performance problems inside Consul, a system used for service discovery, health checks, locking, and key-value data. The impact became much larger because one Consul cluster supported many workloads. Nomad and Vault also depended on it, while critical monitoring depended on affected systems and provided less help when it was most needed. (Roblox postmortem)
Roblox’s databases were not the root problem. The platform lost the shared coordination required for services to find one another, schedule containers, and retrieve production secrets.
This is the hidden risk behind many “global” architectures. Compute may be distributed while the system still has one logical brain: one service-discovery cluster, global database writer, identity provider, KMS hierarchy, DNS account, configuration service, or observability stack.
The response is not to duplicate every component blindly. It is to divide the platform into cells with bounded failure domains.
A sovereignty-aware regional cell should be able to run the essential player journey with limited dependence on other cells. Depending on the game, a cell may contain:
- local APIs and backend workers;
- session and game-server capacity;
- queues and replayable event logs;
- the authoritative data assigned to that region or shard;
- caches and rebuildable read models;
- local secrets and key access;
- monitoring and operational access that do not disappear with the central platform;
- a deliberate reduced-service mode.
The global layer should coordinate only what truly must be global: routing, a directory of account or shard ownership, policy distribution, aggregated telemetry, and carefully designed cross-region workflows.
This makes the boundary explicit. A failure in one cell should not require every other cell to stop. A central control-plane failure should not automatically terminate already-running matches or invalidate every active session. Observability needed to diagnose a cell should not depend entirely on the cell being healthy.
Roblox later described work toward multiple Availability Zones and data centers, additional Consul clusters, workload separation, and removal of circular observability dependencies. The larger lesson is simple: redundancy inside a shared dependency does not remove the shared dependency itself.
5. Laws and Product Policies Are Runtime Dependencies Too
In December 2022, the US Federal Trade Commission announced agreements requiring Epic Games to pay $520 million over allegations involving children’s privacy and unwanted charges. The orders included stronger privacy defaults, consent requirements, and deletion obligations. Epic’s Cabined Accounts show what policy-aware product behavior looks like in practice: a player below 13 or below the age of digital consent in their country can continue to play supported games, while features such as voice and free-text chat, real-money purchases, recommendations, marketing, and some account linking remain restricted until consent is provided. (FTC, Epic Games)
This is not itself a data-residency case. It illustrates a closely related architectural point: country, age, consent, and policy can change what an account is allowed to do while the core game remains available.
Cross-border data rules can change just as materially. In July 2020, the Court of Justice of the European Union invalidated the EU-US Privacy Shield while leaving Standard Contractual Clauses valid subject to the required assessment and safeguards. In July 2023, the European Commission adopted a new adequacy decision for the EU-US Data Privacy Framework. (CJEU, European Commission)
The legal basis for a data flow is therefore not a permanent infrastructure constant.
For a global game, policy should be executable rather than buried in documents and hard-coded assumptions. The architecture needs to know:
- which region is authoritative for an account or dataset;
- which fields are personal, sensitive, financial, or operational;
- which countries, services, and staff roles may store or access them;
- which events may cross a border, and in what minimized or pseudonymized form;
- how consent, deletion, retention, export, and legal-hold workflows propagate;
- how a transfer can be stopped without taking the whole game offline.
Global distribution does not mean copying all data everywhere. A better pattern is to keep regulated authoritative data within its allowed boundary, move only approved tokens or events, and make shared services operate on the minimum information they need.
The legal interpretation belongs with qualified counsel. The engineering responsibility is to make the resulting policy enforceable, observable, and changeable.
Start With Capabilities, Not Regions
The architecture decision becomes clearer when a team classifies game capabilities before choosing a recovery pattern.
Integrity-critical: purchases, entitlements, inventory, currency, competitive results, and security keys. During uncertainty, it may be safer to stop risky writes than to create duplicate grants or an unreconcilable economy.
Continuity-critical: authentication, account lookup, session allocation, matchmaking, and the core game loop. These may justify warm standby or active capacity in more than one location.
Deferrable: chat history, notifications, leaderboards, recommendations, non-critical social updates, and some telemetry. These can often queue, degrade, or become temporarily read-only.
Rebuildable: caches, search indexes, derived analytics, and materialized views. These should be recreated from authoritative sources rather than protected as if they were the source of truth.
Each capability should have its own recovery target:
- RTO: how long it may remain unavailable;
- RPO: how much recent state may be lost;
- integrity rule: which writes must stop if authority is uncertain;
- degraded mode: what players can safely do before full recovery;
- data boundary: where the data may live and who may access it;
- recovery evidence: the last successful restore or failover result.
A single RTO and RPO for “the backend” hide the decisions that matter. The store, a live match, a leaderboard, and an analytics pipeline do not carry the same player promise or business risk.
Replication, Backup, and Portability Solve Different Problems
Replication helps a service stay current and available. It is not protection from every kind of loss. A destructive command, corrupted update, bad economy grant, or compromised administrator can propagate quickly to healthy replicas.
Recovery history should therefore be independently protected: point-in-time recovery, immutable retention, separate accounts and credentials, and—where justified—another region or provider. The recovery path must not require the same identity, KMS, DNS, or control plane that may be unavailable or compromised.
Portability is another layer. It does not require every game to run active-active across multiple clouds. It means preserving affordable options:
- declarative infrastructure and repeatable account bootstrap;
- versioned schemas and documented ownership boundaries;
- portable data exports and regularly tested imports;
- deployment artifacts stored outside the only production path;
- adapters around deeply provider-specific services where change is plausible;
- pre-agreed quotas, certificates, network routes, and operating procedures;
- idempotent jobs and replayable events that can survive movement and retries.
The correct recovery pattern can differ by capability. Backup and restore may be enough for a rebuildable internal tool. A pilot light may protect a critical data service. Warm standby may fit login or LiveOps APIs. Active-active may be justified for a very small set of latency-sensitive or interruption-intolerant functions. Cross-cloud recovery may be appropriate where loss of one provider is a credible business-level scenario.
The goal is not maximum duplication. It is enough independence to preserve the choices the business may realistically need.
Resilience Exists Only After It Has Been Exercised
A diagram is not evidence of recovery.
The useful evidence is operational: a backup restored into a clean environment, a regional cell started from code and artifacts, a DNS or identity fallback used under controlled conditions, a queue replayed without duplicate rewards, and a game economy reconciled after a simulated outage.
GameDays should test product behavior as well as infrastructure behavior. For example:
- remove a region from matchmaking and measure player reassignment, latency, queue time, and match cancellation;
- make the central configuration service unavailable and confirm that cells continue with a safe cached policy;
- disable purchases and trading while keeping owned content and core play available;
- restore player inventory to a point in time and reconcile transactions that occurred around the boundary;
- operate from the recovery credentials and communication channel rather than the normal admin path;
- rebuild observability before declaring the service recovered.
The most useful scorecard is not only monthly uptime. It includes restore success rate, measured recovery time, achieved recovery point, age of the last verified backup, failover completion rate, time spent in degraded mode, player-impact metrics by geography, and the concentration of critical dependencies.
This also changes the executive conversation. Resilience becomes a set of tested business capabilities with known cost and coverage, not an abstract claim that the cloud is redundant.
A Practical Way to Begin
Most studios do not need to rebuild their entire platform or adopt full multi-cloud active-active operation. They need to remove the most dangerous concentrations in a deliberate order.
First, map and classify. Trace the player journey from login to session, progression, rewards, purchases, support, and LiveOps. Mark the authoritative data, legal boundary, common dependencies, and business owner for each step. Agree on capability-specific RTO, RPO, and degraded behavior.
Second, make recovery independent. Protect immutable history outside the production administrator’s direct control. Reproduce infrastructure from code. Preserve artifacts and key recovery procedures. Perform one clean-room restore and record the real time, missing dependencies, and manual steps.
Third, isolate and exercise. Split the highest-risk shared dependency or create the first independent regional cell. Add the minimum routing and policy layer needed to move traffic safely. Run a scenario that includes technical failure, player communication, operational authority, and reconciliation.
Each step should create a real option: recover data, continue in a reduced mode, move a capability, or enter a new region without redesigning the entire service.
Designing for What Comes Next
No architecture can remove every risk. Nor should every game pay for the most complex possible design.
But teams can decide where failures stop, which data remains authoritative, what continues in degraded mode, how recovery is kept independent, and which future choices remain open. Those decisions make a service safer not only during disasters, but also during ordinary growth, launches, migrations, acquisitions, provider changes, and new-market expansion.
This work requires more than an architecture diagram. It connects backend design, real-time multiplayer, networking, databases, cloud infrastructure, security, LiveOps, observability, and recovery operations. It must be implemented and then tested under realistic conditions.
CosmoUniverse brings that design-to-operation experience to global games and live services. We can assess an existing architecture, expose hidden failure domains, define a practical resilience roadmap, and implement the backend, regional-cell, data, infrastructure, LiveOps, and automation work required to make it real. Our experience spans more than 30 years of online-game development and operation, systems serving more than 8 million peak concurrent users, and games reaching more than 800 million players.
Our goal is not to promise that nothing will fail. It is to help clients build systems that can recover, move, and continue to evolve when assumptions change.
Is your service ready for the assumptions to change?
References and Further Reading
Incident status and linked sources last checked on 20 September 2026.
- Reuters — AWS unable to restore Bahrain region and one UAE cloud zone after war damage (15 September 2026)
- WIRED — Customer Data Permanently Lost in Iran Strikes on Amazon Data Centers (17 September 2026)
- Game Developer — Data center blaze wipes out Rust player data across 25 EU servers
- Internet Society — 2024 West Africa Submarine Cable Outage Report
- Riot Games — Leveling Up Networking for a Multi-game Future
- Axios — Riot Games says League of Legends source code was stolen in cyberattack
- Roblox — Roblox Return to Service: 28–31 October 2021
- US Federal Trade Commission — Epic Games privacy and billing settlements
- Epic Games — Cabined Accounts
- Court of Justice of the European Union — Schrems II press release
- European Commission — EU-US data transfers
- AWS Well-Architected — Plan for Disaster Recovery
- Video reference — 데이터를 3곳에 나눠도 안전하지 않은 이유