Compliant, and completely down

Most disaster-recovery stories are about a drill. This one isn't. In early March 2026 the cloud region our entire platform lived in started failing for real — one availability zone, then the next, then all of it — and it stayed down long enough that "we'll fail over" stopped being a plan and started being a wish.

Here's the part that still sits with me: we weren't caught out because we cut corners. We were caught out because we followed the rules. Our biggest customers — several of them government — required their data to live inside their region, so everything ran in a single region by mandate, not by laziness. When that region went down, and took its own backups with it, we had nowhere to run. The easy story is that compliance did this to us. The truer one — which I'll get to — is less comfortable, and more on us.

This is what happened — honestly, and it was very real for us, even if you'll struggle to find it in any headline (more on that near the end). Then a genuine ask: I'll tell you what we did; I want to hear what you did.

Why everything lived in one region

We run an energy-IoT platform — thousands of field devices streaming telemetry through an MQTT broker into databases, an ETL pipeline, dashboards, and an admin plane. All of it sat in one cloud region, and that wasn't an accident of architecture.

Our customers put it there — not the statute. The telemetry we collect doesn't, on its own, trip any of the region's data-residency laws. But the customers we serve are large, and several are government, and for them hosting their data inside their own region isn't a preference — it's a condition of the contract. Meet it or don't win the work. So we met it, and that single requirement quietly decided our whole topology: everything, in one region. "Single-region" wasn't a shortcut we'd get around to fixing — it was the compliant answer to what our customers demanded. On paper, we were doing it right.

And here's the detail that matters more than it should: they required the live data to sit in-region. Nobody ever said a word about backups. Hold that thought.

And here's the part that makes it sharper: just months earlier, we had deliberately migrated the whole stack into that region to become compliant for the customers there. Not only AWS — the relational database, object storage, compute — but every managed piece we lean on: EMQX (the MQTT broker that ingests device telemetry), MongoDB, and ClickHouse (our analytics store). We moved all of it in-region, on purpose, to satisfy residency. Storage, messaging, databases, analytics — every vendor, co-located, fully within the law.

And it wasn't only the live system at stake. Behind it sat everything — the entire history we'd captured, processed, and analysed for those regional customers, and its backups — in the same infrastructure. The migration that made us correct is the same migration that put every egg in one basket.

What "the region went down" actually meant

It didn't fail all at once, which almost made it worse. First one availability zone got unstable. Then a second. Then the whole region became inaccessible — for us, completely.

The blast radius was the entire stack:

The relational database (AWS RDS) went down and stayed down. That alone takes out the APIs, the ETL, and the dashboards — it's the spine.
Compute went offline. EMQX, MongoDB, and ClickHouse all dropped into a degraded state that could die at any moment. Device-to-cloud ingestion stopped — the telemetry had nowhere to land.
And the two things you reach for in exactly this moment were also inside the fire: the backups — and with them the entire historic archive — and the migration tools you'd use to escape. The snapshot storage was in the same region. The managed migration tooling was impacted by the same outage. Your fire escape was on fire.

From AWS: no ETA. No SLA that meant anything. On the first day, I put it to our enterprise account manager in writing — plainly, and on the record:

Confirm the scope and root cause of the disruption.

An estimated time to resolution.

Which SLA actually applies to our services in this region.

The SLA credit process, if applicable.

Any mitigation from your side to limit the damage.

The two that mattered — the ETR and the SLA — never really came back. The guidance amounted to "move to another region." A month later, we still couldn't cleanly stop or fully vacate the impacted region's services. The one concrete thing that ever materialised was that we wouldn't be billed for the affected region. That was it.

I don't write that to dunk on a vendor we still run on. I write it because it's the load-bearing lesson: a single-region dependency plus an SLA is not a resilience strategy — it's a promise that isn't there when you actually need it.

What we did — and who actually did it

I need to be precise about this, because it's the part I most want to get right. The people who pulled us through were the engineering team in Bangalore. They ran the war-room. They got on the phone with every vendor — AWS, EMQX, MongoDB, ClickHouse — and pushed for answers. And they executed the disaster recovery itself, hour after hour, for a week: the rebuild, the restores, the site-by-site data recovery. That was them.

My own part was smaller, and I won't dress it up. I coordinated, I made the calls that needed a decision, and I became the single voice for the updates — the person stakeholders and customers heard from while the team kept their heads down in the actual work.

And I did most of that from a hospital. This landed during the hardest stretch of my life: I was beside my mother's hospital bed, and nearly all of the coordination happened from there — in the gaps, walking between her room and the corridor. I mention it not for sympathy but because it's the honest shape of that week, and because it's exactly why the credit belongs to the team. When I couldn't be fully present, they carried it. That is what a good team is.

Here's what they did, in order.

The first message, at 1:40 AM, was an honest one: the database is down, there's no ETA, and migrating out isn't even feasible right now because the tools are down too. That's the moment you stop waiting for the region and start rebuilding elsewhere. The call went out — rebuild a temporary production pipeline in a different region (one we were permitted to use for recovery) — while escalating hard with every vendor to get the source accessible enough to pull what we could.

The rebuild itself was fast — infrastructure back up in about a day. Infrastructure-as-code earns its keep on exactly this day; you're not clicking through consoles, you're re-running your setup somewhere new. The dashboards and admin app came back first. The broker and document store were restarted in the new region and telemetry started flowing again.

Then came the long part. The infra took a day; the data took a week. We imported the database table by table — one raw-data table alone was well over 400 GB, and the database fell over on memory partway through, so we sized it up and kept going. Raw device data came back site by site — 207, then 330, then 500, then all of them — while I sent a structured update at nearly every milestone, day and night, because in an outage silence is its own kind of damage. We restored live data first, pausing the historical backfill so customers saw current readings as early as possible, then reconciled the gap in phases. By the seventh day, everything was back — in a different region than the one we'd been pinned to.

Customers were told, plainly, while it was happening. Not after.

The hardest part: your backups were in the same fire

If there's one thing to take from the mechanics of it: the backup availability zones went down too. The whole discipline of "take snapshots, keep backups" quietly assumes the backups survive the thing that kills production. Co-locate them in the same region — which is exactly what a residency requirement nudges you toward — and a regional failure takes the primary and the safety net in one move. Keeping recovery alive when the recovery data itself is trapped in the dead region was the hardest engineering of the week.

The uncomfortable lesson

Here's the thing I keep turning over, and it's less flattering than "the rules trapped us." Pull the incident apart and the single point of failure wasn't one thing — it was three, stacked:

We bet everything on one platform. Every layer — database, storage, compute, broker, analytics — on a single cloud, in a single region. Vendor concentration is the risk; we'd built the whole house on one plot of land.
We read "residency" more strictly than anyone required. The contracts said the live data stays in-region. They said nothing about an encrypted backup living somewhere else — and, as it turns out, the general data-protection law would have allowed one with the right safeguards. A compliant escape hatch was available to us the entire time. We just never built it.
Not because we didn't know — because we didn't have the hands. This is the part founders will recognise: the blocker wasn't intent, it was capacity. Money and people. A small team fighting to ship product does not have a spare quarter to stand up and maintain a second, cross-provider, encrypted-backup DR path — until the week it desperately wishes it had.

But here's what stops me short of a tidy moral. Say we had built all of it — the second provider, the out-of-region backup, the failover. Would it have saved us? Maybe not. The fallback we did have — a footprint in a neighbouring region — went down in the same event. When the trigger is geographic or geopolitical, the blast radius isn't one data centre; it's a whole neighbourhood of them. Multi-AZ, a nearby region, even a second cloud provider in the same country — all of it can go at once. In-geography redundancy is comfort, not a guarantee.

So the honest question this left me with isn't "why weren't you multi-region?" It's the one I still can't answer cleanly: what is the plan when the entire geography you're legally pinned to goes down at the same time? The only true escape is out-of-geography — the one thing residency won't allow. There's no neat fix. You mitigate — keep an encrypted backup far enough away that you can at least rebuild from cold — you name the residual risk out loud, and you stop pretending an SLA covers any of it.

And blaming "compliance" for this would be the easy way out — easy, and wrong. Compliance didn't fail us. We made choices, we skipped options we could have taken, and beneath them sat a failure mode that perhaps nobody fully solves. All three of those are truer than a villain.

The laws are more permissive than we treated them

This is the correction I owe anyone who takes the story above as gospel, because I got it wrong in my own head for a long time. The data-localization laws did not actually forbid our escape.

Across the GCC, the general data-protection statutes mostly don't ban data leaving the country — they permit cross-border transfer with safeguards. The UAE's federal law (Personal Data Protection Law, Decree-Law 45/2021, Arts. 22–23) allows transfers to "adequate" destinations, or under derogations like explicit consent and contract-necessity. Saudi Arabia's regime (SDAIA's 2024 Data Transfer Regulation under the PDPL) permits transfer via an adequacy list or standard contractual clauses. Qatar's financial-centre rules, Oman (Royal Decree 6/2022, Art. 23) and Kuwait's CITRA regulations are the same shape — permitted, with conditions. Kuwait's rules even require you to have disaster recovery.

The genuinely hard "data may not leave" mandates are sector-specific — most sharply UAE health data (Federal Law No. 2 of 2019, Art. 13, a default prohibition with carve-outs), plus certain banking and government rules. Our energy telemetry was none of those.

So what actually pinned us wasn't a statute. It was customer contracts (some of them government) demanding the live data stay in-region — and one stubborn architectural fact: on our cloud there is exactly one in-country region. One. So "in-country DR" could only ever mean cross-availability-zone, never cross-region — which is no protection at all the day the whole region goes dark. The law left a door open. Our contracts, our architecture, and our own caution had quietly bricked it up.

The ground is shifting, at least: sovereign-cloud offerings (dedicated in-country infrastructure, local-partner clouds) and formal adequacy mechanisms are arriving to let residency and resilience finally coexist. Slowly.

What we took away

If I compress the whole week into what we actually walked away believing:

Compliant is not the same as resilient. We met every requirement and still went fully dark. Compliance is a floor, not a safety net — necessary, and nowhere near sufficient.
Never bet the whole company on one platform. The biggest miss wasn't a region — it was leaning the entire stack on a single provider. A hybrid, multi-provider posture is the hedge we should have paid for.
The escape hatch was cheaper than we assumed. One encrypted backup, out of the region, would have broken no contract and no law. We simply hadn't built it.
The blocker was capacity, not knowledge. We knew better in theory; a small team shipping product didn't have the hands to build and run it. Naming that honestly is the first step to funding it.
Some failures don't have a clean fix. When a whole geography can go at once, "never fail" isn't on the menu. Plan to rebuild fast from cold, not to be invincible.

What I'd tell a founder with no budget for a full replica

A hot, always-on second region is the textbook answer, and out of reach for most early companies. The good news: you don't need it. AWS frames disaster recovery as a spectrum — backup-and-restore, pilot light, warm standby, multi-site — and surviving a whole-region loss takes only one rung up, to pilot light. On a startup budget, in priority order:

Make rebuilding boring. If your whole environment is infrastructure-as-code, "start over in a new region" is a script, not a project. The highest-return item here — it's what turned our infra recovery into a single day.
Get one encrypted backup out of the blast radius — a different region, or better, a different provider — encrypted so residency is respected and (as I learned the hard way) legal under the safeguards these laws already give you. This is the cheap escape hatch we didn't build. Don't skip it.
Pilot light, not hot standby. Keep the recovery target defined and dormant — templates, replicated backups, DNS ready — not running. Cheap to hold, fast to ignite.
Don't marry one platform. A second in-country provider — another cloud's local region — can be your compliant DR site when your primary's whole region is gone. Concentration is the risk; a little diversity is the hedge.
Write the RTO/RPO down before the incident, not during. "We can lose an hour of data and be back in a day" changes every decision you make at 2 AM.
Practise the restore, not just the backup. A backup you've never restored is a hope, not a plan.

And the honest caveat, founder to founder: none of this is free, and the real obstacle usually isn't knowing — it's capacity. Money and hands you don't have while you're fighting to ship. So don't try to do all of it. Do the top two — infrastructure-as-code, and one encrypted out-of-region backup. They're cheap, they're quick, and together they're the difference between a bad week and a lost company.

And no one ever wrote it down

Here's the part I still can't shake. For an outage that took real businesses offline for the better part of a week, there was — as far as I can find — no public post-mortem, no root-cause analysis, no acknowledgement that a region had a week like that. Nothing. A cloud region can fail that hard and leave essentially no public trace.

The missing SLA was one thing. The missing record is worse: you cannot hold anyone to account for a failure that, officially, never happened. If you're betting your company on someone else's infrastructure, that opacity is part of the deal — and almost nobody prices it in.

The ask — genuinely

That's our part, told straight. Now I actually want to learn from yours. Three questions, and I mean them:

If you run under data-residency requirements — statutory or, like us, contractual — how do you actually do disaster recovery? What does it look like when your customers won't let the live data leave the region?
What are regulators and cloud providers doing about the resilience-vs-residency gap? Sovereign cloud, in-country second regions, cleaner backup carve-outs — what's genuinely working versus just announced?
For founders without the budget or the hands for a full replica — what actually worked for you? The scrappy, real thing, not the reference architecture.

Tell me in the comments, or write to me. I'll fold the best of it into a follow-up. We shared what a real regional failure looked like from the inside. I'd like to know what it looked like from yours.