The Problem
Disaster recovery is the only system you pay for every month hoping never to use. The standard answer — pilot light — keeps a shrunken copy of production permanently running in a second region or cloud: a small standby database, a standby cache, maybe a scaled-to-one service. It recovers fast. It also bills you 24/7/365 for an outage that may never come.
The setup I inherited had a second problem on top of the first. Failover wasn't just expensive — it was a hand-run runbook. A human followed numbered steps under pressure, at the exact moment when humans are worst at following numbered steps. And like every runbook, it silently rotted as the infrastructure it described drifted away from it.
There was a third wrinkle specific to this platform: primary wasn't uniform. Most core services ran on AWS. A separate group of AI services ran on GCP Cloud Run. So "our DR plan" actually needed to be two plans pointing in opposite directions.
This post walks through the design I landed on: a cold-start model where nothing DR-sized runs day to day, failover is a single Terraform apply, and the whole thing works in both directions.
Picking a DR Posture
Every DR conversation is the same trade: how much do you pay while nothing is wrong, versus how long you're down when something is. Four postures, roughly:
| Posture | Running day-to-day | Recovery time | Standing cost |
|---|---|---|---|
| Backup & restore | Backups only | Hours to days | Near zero |
| Cold start | Replicated data only; no compute, no DB | Tens of minutes | Storage only |
| Pilot light | Shrunken DB + cache, services at zero | Minutes | Continuous |
| Warm / hot standby | Scaled-down or full copy of prod | Seconds to minutes | Substantial |
Pilot light is the default recommendation, and for a payments platform or anything with a contractual RTO in single-digit minutes, it's the right one. But it's worth actually asking what your recovery objective is rather than inheriting one. If the business can tolerate tens of minutes of downtime in a genuine regional disaster — a rare, already-catastrophic event — then cold start buys you most of pilot light's recovery speed at storage-only cost.
Cold start trades recovery minutes for every single month's standby bill. That trade is good when disasters are rare and your RTO is measured in tens of minutes, not seconds. It is a bad trade if you can't state your RTO — because then you haven't made a decision, you've just made it cheaper.
Architecture: Two Directions, One State
Because primary was split across two clouds, the design has two independent failover directions living in a single Terraform state. They share no resources and can be planned or applied separately.
Two failover directions, opposite primaries, different activation levers.
Note that the two directions use different switches, and that's deliberate. Direction 1 is gated by a boolean — nothing exists until you declare a disaster, so the lever has to create infrastructure. Direction 2's services stay registered day to day because routing already points at them permanently; there the lever is just a desired count. Scaling from 0 to 1 is faster and far less risky than creating a service from nothing.
Terraform As The Runbook
The single biggest win here isn't cost — it's that the runbook became executable.
A markdown runbook describes intent. It drifts the moment someone changes a security group by hand. Terraform is the intent, and terraform plan tells you, before the disaster, whether your DR plan still matches reality. You can run that plan on a schedule. You cannot run a markdown file on a schedule.
Declaring a disaster collapses to one variable flip:
terraform plan -var dr_active=true -out=tfplan
terraform apply tfplanThat single apply restores the latest dump into a freshly created managed database, stands up the cache, deploys every service from its current image tag, and patches the new endpoints into each service's secret. The steps a human used to perform in order, under stress, are now a dependency graph the tool resolves.
Keep the disaster switch as a variable, not a separate workspace or a separate repo. One state file means plan always compares your DR definition against live reality. Split them and you get two things that drift apart — which is the exact failure mode you were trying to eliminate.
Keyless Cross-Cloud Auth — And The Trap In It
The always-on piece of Direction 1 is a database dump shipped from an AWS host into GCP object storage. The obvious way to authenticate that is a GCP service-account JSON key file sitting on the EC2 box. Don't. That key is long-lived, it doesn't rotate, it's exfiltratable, and it grants its permissions to anyone who can read that file.
Workload Identity Federation removes the key entirely. GCP is configured to trust identities from the AWS account directly: the instance presents its own AWS identity, GCP validates it and mints a short-lived token. No secret material ever lands on disk.
Here's the part that's easy to get wrong. The natural way to write that trust is "trust this IAM role." And that's usually too broad.
The general lesson: on AWS, an IAM role is not an identity — it's a hat that many machines can wear. If your federation condition names only the role, you've granted access to every current and future wearer of that hat. Scope the condition to the assumed-role ARN so it pins the individual workload.
Worth checking explicitly: the role name and the instance-profile name are frequently not the same string, and the console tends to show you the profile. Query instance metadata on the actual host to find out which one you're really assuming before you write the condition against it.
A Debugging Detour Worth Documenting
Getting that keyless flow working surfaced a genuinely confusing failure, and it's the kind of thing that costs an afternoon if nobody wrote it down.
The host enforced IMDSv2-only on its instance metadata service — a good hardening setting you should not weaken. The cloud CLI's AWS credential lookup, however, omits the IMDSv2 session token unless you explicitly enable it when generating the credential config. Without that flag, the metadata request comes back 401, which surfaces as:
Unable to retrieve AWS role name
Which sends you hunting through IAM for a role problem that doesn't exist. The error is about metadata access, not roles at all.
Before concluding it was a CLI bug I reproduced it across a wide span of CLI versions and against a freshly installed auth library. Identical behaviour throughout — so, not a regression, just a flag you have to pass. Ruling out "it's a version bug" is worth the twenty minutes; otherwise you burn a day upgrading things.
Two smaller traps from the same pipeline, both cron-specific:
- Client libraries and the CLI don't read credentials the same way. Setting the standard credentials environment variable satisfies the SDKs but not the CLI binary — that needs an explicit auth-login against the credential file.
- That auth step needs a quiet flag or it hangs forever. Re-authenticating an already-cached principal prompts "overwrite existing credentials?" with no default. Interactively you'd never notice. Under cron there's no TTY to answer, so the job blocks indefinitely — the worst failure mode, because nothing errors and nothing alerts.
Anything you intend to run under cron must be tested as cron — no TTY, minimal environment, absolute paths. A script that works perfectly in your SSH session and hangs silently at 3am is not a working script.
Size The Standby From Telemetry, Not From Limits
This is the part I'd most want a reviewer to take away, because it's the difference between DR that works and DR that only looks like it works.
The tempting way to size failover infrastructure is to copy the primary's configured limits. That's not sizing — it's copying someone else's guess. Configured limits are ceilings, often set once at launch and never revisited. They tell you nothing about actual consumption.
So I pulled 24-hour CPU and memory peaks per service from real production telemetry and sized against those. The results were not what the configuration implied:
| Service | Configured memory | Real peak usage | Outcome |
|---|---|---|---|
| AI inference service | 2 GB | ~10.2 GB | Grew to 12 GB |
| Business engine | 1 GB | ~235 MB | Reduced |
| Call-flow service | 512 MB | ~205 MB | Reduced |
The first row is the important one. Had failover ever fired with the copied configuration, that service would have OOM-looped immediately — crash, restart, crash — during the exact incident DR exists to survive. The standby would have been green in the console and useless in reality.
The other two rows paid for the first: both were significantly over-provisioned, so right-sizing the whole set was close to cost-neutral while removing the failure.
A cold standby is never exercised, so nothing tells you it's wrong until the day it matters. If you have DR infrastructure you've never load-tested and never sized from measurements, you don't have a recovery plan — you have an untested hypothesis with a monthly bill.
Patch Secrets Structurally, Not Wholesale
Failover changes endpoints. The DR database and cache have different hostnames than production, and every service needs the new values. The blunt approach — template out a fresh environment file per service — means Terraform now owns every secret in that file, including live production credentials it has no business holding.
Better: read the existing secret, parse it, replace only the specific keys that changed, and write it back. Everything else passes through untouched. Terraform manages the endpoints and never learns the credentials.
Two details that made this work in practice:
- Reuse the production database credentials on the DR instance. Creating the restored database with the same username and password means only the host key ever needs patching. A new random credential would mean patching three keys instead of one, in every service, for no security gain — it's the same data either way.
- Applications disagree about connection-string shape. One service read a bare
host:portand prefixed the scheme itself; two others expected a full URL. Confirm the shape from each service's source rather than assuming a convention. A patched-in value of the wrong shape fails at connect time — during failover.
One more: patching the stored secret only fixes future task launches. Already-running tasks keep the old value. If an endpoint can change while services are up, pair the patch with a forced redeployment triggered on the same change, or the running fleet quietly keeps talking to an address that no longer exists.
What Cold Start Actually Costs You
Everything above is the case for this design. Here's the honest other side.
Your RPO is your replication interval. Periodic dumps mean you can lose up to one interval's writes. Continuous replication — a managed migration service, or an external read replica in the target cloud — gives a dramatically better RPO. I chose dumps for operational simplicity and cost, and that is a real trade with real data loss attached. If the workload can't tolerate it, this is the wrong design and you should pay for replication.
Your RTO includes creating a database from scratch. Provisioning a managed instance plus restoring a full dump is the long pole, and it scales with data size. Time it. A DR design whose author can't state its RTO isn't finished, and "tens of minutes" is only an acceptable answer if you measured it and the business agreed to it.
Traffic cutover may still be manual. In my case DNS and routing changes were deliberately left outside the automation. That's defensible — you rarely want a machine unilaterally deciding to move production traffic between clouds — but it means the true end-to-end recovery time includes a human step, and your RTO must count it.
Untested DR is a hypothesis. Cold start is harder to verify than warm standby precisely because nothing runs. The mitigation is game days: actually apply it into an isolated environment, hit the health endpoints, verify data freshness, and time the whole thing. Terraform makes this cheap — stand it up, verify, destroy — which is itself an argument for the model.
Takeaways
- Pick a posture from a stated RTO, don't inherit pilot light by default. Cold start gets most of the recovery speed for storage-only cost — when tens of minutes is genuinely acceptable.
- Make the runbook executable. Terraform's
planverifies your DR still matches reality before the disaster. Markdown can't. - Keep the disaster switch in the same state as everything else so the two can't drift apart.
- Federate identity instead of shipping keys — and scope the trust condition to the workload, not to a role several machines share.
- Size standby infrastructure from measured peaks. Copied limits hide under-provisioning that only surfaces during the incident.
- Patch secrets key-by-key so your IaC never takes custody of production credentials.
- Test cron jobs as cron. No TTY, no interactive prompts, or they hang silently forever.
- State your RPO and RTO out loud. If you can't, you haven't designed DR — you've just made it cheaper.
Cold-start DR isn't right for every workload. But for a lot of teams quietly paying a pilot-light bill every month against an RTO nobody ever wrote down, it's worth asking whether you're buying recovery speed you don't actually need — and whether the standby you're paying for would even survive first contact with a real failover.