Skip to content

Disaster recovery + chaos

Recovery posture, backup inventory, autoscale policy, and chaos schedule for cashless-v2. This page is a runbook index — sections link back to the planning doc that is the source of truth.

Source planning docs:

Pre-launch scaffolding

The DR + chaos scripts under scripts/dr/ and scripts/chaos/ have not yet been executed against any cluster. Every script hard-aborts unless DR_ENV / CHAOS_ENV is sandbox or staging.

RPO / RTO targets

Tier-0 is the auth + payment path. Tier-2 is operator-facing surfaces where minutes of downtime are tolerable. Targets are aspirational until the first drills land observed numbers.

Tier Services RPO RTO
0 cashless-auth, payment writes on pa_v4 5 min 30 min
1 pa_v4 non-payment, cm_v2 writes, MySQL primary 15 min 1 h
2 cm_v2 reads + reports, frontends, async workers 1 h 4 h

Combined cluster-plus-database failure (Tier-0 worst case): 40-minute end-to-end target via parallel restore paths (dr-runbook §3.8).

Backup matrix

The MySQL instance is mycashless-pa-development:us-central1:cm1 — every backend service uses it via a distinct logical schema.

Target Tool Cadence Retention Restore command
Cloud SQL (MySQL 8.0) Cloud SQL automated backup Daily 7 days gcloud sql backups restore <id> --restore-instance=<new>
Cloud SQL transaction logs (PITR) Cloud SQL binary-log retention Continuous 7 days gcloud sql instances clone <src> <new> --point-in-time=<ts>
App-level mysqldump → GCS cashless-db-backup CronJob (03:00 UTC) Daily 30 days Fresh Cloud SQL instance + mysql < dump.sql.gz
User uploads (GCS) Object versioning On write 90 days gsutil cp gs://<bucket>/<obj>#<gen> gs://<bucket>/<obj>
Event assets (GCS) Object versioning On write 180 days Same gsutil cp ...#<gen> pattern
CSV / finance exports (GCS) Object versioning On write 7 days Same gsutil cp ...#<gen> pattern
Secrets (GCP Secret Manager) Secret Manager versioning Continuous Until destroyed gcloud secrets versions enable <prior> + ESO force-sync
ArgoCD Application CRs + values Git (parent monorepo) Per commit Forever kubectl apply -f deploy/argocd/*.yaml after bootstrap
Helm charts Git (each submodule) Per commit Forever ArgoCD reconciles from cashless-v2 HEAD

The app-level mysqldump is defence-in-depth against losing the Cloud SQL control plane itself (IAM revocation, billing suspension, regional outage). The CronJob lives at services/cashless-db-backup/ with schedule: "0 3 * * *" and retentionDays: 30. Daily verification is gated on scripts/dr/dr-verify-backup.sh.

Restore procedures

Scenario 1 — MySQL instance is gone

Tier-1 / Tier-2 flow. Restore from the most recent SUCCESSFUL automated backup into a new Cloud SQL instance, then repoint the cluster at it.

  1. Confirm sandbox/staging context: gcloud config configurations activate dr-drill then verify the project.

  2. Run the restore script (defaults to --dry-run; add --execute):

    DR_ENV=sandbox \
  scripts/dr/dr-restore-sandbox.sh \
  --project=mycashless-pa-development \
  --instance=cm1 \
  --source=backup \
  --tier-label=tier-1 \
  --execute

    Phases: A — provision target (~8–15 min), B — gcloud sql backups restore (~10–25 min), C — wait for state=RUNNABLE, D — verification, E — teardown (EXIT trap).

  3. Verify the restored instance — SELECT 1 returns 1, schema fingerprint MD5 matches source, smoke row counts on alembic_version, user, event, transaction within ±1%, cashless-auth /healthz returns 200 within 30s.

  4. Repoint the cluster — commit a change to deploy/argocd/values/*.yaml setting cloudSqlProxy.instanceConnectionName, push to cashless-v2, let ArgoCD reconcile.

For Tier-0 use the PITR variant: same script with --source=pitr --pitr-timestamp=<RFC-3339>. Phase B runs gcloud sql instances clone instead; typical wall-time 5–20 min.

Scenario 2 — Single Application went bad

Cluster is fine; one ArgoCD Application is Degraded or stuck on a bad image. No DB restore needed.

  1. Identify the bad app and prior good revision:

    kubectl -n argocd get app <name> -o jsonpath='{.status.conditions}'
kubectl -n argocd get app <name> -o jsonpath='{.status.history}' | jq .

  2. Pin image.tag back to the prior sha-<12 hex> in deploy/argocd/values/<service>.yaml, push to cashless-v2. Pin the tag in the values file directly — the helm.parameters override layer is bypass-prone.

  3. Force a sync: kubectl -n argocd patch app <name> --type merge -p '{"operation":{"sync":{}}}'.
  4. Watch the rollout: kubectl -n cashless rollout status deploy/<svc>.

Never kubectl edit the live Deployment — ArgoCD reverts it on the next sync.

Scenario 3 — Region outage

We do not have multi-region

Every component runs in us-central1: GKE, Cloud SQL, GCS, Secret Manager. A full us-central1 outage takes the platform down for the duration of the region outage. Backup buckets are single-region; cross-region replication is open question §6.1 in the DR runbook (due 2026-06-15). In-region recovery only — there is no manual failover plan at this writing.

Scenario 4 — Full cluster rebuild

Cluster is gone; Cloud SQL + Secret Manager + GCS are intact.

  1. Re-create GKE:

    gcloud container clusters create-auto mycashless-development \
  --location=us-central1 \
  --project=mycashless-pa-development \
  --release-channel=regular

  2. Fetch credentials, bootstrap secrets (skip if project state intact), run services/cluster-bootstrap/scripts/bootstrap.sh.

  3. Apply bootstrap manifests (clustersecretstore.yaml, argocd-repo-creds*.yaml, ghcr-pull-secret.yaml) and patch every cashless namespace ServiceAccount with imagePullSecrets: [{name: ghcr-pull}].
  4. Wait for ArgoCD root sync — App-of-Apps discovers every deploy/argocd/*.yaml in sync-wave order (cashless-secrets → cashless-migrate + argocd-image-updater → everything else). All apps Synced + Healthy within ~30 min once images land.

End-to-end target: 1 h for a Tier-2 rebuild with the database intact. Full procedure: dr-runbook §3.7.

Autoscale policy

HPA is disabled across the board

Every cashless-v2 chart ships autoscaling.enabled: false. The HPA templates exist and are well-formed but the gate is off — the cluster will not horizontally scale at launch unless ops enables each chart's HPA with the targets below.

Recommended HPA values (audit-only proposal — not yet landed in chart values). Current min is the static replicas: 2.

Service Current Recommended min / max Metric Notes
cashless-auth 2 5 / 15 CPU 60% Argon2id CPU-bound; trip earlier than 70%.
pa_v4 2 4 / 20 CPU 65% Plus 2 vCPU / 2 GiB pods, UVICORN_WORKERS=4.
cm_v2 2 4 / 12 CPU 65% Add targetMemoryUtilizationPercentage: 70.
cashless-admin-ui 2 2 / 10 CPU 70% nginx + SPA — chart defaults fine.
cashless-refund-ui 2 2 / 10 CPU 70% Same.
marketplace-user-ui 2 2 / 10 CPU 70% Same.
personalaccount-ui 2 2 / 10 CPU 70% Same.
cashless-redis 1 1 (singleton) n/a Standalone bitnami subchart; no horizontal scaling.

Scale-up: aggressive (+100% / 30s, 0s stabilization). Scale-down: soft (-50% / 60s, 300s stabilization). PDBs ship minAvailable: 1 everywhere — bump to floor(minReplicas × 0.5) when raising minReplicas. Pre-warm the fleet 30 min before event-start via a one-shot kubectl scale to avoid HPA cold-start lag.

Chaos drills

Seven scenarios scaffolded; none have executed yet. First run is gated on staging being SLO-validated under k6 baseline load.

Scenario Inject method Target RTO
3.1 Primary DB failover gcloud sql instances failover or toxiproxy down toxic < 90s
3.2 Redis unavailability NetworkPolicy deny-all-ingress for 60s < 90s
3.3 Stripe API 503 Toxiproxy in front of api.stripe.com, http 503 toxic < 30s
3.4 Workload Identity / GCP auth failure Manual SA disable in IAM for 5 min < 60s creds
3.5 Pod kill (cashless-backend-v2) kubectl delete pod (optionally repeat) < 30s
3.6 Slow disk (DB connection) Toxiproxy latency toxic, 500ms for 5 min < 30s
3.7 DNS failure (CoreDNS) kubectl scale deploy/coredns --replicas=0 for 30s < 30s

Cadence. Pre-launch hardening: each scenario weekly until two consecutive green runs. Launch readiness: full suite within 7 days of sign-off, re-run within 24h. Post-launch: quarterly full-suite plus single-scenario re-run after any related infra change.

Operators. Ops on-call + the service tech lead for the target. WI revoke (3.4) and DNS failure (3.7) require two operators. Never scheduled during a live event window, an active Sev-2+ incident in the last 4h, the ArgoCD sync window (Mon–Fri 09:00–11:00 UTC), or the 24h before a planned launch.

DR drill cadence (shares the chaos rotation): dr-verify-backup.sh daily, Tier-1 restore drill quarterly, Tier-0 PITR drill quarterly, object-storage restore semi-annual per bucket, secrets rollback after every credential rotation, combined cluster+DB drill semi-annual.

Single points of failure

Honest SPOF inventory

The platform is single-region and several stateful tiers are single-instance. Pre-launch this is accepted posture; post-launch the items in this list are the backlog priority.

  1. us-central1 region. Everything — GKE, Cloud SQL, GCS, Secret Manager — runs in one region. No multi-region failover.
  2. Cloud SQL instance (cm1). Single primary backing every backend service via separate logical schemas. HA replica handles zonal failure (30–90s, not yet drilled); full instance loss requires the restore procedures above.
  3. cashless-redis (standalone). Single bitnami pod with AOF persistence on an 8 GiB PVC. Crash recovery only — no failover. Sentinel + replicas is the follow-up before traffic doubles.
  4. GHCR (ghcr.io/cashless-media/*). A GHCR outage breaks image pull for any new pod. Running pods keep their cache; a cluster rebuild during a GHCR outage is blocked. No backup registry today.
  5. HPA disabled. Static replicas: 2 is 2–8× under the safe minimum for event-start peak. Operator pre-warm is the bypass; the runbook gap is the autoscale policy not yet applied.
  6. No Terraform. GKE + Cloud SQL + GCS + IAM are provisioned imperatively. Cluster destruction recovery depends on the operator following services/cluster-bootstrap/README.md exactly. Migration tracked for 2026-Q3.
  7. GCS buckets are single-region. Object versioning protects against accidental delete + overwrite, not region loss. Cross-region replication is DR open question §6.1.

Incident response

TBD — channels + paging wiring

The incident-response runbook at deploy/runbooks/incident-response.md is the source of truth for severity classification, response-time SLOs, and the post-mortem template. Concrete Slack channel names, PagerDuty service IDs, and the escalation matrix are still being finalised and will be filled in here once the on-call rotation is live.

Severity ladder summary (full detail in the runbook):

Severity Trigger Pager ack
Sev-1 Platform-wide outage; payments lost or login fully down 5 min
Sev-2 Significant partial impact; one processor down 15 min
Sev-3 Degraded but contained; SLO budget burn elevated 1 h
Sev-4 Cosmetic / non-customer-facing next day

Chaos and DR runs post to operator Slack channels (#chaos-runs / #dr-runs) before injection and on completion via CHAOS_SLACK_WEBHOOK / DR_SLACK_WEBHOOK.

Cross-references