Reference only. This is the v0 prototype. The current plan is DESIGN.md, which replaces the setup binary, shared credentials and monthly rotation described below.

Backup one tenant namespace. Not restore. Not retention. Not PITR.

Nightly CronJob covering CNPG dumps, opt-in S3 buckets, and PVCs (direct mount or snapshot-clone). One restic repo per namespace on the off-site RGW.

Two binaries, same split as harmony-fleet vs harmony-fleet-deploy:

harmony-tenant-backup run — operate (in-cluster, or a laptop dry-run)
harmony-tenant-backup-setup — install RBAC + CronJob (workstation)

Out of scope: OpenBao, Ceph, K8s objects, Zitadel.

Data layout

Repo: s3:https://<target-endpoint>/<target-bucket>/<cluster-id>/<ns>, or .../<YYYY-MM> with rotate_repo_monthly (expiry = delete a whole prefix).
Host pinned to <cluster-id>-<ns>. Tags: run:, ns:, surface:pg|s3|pvc, plus cluster: / bucket: / pvc:.

Runbook

Build and push the image:

cargo build --release -p harmony-tenant-backup
docker build -f examples/harmony_tenant_backup/Dockerfile \
  -t harbor.example/harmony/ns-backup:v0 target/release
docker push harbor.example/harmony/ns-backup:v0

Config at <mount>/data/harmony/<ns>/TenantBackup (NAMESPACE maps to store path harmony/<ns>):
```
 bao kv put -mount=secret harmony/tenant-x/TenantBackup \
   target_endpoint=https://rgw.site-b target_bucket=tenant-backups \
   target_access_key=... target_secret_key=... target_region=default \
   source_endpoint=https://rgw.site-A source_access_key=... \
   source_secret_key=... rotate_repo_monthly=true
```
target_*: append-only at the target. source_*: read-only site-A, only for buckets. target_ca_pem: private-CA target only. Never skip TLS. Do not set restic_password: setup mints it into OpenBao for the job and prints it once for the offsite kit (KeepassX). Never rotate it.

One vault credential in the namespace, plus the bucket opt-in:

kubectl create secret generic harmony-tenant-backup-zitadel-key \
  -n tenant-x --from-file=key.json=<machine-key.json>
kubectl annotate ns tenant-x backup.platform/buckets=tenant-x-assets,tenant-x-uploads

Install:

OPENBAO_URL=... OPENBAO_JWT_ROLE=... HARMONY_SSO_URL=... HARMONY_ZITADEL_AUDIENCE=... \
HARMONY_ZITADEL_KEY_JSON="$(cat machine-key.json)" \
cargo run -p harmony-tenant-backup --bin harmony-tenant-backup-setup -- \
  --namespace tenant-x --cluster-id site-a-prod \
  --schedule "17 3 * * *" \
  --image harbor.example/harmony/ns-backup@sha256:<digest>

Image must be digest-pinned. Stagger schedules per namespace.

Run now / watch:

kubectl create job backup-now --from=cronjob/harmony-tenant-backup -n tenant-x
kubectl logs -n tenant-x job/backup-now -f

Laptop rehearsal needs the same OpenBao login as setup, plus the machine key as HARMONY_ZITADEL_KEY_JSON (or HARMONY_ZITADEL_KEY_PATH):

HARMONY_ZITADEL_KEY_JSON="$(cat machine-key.json)" \
harmony-tenant-backup run --namespace tenant-x --cluster-id site-a-prod \
  --dry-run --scratch /tmp/htb

Exit codes: 0 clean, 2 fatal (no repo, no config), 3 partial. Setup exits 1.

Restore

Fire restore uses the offsite kit (KeepassX), never OpenBao or the burned cluster. Same kit as OpenBao unseal keys. Target S3 creds + restic ≥ 0.18:

export RESTIC_REPOSITORY="s3:https://rgw.site-b/tenant-backups/site-a-prod/tenant-x/2026-08"
export RESTIC_PASSWORD='<restic_password from KeepassX>'
export AWS_ACCESS_KEY_ID=... AWS_SECRET_ACCESS_KEY=... AWS_DEFAULT_REGION=default
R=(restic -o s3.bucket-lookup=path)

"${R[@]}" snapshots --tag "ns:tenant-x"

# one database, into a NEW CNPG cluster. Monthly rotation: repo path ends /YYYY-MM;
# "latest" is only that month. --tag cluster:<cnpg-cluster-name>
"${R[@]}" restore latest --tag surface:pg --tag cluster:zito --target /restore/pg/zito
pg_restore -h new-cluster-rw -U postgres -d app /restore/pg/zito/app.dump
psql -h new-cluster-rw -U postgres -f /restore/pg/zito/globals.sql

"${R[@]}" restore latest --tag bucket:tenant-x-assets --target /restore/assets
"${R[@]}" restore latest --tag pvc:media --target /restore/media

globals.sql is pg_dumpall --globals-only: without it, app roles are gone. State the RPO to the tenant before starting.

Retention

No restic forget/prune. Monthly rotation expires a whole prefix, by hand: rclone delete --rmdirs ":s3:<bucket>/<cluster>/<ns>/2026-05". A human on the delete button is the design.

Pilot checklist

harmony-tenant-backup run --namespace <ns> --cluster-id ... --dry-run --scratch /tmp/htb
NetworkPolicy on CNPG: planner cannot reach <cluster>-rw. First failure.
SigV4 = region mismatch; SecondLevelDomainDenied = path-style stripped.
emptyDir peak = largest dump or bucket (CronJob ships 5Gi).
restic unlock if a killed Job left a lock (--retry-lock 5m covers blips).
Restore one database and one PVC into a scratch namespace. Run the app.

Development

cargo test -p harmony-tenant-backup. Acceptance is the pilot. src/common.rs is the names both binaries share.