harmony/examples/harmony_tenant_backup/README.md

# harmony-tenant-backup

> **Reference only.** This is the v0 prototype. The current plan is
> [DESIGN.md](DESIGN.md), which replaces the setup binary, shared credentials
> and monthly rotation described below.

Backup one tenant namespace. Not restore. Not retention. Not PITR.

Nightly CronJob covering CNPG dumps, opt-in S3 buckets, and PVCs (direct
mount or snapshot-clone). One restic repo per namespace on the off-site RGW.

Two binaries, same split as `harmony-fleet` vs `harmony-fleet-deploy`:

- `harmony-tenant-backup run` — operate (in-cluster, or a laptop dry-run)
- `harmony-tenant-backup-setup` — install RBAC + CronJob (workstation)

Out of scope: OpenBao, Ceph, K8s objects, Zitadel.

## Data layout

- Repo: `s3:https://<target-endpoint>/<target-bucket>/<cluster-id>/<ns>`, or
  `.../<YYYY-MM>` with `rotate_repo_monthly` (expiry = delete a whole prefix).
- Host pinned to `<cluster-id>-<ns>`. Tags: `run:`, `ns:`, `surface:pg|s3|pvc`,
  plus `cluster:` / `bucket:` / `pvc:`.

## Runbook

1. Build and push the image:

   ```bash
   cargo build --release -p harmony-tenant-backup
   docker build -f examples/harmony_tenant_backup/Dockerfile \
     -t harbor.example/harmony/ns-backup:v0 target/release
   docker push harbor.example/harmony/ns-backup:v0
   ```

2. Config at `<mount>/data/harmony/<ns>/TenantBackup` (`NAMESPACE` maps to
   store path `harmony/<ns>`):

   ```bash
    bao kv put -mount=secret harmony/tenant-x/TenantBackup \
      target_endpoint=https://rgw.site-b target_bucket=tenant-backups \
      target_access_key=... target_secret_key=... target_region=default \
      source_endpoint=https://rgw.site-A source_access_key=... \
      source_secret_key=... rotate_repo_monthly=true
   ```

   `target_*`: append-only at the target. `source_*`: read-only site-A, only
   for buckets. `target_ca_pem`: private-CA target only. Never skip TLS.
   Do not set `restic_password`: setup mints it into OpenBao for the job
   and prints it once for the offsite kit (KeepassX). Never rotate it.

3. One vault credential in the namespace, plus the bucket opt-in:

   ```bash
   kubectl create secret generic harmony-tenant-backup-zitadel-key \
     -n tenant-x --from-file=key.json=<machine-key.json>
   kubectl annotate ns tenant-x backup.platform/buckets=tenant-x-assets,tenant-x-uploads
   ```

4. Install:

   ```bash
   OPENBAO_URL=... OPENBAO_JWT_ROLE=... HARMONY_SSO_URL=... HARMONY_ZITADEL_AUDIENCE=... \
   HARMONY_ZITADEL_KEY_JSON="$(cat machine-key.json)" \
   cargo run -p harmony-tenant-backup --bin harmony-tenant-backup-setup -- \
     --namespace tenant-x --cluster-id site-a-prod \
     --schedule "17 3 * * *" \
     --image harbor.example/harmony/ns-backup@sha256:<digest>
   ```

   Image must be digest-pinned. Stagger schedules per namespace.

5. Run now / watch:

   ```bash
   kubectl create job backup-now --from=cronjob/harmony-tenant-backup -n tenant-x
   kubectl logs -n tenant-x job/backup-now -f
   ```

    Laptop rehearsal needs the same OpenBao login as setup, plus the
    machine key as `HARMONY_ZITADEL_KEY_JSON` (or `HARMONY_ZITADEL_KEY_PATH`):

    ```bash
    HARMONY_ZITADEL_KEY_JSON="$(cat machine-key.json)" \
    harmony-tenant-backup run --namespace tenant-x --cluster-id site-a-prod \
      --dry-run --scratch /tmp/htb
    ```

Exit codes: 0 clean, 2 fatal (no repo, no config), 3 partial. Setup exits 1.

## Restore

Fire restore uses the offsite kit (KeepassX), never OpenBao or the burned
cluster. Same kit as OpenBao unseal keys. Target S3 creds + restic ≥ 0.18:

```bash
export RESTIC_REPOSITORY="s3:https://rgw.site-b/tenant-backups/site-a-prod/tenant-x/2026-08"
export RESTIC_PASSWORD='<restic_password from KeepassX>'
export AWS_ACCESS_KEY_ID=... AWS_SECRET_ACCESS_KEY=... AWS_DEFAULT_REGION=default
R=(restic -o s3.bucket-lookup=path)

"${R[@]}" snapshots --tag "ns:tenant-x"

# one database, into a NEW CNPG cluster. Monthly rotation: repo path ends /YYYY-MM;
# "latest" is only that month. --tag cluster:<cnpg-cluster-name>
"${R[@]}" restore latest --tag surface:pg --tag cluster:zito --target /restore/pg/zito
pg_restore -h new-cluster-rw -U postgres -d app /restore/pg/zito/app.dump
psql -h new-cluster-rw -U postgres -f /restore/pg/zito/globals.sql

"${R[@]}" restore latest --tag bucket:tenant-x-assets --target /restore/assets
"${R[@]}" restore latest --tag pvc:media --target /restore/media
```

`globals.sql` is `pg_dumpall --globals-only`: without it, app roles are gone.
State the RPO to the tenant before starting.

## Retention

No `restic forget`/`prune`. Monthly rotation expires a whole prefix, by hand:
`rclone delete --rmdirs ":s3:<bucket>/<cluster>/<ns>/2026-05"`. A human on
the delete button is the design.

## Pilot checklist

1. `harmony-tenant-backup run --namespace <ns> --cluster-id ... --dry-run --scratch /tmp/htb`
2. NetworkPolicy on CNPG: planner cannot reach `<cluster>-rw`. First failure.
3. SigV4 = region mismatch; `SecondLevelDomainDenied` = path-style stripped.
4. emptyDir peak = largest dump or bucket (CronJob ships 5Gi).
5. `restic unlock` if a killed Job left a lock (`--retry-lock 5m` covers blips).
6. Restore one database and one PVC into a scratch namespace. Run the app.

## Development

`cargo test -p harmony-tenant-backup`. Acceptance is the pilot.
`src/common.rs` is the names both binaries share.