harmony/examples/harmony_tenant_backup/DESIGN.md

Tenant backup design

Status: this is the current plan. The code in this folder is an earlier prototype (v0), kept for reference only. It does not implement this design; see "What changes from v0".

A tenant on a shared cluster opts in to offsite backups of their own data. The backup job runs in the tenant's namespace with an identity that can write backups and can never delete them through the S3 API.

Backup instance

The unit is a backup instance. It is created when a tenant enables backup and owns one config, one bucket, one restic password and one writer identity. A second instance, or a replacement, gets all four new. Nothing is shared between instances or between tenants.

Who	Does
Platform	BackupInstanceScore: bucket on the offsite RGW with versioning and Object Lock, a writer scoped to it, writer credentials into OpenBao at the tenant's path. Run by the platform team on request today, triggered by tenant self-service later.
Tenant	TenantBackupScore in their deploy code: schedule, CNPG clusters, buckets, PVCs. Data they can regenerate stays out. Generates the restic password in their own password manager.
Job	CronJob in the tenant namespace. Reads its instance config from OpenBao (see "Job identity").

Tenants are admins of their namespace. They can read the job's secrets, edit or delete the CronJob. That only affects their own backups, which is acceptable because backup is their choice.

Storage and security

• The bucket is owned by the platform's RGW account. The writer is an IAM user in that account with an allow-only policy: GetObject, PutObject, ListBucket, GetBucketLocation, and DeleteObject on locks/*. Everything else is denied by default. The bucket owner is never the writer: an owner keeps full control through ACLs whatever the policy says.
• PutObject can overwrite, so the bucket has versioning and Object Lock in compliance mode, 90 days by default. An overwrite or delete only adds a version or a delete marker; the locked version stays. restic never rewrites a key except locks/*. The 2025 median intrusion dwell time is 14 days, 26 when an outside party reports it (Mandiant M-Trends 2026); 90 days leaves room for the slow tail. Cost: every locked version is stored for 90 days, and a bucket can only be deleted 90 days after its last write. Compliance retention can be extended, never shortened.
• One lifecycle rule, on locks/ only, expires noncurrent lock versions and stale delete markers after the retention period. No lifecycle rule ever touches other prefixes; their old versions are what a rollback reads.
• A bucket quota caps growth. A full bucket fails the job loudly.
• Object Lock binds the S3 API only. The Admin Ops bucket purge, radosgw-admin bucket rm --purge-objects, radosgw-admin object rm and raw rados rm all delete without checking retention (only rgw_op.cc's S3 delete paths call verify_object_lock). On the offsite cluster, Ceph keyrings that reach the RGW pools and RGW users with buckets=write caps, including the one Rook creates, are held outside routine operator access. Object Lock protects against tenants and stolen S3 keys, not against the storage operator.
• RGW accounts and IAM users need Ceph Squid (19.2) or later. On Reef the writer is a plain RGW user with an inline policy on a bucket owned by another user.
• Writer credentials reach one bucket. A tenant who reads them learns nothing about another tenant.
• An offsite breach alone yields ciphertext without the key.
• The job never runs restic forget or prune.
• TLS is never skipped: sslmode=verify-full to Postgres, the RGW CA mounted if it is private.

BackupInstanceScore proves its own policy before returning: the writer can put an object, and cannot delete an object version or change the bucket policy.

Not covered: whoever can read the instance's OpenBao path (OpenBao root, or cluster-admin acting as the job) gets the password and the writer credentials, and with them all history. They could already read the live data. restic is symmetric (no write-only key exists in restic, rustic, kopia or borg), so the job necessarily holds a key that reads all history.

Restic password

The tenant generates the password in their own password manager, so the offline copy exists before the backup does. Nothing on the platform side generates it.

The password is a #[config(secret)] field of the instance's config, stored by harmony_config like any other secret. On first apply, TenantBackupScore calls get_or_prompt and the tenant pastes it. In CI a missing password fails with "apply once interactively to set the restic password". On every apply the Score checks that the password opens the existing repo, so a wrong paste never creates a second repo or orphans history.

Event	Outcome
Password lost from the tenant's vault, OpenBao alive	Read it back with harmony_config. Same instance.
Lost from both	Backups are unreadable, permanently.
Password leaked	Read exposure only. New instance; the old bucket is deleted once the new one has a verified snapshot.
Writer credentials reissued by the platform	Update the tenant's vault copy; the Score picks them up on next apply.

The tenant's vault entry holds endpoint, bucket, region, writer credentials and password: everything the restore below needs. The password is never rotated in place and never escrowed by the platform.

Job identity (open)

The job reads its config from OpenBao at run time, using its own short-lived token rather than a stored secret. All auth goes through Zitadel today, and Zitadel cannot exchange a Kubernetes ServiceAccount token for its own: token exchange only accepts Zitadel-issued subject tokens, and the JWT IdP is a browser flow. So one of these:

	Holds	OpenBao trusts	Revocation	Cost
A. Second OpenBao JWT mount on the cluster's service-account issuer, static jwt_validation_pubkeys	Projected token, audience openbao, about 1 h	Zitadel, plus each cluster issuer limited to backup roles	Token TTL	Mount per cluster; one role per instance bound to system:serviceaccount:<ns>:<sa> and one path; update keys when the cluster rotates its signing key. harmony_secret reads the token file instead of signing with a Zitadel key.
B. Zitadel machine key per instance (v0)	Long-lived key in a namespace Secret	Zitadel only	Manual	None, but the stored secret returns
C. OpenBao kubernetes auth (TokenReview)	Projected token	Zitadel, plus each cluster API server	Immediate	OpenBao reaches each API server; new login method

Leaning A. Revisit if Zitadel ships workload identity federation. Rejected: copying the config into a namespace Secret at apply time, or External Secrets Operator; both leave the secret in etcd, and ESO adds a cluster-wide operator.

The job

Two CronJobs, so a PVC that cannot mount never blocks the databases:

• pg+s3: Postgres dumps and tenant buckets.
• pvc: declared RWX volumes. Only created when the Score lists any.

Settings: concurrencyPolicy: Forbid, startingDeadlineSeconds: 3600, backoffLimit: 0 (a retry duplicates snapshots), restartPolicy: Never, activeDeadlineSeconds, failedJobsHistoryLimit: 3. Scratch is an emptyDir with sizeLimit and a matching ephemeral-storage request, staged one item at a time. Memory leaves room for the restic index, which grows forever without prune.

The Score emits one NetworkPolicy: egress from the job to the CNPG pods on 5432, the RGW on 443 and DNS; matching ingress on the CNPG pods. Shared OKD projects often default-deny.

Every run ends with restic check --read-data-subset=<weekday>/7, so the whole repo is read once a week. It is the only thing that notices a pack overwritten with the writer's credentials. A killed run leaves a stale non-exclusive lock that does not block the next backup, and orphan packs that stay forever (bounded per run).

Exit is zero or non-zero; the log line says fatal or partial and why.

Surfaces

• CNPG: the Score names the credential Secret per cluster, <cluster>-app by default (CNPG disables superuser access by default). pg_dumpall --globals-only --no-role-passwords, then pg_dump --format=custom --compress=0 --lock-wait-timeout=60s per database over <cluster>-rw, then pg_restore --list to reject an unreadable archive. No compression because restic compresses and compressed dumps defeat deduplication. Superuser is required only for a cluster with several databases owned by different roles; the Score checks this at apply time. The image's pg_dump is the highest Postgres major on the platform. RPO is the last successful run; there is no PITR.
• Tenant S3 buckets: rclone copy to scratch with read-only source credentials, then restic. Old versions on a versioned source bucket are not copied.
• PVCs: RWX only, decided at apply time. A declared RWO PVC is a Score error; on Ceph most PVCs are RBD RWO, so this surface mostly covers CephFS. The volume is readOnly at the persistentVolumeClaim source, not just the mount, with fsGroupChangePolicy: OnRootMismatch, so kubelet never chowns the tenant's live data. Files the job's UID cannot read make the run partial, not fatal. CNPG data PVCs are never listed; the dump covers them.

Each instance is one restic repo at the root of its bucket, with --host pinned to <cluster-id>-<ns>. Tags on every snapshot: run:, ns:, surface:pg|s3|pvc, and cluster:, bucket: or pvc:.

Deploy and failure visibility

TenantBackupScore returns only after one Job created from the CronJob template exits zero and its snapshot is listed. NetworkPolicy, region, path-style and TLS mistakes fail the deploy with an actionable message instead of a 3 AM surprise.

The Score emits a PrometheusRule for user-workload monitoring: alert when time() - kube_cronjob_status_last_successful_time exceeds twice the schedule interval. An optional dead-man's-switch URL is pinged on success with the snapshot IDs, which also catches a deleted CronJob or a burned cluster.

Schedule, growth and retention

The schedule is a typed Score field: Daily { at } (default), Hourly, or Cron(String) for anything else. There is no pruning, so the bucket only grows; restic deduplication keeps each run to the chunks that changed, which is why dumps are uncompressed. Each run logs the bytes it added, so the tenant sees the real growth rate instead of an estimate.

To drop old backups the tenant opens a ticket. The platform disables the writer key; the grace period, no shorter than the Object Lock retention, starts then. The tenant provisions a new instance, and the platform deletes the old bucket after the grace period.

Restore

Restore needs only the tenant's vault entry, restic and rclone, never the burned cluster. Pick the snapshot by the S3 LastModified of snapshots/<id>, which the server sets and Object Lock preserves, never by latest or restic's own timestamp: someone holding the writer credentials and the password can write a fake newer or backdated snapshot. restic 0.18 or later:

export RESTIC_REPOSITORY="s3:https://<target-endpoint>/<instance-bucket>"
export RESTIC_PASSWORD='<instance restic password>'
export AWS_ACCESS_KEY_ID=... AWS_SECRET_ACCESS_KEY=... AWS_DEFAULT_REGION=default
R=(restic -o s3.bucket-lookup=path)

"${R[@]}" snapshots --tag "ns:tenant-x"

# one database, into a NEW CNPG cluster, never over the broken one
"${R[@]}" restore <snapshot-id> --target /restore
psql -h new-cluster-rw -U postgres -f /restore/pg/app/globals.sql   # first; "role exists" is fine
pg_restore -h new-cluster-rw -U postgres -C -d postgres /restore/pg/app/app.dump
# then reset app role passwords from the new cluster's -app Secret

"${R[@]}" restore <snapshot-id> --target /restore   # bucket or PVC snapshot

The restore target needs superuser enabled on the new cluster. Dumps are staged at pg/<cluster>/<db>.dump and pg/<cluster>/globals.sql; the job keeps that layout fixed.

If the repo was overwritten, read it as it was before the incident without changing anything in the bucket:

RCLONE_S3_VERSION_AT='<time before incident>' \
  restic --no-lock -r rclone:offsite:<instance-bucket> restore <snapshot-id> --target /restore

Tell the tenant the cut time before starting: everything after the last dump is lost. A Zitadel database is useless without its master key, which must come from outside the cluster.

Drill: once before the schedule is trusted, then yearly or when the vault owner changes. On a machine with no cluster access, using only the vault entry: snapshots, restore one dump, restore it into a scratch CNPG cluster, run the app against it.

Code layout

tenant_backup/          runtime binary, no harmony dep: dump, copy, restic, check
tenant_backup_deploy/   TenantBackupScore<T: K8sclient>, via harmony_cli
harmony modules/storage BackupInstanceScore<T: ObjectStorageAdmin>
harmony infra/rgw       ObjectStorageAdmin for Ceph RGW (accounts, IAM users, Object Lock)

ObjectStorageAdmin is a capability, not an RGW binding. The existing ObjectBucketScore cannot be reused: an ObjectBucketClaim cannot set Object Lock. Estimated size 1.3 to 1.6k lines; v0's restic/rclone/pg wrappers and naming helpers carry over.

What changes from v0

The prototype gets the job shape right (stage locally, restic offsite, no delete on the happy path). These parts go:

v0	This design
Writer credentials shared by every tenant on a cluster (<cluster-id>/*)	One writer per instance, IAM user in the platform account
Append-only by IAM alone	Versioning and Object Lock
-setup binary from a workstation	TenantBackupScore in the tenant's deploy code
Password minted by setup	Generated in the tenant's password manager, pasted once through harmony_config
Job logs in to OpenBao with a long-lived Zitadel key in the namespace	Short-lived projected token (pending)
<cluster>-superuser assumed	Named Secret, <cluster>-app by default
Compressed -Fc dumps	--compress=0, restic compresses
Monthly repo rotation	New instance, old bucket deleted after the grace period
Planner/worker split, snapshot-to-clone for RWO	Two CronJobs; RWX only
Runtime binary depends on harmony	Runtime and deploy crates split

Rejected

• One cluster-wide job with one repo and one password, reading every tenant's secrets through a ClusterRole. It fits a single organisation, not tenants who must not see each other's data, and one failing tenant fails everyone.
• An operator watching a backup CRD in each namespace. It reimplements what CronJob already schedules. It may come back with tenant self-service or per-tenant coverage status, behind the same Score.
• age per artifact to a tenant public key, instead of restic. The job would hold no key that reads history, but every run is a full copy and hourly runs become unaffordable. Anyone who controls the cluster already has the live data.
• Password escrow, Shamir splits, HSM wrapping. The job must hold the plaintext key at runtime, and each adds a key or a ceremony to lose.
• Dumping from the -ro replica, co-mounting RWO volumes by pod affinity, rclone check after rclone copy: more failure modes than they remove at this size.

Open questions

1. Job identity (above).

CNPG's generated server certificate lists <cluster>-rw, .<ns>, .<ns>.svc and .<ns>.svc.<cluster-domain> (api/v1/cluster_funcs.go GetClusterAltDNSNames), so verify-full against <cluster>-rw.<ns>.svc with <cluster>-ca/ca.crt works without extra configuration.