The OpenBao chart's StatefulSet is OnDelete, so `rollout restart` silently no-ops on config changes — document deleting the pod + re-run to unseal instead. Replace the audit jq recipe's literal-path filter (can't match HMAC-hashed paths) with identity-field attribution, matching observed OpenBao 2.5.4 output. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Harmony SSO Example
Deploys Zitadel (identity provider) and OpenBao (secrets management) on a local k3d cluster, then demonstrates using them as harmony_config backends for shared config and secret management.
Prerequisites
- Docker running
- Ports 8080 and 8200 free
/etc/hostsentries (or use a local DNS resolver):127.0.0.1 sso.harmony.local 127.0.0.1 bao.harmony.local
Usage
Full deployment
# Deploy everything (OpenBao + Zitadel)
cargo run -p example-harmony-sso
# OpenBao only (faster, skip Zitadel)
cargo run -p example-harmony-sso -- --skip-zitadel
No env.sh to source: the example sets HARMONY_SECRET_STORE=file and
HARMONY_SECRET_NAMESPACE=harmony-sso-example at startup so the global
harmony_secret manager uses the local-file backend (writing to
~/.local/share/harmony/secrets/). Override either variable in your
shell if you want different behaviour.
Config storage demo (token auth)
After deployment, run the config demo to verify harmony_config works with OpenBao:
cargo run -p example-harmony-sso -- --demo
This writes and reads a SsoExampleConfig through the ConfigClient chain (EnvSource -> StoreSource<OpenbaoSecretStore>), demonstrating environment variable overrides and persistent storage in OpenBao KV v2.
SSO device flow demo
Requires a Zitadel application configured for device code grant:
HARMONY_SSO_CLIENT_ID=<zitadel-app-client-id> \
cargo run -p example-harmony-sso -- --sso-demo
Cleanup
cargo run -p example-harmony-sso -- --cleanup
What gets deployed
| Component | Namespace | Access |
|---|---|---|
| OpenBao (standalone, file storage) | openbao |
http://bao.harmony.local:8200 |
| Zitadel (with CNPG PostgreSQL) | zitadel |
http://sso.harmony.local:8080 |
OpenBao configuration
- Auth methods: userpass, JWT
- Secrets engine: KV v2 at
secret/ - Policy:
harmony-devgrants CRUD onsecret/data/harmony/* - Userpass credentials:
harmony/harmony-dev-password - JWT auth: configured with Zitadel as OIDC provider, role
harmony-developer - Unseal keys: saved to
~/.local/share/harmony/openbao/unseal-keys.json - Audit device: file audit enabled at
/openbao/audit/audit.log(see below)
Audit log: who changed what
OpenBao's KV v2 version history shows when a value changed and which version, but never who — the caller identity is not stored in KV metadata, so the UI version list can't show the modifier. The modifier is recorded only in the audit log.
A file audit device is declared at /openbao/audit/audit.log (on the
auditStorage PVC). Recent OpenBao refuses runtime audit enablement
via the API (cannot enable audit device via API), so OpenbaoScore
configures it declaratively in the OpenBao server config (an
audit "file" "file" stanza in the helm values). The device logs
every API request/response with the authenticated identity. Because
the SSO flow mints the OpenBao token from the Zitadel id_token with
user_claim=email, each write is attributed to the user's email.
Because the device is loaded from the server config at startup, it
applies only after the OpenBao pod (re)starts with the new config.
The OpenBao helm chart sets the server StatefulSet's updateStrategy
to OnDelete (by design — restarting OpenBao requires a manual
unseal), so a helm upgrade that changes the server config does
not recreate the pod. kubectl rollout restart statefulset/openbao
reports success but does nothing here — OnDelete only annotates the
template. You must delete the pod yourself, then re-run the example to
unseal the fresh (sealed) process:
kubectl delete pod openbao-0 -n openbao
cargo run -p example-harmony-sso # unseals the freshly-restarted pod
Confirm the device loaded with bao audit list below.
The sys/audit endpoint and reading the audit log both require the
root token — the harmony-dev policy on day-to-day tokens can't
touch them. Grab the root token from the unseal-keys cache:
ROOT=$(jq -r .root_token ~/.local/share/harmony/openbao/unseal-keys.json)
Confirm the device is on:
kubectl exec -n openbao openbao-0 -- \
sh -c "export VAULT_TOKEN=$ROOT && bao audit list"
# Path Type Description
# file/ file n/a
(bao audit list with no token returns 403 permission denied — that
means "you didn't authenticate", not "audit is off".)
See who performed each operation (the audit log is JSON-lines on the audit PVC; reading the file needs no OpenBao token, just pod access). Because paths/values are HMAC-hashed by default you can't filter by a literal key path — read the identity fields instead:
kubectl exec -n openbao openbao-0 -- cat /openbao/audit/audit.log | \
jq -c 'select(.type == "response" and .auth.display_name != null)
| {time, op: .request.operation,
who: .auth.display_name, role: .auth.metadata.role,
entity: .auth.entity_id}' | tail -n 15
SSO-authenticated operations are attributed to the user, e.g.:
{"time":"...","op":"update","who":"jwt-admin@zitadel.sso.harmony.local","role":"harmony-developer","entity":"81ddf1d7-..."}
The jwt- prefix is the JWT auth mount name; the rest is the email
from the Zitadel id_token (user_claim=email). Setup/unseal steps run
with the root token and show up as "who":"root".
Notes:
- Audit entries HMAC-hash paths/values by default, so the identity
fields (
auth.display_name,auth.metadata,auth.entity_id) are what you rely on for attribution. If you need raw paths/values, tune the audit device (log_raw,hmac_accessor) — not done here. - This is the OpenBao-native answer; per-version "who" is not surfaced in the UI.
Architecture
Developer CLI
|
|-- harmony_config::ConfigClient
| |-- EnvSource (HARMONY_CONFIG_* env vars)
| |-- StoreSource<OpenbaoSecretStore>
| |-- Token auth (OPENBAO_TOKEN)
| |-- Cached token validation
| |-- Zitadel OIDC device flow (RFC 8628)
| |-- Userpass fallback
|
v
k3d cluster (harmony-example)
|-- OpenBao (KV v2 secrets engine)
| |-- JWT auth -> validates Zitadel id_tokens
| |-- userpass auth -> dev credentials
|
|-- Zitadel (OpenID Connect IdP)
|-- Device authorization grant
|-- Federated login (Google, GitHub, Entra ID)