3 changed files with 173 additions and 140 deletions
--- a/examples/harmony_sso/README.md
+++ b/examples/harmony_sso/README.md
@@ -1,12 +1,12 @@
 # Harmony SSO Example

-Deploys Zitadel (identity provider) and OpenBao (secrets management) on a local k3d cluster, then demonstrates using them as `harmony_config` backends for shared config and secret management.
+Deploys Zitadel (identity provider) and OpenBao (secrets management) on a local k3d cluster, then round-trips two `harmony_config` structs (Secret + Standard class) through OpenBao via SSO-authenticated access.

 ## Prerequisites

 - Docker running
 - Ports 8080 and 8200 free
- `/etc/hosts` entries (or use a local DNS resolver):
+- `/etc/hosts` entries (or a local DNS resolver):
  ```
  127.0.0.1  sso.harmony.local
  127.0.0.1  bao.harmony.local
@@ -14,41 +14,19 @@ Deploys Zitadel (identity provider) and OpenBao (secrets management) on a local

 ## Usage

-### Full deployment
-
 ```bash
-# Deploy everything (OpenBao + Zitadel)
+# Deploy everything (OpenBao + Zitadel) and round-trip the example configs
 cargo run -p example-harmony-sso

 # OpenBao only (faster, skip Zitadel)
 cargo run -p example-harmony-sso -- --skip-zitadel
-```

-### Config storage demo (token auth)
-
-After deployment, run the config demo to verify `harmony_config` works with OpenBao:
-
-```bash
-cargo run -p example-harmony-sso -- --demo
-```
-
-This writes and reads a `SsoExampleConfig` through the `ConfigManager` chain (`EnvSource -> StoreSource<OpenbaoSecretStore>`), demonstrating environment variable overrides and persistent storage in OpenBao KV v2.
-
-### SSO device flow demo
-
-Requires a Zitadel application configured for device code grant:
-
-```bash
-HARMONY_SSO_CLIENT_ID=<zitadel-app-client-id> \
-  cargo run -p example-harmony-sso -- --sso-demo
-```
-
-### Cleanup
-
-```bash
+# Teardown
 cargo run -p example-harmony-sso -- --cleanup
 ```

+The example sets `HARMONY_SECRET_STORE=file` and `HARMONY_SECRET_NAMESPACE=harmony-sso-example` at startup so the global `harmony_secret` manager uses the local-file backend (`~/.local/share/harmony/secrets/`). Override either in your shell to change behaviour.
+
 ## What gets deployed

 | Component | Namespace | Access |
@@ -62,15 +40,60 @@ cargo run -p example-harmony-sso -- --cleanup
 - **Secrets engine:** KV v2 at `secret/`
 - **Policy:** `harmony-dev` grants CRUD on `secret/data/harmony/*`
 - **Userpass credentials:** `harmony` / `harmony-dev-password`
- **JWT auth:** configured with Zitadel as OIDC provider, role `harmony-developer`
- **Unseal keys:** saved to `~/.local/share/harmony/openbao/unseal-keys.json`
+- **JWT auth:** Zitadel as OIDC provider, role `harmony-developer`
+- **Unseal keys:** `~/.local/share/harmony/openbao/unseal-keys.json`
+- **Audit device:** file audit at `/openbao/audit/audit.log` (see below)
+
+## Audit log: who changed what
+
+OpenBao's KV v2 version history shows *when* a value changed and which version, but never *who* — the caller identity isn't in KV metadata. It lives in the **audit log**.
+
+A file audit device is enabled at `/openbao/audit/audit.log` (`auditStorage` PVC), logging every request/response with the authenticated identity. SSO mints the OpenBao token from the Zitadel id_token with `user_claim=email`, so each operation is attributed to the user.
+
+**Applying audit config changes:** the device loads at OpenBao startup, and the helm chart's StatefulSet uses `updateStrategy: OnDelete` (a Vault/OpenBao restart needs a manual unseal — auto-rolling is intentionally disabled). So `helm upgrade` updates the server config but does *not* recreate the pod; `kubectl rollout restart` reports success but does nothing here. Delete the pod yourself and re-run to unseal:
+
+```bash
+kubectl delete pod openbao-0 -n openbao
+cargo run -p example-harmony-sso   # unseals the fresh pod
+```
+
+Reading the audit needs the **root token** (the `harmony-dev` policy can't touch `sys/audit`):
+
+```bash
+ROOT=$(jq -r .root_token ~/.local/share/harmony/openbao/unseal-keys.json)
+
+# Confirm device is on:
+kubectl exec -n openbao openbao-0 -- sh -c "export VAULT_TOKEN=$ROOT && bao audit list"
+# Path     Type    Description
+# file/    file    n/a
+```
+
+(`bao audit list` with no token returns `403 permission denied` — that means "you didn't authenticate", not "audit is off".)
+
+See who performed each operation. Paths and values are HMAC-hashed by default, so filter on the *identity* fields:
+
+```bash
+kubectl exec -n openbao openbao-0 -- cat /openbao/audit/audit.log | \
+  jq -c 'select(.type == "response" and .auth.display_name != null)
+         | {time, op: .request.operation,
+            who: .auth.display_name, role: .auth.metadata.role,
+            entity: .auth.entity_id}' | tail -n 15
+```
+
+SSO-authenticated operations are attributed to the user:
+
+```
+{"time":"...","op":"update","who":"jwt-admin@zitadel.sso.harmony.local","role":"harmony-developer","entity":"81ddf1d7-..."}
+```
+
+The `jwt-` prefix is the JWT auth-mount name; the rest is the `user_claim=email` value from the id_token. Setup/unseal steps run with the root token and show as `"who":"root"`. For raw paths/values, tune the audit device (`log_raw`, `hmac_accessor`) — out of scope here.

 ## Architecture

 ```
 Developer CLI
  |
-  |-- harmony_config::ConfigManager
+  |-- harmony_config::ConfigClient
  |     |-- EnvSource (HARMONY_CONFIG_* env vars)
  |     |-- StoreSource<OpenbaoSecretStore>
  |           |-- Token auth (OPENBAO_TOKEN)
--- a/examples/harmony_sso/harmony_sso_plan.md
+++ b/examples/harmony_sso/harmony_sso_plan.md
@@ -11,7 +11,7 @@ Deploy Zitadel and OpenBao on a local k3d cluster, use them as `harmony_config`
 - [x] A.1 -- CLI argument parsing (`--demo`, `--sso-demo`, `--skip-zitadel`, `--cleanup`)
 - [x] A.2 -- Zitadel deployment via `ZitadelScore` (`external_secure: false` for k3d)
 - [x] A.3 -- OpenBao JWT auth method + `harmony-dev` policy configuration
- [x] A.4 -- `--demo` flag: config storage demo with token auth via `ConfigManager`
+- [x] A.4 -- `--demo` flag: config storage demo with token auth via `ConfigClient`
 - [x] A.5 -- Hardening: retry loops for pod readiness, HTTP readiness checks, `--cleanup`
 - [x] A.6 -- README with prerequisites, usage, and architecture

@@ -35,9 +35,11 @@ The Zitadel OIDC device flow code exists (`harmony_secret/src/store/zitadel.rs`)
 - `OPENBAO_JWT_AUTH_MOUNT` (default: `jwt`)
 - `OPENBAO_JWT_ROLE` (default: `harmony-developer`)

-**B.4 -- Silent refresh:**
- Add `refresh_token()` method to `ZitadelOidcAuth`
- Update auth chain in `openbao.rs`: cached session -> silent refresh -> device flow
+**B.4 -- Silent refresh -- DONE:**
+- `ZitadelOidcAuth::silent_refresh()` + `refresh_id_token()` added in `harmony_secret/src/store/zitadel.rs`
+- `authenticate()` chain is now: cached session (valid) -> silent refresh (when expired + refresh_token present) -> device flow
+- Refresh response that omits a new `refresh_token` preserves the previous one (OAuth allows rotation to be optional)
+- Refreshed session is persisted to `oidc_session_*` so subsequent runs pick up the new tokens

 **B.5 -- `--sso-demo` flag:**
 - Already stubbed in `examples/harmony_sso/src/main.rs`
@@ -56,7 +58,7 @@ The Zitadel OIDC device flow code exists (`harmony_secret/src/store/zitadel.rs`)
 - `test_openbao_health` -- health endpoint
 - `test_zitadel_openid_config` -- OIDC discovery
 - `test_openbao_userpass_auth` -- write/read secret
- `test_config_manager_openbao_backend` -- full ConfigManager chain
+- `test_config_manager_openbao_backend` -- full ConfigClient chain
 - `test_openbao_jwt_auth_configured` -- verify JWT auth method + role exist

 **C.2 -- Zitadel application automation** (`examples/harmony_sso/src/zitadel_setup.rs`):
@@ -145,7 +147,7 @@ The user flagged several areas that should use `harmony-k8s` instead of raw `kub
 - `cargo run -p example-harmony-sso` -> deploys k3d + OpenBao + Zitadel (with retry for CNPG CRD + PG readiness)
 - `curl -H "Host: bao.harmony.local" http://127.0.0.1:8080/v1/sys/health` -> OpenBao healthy (initialized, unsealed)
 - `curl -H "Host: sso.harmony.local" http://127.0.0.1:8080/.well-known/openid-configuration` -> Zitadel OIDC config with device_authorization_endpoint
- `cargo run -p example-harmony-sso -- --demo` -> writes/reads config via ConfigManager + OpenbaoSecretStore, env override works
+- `cargo run -p example-harmony-sso -- --demo` -> writes/reads config via ConfigClient + OpenbaoSecretStore, env override works

 **Phase B:**
 - `HARMONY_SSO_URL=http://sso.harmony.local HARMONY_SSO_CLIENT_ID=<id> cargo run -p example-harmony-sso -- --sso-demo`
--- a/examples/harmony_sso/src/main.rs
+++ b/examples/harmony_sso/src/main.rs
@@ -10,7 +10,7 @@ use harmony::modules::zitadel::{
 };
 use harmony::score::Score;
 use harmony::topology::{K8sclient, Topology};
-use harmony_config::{Config, ConfigClient, EnvSource, StoreSource};
+use harmony_config::{Config, ConfigClass, ConfigClient, EnvSource, StoreSource};
 use harmony_k8s::K8sClient;
 use harmony_secret::OpenbaoSecretStore;
 use k3d_rs::{K3d, PortMapping};
@@ -44,15 +44,13 @@ struct Args {
    cleanup: bool,
 }

-// ---------------------------------------------------------------------------
-// Config type stored via SSO-authenticated OpenBao
-// ---------------------------------------------------------------------------
-
-#[derive(Debug, Clone, Serialize, Deserialize, JsonSchema, PartialEq)]
+#[derive(Debug, Clone, Serialize, Deserialize, JsonSchema, PartialEq, Config)]
 struct SsoExampleConfig {
    team_name: String,
    environment: String,
    max_replicas: u16,
+    #[config(secret)]
+    deploy_token: String,
 }

 impl Default for SsoExampleConfig {
@@ -61,17 +59,69 @@ impl Default for SsoExampleConfig {
            team_name: "platform-team".to_string(),
            environment: "staging".to_string(),
            max_replicas: 3,
+            deploy_token: "dev-token-replace-me".to_string(),
        }
    }
 }

-impl Config for SsoExampleConfig {
-    const KEY: &'static str = "SsoExampleConfig";
+#[derive(Debug, Clone, Serialize, Deserialize, JsonSchema, PartialEq, Config)]
+struct TeamMetadataConfig {
+    display_name: String,
+    slack_channel: String,
+    oncall_schedule: String,
 }

-// ---------------------------------------------------------------------------
-// Helpers
-// ---------------------------------------------------------------------------
+impl Default for TeamMetadataConfig {
+    fn default() -> Self {
+        Self {
+            display_name: "Platform Team".to_string(),
+            slack_channel: "#platform".to_string(),
+            oncall_schedule: "platform-primary".to_string(),
+        }
+    }
+}
+
+async fn round_trip<T>(manager: &ConfigClient, default_value: T) -> anyhow::Result<()>
+where
+    T: Config + std::fmt::Debug + Clone + PartialEq,
+{
+    match manager.get::<T>().await {
+        Ok(found) => {
+            info!(
+                "[{key}] loaded existing value (class={class:?}): {found:?}",
+                key = T::KEY,
+                class = T::CLASS,
+            );
+            Ok(())
+        }
+        Err(harmony_config::ConfigError::NotFound { .. }) => {
+            info!(
+                "[{key}] not found, storing default (class={class:?})",
+                key = T::KEY,
+                class = T::CLASS,
+            );
+            manager.set(&default_value).await?;
+            let retrieved: T = manager.get().await?;
+            anyhow::ensure!(
+                retrieved == default_value,
+                "Round-trip mismatch for {}: stored != retrieved",
+                T::KEY
+            );
+            info!("[{key}] round-trip verified", key = T::KEY);
+            Ok(())
+        }
+        Err(e) => Err(e.into()),
+    }
+}
+
+fn set_env_default(key: &str, value: &str) {
+    if std::env::var_os(key).is_none() {
+        // SAFETY: called from main() before any tokio task is spawned.
+        unsafe {
+            std::env::set_var(key, value);
+        }
+    }
+}

 fn harmony_data_dir() -> PathBuf {
    directories::BaseDirs::new()
@@ -90,6 +140,8 @@ fn create_topology(k3d: &K3d) -> harmony::topology::K8sAnywhereTopology {
    let context = k3d
        .context_name()
        .unwrap_or_else(|| format!("k3d-{}", CLUSTER_NAME));
+    // SAFETY: called from main() before any tokio task is spawned; the topology
+    // reads these vars immediately below via from_env().
    unsafe {
        std::env::set_var("HARMONY_USE_LOCAL_K3D", "false");
        std::env::set_var("HARMONY_AUTOINSTALL", "false");
@@ -98,19 +150,24 @@ fn create_topology(k3d: &K3d) -> harmony::topology::K8sAnywhereTopology {
    harmony::topology::K8sAnywhereTopology::from_env()
 }

-fn harmony_dev_policy() -> OpenbaoPolicy {
-    OpenbaoPolicy {
-        name: "harmony-dev".to_string(),
-        hcl: r#"path "secret/data/harmony/*" { capabilities = ["create","read","update","delete","list"] }
+fn openbao_setup(jwt: Option<OpenbaoJwtAuth>) -> OpenbaoSetupScore {
+    OpenbaoSetupScore {
+        policies: vec![OpenbaoPolicy {
+            name: "harmony-dev".to_string(),
+            hcl: r#"path "secret/data/harmony/*" { capabilities = ["create","read","update","delete","list"] }
 path "secret/metadata/harmony/*" { capabilities = ["list","read"] }"#
-            .to_string(),
+                .to_string(),
+        }],
+        users: vec![OpenbaoUser {
+            username: "harmony".to_string(),
+            password: "harmony-dev-password".to_string(),
+            policies: vec!["harmony-dev".to_string()],
+        }],
+        jwt_auth: jwt,
+        ..Default::default()
    }
 }

-// ---------------------------------------------------------------------------
-// Zitadel deployment (with CNPG retry)
-// ---------------------------------------------------------------------------
-
 async fn deploy_zitadel(k3d: &K3d) -> anyhow::Result<()> {
    info!("Deploying Zitadel (this may take several minutes)...");

@@ -171,10 +228,6 @@ async fn wait_for_zitadel_ready() -> anyhow::Result<()> {
    anyhow::bail!("Timed out waiting for Zitadel")
 }

-// ---------------------------------------------------------------------------
-// Cluster lifecycle
-// ---------------------------------------------------------------------------
-
 async fn ensure_k3d_cluster(k3d: &K3d) -> anyhow::Result<()> {
    info!("Ensuring k3d cluster '{}' is running...", CLUSTER_NAME);
    k3d.ensure_installed()
@@ -209,13 +262,16 @@ async fn cleanup_openbao_webhook(k8s: &K8sClient) -> anyhow::Result<()> {
    Ok(())
 }

-// ---------------------------------------------------------------------------
-// Main
-// ---------------------------------------------------------------------------
-
 #[tokio::main]
 async fn main() -> anyhow::Result<()> {
    env_logger::Builder::from_env(env_logger::Env::default().default_filter_or("info")).init();
+
+    // harmony_secret defaults to Infisical and panics if its env vars are missing.
+    // Set the file-backend defaults BEFORE the lazy SECRET_MANAGER touches them;
+    // an operator override from the shell still wins.
+    set_env_default("HARMONY_SECRET_STORE", "file");
+    set_env_default("HARMONY_SECRET_NAMESPACE", "harmony-sso-example");
+
    let args = Args::parse();
    let k3d = create_k3d();

@@ -223,11 +279,7 @@ async fn main() -> anyhow::Result<()> {
        return cleanup_cluster(&k3d);
    }

-    info!("===========================================");
-    info!("Harmony SSO Example");
-    info!("===========================================");
-
-    // --- Phase 1: Infrastructure ---
+    info!("=== Harmony SSO Example ===");

    ensure_k3d_cluster(&k3d).await?;

@@ -242,7 +294,6 @@ async fn main() -> anyhow::Result<()> {
        .await
        .map_err(|e| anyhow::anyhow!("K8s client: {}", e))?;

-    // Deploy + configure OpenBao (no JWT auth yet -- Zitadel isn't up)
    cleanup_openbao_webhook(&k8s).await?;
    OpenbaoScore {
        host: OPENBAO_HOST.to_string(),
@@ -252,28 +303,16 @@ async fn main() -> anyhow::Result<()> {
    .await
    .context("OpenBao deploy failed")?;

-    OpenbaoSetupScore {
-        policies: vec![harmony_dev_policy()],
-        users: vec![OpenbaoUser {
-            username: "harmony".to_string(),
-            password: "harmony-dev-password".to_string(),
-            policies: vec!["harmony-dev".to_string()],
-        }],
-        jwt_auth: None, // Phase 2 adds JWT after Zitadel is ready
-        ..Default::default()
-    }
-    .interpret(&Inventory::autoload(), &topology)
-    .await
-    .context("OpenBao setup failed")?;
+    openbao_setup(None)
+        .interpret(&Inventory::autoload(), &topology)
+        .await
+        .context("OpenBao setup failed")?;

    if args.skip_zitadel {
-        info!("=== Skipping Zitadel (--skip-zitadel) ===");
-        info!("OpenBao: http://{}:{}", OPENBAO_HOST, HTTP_PORT);
+        info!("Skipping Zitadel (--skip-zitadel). OpenBao: http://{OPENBAO_HOST}:{HTTP_PORT}");
        return Ok(());
    }

-    // --- Phase 2: Identity + SSO Wiring ---
-
    CoreDNSRewriteScore {
        rewrites: vec![
            CoreDNSRewrite {
@@ -293,7 +332,6 @@ async fn main() -> anyhow::Result<()> {
    deploy_zitadel(&k3d).await?;
    wait_for_zitadel_ready().await?;

-    // Provision Zitadel project + device-code application
    ZitadelSetupScore {
        host: ZITADEL_HOST.to_string(),
        scheme: Default::default(),
@@ -315,7 +353,6 @@ async fn main() -> anyhow::Result<()> {
    .await
    .context("Zitadel setup failed")?;

-    // Read the client_id from the cache written by ZitadelSetupScore
    let zitadel_config =
        ZitadelClientConfig::load().context("ZitadelSetupScore did not produce a client config")?;
    let client_id = zitadel_config
@@ -323,37 +360,23 @@ async fn main() -> anyhow::Result<()> {
        .context("No client_id for harmony-cli app")?
        .clone();

-    info!("Zitadel app '{}' client_id: {}", APP_NAME, client_id);
+    info!("Zitadel app '{APP_NAME}' client_id: {client_id}");

-    // Now configure OpenBao JWT auth with the real client_id
-    OpenbaoSetupScore {
-        policies: vec![harmony_dev_policy()],
-        users: vec![OpenbaoUser {
-            username: "harmony".to_string(),
-            password: "harmony-dev-password".to_string(),
-            policies: vec!["harmony-dev".to_string()],
-        }],
-        jwt_auth: Some(OpenbaoJwtAuth {
-            oidc_discovery_url: format!("http://{}:{}", ZITADEL_HOST, HTTP_PORT),
-            bound_issuer: format!("http://{}:{}", ZITADEL_HOST, HTTP_PORT),
-            role_name: "harmony-developer".to_string(),
-            bound_audiences: client_id.clone(),
-            user_claim: "email".to_string(),
-            policies: vec!["harmony-dev".to_string()],
-            ttl: "4h".to_string(),
-            max_ttl: "24h".to_string(),
-        }),
-        ..Default::default()
-    }
+    openbao_setup(Some(OpenbaoJwtAuth {
+        oidc_discovery_url: format!("http://{ZITADEL_HOST}:{HTTP_PORT}"),
+        bound_issuer: format!("http://{ZITADEL_HOST}:{HTTP_PORT}"),
+        role_name: "harmony-developer".to_string(),
+        bound_audiences: client_id.clone(),
+        user_claim: "email".to_string(),
+        policies: vec!["harmony-dev".to_string()],
+        ttl: "4h".to_string(),
+        max_ttl: "24h".to_string(),
+    }))
    .interpret(&Inventory::autoload(), &topology)
    .await
    .context("OpenBao JWT auth setup failed")?;

-    // --- Phase 3: Config via SSO ---
-
-    info!("===========================================");
-    info!("Storing config via SSO-authenticated OpenBao");
-    info!("===========================================");
+    info!("=== Storing config via SSO-authenticated OpenBao ===");

    let _pf = k8s
        .port_forward(OPENBAO_POD, OPENBAO_NAMESPACE, 8200, 8200)
@@ -362,7 +385,7 @@ async fn main() -> anyhow::Result<()> {
    tokio::time::sleep(tokio::time::Duration::from_secs(1)).await;

    let openbao_url = format!("http://127.0.0.1:{}", _pf.port());
-    let sso_url = format!("http://{}:{}", ZITADEL_HOST, HTTP_PORT);
+    let sso_url = format!("http://{ZITADEL_HOST}:{HTTP_PORT}");

    let store = OpenbaoSecretStore::new(
        openbao_url,
@@ -385,31 +408,16 @@ async fn main() -> anyhow::Result<()> {
        Arc::new(StoreSource::new("harmony".to_string(), store)),
    ]);

-    // Try to load existing config (succeeds on re-run)
-    match manager.get::<SsoExampleConfig>().await {
-        Ok(config) => {
-            info!("Config loaded from OpenBao: {:?}", config);
-        }
-        Err(harmony_config::ConfigError::NotFound { .. }) => {
-            info!("No config found, storing default...");
-            let config = SsoExampleConfig::default();
-            manager.set(&config).await?;
-            info!("Config stored: {:?}", config);
+    assert_eq!(SsoExampleConfig::CLASS, ConfigClass::Secret);
+    assert_eq!(TeamMetadataConfig::CLASS, ConfigClass::Standard);

-            let retrieved: SsoExampleConfig = manager.get().await?;
-            info!("Config verified: {:?}", retrieved);
-            assert_eq!(config, retrieved);
-        }
-        Err(e) => return Err(e.into()),
-    }
+    round_trip::<SsoExampleConfig>(&manager, SsoExampleConfig::default()).await?;
+    round_trip::<TeamMetadataConfig>(&manager, TeamMetadataConfig::default()).await?;

-    info!("===========================================");
-    info!("Success! Config managed via Zitadel SSO + OpenBao");
-    info!("===========================================");
-    info!("OpenBao: http://{}:{}", OPENBAO_HOST, HTTP_PORT);
-    info!("Zitadel: http://{}:{}", ZITADEL_HOST, HTTP_PORT);
-    info!("Run again to verify cached session works.");
-    info!("cargo run -p example-harmony-sso -- --cleanup  # teardown");
+    info!("=== Success — config managed via Zitadel SSO + OpenBao ===");
+    info!("OpenBao: http://{OPENBAO_HOST}:{HTTP_PORT}");
+    info!("Zitadel: http://{ZITADEL_HOST}:{HTTP_PORT}");
+    info!("Run again to verify cached session; --cleanup to teardown.");

    Ok(())
 }