harmony/OPENBAO_POLICY_MIGRATION.md

OpenBao Entity Policy Migration Guide

Moving fleet device auth from Design A (JWT deployments claim) to Design B (OpenBao entity policies).

---

Background: how the pieces fit together

Before touching any code, here is the data flow you need to hold in your head.

The three systems

Zitadel is the identity provider. Every fleet device has a machine user in Zitadel with a key file (a JSON file containing an RSA private key). The device uses that key to mint a short-lived JWT, signed by Zitadel, proving "I am device X."

OpenBao is the secret store (an open-source Vault fork). It holds per-deployment secrets that devices need at runtime (e.g. API keys, certificates). A device can only read secrets for the deployments it is currently assigned to.

Harmony operator (fleet/harmony-fleet-operator) is the orchestration process running in the cluster. It watches Kubernetes Deployment CRDs and Device CRDs, and its job is to keep everything in sync: NATS desired-state KV entries, OpenBao entity policies, Kubernetes status patches.

How a device reads a secret (the auth flow)

Device                    Zitadel                  OpenBao
  |                          |                        |
  |-- sign JWT with key ---→ |                        |
  |←-- short-lived JWT ------                         |
  |                                                   |
  |-- POST /v1/auth/jwt/login (jwt=<token>) --------→ |
  |←-- OpenBao token (scoped by entity policies) ---- |
  |                                                   |
  |-- GET /v1/secret/data/deployment-a/... ---------→ |
  |←-- secret value --------------------------------- |

OpenBao verifies the JWT against Zitadel's JWKS endpoint, then looks up the entity associated with the JWT's sub claim. The entity carries a policy list. The issued OpenBao token inherits those policies.

How deployment assignment works (the operator flow)

Operator
  |
  |-- watches k8s Deployment CR (spec.targetSelector)
  |-- watches k8s Device CR (metadata.labels)
  |
  | When device labels match a deployment's selector:
  |-- writes desired-state.<device>.<deployment> to NATS KV  ← tells the agent what to run
  |-- calls PUT /v1/identity/entity/name/<device> on OpenBao ← grants access to secrets
  |
  | When the match goes away:
  |-- deletes the NATS KV entry
  |-- updates entity policies (removing that deployment)

The two writes always happen together. A device can run a deployment task only if the NATS KV entry exists (so it knows what to do) AND the entity has the policy (so it can read the secrets).

---

Why we are changing (Design A → Design B)

Design A encoded the device's current deployments as a deployments: ["a", "b"] claim inside the Zitadel JWT. OpenBao read that claim via groups_claim=deployments and mapped it to policies at login time. This meant:

• Every time a device was reassigned to a new deployment, it had to log out of OpenBao and back in to get a new token with updated scope.
• At 50k devices all receiving a new deployment simultaneously, that is a thundering herd hitting both Zitadel (JWT mint) and OpenBao (JWT login) at the same time.

Design B stores the policy list directly on the OpenBao entity. The device logs in once and gets a token. When the operator changes the assignment, it updates the entity in OpenBao. The same device token, on its next API call, sees the updated policy. No re-auth required.

The deployments JWT claim and the Zitadel action that set it are both deleted.

---

Code map: where each concept lives

Concept	File
OpenBao Helm deploy	harmony/src/modules/openbao/mod.rs — OpenbaoScore
OpenBao init, unseal, JWT auth config	harmony/src/modules/openbao/setup.rs — OpenbaoSetupScore
Zitadel Helm deploy + PG	harmony/src/modules/zitadel/mod.rs — ZitadelScore
Zitadel post-deploy config (machine users, apps)	harmony/src/modules/zitadel/setup.rs — ZitadelSetupScore
[DELETE] Zitadel action that set deployments claim	harmony/src/modules/zitadel/action.rs
Device secret reading (harmony_secret client)	harmony_secret/src/store/openbao.rs — OpenbaoSecretStore
Operator: deployment assignment → NATS KV	fleet/harmony-fleet-operator/src/fleet_aggregator.rs
Operator: wires up k8s CRD watch + NATS	fleet/harmony-fleet-operator/src/main.rs

---

Step 1 — Delete action.rs

File: harmony/src/modules/zitadel/action.rs

This file implements ZitadelDeploymentsClaimActionScore. A "Zitadel Action" is a JavaScript snippet that Zitadel executes during token issuance to add custom claims. This one read the machine user's deployments metadata and injected it as a JWT claim. With Design B, OpenBao no longer reads this claim — delete the file entirely.

git rm harmony/src/modules/zitadel/action.rs

---

Step 2 — Remove the action module from zitadel/mod.rs

File: harmony/src/modules/zitadel/mod.rs, line 1

Remove:

pub mod action;

Nothing else in mod.rs re-exports types from action, so this is the only line to touch.

Why it is a pub mod at all: In Rust, modules must be explicitly declared. pub mod action both declares the submodule and makes it visible to crates that depend on harmony. Since the file is gone, the declaration must go too, or the compiler will error looking for action.rs.

---

Step 3 — Strip Design A from harmony_secret/src/store/openbao.rs

This file is the device-side client that authenticates to OpenBao and reads secrets. It currently carries leftover Design A logic: decoding the deployments JWT claim to track which secrets the current token is scoped to. In Design B the scope is stored in OpenBao itself, so the device code doesn't need to know about it.

3a. Remove imports

Lines 3 and 7 — remove:

use base64::Engine;
use std::collections::HashSet;

base64 is only used to decode the JWT body for decode_deployments_claim. HashSet is only used to hold the scope. Both go away.

3b. Remove decode_deployments_claim

Lines 37–56 — remove the entire function:

fn decode_deployments_claim(jwt: &str) -> Result<HashSet<String>, SecretStoreError> {
    // ... base64-decode the JWT body, parse the `deployments` array
}

3c. Remove scope from the Inner struct

Line 71 — the Inner struct holds the live VaultClient plus a cached copy of what deployments the current token can access. Remove scope:

// Before
struct Inner {
    client: VaultClient,
    scope: HashSet<String>,
}

// After
struct Inner {
    client: VaultClient,
}

3d. Update refresh_auth

This method re-mints the Zitadel JWT and exchanges it for a new OpenBao token. Remove the scope decoding line and update the Inner construction:

// Remove this line:
let scope = decode_deployments_claim(&jwt)?;

// Change this:
*self.inner.lock().await = Inner { client, scope };
// To:
*self.inner.lock().await = Inner { client };

3e. Remove cached_scope

Lines 336–338 — remove:

pub async fn cached_scope(&self) -> HashSet<String> {
    self.inner.lock().await.scope.clone()
}

3f. Update with_token_and_jwt

Line 466 — remove scope: HashSet::new() from the Inner construction:

// Before
let inner = Mutex::new(Inner { client, scope: HashSet::new() });
// After
let inner = Mutex::new(Inner { client });

3g. Remove the tests

Remove fn unsigned_jwt (the test helper that constructs a fake JWT using base64) and both decode_deployments_claim_* test functions at lines 708–750. The two remaining tests (test_hash_url_consistency and test_hash_url_uniqueness) are unrelated and stay.

---

Step 4 — Remove base64 from harmony_secret/Cargo.toml

File: harmony_secret/Cargo.toml, line 16

Remove:

base64.workspace = true

base64 was a workspace dependency (version pinned in the root Cargo.toml). It was only used in openbao.rs for decode_deployments_claim and the test helper. After Step 3 it is unused; leaving it would cause a cargo clippy warning about unused deps.

---

Step 5 — Update OpenbaoJwtAuth in setup.rs

File: harmony/src/modules/openbao/setup.rs

OpenbaoSetupScore handles the one-time OpenBao bootstrap: init, unseal, KV engine, userpass auth, and JWT auth. OpenbaoJwtAuth is the configuration struct for the JWT auth method.

5a. Remove policies from OpenbaoJwtAuth

// Before
pub struct OpenbaoJwtAuth {
    pub oidc_discovery_url: String,
    pub bound_issuer: String,
    pub role_name: String,
    pub bound_audiences: String,
    pub user_claim: String,
    pub policies: Vec<String>,   // ← remove
    pub ttl: String,
    pub max_ttl: String,
}

In Design A, the JWT role itself carried policies so every device that logged in got those policies unconditionally (or via groups_claim). In Design B, the role grants no static policies — policies come from the entity. OpenBao will still attach the built-in default policy automatically.

5b. Update configure_jwt in OpenbaoSetupInterpret

Remove the two lines that format and inject policies into the bao write auth/jwt/role/... command:

// Remove:
let policies = jwt.policies.join(",");
// Remove from the &[...] slice:
&format!("policies={}", policies),

The user_claim field stays — this is what tells OpenBao which JWT field to use as the entity alias name (you'll set it to sub, the Zitadel machine user's client ID). OpenBao will auto-create an entity alias on first login and link it to whatever entity already has that alias name.

---

Step 6 — Add entity management to setup.rs

File: harmony/src/modules/openbao/setup.rs

How OpenBao identity entities work

OpenBao's identity engine has three concepts:

• Entity: a named principal with an attached policy list. Think of it as the device's identity record inside OpenBao.
• Entity alias: links an external identity (e.g. the sub value from a Zitadel JWT) to an entity. One entity can have aliases from multiple auth methods.
• Policy: an HCL document granting read/write access to specific KV paths. You'll have one per deployment, e.g. deployment-a grants read on secret/data/deployment-a/*.

When a device logs in via JWT auth, OpenBao looks up the entity alias matching (jwt_mount_accessor, jwt_sub_value). The entity that alias points to defines what the token can do.

6a. Add OpenbaoEntity

#[derive(Debug, Clone, Serialize)]
pub struct OpenbaoEntity {
    /// Name of the entity in OpenBao (typically the device ID or machine user name).
    pub name: String,
    /// The JWT `sub` value for this device — the Zitadel machine user's client ID.
    /// This is what OpenBao uses as the entity alias name on the JWT auth mount.
    pub alias_name: String,
    /// Initial policies to attach. Usually empty at setup time; the operator
    /// updates this when deployment assignment happens.
    pub policies: Vec<String>,
}

6b. Add entities to OpenbaoSetupScore

pub struct OpenbaoSetupScore {
    // ... existing fields ...
    #[serde(default)]
    pub entities: Vec<OpenbaoEntity>,
}

6c. Add create_entities to OpenbaoSetupInterpret

This method runs after configure_jwt. For each entity it:

1. Creates/updates the entity with its policies:

   bao write identity/entity name=<name> policies=<csv>
   

2. Reads back the entity ID (needed to create the alias):

   bao read -format=json identity/entity/name/<name>
   

   Parse data.id from the JSON output.

3. Gets the JWT auth mount's accessor (a stable ID OpenBao assigns to each auth method mount):

   bao auth list -format=json
   

   Parse the accessor field for the jwt/ key.

4. Creates the entity alias (idempotent — same alias on same mount accessor is an upsert):

   bao write identity/entity-alias \
     name=<alias_name> \
     mount_accessor=<accessor> \
     canonical_id=<entity_id>
   

Call create_entities in execute() after configure_jwt().

6d. Add OpenbaoEntityPoliciesScore

This is the Score the operator runs (via OpenbaoEntityPoliciesScore::interpret) when a device's deployment assignment changes. It's separate from OpenbaoSetupScore because it is a runtime operation, not a one-time setup step.

#[derive(Debug, Clone, Serialize)]
pub struct OpenbaoEntityPoliciesScore {
    pub instance: OpenbaoInstance,
    pub entity_name: String,
    pub policies: Vec<String>,
}

Its Interpret::execute runs one command:

bao write identity/entity/name/<entity_name> policies=<csv>

The bao write identity/entity/name/... endpoint is an upsert — it creates or updates. Passing an empty policies list clears all policies. This is idempotent and safe to retry.

---

Step 7 — Export new types from harmony/src/modules/openbao/mod.rs

File: harmony/src/modules/openbao/mod.rs

Add to the pub use setup::... line:

pub use setup::{
    OpenbaoEntity,
    OpenbaoEntityPoliciesScore,
    OpenbaoJwtAuth,
    OpenbaoPolicy,
    OpenbaoSetupScore,
    OpenbaoUser,
    cached_root_token,
};

---

Step 8 — New file: fleet/harmony-fleet-operator/src/openbao.rs

Why this file exists: The operator needs to call OpenBao's HTTP API directly (not via pod exec like OpenbaoSetupScore does — that requires k8s exec access and is too slow for a hot reconcile loop). reqwest is already in the operator's Cargo.toml.

use anyhow::Result;

pub struct OpenBaoIdentityClient {
    client: reqwest::Client,
    base_url: String,
    token: String,
}

impl OpenBaoIdentityClient {
    pub fn new(base_url: String, token: String) -> Self {
        Self {
            client: reqwest::Client::new(),
            base_url,
            token,
        }
    }

    /// Idempotent upsert: sets the entity's policy list to exactly `policies`.
    /// Passing an empty slice clears all policies (device loses all secret access).
    pub async fn set_entity_policies(&self, entity: &str, policies: &[String]) -> Result<()> {
        let url = format!(
            "{}/v1/identity/entity/name/{}",
            self.base_url.trim_end_matches('/'),
            entity
        );
        let body = serde_json::json!({ "policies": policies });
        let resp = self.client
            .post(&url)
            .header("X-Vault-Token", &self.token)
            .json(&body)
            .send()
            .await?;
        let status = resp.status();
        if !status.is_success() {
            let text = resp.text().await.unwrap_or_default();
            anyhow::bail!("OpenBao entity update {entity} returned {status}: {text}");
        }
        Ok(())
    }
}

Note on the HTTP method: OpenBao's identity entity endpoint accepts both POST and PUT. POST /v1/identity/entity/name/<name> is the create-or-update form (no entity ID required). If the entity doesn't exist yet, OpenBao creates it; if it does, it updates it.

---

Step 9 — Update fleet/harmony-fleet-operator/src/fleet_aggregator.rs

This is the biggest conceptual change. Read the existing file top-to-bottom once before making changes — understanding the FleetState struct and owned_targets is key.

How owned_targets works

owned_targets: HashMap<DeploymentName, HashSet<device_id>> is the aggregator's in-memory truth of "which devices currently match which deployments." It is the diff source for both the NATS KV writes and (after this change) the OpenBao entity updates.

When a device gains a deployment: owned_targets[deployment].insert(device_id) + KV put. When a device loses a deployment: owned_targets[deployment].remove(device_id) + KV delete.

To compute the full policy list for a device at any point, you iterate over owned_targets and collect every deployment name where the device appears.

9a. Update run signature

pub async fn run(
    client: Client,
    js: async_nats::jetstream::Context,
    openbao: Option<Arc<OpenBaoIdentityClient>>,
) -> anyhow::Result<()>

Thread openbao down to the internal event handlers by adding it as a parameter or by wrapping it in the existing SharedFleetState (the latter is cleaner since you avoid changing every function signature — add openbao: Option<Arc<OpenBaoIdentityClient>> to FleetState).

9b. Add the policy helper

/// Returns the full set of OpenBao policies for `device_id` given the
/// current `owned_targets`. One policy per matched deployment, named
/// `deployment-<name>` to match the HCL policies created by OpenbaoSetupScore.
fn device_deployment_policies(state: &FleetState, device_id: &str) -> Vec<String> {
    state
        .owned_targets
        .iter()
        .filter(|(_, devices)| devices.contains(device_id))
        .map(|(dn, _)| format!("deployment-{}", dn.as_str()))
        .collect()
}

Why compute the full set instead of add/remove: If you add/remove incrementally you need a read-before-write (race condition) or local state that can drift. Writing the full current set is idempotent and always converges to the right answer, even after an operator restart.

9c. Add OpenBao sync to on_device_upsert

After the match (was, now) block that does the KV writes, add:

// After all KV writes/deletes for this device, sync entity policies.
if let Some(bao) = &openbao {
    let policies = {
        let guard = state.lock().await;
        device_deployment_policies(&guard, &name)
    };
    if let Err(e) = bao.set_entity_policies(&name, &policies).await {
        tracing::warn!(device = %name, error = %e, "aggregator: OpenBao entity policy sync failed");
    }
}

The OpenBao call is outside the lock and non-fatal. Matching the existing pattern for KV write failures — log and continue, don't abort the reconcile loop.

9d. Add OpenBao sync to on_deployment_delete

After the loop that deletes KV entries for previously-owned devices, update each device's entity:

for device in &previous {
    // ... existing KV delete ...
    if let Some(bao) = &openbao {
        let policies = {
            let guard = state.lock().await;
            device_deployment_policies(&guard, device)
        };
        if let Err(e) = bao.set_entity_policies(device, &policies).await {
            tracing::warn!(device = %device, error = %e, "aggregator: OpenBao policy sync on deployment delete failed");
        }
    }
}

9e. Add OpenBao sync to on_device_delete

When a device CR is deleted (device decommissioned), clear all its policies:

if let Some(bao) = &openbao {
    if let Err(e) = bao.set_entity_policies(&name, &[]).await {
        tracing::warn!(device = %name, error = %e, "aggregator: OpenBao policy clear on device delete failed");
    }
}

---

Step 10 — Update fleet/harmony-fleet-operator/src/main.rs

10a. Add CLI args

Add to the Cli struct:

#[arg(long, env = "OPENBAO_URL", global = true)]
openbao_url: Option<String>,

#[arg(long, env = "OPENBAO_TOKEN", global = true)]
openbao_token: Option<String>,

Both are optional. If either is absent, the operator runs without OpenBao policy sync (same as today — useful for dev environments without OpenBao).

10b. Build the client and pass it to the aggregator

In the run() function:

let openbao = match (&cli.openbao_url, &cli.openbao_token) {
    (Some(url), Some(token)) => {
        tracing::info!(url = %url, "OpenBao identity sync enabled");
        Some(Arc::new(OpenBaoIdentityClient::new(url.clone(), token.clone())))
    }
    _ => {
        tracing::warn!("OPENBAO_URL or OPENBAO_TOKEN not set — entity policy sync disabled");
        None
    }
};

// Pass to the aggregator:
r = fleet_aggregator::run(client, js, openbao) => r,

The admin token (OPENBAO_TOKEN) should be mounted from a Kubernetes Secret in the operator's Pod spec — same pattern as FLEET_OPERATOR_CREDENTIALS_TOML. The token needs a policy that allows writes to identity/entity/name/*.

---

Complete change list

File	What changes
harmony/src/modules/zitadel/action.rs	Delete — no longer needed
harmony/src/modules/zitadel/mod.rs	Remove pub mod action; (1 line)
harmony_secret/src/store/openbao.rs	Remove decode_deployments_claim, scope field, cached_scope, base64/HashSet imports, 2 tests
harmony_secret/Cargo.toml	Remove base64 dep (1 line)
harmony/src/modules/openbao/setup.rs	Remove policies from OpenbaoJwtAuth; add OpenbaoEntity, entities field, create_entities(), OpenbaoEntityPoliciesScore
harmony/src/modules/openbao/mod.rs	Export OpenbaoEntity, OpenbaoEntityPoliciesScore
fleet/harmony-fleet-operator/src/openbao.rs	New — OpenBaoIdentityClient::set_entity_policies
fleet/harmony-fleet-operator/src/fleet_aggregator.rs	Add openbao param, device_deployment_policies helper, 3 call sites
fleet/harmony-fleet-operator/src/main.rs	Add OPENBAO_URL/OPENBAO_TOKEN args; build and thread OpenBaoIdentityClient

---

Verification checklist

After all changes:

cargo check --all-targets --all-features --keep-going
cargo fmt --check
cargo clippy
cargo test

Manual smoke test (requires a running local stack):

1. Assign a device to deployment-a via the k8s CR → operator logs should show OpenBao entity policy sync for that device → bao read identity/entity/name/<device> should show policies: ["deployment-a"].
2. Reassign the device to deployment-b → entity should update to ["deployment-b"] without the device re-authenticating to OpenBao.
3. Remove the device CR → entity policies should be cleared.