harmony/docs/adr/025-fleet-device-secret-access.md

Architecture Decision Record: Fleet Device Secret Access

Initial Author: Jean-Gabriel Gill-Couture

Initial Date: 2026-06-01

Last Updated Date: 2026-06-11

Status

Accepted. User-facing documentation: Fleet device secrets guide.

This ADR is the third design iteration; the trail is preserved in 025-device-secret-access/: 025-1 (JWT claim scoping, superseded) → 025-2 (entity-by-policy, first implementation, retired) → 025-3 (the scale and security analysis that produced this design).

Context

Fleet agents must read per-deployment secrets (image-pull credentials, application secrets) from OpenBao. Constraints:

1. One root of trust per device. The agent already holds a Zitadel machine keyfile for NATS auth; secret access must derive from the same key, never a second secret.
2. Store-enforced isolation. The secret store decides what a device may read; a device cannot widen its own scope.
3. Cross-project safety. A token minted in another Zitadel project must be rejected outright.
4. Fleet scale. The target is millions of devices. Control-plane write cost must track membership churn (rare, per-device), never deployment churn × targeted devices — a deployment rolling out to a large group must not trigger a per-device write storm.

Decision

Groups are the security boundary; labels only place workloads.

Identity: one key, one login, short batch tokens

The device mints a Zitadel access token from its machine keyfile (RFC 7523 JWT-bearer, shared with NATS auth via harmony_zitadel_jwt) and logs into OpenBao's JWT auth method through harmony_config/OpenbaoSecretStore. The auth role pins bound_issuer and bound_audiences = <Zitadel project ID> — the project is the trust boundary — with user_claim = sub and groups_claim = groups. The role issues batch tokens (no storage writes per login) with TTL ≈ 1 h, no renewal. Expiry forces a fresh login (refresh_auth()), and that login is the group-membership sync — there is no separate re-login flow, signal, or membership watcher.

Groups: admin-managed in Zitadel, mirrored as OpenBao external groups

A fleet group (e.g. edge-a, customer-x) is membership a device can never self-assert: admins manage it in Zitadel, and it reaches OpenBao inside the signed token's groups array claim. Each fleet group has an OpenBao external group whose alias matches the JWT mount, so a login binds the device's entity to its current groups. Interim claim source until Zitadel's first-class groups are GA: one project role per group plus the standard flattening Action (Zitadel's roles claim is a map; groups_claim requires a string array).

Authorization and scheduling: groups gate, labels place

A Deployment declares allowedGroups — the groups permitted to view it (run it and read its secrets). The operator schedules onto members of the allowed groups ∩ the label selector. Labels remain device-reported placement vocabulary; they can narrow a deployment's spread, never widen access. (This is Kubernetes' RBAC-vs-scheduling split.)

Grants: one policy per deployment, attached to groups

Secrets live under the deployment's subtree of the fleet KV mount; the policy deployment-<name> grants read on exactly that subtree. When a Deployment appears, the operator creates the policy and attaches it to each allowed group's external group — O(allowed groups) writes, regardless of fleet size. Identity policies resolve at request time, so every already-issued token of every member device gains (or loses) access immediately when a policy is attached (or detached).

Convergence and revocation

Change	Mechanism	Latency
Deployment created / allowedGroups edited	attach/detach policy on group	immediate (request time)
Device added to / removed from a group	next login (token expiry)	≤ token TTL
Emergency: cut one device	disable its OpenBao entity	immediate
Emergency: cut one deployment or group	detach the policy	immediate
Device key compromised	rotate/revoke machine key in Zitadel + disable entity	immediate

TTL is therefore an operational tolerance for routine de-grouping, not a security boundary — emergencies never wait on it.

Rationale

The full analysis with sources is 025-3; the load-bearing points:

• Write cost aligned with event granularity. The retired entity-by-policy design (025-2) wrote every targeted device's entity per deployment change; each identity write rewrites 1/256 of all entities, so cost grows with fleet size — gigabytes of Raft churn and hours of convergence at 10⁵–10⁶ targets. Group attachment is one write and binds instantly.
• Groups close the label-escalation hole. Labels are device-reported; with labels gating secrets, a compromised device relabels itself to attract any deployment (Kubernetes needs NodeRestriction for exactly this). Group membership is signed by Zitadel and admin-only.
• Renewal does not re-sync groups. Verified in the JWT auth plugin source (Vault and OpenBao): renew only extends TTLs. Hence short-lived, non-renewed tokens whose natural expiry is the sync — batch tokens make this free (1M devices at 1 h TTL ≈ 280 stateless logins/s; the Zitadel-side token is reused across many logins).
• Precedent. AWS IoT Core authorizes fleets exactly this way: policies attach to thing groups and "are dynamically applied to things when they change group membership"; Greengrass deploys secrets by targeting thing groups. There is no Vault-ecosystem pattern for per-device grant materialization at fleet scale.
• Cell ceiling is independent of this choice. OpenBao's identity store is memory-resident and packs entities into 256 storage entries (≈0.5–1.25M entities max), and JWT login creates one entity per device under any design. Past ~10⁵ devices, fleets shard into cells (one Zitadel project + OpenBao pair each); groups are scoped within a cell.

Consequences

Pros

• New deployment: no device session touched, no per-device writes, no admin permission work beyond declaring allowedGroups.
• One auth root per device covers NATS and OpenBao; rotation, inventory, and revocation stay centralized in Zitadel.
• The security boundary is legible: group rosters in Zitadel, group→deployment grants on the OpenBao group object, both human-auditable.
• Batch tokens keep steady-state load CPU-bound (JWKS verification), with zero token-storage writes.

Cons / accepted trade-offs

• Group-granular visibility. Every device in an allowed group can read the deployment's secrets, scheduled onto it or not. Isolation granularity is group sizing — need finer isolation, define finer groups. (Per-workload identity via SPIFFE is the future evolution that lifts this; the policy layer survives that migration.)
• De-grouping converges at TTL (≤ 1 h). Emergencies use the immediate levers above instead.
• Zitadel Action required until first-class groups GA — small, version-controlled server-side JS in the token path.
• Batch tokens are not individually revocable; per-device cutoff is entity disable, which blocks the token at request time.
• One entity + alias per device still accumulates in OpenBao (the cell ceiling above).

Threat model

Attacker	Capability	Defense
External (no Zitadel identity)	None	JWKS signature validation.
Other Zitadel project	Mints valid tokens for its own project	bound_audiences rejects before any claim is read.
Compromised device (key theft)	Reads secrets of deployments its groups are allowed to view — nothing more. Cannot self-widen: labels don't grant, groups are Zitadel-signed.	Disable entity + rotate key; blast radius bounded by group sizing.
Compromised operator	Full control of its fleet's grants (policy attach, scheduling)	Blast radius = one cell (one project, one Bao); operator creds under a separate Bao auth path.
Compromised OpenBao	Everything in its cell	Out of scope — root of secret trust (ADR-006); cells bound the radius.

Alternatives considered

Full detail in the iteration trail. In brief: JWT deployments claim (025-1 — scope frozen at mint time, two-token invalidation per change, Zitadel metadata as a second source of truth); entity-by-policy (025-2, first implementation — instant per-device grants but O(targeted devices) writes per deployment, disqualifying at fleet scale); one templated policy on alias metadata (zero writes ever, but policy templating cannot iterate, so exactly one group per device — kept as a fallback simplification); static per-device Bao tokens (second long-lived secret, no scoping); AppRole / TLS-cert / SPIFFE auth (solve identity, which Zitadel already provides; SPIFFE noted as the per-workload evolution path); request-time authorization plugin (true NodeAuthorizer equivalent, but couples secret reads to control-plane liveness).

Relationship to other ADRs

ADR-016: the Zitadel machine identity reused here. ADR-020/020-1: harmony_config as the single secret entry point; the human counterpart of this machine auth path. ADR-023: all OpenBao/Zitadel configuration lands via Scores.