harmony/docs/adr/025-fleet-device-secret-access.md

# Architecture Decision Record: Fleet Device Secret Access

Initial Author: Jean-Gabriel Gill-Couture

Initial Date: 2026-06-01

Last Updated Date: 2026-06-11

## Status

Accepted. User-facing documentation:
[Fleet device secrets guide](../guides/fleet-device-secrets.md).

This ADR is the third design iteration; the trail is preserved in
[`025-device-secret-access/`](./025-device-secret-access/):
[025-1](./025-device-secret-access/025-1-jwt-deployments-claim.md)
(JWT claim scoping, superseded) →
[025-2](./025-device-secret-access/025-2-entity-by-policy.md)
(entity-by-policy, first implementation, retired) →
[025-3](./025-device-secret-access/025-3-groups-as-security-boundary.md)
(the scale and security analysis that produced this design).

## Context

Fleet agents must read per-deployment secrets (image-pull credentials,
application secrets) from OpenBao. Constraints:

1. **One root of trust per device.** The agent already holds a Zitadel
   machine keyfile for NATS auth; secret access must derive from the
   same key, never a second secret.
2. **Store-enforced isolation.** The secret store decides what a device
   may read; a device cannot widen its own scope.
3. **Cross-project safety.** A token minted in another Zitadel project
   must be rejected outright.
4. **Fleet scale.** The target is millions of devices. Control-plane
   write cost must track *membership churn* (rare, per-device), never
   *deployment churn × targeted devices* — a deployment rolling out to
   a large group must not trigger a per-device write storm.

## Decision

**Groups are the security boundary; labels only place workloads.**

### Identity: one key, one login, short batch tokens

The device mints a Zitadel access token from its machine keyfile (RFC
7523 JWT-bearer, shared with NATS auth via `harmony_zitadel_jwt`) and
logs into OpenBao's JWT auth method through
`harmony_config`/`OpenbaoSecretStore`. The auth role pins
`bound_issuer` and `bound_audiences = <Zitadel project ID>` — the
project is the trust boundary — with `user_claim = sub` and
`groups_claim = groups`. The role issues **batch tokens** (no storage
writes per login) with **TTL ≈ 1 h, no renewal**. Expiry forces a fresh
login (`refresh_auth()`), and that login is the group-membership sync —
there is no separate re-login flow, signal, or membership watcher.

### Groups: admin-managed in Zitadel, mirrored as OpenBao external groups

A fleet group (e.g. `edge-a`, `customer-x`) is membership a device can
never self-assert: admins manage it in Zitadel, and it reaches OpenBao
inside the signed token's `groups` array claim. Each fleet group has an
OpenBao **external group** whose alias matches the JWT mount, so a
login binds the device's entity to its current groups. Interim claim
source until Zitadel's first-class groups are GA: one project role per
group plus the standard flattening Action (Zitadel's roles claim is a
map; `groups_claim` requires a string array).

### Authorization and scheduling: groups gate, labels place

A `Deployment` declares `allowedGroups` — the groups permitted to view
it (run it and read its secrets). The operator schedules onto
**members of the allowed groups ∩ the label selector**. Labels remain
device-reported placement vocabulary; they can narrow a deployment's
spread, never widen access. (This is Kubernetes' RBAC-vs-scheduling
split.)

### Grants: one policy per deployment, attached to groups

Secrets live under the deployment's subtree of the fleet KV mount; the
policy `deployment-<name>` grants read on exactly that subtree. When a
`Deployment` appears, the operator creates the policy and attaches it
to each allowed group's external group — **O(allowed groups) writes,
regardless of fleet size**. Identity policies resolve at request time,
so every already-issued token of every member device gains (or loses)
access immediately when a policy is attached (or detached).

### Convergence and revocation

| Change | Mechanism | Latency |
|---|---|---|
| Deployment created / `allowedGroups` edited | attach/detach policy on group | immediate (request time) |
| Device added to / removed from a group | next login (token expiry) | ≤ token TTL |
| Emergency: cut one device | disable its OpenBao entity | immediate |
| Emergency: cut one deployment or group | detach the policy | immediate |
| Device key compromised | rotate/revoke machine key in Zitadel + disable entity | immediate |

TTL is therefore an operational tolerance for *routine* de-grouping,
not a security boundary — emergencies never wait on it.

## Rationale

The full analysis with sources is
[025-3](./025-device-secret-access/025-3-groups-as-security-boundary.md);
the load-bearing points:

- **Write cost aligned with event granularity.** The retired
  entity-by-policy design (025-2) wrote every targeted device's entity
  per deployment change; each identity write rewrites 1/256 of all
  entities, so cost grows with fleet size — gigabytes of Raft churn
  and hours of convergence at 10⁵–10⁶ targets. Group attachment is one
  write and binds instantly.
- **Groups close the label-escalation hole.** Labels are
  device-reported; with labels gating secrets, a compromised device
  relabels itself to attract any deployment (Kubernetes needs
  `NodeRestriction` for exactly this). Group membership is signed by
  Zitadel and admin-only.
- **Renewal does not re-sync groups.** Verified in the JWT auth plugin
  source (Vault and OpenBao): renew only extends TTLs. Hence
  short-lived, non-renewed tokens whose natural expiry is the sync —
  batch tokens make this free (1M devices at 1 h TTL ≈ 280 stateless
  logins/s; the Zitadel-side token is reused across many logins).
- **Precedent.** AWS IoT Core authorizes fleets exactly this way:
  policies attach to thing groups and "are dynamically applied to
  things when they change group membership"; Greengrass deploys
  secrets by targeting thing groups. There is no Vault-ecosystem
  pattern for per-device grant materialization at fleet scale.
- **Cell ceiling is independent of this choice.** OpenBao's identity
  store is memory-resident and packs entities into 256 storage entries
  (≈0.5–1.25M entities max), and JWT login creates one entity per
  device under any design. Past ~10⁵ devices, fleets shard into cells
  (one Zitadel project + OpenBao pair each); groups are scoped within
  a cell.

## Consequences

**Pros**

- New deployment: no device session touched, no per-device writes, no
  admin permission work beyond declaring `allowedGroups`.
- One auth root per device covers NATS and OpenBao; rotation,
  inventory, and revocation stay centralized in Zitadel.
- The security boundary is legible: group rosters in Zitadel,
  group→deployment grants on the OpenBao group object, both
  human-auditable.
- Batch tokens keep steady-state load CPU-bound (JWKS verification),
  with zero token-storage writes.

**Cons / accepted trade-offs**

- **Group-granular visibility.** Every device in an allowed group can
  read the deployment's secrets, scheduled onto it or not. Isolation
  granularity is group sizing — need finer isolation, define finer
  groups. (Per-workload identity via SPIFFE is the future evolution
  that lifts this; the policy layer survives that migration.)
- **De-grouping converges at TTL** (≤ 1 h). Emergencies use the
  immediate levers above instead.
- **Zitadel Action required** until first-class groups GA — small,
  version-controlled server-side JS in the token path.
- **Batch tokens are not individually revocable**; per-device cutoff
  is entity disable, which blocks the token at request time.
- One entity + alias per device still accumulates in OpenBao (the
  cell ceiling above).

## Threat model

| Attacker | Capability | Defense |
|---|---|---|
| External (no Zitadel identity) | None | JWKS signature validation. |
| Other Zitadel project | Mints valid tokens for its own project | `bound_audiences` rejects before any claim is read. |
| Compromised device (key theft) | Reads secrets of deployments its groups are allowed to view — nothing more. Cannot self-widen: labels don't grant, groups are Zitadel-signed. | Disable entity + rotate key; blast radius bounded by group sizing. |
| Compromised operator | Full control of its fleet's grants (policy attach, scheduling) | Blast radius = one cell (one project, one Bao); operator creds under a separate Bao auth path. |
| Compromised OpenBao | Everything in its cell | Out of scope — root of secret trust (ADR-006); cells bound the radius. |

## Alternatives considered

Full detail in the
[iteration trail](./025-device-secret-access/). In brief: **JWT
`deployments` claim** (025-1 — scope frozen at mint time, two-token
invalidation per change, Zitadel metadata as a second source of
truth); **entity-by-policy** (025-2, first implementation — instant
per-device grants but O(targeted devices) writes per deployment,
disqualifying at fleet scale); **one templated policy on alias
metadata** (zero writes ever, but policy templating cannot iterate, so
exactly one group per device — kept as a fallback simplification);
**static per-device Bao tokens** (second long-lived secret, no
scoping); **AppRole / TLS-cert / SPIFFE auth** (solve identity, which
Zitadel already provides; SPIFFE noted as the per-workload evolution
path); **request-time authorization plugin** (true NodeAuthorizer
equivalent, but couples secret reads to control-plane liveness).

## Relationship to other ADRs

ADR-016: the Zitadel machine identity reused here. ADR-020/020-1:
`harmony_config` as the single secret entry point; the human
counterpart of this machine auth path. ADR-023: all OpenBao/Zitadel
configuration lands via Scores.