harmony/docs/guides/fleet-device-secrets.md

Fleet device secrets

How fleet devices read their deployments' secrets — image-pull credentials, application config, API keys — from OpenBao, and how you control who reads what. Design reasoning lives in ADR-025; this page is the operating manual.

The architecture in one minute

A device proves who it is with the Zitadel machine key it already uses for NATS. What it may read is decided by group membership, which only admins can change. Where it runs is decided by labels. The three never mix:

 Admin                          Zitadel                       OpenBao
   │ assign device to groups ──▶ device ∈ {edge-a, …}           │
   │ write secret ──────────────────────────────────────────────▶ secret/<fleet>/<deployment>/…
   │                                                            │
 Deployment CR                                                  │
   │ allowedGroups: [edge-a] ──▶ operator attaches policy ──────▶ group edge-a may read
   │ targetSelector: labels  ──▶ operator schedules within      │   deployment-<name> subtree
   │                             authorized devices             │
   │                                                            │
 Device                                                         │
   └─ machine key ─▶ Zitadel token (groups: [edge-a]) ─▶ JWT login ─▶ reads its secrets

Three rules to remember:

Concept	Who sets it	What it controls
Group	Admin, in Zitadel	Security boundary: which deployments' secrets a device may read
Label	Device config (self-reported)	Placement only: which authorized devices actually run the workload — narrows, never widens
allowedGroups	Deployment author	Which groups may view (run + read secrets of) the deployment

Storing a secret for a deployment

Write it under the deployment's subtree of the fleet KV mount:

bao kv put secret/<fleet-ns>/<deployment-name>/db-credentials \
    username=app password=…

Nothing else to configure — access follows from the deployment's allowedGroups below. Every device in an allowed group can read the whole <deployment-name>/ subtree, whether or not the workload landed on it; size your groups to the audience the secret may have.

Authorizing and placing a deployment

apiVersion: fleet.nationtech.io/v1alpha1
kind: Deployment
metadata:
  name: hello-web
spec:
  allowedGroups: [edge-a]        # security: who may view this deployment
  targetSelector:
    matchLabels:
      hw: pi5                    # placement: which authorized devices run it
  score: …

The operator schedules onto devices that are members of an allowed group AND match the selector, then attaches the deployment's read policy to each allowed group. Existing device sessions pick the grant up immediately — rolling out a new deployment never touches logins, tokens, or per-device configuration.

Managing device group membership

Until Zitadel's first-class groups are GA, a fleet group is a Zitadel project role (one role per group, surfaced as a groups claim by the fleet's token Action — applied by the fleet bootstrap, not by hand). Grant or remove the role on the device's machine user to move it between groups.

Membership changes take effect at the device's next login, at most one token TTL later (≈ 1 h). This is the only eventually-consistent operation in the system — everything else binds immediately. If that window matters, use the emergency levers below.

Reading secrets on the device

Nothing to do: the agent reads through harmony_config, which authenticates to OpenBao with the machine keyfile automatically (JWT-bearer rung of OpenbaoSecretStore). Tokens are short-lived batch tokens; the store re-logins transparently on expiry.

Revocation runbook

Situation	Action	Effect
Device leaves a group (routine)	Remove the role/group in Zitadel	Converges at next login, ≤ 1 TTL
Device compromised	Disable its OpenBao identity entity, then rotate/delete its Zitadel machine key	Token dead at next request; no new logins
Secret/deployment compromised	Detach deployment-<name> policy from its groups (or rotate the secret)	All devices lose read immediately
Whole group compromised	Detach the group's policies	Immediate, group-wide

Guarantees and limits

• A device can only ever read secrets of deployments whose allowedGroups intersect its groups. Labels, CR edits, or a compromised agent cannot widen that — membership is signed into the Zitadel token and validated against the project (bound_audiences), so other projects' tokens are rejected outright.
• Visibility is group-granular: being scheduled is not required to read. Finer isolation ⇒ finer groups.
• One Zitadel project + OpenBao pair (a cell) comfortably serves ~10⁵ devices; beyond that, shard fleets into cells (ADR-025, Rationale).