Jean-Gabriel Gill-Couture
397f266d31
Rewrite ADR-025 around the group-based design: groups gate (admin- managed identity), labels place, OpenBao enforces via external groups; batch tokens with 1h TTL make expiry re-login the membership sync. Iteration trail preserved under 025-device-secret-access/ (025-1 JWT claim scoping, 025-2 entity-by-policy as built, 025-3 the scale and security analysis with verified sources). Add the user-facing guide and the implementation + branch-consolidation rollout plan.
114 lines
5.1 KiB
Markdown
114 lines
5.1 KiB
Markdown
# Fleet device secrets
|
|
|
|
How fleet devices read their deployments' secrets — image-pull
|
|
credentials, application config, API keys — from OpenBao, and how you
|
|
control who reads what. Design reasoning lives in
|
|
[ADR-025](../adr/025-fleet-device-secret-access.md); this page is the
|
|
operating manual.
|
|
|
|
## The architecture in one minute
|
|
|
|
A device proves *who it is* with the Zitadel machine key it already
|
|
uses for NATS. *What it may read* is decided by **group membership**,
|
|
which only admins can change. *Where it runs* is decided by labels.
|
|
The three never mix:
|
|
|
|
```
|
|
Admin Zitadel OpenBao
|
|
│ assign device to groups ──▶ device ∈ {edge-a, …} │
|
|
│ write secret ──────────────────────────────────────────────▶ secret/<fleet>/<deployment>/…
|
|
│ │
|
|
Deployment CR │
|
|
│ allowedGroups: [edge-a] ──▶ operator attaches policy ──────▶ group edge-a may read
|
|
│ targetSelector: labels ──▶ operator schedules within │ deployment-<name> subtree
|
|
│ authorized devices │
|
|
│ │
|
|
Device │
|
|
└─ machine key ─▶ Zitadel token (groups: [edge-a]) ─▶ JWT login ─▶ reads its secrets
|
|
```
|
|
|
|
Three rules to remember:
|
|
|
|
| Concept | Who sets it | What it controls |
|
|
|---|---|---|
|
|
| **Group** | Admin, in Zitadel | Security boundary: which deployments' secrets a device may read |
|
|
| **Label** | Device config (self-reported) | Placement only: which authorized devices actually run the workload — narrows, never widens |
|
|
| **`allowedGroups`** | Deployment author | Which groups may view (run + read secrets of) the deployment |
|
|
|
|
## Storing a secret for a deployment
|
|
|
|
Write it under the deployment's subtree of the fleet KV mount:
|
|
|
|
```bash
|
|
bao kv put secret/<fleet-ns>/<deployment-name>/db-credentials \
|
|
username=app password=…
|
|
```
|
|
|
|
Nothing else to configure — access follows from the deployment's
|
|
`allowedGroups` below. Every device in an allowed group can read the
|
|
whole `<deployment-name>/` subtree, whether or not the workload landed
|
|
on it; size your groups to the audience the secret may have.
|
|
|
|
## Authorizing and placing a deployment
|
|
|
|
```yaml
|
|
apiVersion: fleet.nationtech.io/v1alpha1
|
|
kind: Deployment
|
|
metadata:
|
|
name: hello-web
|
|
spec:
|
|
allowedGroups: [edge-a] # security: who may view this deployment
|
|
targetSelector:
|
|
matchLabels:
|
|
hw: pi5 # placement: which authorized devices run it
|
|
score: …
|
|
```
|
|
|
|
The operator schedules onto devices that are **members of an allowed
|
|
group AND match the selector**, then attaches the deployment's read
|
|
policy to each allowed group. Existing device sessions pick the grant
|
|
up immediately — rolling out a new deployment never touches logins,
|
|
tokens, or per-device configuration.
|
|
|
|
## Managing device group membership
|
|
|
|
Until Zitadel's first-class groups are GA, a fleet group is a Zitadel
|
|
project role (one role per group, surfaced as a `groups` claim by the
|
|
fleet's token Action — applied by the fleet bootstrap, not by hand).
|
|
Grant or remove the role on the device's machine user to move it
|
|
between groups.
|
|
|
|
Membership changes take effect at the device's **next login, at most
|
|
one token TTL later (≈ 1 h)**. This is the only eventually-consistent
|
|
operation in the system — everything else binds immediately. If that
|
|
window matters, use the emergency levers below.
|
|
|
|
## Reading secrets on the device
|
|
|
|
Nothing to do: the agent reads through `harmony_config`, which
|
|
authenticates to OpenBao with the machine keyfile automatically
|
|
(JWT-bearer rung of `OpenbaoSecretStore`). Tokens are short-lived
|
|
batch tokens; the store re-logins transparently on expiry.
|
|
|
|
## Revocation runbook
|
|
|
|
| Situation | Action | Effect |
|
|
|---|---|---|
|
|
| Device leaves a group (routine) | Remove the role/group in Zitadel | Converges at next login, ≤ 1 TTL |
|
|
| Device compromised | Disable its OpenBao identity entity, then rotate/delete its Zitadel machine key | Token dead at next request; no new logins |
|
|
| Secret/deployment compromised | Detach `deployment-<name>` policy from its groups (or rotate the secret) | All devices lose read immediately |
|
|
| Whole group compromised | Detach the group's policies | Immediate, group-wide |
|
|
|
|
## Guarantees and limits
|
|
|
|
- A device can only ever read secrets of deployments whose
|
|
`allowedGroups` intersect its groups. Labels, CR edits, or a
|
|
compromised agent cannot widen that — membership is signed into the
|
|
Zitadel token and validated against the project (`bound_audiences`),
|
|
so other projects' tokens are rejected outright.
|
|
- Visibility is **group-granular**: being scheduled is not required to
|
|
read. Finer isolation ⇒ finer groups.
|
|
- One Zitadel project + OpenBao pair (a *cell*) comfortably serves
|
|
~10⁵ devices; beyond that, shard fleets into cells
|
|
([ADR-025](../adr/025-fleet-device-secret-access.md), Rationale).
|