harmony/docs/adr/025-device-secret-access/025-3-groups-as-security-boundary.md

ADR 025-3: Groups as the Security Boundary — Scale & Security Analysis

Initial Author: Jean-Gabriel Gill-Couture

Initial Date: 2026-06-11

Status

Accepted — this analysis produced the design in the main ADR-025. Kept as the record of the entity-vs-groups comparison, the verification work behind it, and the state-of-the-art survey.

Context

025-2 (entity-by-policy) shipped first and is correct at small scale, but two concerns drove a re-design:

1. Privilege escalation via labels. Scheduling — and therefore secret grants — keyed on a label selector over device-reported labels. A compromised device relabels itself to match any deployment's selector and receives its secrets. Kubernetes has the same structural risk and closes it with the NodeRestriction admission plugin; we had no equivalent, and no operator-facing way to say "this deployment may only ever be seen by the lab fleet."
2. Write amplification. The operator rewrote one OpenBao identity entity per targeted device per deployment change — O(targeted devices) Raft-backed writes. The fleet ambition is millions of devices.

The proposal: split authorization from placement, as Kubernetes splits RBAC from scheduling. Groups (admin-managed, never self-asserted) gate who may view a deployment; labels only place workloads within the authorized set. Two enforcement variants were compared:

• A — external groups: Zitadel group membership rides in the token (groups_claim); OpenBao maps it to external identity groups; each deployment's policy attaches to its allowed groups.
• B — entity-by-policy + group gate: keep 025-2's per-device entity policy lists; groups exist only in the control plane as a scheduling filter.

An initial review recommended B (finer granularity, no re-login, instant server-side revocation). Verification of the underlying facts — plugin source, documented limits, at-scale precedent — reversed that recommendation. A wins at the stated scale.

Verified facts (with sources)

1. Token renewal does NOT re-sync external-group membership for JWT auth. The plugin's renew handler only checks the role still exists and extends TTLs; claims are never re-read, GroupAliases never rebuilt. Membership converges only at fresh login. HashiCorp's KB wording ("login or renewal") is true for methods whose renew path re-queries the IdP (e.g. LDAP), not JWT. Source: vault-plugin-auth-jwt/path_login.go and OpenBao's fork builtin/credential/jwt/path_login.go.
2. Identity policies resolve at request time. "The evaluation of policies applicable to the token through its identity will happen at request time" (Vault/OpenBao identity concepts). Attaching or detaching a policy on a group is visible to every member's existing tokens immediately.
3. Batch tokens carry identity. OpenBao marshals EntityID into the encrypted batch-token blob (vault/token_store.go), so login-issued batch tokens resolve entity and external-group policies per request — with zero storage writes per login. Limits: no renewal, no accessor, not individually revocable. (Token-backend-created batch tokens have no entity — the source of conflicting community lore.)
4. Entity disable is the per-device kill switch. identity/entity disabled=true: "associated tokens cannot be used, but are not revoked" — request-time blocking, no device cooperation, works for batch tokens.
5. Identity store ceiling. Entities pack into 256 storage entries (hard cap ≈ 0.5–1.25M entities at default limits, ~200 B/entity), the store is memory-resident (go-memdb), and each entity upsert rewrites its whole shard — per-write cost grows with fleet size. Community reports: degradation from ~50k identities; 5.7M entities → ~110 GB RAM and >10 min failover (hashicorp/vault#8761). JWT login unavoidably creates one entity + alias per distinct user_claim — under any design.
6. Write throughput. Reference-architecture benchmark ≈ 720 storage writes/s; >1k req/s is "high volume" guidance for integrated storage. No bulk-entity-update API exists.
7. Zitadel. Machine-JWT token issuance benchmarked ≈ 437/s (v2.66) to ≈ 852/s (v4-rc); roles claim (urn:zitadel:iam:org:project:roles) is a map, while groups_claim requires a string array → flattening Action needed until first-class groups (epic zitadel#9702, in flight, machine users included) ship a documented array claim.
8. At-scale precedent — AWS IoT Core. Policies attach to thing groups; "policies are dynamically applied to things when they change group membership"; Greengrass deploys secrets by targeting thing groups. Group-attached policy is the canonical hyperscale IoT authorization model. Azure scopes to per-device/module identities; no per-workload grant materialization anywhere.

The decisive arithmetic

Let a deployment target M devices in a fleet cell.

	B: entity-by-policy	A: groups
Writes per deployment change	M entity upserts; each rewrites ~1/256 of all entities → at M = 10⁵, gigabytes of Raft churn, minutes–hours of convergence, partial-failure management	1 policy attach per allowed group, instant for all existing tokens (fact 2)
Writes per membership change	1 (instant)	1 Zitadel write; converges at next login ≤ TTL (fact 1)
Steady-state auth load	~0 (long-lived tokens viable)	N/TTL logins/s — 1M devices @ 1 h ≈ 280/s, stateless with batch tokens (fact 3); Zitadel-side token reused across logins (~23 mints/s @ 12 h validity)
Per-device emergency cutoff	strip entity policies	disable entity (fact 4) — equivalent
Compromised-device blast radius	deployments scheduled on it	all deployments its groups may view — by declaration; granularity = group sizing
IdP machinery	none	flattening Action until Zitadel groups GA (fact 7)

B's write cost scales with deployment churn × fleet size; A's with membership churn. Membership is the rare, per-device event — deployments are the frequent, fleet-wide one. The data plane already fans out O(M) per deployment (NATS desired-state KV), but NATS KV sustains ~10⁵ writes/s where OpenBao identity writes run ~10²–10³/s and degrade as the store grows: per-device instructions scale; per-device grants don't. B's apparent security edge largely dissolved under verification: per-device hard revocation exists in A (fact 4), routine de-grouping ≤ TTL is an operational tolerance, and finer isolation is available by defining finer groups. What remains is A's group-wide secret visibility — accepted explicitly, since allowedGroups is the declared audience.

A corollary of fact 1: don't renew tokens at all. Short-TTL batch tokens whose natural expiry forces re-login make the login itself the membership sync — deleting the re-login-signal mechanism the first sketch of this proposal included.

Independent of A vs B, fact 5 caps any single OpenBao at ~10⁵–10⁶ device entities: "millions of devices" means cell sharding (one Zitadel project + OpenBao pair per cell) under either design. Groups shard naturally with cells.

State of the art surveyed

• Vault-ecosystem fleet patterns — AppRole + trusted orchestrator, TLS-cert + SCEP (Vault 1.20), SPIFFE auth (Vault 1.21, not in OpenBao) — all solve device identity, which Zitadel already provides; none address per-workload scoping. Nothing resembling "a deployment declares which device groups may read its secrets" exists in that ecosystem; the model is uniformly identity → policy → path.
• Kubernetes NodeAuthorizer — per-node grants derived from pod placement, the inspiration for B — materializes nothing: it is an in-memory request-time graph in the API server. The faithful OpenBao equivalent would be a custom request-time authorization plugin, rejected for coupling secret reads to control-plane liveness.
• Templated policies (claim_mappings → alias metadata → one fleet-wide policy …/{{identity.entity.aliases.<accessor>.metadata.group}}/*) achieve zero grant writes ever — AWS IoT's policy-variable trick — but templating cannot iterate, forcing exactly one group per device and group-scoped secret paths. Kept as a fallback simplification.
• SPIFFE/SPIRE — workload-level (not device-level) identity would let each deployment's process authenticate as itself, shrinking the blast radius below anything device-scoped and making grants O(deployments). Deferred: SPIRE server + per-device agents, weak node attestation on TPM-less hardware, and no OpenBao SPIFFE auth method yet. The group/policy layer chosen here survives that migration — only the auth mount changes.

Resolved during review

• Re-login signal on membership change — unnecessary; TTL expiry is the sync.
• Where membership lives — Zitadel (roles-as-groups interim); Device CR fields would create a second authority.
• Vocabulary: the staging fleet's device label named group must be renamed or clearly distinguished — labels are placement vocabulary and device-writable; the authorization group is admin-only.

Open items for implementation

1. Confirm in staging that a disabled entity blocks an in-flight batch token at request time (fact 4 is documented; prove it e2e).
2. TTL default 1 h — revisit with measured login load.
3. Per-cell sizing guidance once identity-store memory is measured on OpenBao (Vault-lineage numbers above are the planning basis).