harmony/docs/adr/025-device-secret-access/025-3-groups-as-security-boundary.md

# ADR 025-3: Groups as the Security Boundary — Scale & Security Analysis

Initial Author: Jean-Gabriel Gill-Couture

Initial Date: 2026-06-11

## Status

Accepted — this analysis produced the design in the main
[ADR-025](../025-fleet-device-secret-access.md). Kept as the record of
the entity-vs-groups comparison, the verification work behind it, and
the state-of-the-art survey.

## Context

[025-2](./025-2-entity-by-policy.md) (entity-by-policy) shipped first
and is correct at small scale, but two concerns drove a re-design:

1. **Privilege escalation via labels.** Scheduling — and therefore
   secret grants — keyed on a label selector over *device-reported*
   labels. A compromised device relabels itself to match any
   deployment's selector and receives its secrets. Kubernetes has the
   same structural risk and closes it with the `NodeRestriction`
   admission plugin; we had no equivalent, and no operator-facing way
   to say "this deployment may only ever be seen by the lab fleet."
2. **Write amplification.** The operator rewrote one OpenBao identity
   entity per targeted device per deployment change — O(targeted
   devices) Raft-backed writes. The fleet ambition is millions of
   devices.

The proposal: split authorization from placement, as Kubernetes splits
RBAC from scheduling. Groups (admin-managed, never self-asserted) gate
who may *view* a deployment; labels only place workloads within the
authorized set. Two enforcement variants were compared:

- **A — external groups:** Zitadel group membership rides in the token
  (`groups_claim`); OpenBao maps it to external identity groups; each
  deployment's policy attaches to its allowed groups.
- **B — entity-by-policy + group gate:** keep 025-2's per-device
  entity policy lists; groups exist only in the control plane as a
  scheduling filter.

An initial review recommended B (finer granularity, no re-login,
instant server-side revocation). Verification of the underlying
facts — plugin source, documented limits, at-scale precedent —
reversed that recommendation. **A wins at the stated scale.**

## Verified facts (with sources)

1. **Token renewal does NOT re-sync external-group membership for
   JWT auth.** The plugin's renew handler only checks the role still
   exists and extends TTLs; claims are never re-read, `GroupAliases`
   never rebuilt. Membership converges **only at fresh login**.
   HashiCorp's KB wording ("login or renewal") is true for methods
   whose renew path re-queries the IdP (e.g. LDAP), not JWT.
   Source: `vault-plugin-auth-jwt/path_login.go` and OpenBao's fork
   `builtin/credential/jwt/path_login.go`.
2. **Identity policies resolve at request time.** "The evaluation of
   policies applicable to the token through its identity will happen
   at request time" (Vault/OpenBao identity concepts). Attaching or
   detaching a policy on a group is visible to every member's existing
   tokens immediately.
3. **Batch tokens carry identity.** OpenBao marshals `EntityID` into
   the encrypted batch-token blob (`vault/token_store.go`), so
   login-issued batch tokens resolve entity and external-group
   policies per request — with **zero storage writes per login**.
   Limits: no renewal, no accessor, not individually revocable.
   (Token-backend-created batch tokens have no entity — the source of
   conflicting community lore.)
4. **Entity disable is the per-device kill switch.**
   `identity/entity` `disabled=true`: "associated tokens cannot be
   used, but are not revoked" — request-time blocking, no device
   cooperation, works for batch tokens.
5. **Identity store ceiling.** Entities pack into **256 storage
   entries** (hard cap ≈ 0.5–1.25M entities at default limits,
   ~200 B/entity), the store is memory-resident (go-memdb), and each
   entity upsert rewrites its whole shard — per-write cost grows with
   fleet size. Community reports: degradation from ~50k identities;
   5.7M entities → ~110 GB RAM and >10 min failover
   (hashicorp/vault#8761). **JWT login unavoidably creates one
   entity + alias per distinct `user_claim`** — under any design.
6. **Write throughput.** Reference-architecture benchmark ≈ 720
   storage writes/s; >1k req/s is "high volume" guidance for
   integrated storage. No bulk-entity-update API exists.
7. **Zitadel.** Machine-JWT token issuance benchmarked ≈ 437/s
   (v2.66) to ≈ 852/s (v4-rc); roles claim
   (`urn:zitadel:iam:org:project:roles`) is a **map**, while
   `groups_claim` requires a **string array** → flattening Action
   needed until first-class groups (epic zitadel#9702, in flight,
   machine users included) ship a documented array claim.
8. **At-scale precedent — AWS IoT Core.** Policies attach to thing
   groups; "policies are dynamically applied to things when they
   change group membership"; Greengrass deploys secrets by targeting
   thing groups. Group-attached policy *is* the canonical hyperscale
   IoT authorization model. Azure scopes to per-device/module
   identities; no per-workload grant materialization anywhere.

## The decisive arithmetic

Let a deployment target M devices in a fleet cell.

| | B: entity-by-policy | A: groups |
|---|---|---|
| Writes per deployment change | M entity upserts; each rewrites ~1/256 of all entities → at M = 10⁵, gigabytes of Raft churn, minutes–hours of convergence, partial-failure management | 1 policy attach per allowed group, instant for all existing tokens (fact 2) |
| Writes per membership change | 1 (instant) | 1 Zitadel write; converges at next login ≤ TTL (fact 1) |
| Steady-state auth load | ~0 (long-lived tokens viable) | N/TTL logins/s — 1M devices @ 1 h ≈ 280/s, stateless with batch tokens (fact 3); Zitadel-side token reused across logins (~23 mints/s @ 12 h validity) |
| Per-device emergency cutoff | strip entity policies | disable entity (fact 4) — equivalent |
| Compromised-device blast radius | deployments scheduled on it | all deployments its groups may view — by declaration; granularity = group sizing |
| IdP machinery | none | flattening Action until Zitadel groups GA (fact 7) |

B's write cost scales with *deployment churn × fleet size*; A's with
*membership churn*. Membership is the rare, per-device event —
deployments are the frequent, fleet-wide one. The data plane already
fans out O(M) per deployment (NATS desired-state KV), but NATS KV
sustains ~10⁵ writes/s where OpenBao identity writes run ~10²–10³/s
and degrade as the store grows: per-device *instructions* scale;
per-device *grants* don't. B's apparent security edge largely
dissolved under verification: per-device hard revocation exists in A
(fact 4), routine de-grouping ≤ TTL is an operational tolerance, and
finer isolation is available by defining finer groups. What remains is
A's group-wide secret visibility — accepted explicitly, since
`allowedGroups` *is* the declared audience.

A corollary of fact 1: don't renew tokens at all. Short-TTL batch
tokens whose natural expiry forces re-login make the login itself the
membership sync — deleting the re-login-signal mechanism the first
sketch of this proposal included.

Independent of A vs B, fact 5 caps any single OpenBao at ~10⁵–10⁶
device entities: "millions of devices" means **cell sharding** (one
Zitadel project + OpenBao pair per cell) under either design. Groups
shard naturally with cells.

## State of the art surveyed

- **Vault-ecosystem fleet patterns** — AppRole + trusted orchestrator,
  TLS-cert + SCEP (Vault 1.20), SPIFFE auth (Vault 1.21, not in
  OpenBao) — all solve device *identity*, which Zitadel already
  provides; none address per-workload scoping. Nothing resembling
  "a deployment declares which device groups may read its secrets"
  exists in that ecosystem; the model is uniformly identity → policy
  → path.
- **Kubernetes `NodeAuthorizer`** — per-node grants derived from pod
  placement, the inspiration for B — materializes nothing: it is an
  in-memory request-time graph in the API server. The faithful OpenBao
  equivalent would be a custom request-time authorization plugin,
  rejected for coupling secret reads to control-plane liveness.
- **Templated policies** (`claim_mappings` → alias metadata → one
  fleet-wide policy `…/{{identity.entity.aliases.<accessor>.metadata.group}}/*`)
  achieve zero grant writes ever — AWS IoT's policy-variable trick —
  but templating cannot iterate, forcing exactly one group per device
  and group-scoped secret paths. Kept as a fallback simplification.
- **SPIFFE/SPIRE** — workload-level (not device-level) identity would
  let each deployment's process authenticate as itself, shrinking the
  blast radius below anything device-scoped and making grants
  O(deployments). Deferred: SPIRE server + per-device agents, weak
  node attestation on TPM-less hardware, and no OpenBao SPIFFE auth
  method yet. The group/policy layer chosen here survives that
  migration — only the auth mount changes.

## Resolved during review

- ~~Re-login signal on membership change~~ — unnecessary; TTL expiry
  is the sync.
- ~~Where membership lives~~ — Zitadel (roles-as-groups interim);
  Device CR fields would create a second authority.
- Vocabulary: the staging fleet's device *label* named `group` must be
  renamed or clearly distinguished — labels are placement vocabulary
  and device-writable; the authorization group is admin-only.

## Open items for implementation

1. Confirm in staging that a disabled entity blocks an in-flight
   batch token at request time (fact 4 is documented; prove it e2e).
2. TTL default 1 h — revisit with measured login load.
3. Per-cell sizing guidance once identity-store memory is measured on
   OpenBao (Vault-lineage numbers above are the planning basis).