harmony/ROADMAP/fleet_platform/secret_access_groups_rollout.md

Secret access: groups implementation + branch consolidation

Design: ADR-025 · user guide · analysis Branch: feat/fleet-device-secret-access Written: 2026-06-11

Two efforts tracked here: (1) TDD implementation of the group-based secret access design, (2) consolidation of the six stacked chapter branches that will conflict with it.

Part 1 — implementation (test-first)

Each phase: write the test, watch it fail, implement to green.

Phase 1 — OpenBao group enforcement e2e (the security core)

New fleet/harmony-fleet-e2e/tests/openbao_groups.rs, self-contained (no Zitadel: the JWT auth mount validates against a static test RSA key via jwt_validation_pubkeys; tests mint their own JWTs). Proves every row of ADR-025's convergence/revocation table:

• JWT login with groups: ["edge-a"] claim binds the device to the edge-a external group; batch token issued
• Member device reads deployment-web secrets; non-member (groups: ["edge-b"]) is denied
• Attaching the deployment policy to a group grants existing tokens immediately (request-time)
• Detaching revokes existing tokens immediately
• Membership change binds only at re-login: old token keeps old groups, fresh login sees new groups
• disabled=true on the entity blocks an in-flight batch token (ADR-025-3 open item 1)

Phase 2 — OpenBao group-grant backend

DeviceSecretAccess (device → deployments, retired with 025-2) becomes deployment → allowed-groups:

• Trait: set_deployment_groups(&[(DeploymentName, Vec<GroupName>)]) — absolute, idempotent; empty set detaches everywhere; in-memory impl for operator tests
• OpenBao impl: upsert policy deployment-<name> (read on its subtree), upsert external group + JWT-mount alias per fleet group, set group policy lists (add to allowed, remove from formerly-allowed)
• Phase 1 e2e drives this backend end to end

Phase 3 — JWT auth role: groups, batch, TTL

OpenbaoJwtAuth (harmony/src/modules/openbao/setup.rs):

• groups_claim (default empty → flag omitted)
• token_type (default service; fleet role uses batch)
• jwt_validation_pubkeys as alternative to oidc_discovery_url (what makes Phase 1 self-contained)
• Fleet defaults: ttl=1h, no renewal reliance (ADR-025)

Phase 4 — operator: groups gate scheduling + drive grants

harmony/src/modules/fleet/operator/crd.rs + fleet/harmony-fleet-operator/src/fleet_aggregator.rs:

• allowed_groups: Vec<String> on DeploymentSpec
• Membership source trait (device → groups) with in-memory test impl; Zitadel impl reads role grants via the existing Management API client (users/grants/_search, harmony/src/modules/zitadel/setup.rs) — roles-as-groups interim per ADR-025
• Scheduling = members(allowed_groups) ∩ label selector; unit tests: out-of-group device never targeted even on label match
• On Deployment upsert/delete: sync deployment → groups to the Phase 2 backend (replaces per-device sync_secret_access)
• Rename the group label in test fixtures (placement vocab, device-writable) so it can't be confused with authorization groups
• Operator e2e: deployment with allowedGroups schedules only onto in-group devices

Phase 5 — Zitadel roles→groups Action

• Score upserting the flattening Action (roles map → groups string array claim) + flow trigger on the fleet project; idempotent; unit-tested on request shape
• Staging verification deferred to the integration milestone (needs a live Zitadel)

Phase 6 — verification + push

• ./build/check.sh green
• HARMONY_FLEET_E2E=1 suite green (incl. existing openbao_policy.rs, ping.rs, operator.rs)
• Push feat/fleet-device-secret-access

Part 2 — chapter branch consolidation

Six branches stack linearly from 97af299d (pre-secret-access): fleet-device-exec-logs ⊂ ch1-role-gate ⊂ ch2-operator-recovery ⊂ ch3-log-streaming ⊂ ch4-agent-upgrade ⊂ ch5-graceful-deploy-upgrade — except ch2's three follow-up fixes never propagated to ch3–ch5. All will conflict with this branch (credentials.rs was refactored into harmony_zitadel_jwt; fleet_aggregator.rs changes in Phase 4).

Collapse to two feature-scoped branches, rebased onto feat/fleet-device-secret-access after Part 1 lands:

New branch	Absorbs	Scope
feat/fleet-operator-recovery-and-logs	exec-logs base fixes + ch1 + ch2 (fixes folded in) + ch3	Operator reliability & observability: role gate via OIDC scope, aggregator recovery + orphan GC, log selector + live tail, agent auto-start replay
feat/fleet-upgrade-path	ch4 + ch5	Agent self-upgrade + auto-rollback (ADR-022) and graceful roll-forward with container-ID identity

• Rebase/rewrite branch A; resolve credentials.rs → harmony_zitadel_jwt conflicts; ch2 fixes squashed into the recovery commit
• Rebase branch B on top of A (it builds on ch3 state)
• Both: ./build/check.sh + fleet e2e green = working state
• Push both; old six branches left untouched for reference until the new ones are reviewed

Deferred to staging integration milestone

• Flattening Action applied + token decode shows groups array (flow/trigger ids 2/4,5 assumed from API docs — verify)
• Zitadel membership source verified against live role grants
• TTL = 1 h default revisited with measured login load
• Per-cell sizing guidance (identity-store memory measurement)
• Known gap: a failed grant sync only retries on the next Deployment change (warn-and-continue). Add a periodic resync if staging shows real transient failures.
• Operator e2e for the allowedGroups gate needs operator-pod env injection in the harness (FLEET_DEVICE_GROUPS); gate currently unit-tested, enforcement e2e-tested.