harmony/ROADMAP/fleet_platform/agent_secret_access_e2e.md

Agent secret access: full-stack integration e2e

Design: ADR-025 · user guide · prior work: groups rollout Branch: feat/fleet-device-secret-access Written: 2026-06-12

The groups rollout proved OpenBao-side enforcement against minted test JWTs. This entry closes the loop with the real device path, in one VM e2e:

• agent connects to NATS through the auth callout with its Zitadel machine identity (first time callout runs in the e2e harness),
• a deployment in desired state references a secret,
• the agent logs into OpenBao with the same Zitadel JWT (groups claim from the flattening Action — live verification of the deferred item) and injects the secret into the container,
• removing the OpenBao policy makes the next fetch fail, visibly.

The operator is deliberately absent: CR→KV plumbing and grant sync are already unit/e2e-covered; the test drives desired state via AdminKv and grants via the same OpenBaoDeploymentSecretGrants the operator composes. The axis under test is agent ↔ Zitadel ↔ callout/OpenBao.

Phase 1 — agent secret read path (test-first)

• PodmanService.secret_env: Vec<SecretEnvVar { name, secret }> (serde default, wire-compat with old payloads). Secret lives at <kv>/<prefix>/<deployment>/<secret>, field value — harmony_secret's existing convention.
• OpenbaoSecretStore: constructor taking the agent's existing ZitadelJwtBearer (one identity for NATS + OpenBao, no token cache files); get_raw retries once through refresh_auth on permission-denied (batch tokens expire, re-login IS the sync — ADR-025).
• Agent [openbao] config section (url, role, mounts, prefix); reconciler resolves secret_env → env before interpret; fetch failure → Phase::Failed with the OpenBao error (the observable the negative test asserts).
• Unit tests: wire compat, config parse, resolution + failure mapping via in-memory SecretStore.

Phase 2 — e2e stack: callout mode

AuthMode::Callout in fleet/harmony-fleet-e2e/src/stack.rs, composed from the same Scores as fleet_staging_install / fleet_auth_callout:

• Zitadel + CoreDNS rewrite (sso.fleet-e2e.local → zitadel svc) + ZitadelSetupScore: project, roles-as-groups (edge-a, edge-b), groups_claim_action: true, machine users per device (username = device id) + e2e-admin (fleet-admin role) for the host-side NATS client.
• NATS via FleetNatsScore::callout + NatsAuthCalloutScore (callout image build/sideload joins images.rs).
• OpenBao + JWT auth: oidc_discovery_url = in-cluster issuer, groups_claim=groups, batch, 1h — same shape staging uses. NodePort service so the VM reaches it via the libvirt gateway.
• k3d cluster gains port mappings 30424 (Zitadel NodePort, with a zitadel-e2e Service whose in-cluster port equals the nodePort so the issuer http://sso.fleet-e2e.local:30424 is valid from host, VM and in-cluster; 8080 was rejected — dev hosts commonly occupy it) and 30425 (OpenBao). Existing cluster without them → clear bring-up error telling the user to delete fleet-e2e.
• VM device setup: FleetDeviceAuth Zitadel machine key + hosts_entries (sso → gateway IP) + [openbao] section.
• Host-side admin NATS connect mints a Zitadel token via the Host-header trick (no /etc/hosts edits on the host).

Phase 3 — the tests (tests/vm_secret_access.rs)

Gated by HARMONY_FLEET_VM_E2E=1, shared VM stack:

• Grant + read: device granted edge-a in Zitadel; secret written with the root token; set_deployment_groups([web → [edge-a]]); desired state with secret_env → Phase::Running and SSH ground truth: podman inspect shows the env value.
• Policy removed: delete deployment-web policy in OpenBao, bump the spec to force re-apply → fetch denied → Phase::Failed, error names the denial.

Phase 4 — verification

• ./build/check.sh green
• Existing fleet e2e suite green
• New VM e2e green (x86_64; aarch64 before release)
• Push

Deferred (carried from groups rollout)

• Operator-pod env injection live test (grant sync in-cluster) — still staging-milestone scope.
• TTL tuning, per-cell sizing — unchanged.
• aarch64 VM run of the new test if the manual x86_64 run is the gate today.