Demonstrates the compose-as-source-of-truth path (ADR-026): import an
existing docker-compose.yml into typed Scores, then build/publish/deploy
with the fleet pattern (no ApplicationScore, no ArgoCD).
- compose.rs: docker-compose-types → typed ComposeApp; loud on bind
mounts / unparseable ports, warns on depends_on/command/restart.
- chart.rs: per-service Deployment+Service (unique filenames), RWX PVC
per mounted named volume, RollingUpdate/Recreate strategy. Deploy-only
knobs live here, never in compose or Harmony.toml.
- score.rs: ComposeAppScore = helm upgrade --install + public Ingress.
- publish.rs: docker build/push each built service + helm package/push.
- app/: minimal real Java(SQLite)+React(nginx) compose stack.
- 16 unit tests over import + chart generation.
Two HARMONY_FLEET_VM_E2E-gated tests on a real VM agent, nothing
mocked: (1) agent connects to NATS through the callout with its
Zitadel machine identity, fetches a referenced secret from OpenBao
via the groups claim (flattening Action verified live), and the
container env carries the value (SSH podman inspect ground truth);
(2) deleting the deployment policy denies the next fetch, surfaced
as Phase::Failed naming the denied secret.
AuthMode::Callout now deploys the production auth fabric from the
same Scores fleet_staging_install uses: Zitadel (stable namespace,
groups-claim Action, roles-as-groups) + FleetNatsScore::callout +
NatsAuthCalloutScore + OpenBao with JWT auth bound to the live
issuer. VM devices enroll as Zitadel machine users (username =
device id) and authenticate everywhere with that one identity.
Issuer plumbing: http://sso.fleet-e2e.local:30424 resolves from the
host (k3d port mapping + Host header), in-cluster (CoreDNS rewrite to
a zitadel-e2e NodePort Service whose port equals the nodePort), and
VMs (/etc/hosts → libvirt gateway). 8080 was rejected — dev hosts
commonly occupy it.
Also: Stack::connect_admin for per-test NATS connections (the shared
client's driver dies with the bring-up test's runtime), callout image
joins the generic image builder, harness pins
HARMONY_SECRET_STORE=file + a namespace so credential-persisting
Scores work unconfigured, port-mapping preflight with an actionable
delete-and-recreate error.
PodmanService gains secret_env (env name + secret name under
<kv>/<prefix>/<deployment>/); the agent resolves references at apply
time through harmony_secret's OpenbaoSecretStore, reusing the NATS
path's ZitadelJwtBearer — one machine identity for both consumers.
Resolution failure surfaces as Phase::Failed naming the denied fetch.
- OpenbaoSecretStore::with_zitadel_bearer: no token-file cache; login
is lazy and get_raw retries once through refresh_auth on permission
denied (batch tokens expire without renewal; re-login IS the sync)
- agent [openbao] config section (url + secret_prefix; mounts/role
default to the staging shape)
- FleetDeviceSetupScore renders the [openbao] block when configured
- resolution binds at apply, never on the periodic tick, keeping the
secret store off the 30s hot path (fleet-scale login load is the
ADR-025 constraint)
What harmony applies is declared in code — the source of truth. The
non-forced server-side apply 409'd forever once any other field
manager had touched the resource (e.g. the operator helm chart and
install_crds both installing the fleet CRDs); taking ownership on
conflict is the documented SSA pattern for controllers, and the
dynamic-apply path already supports it.
The harness's child cargo honors CARGO_TARGET_DIR but the staging copy
read <workspace>/target/release unconditionally — image builds fail
with 'No such file or directory' on shared-target setups (e.g. git
worktrees).
The default tools-node mode can fail to copy the image tar (docker:
'mkdirat var/run: file exists') while still exiting 0 — the image
never lands in containerd and every operator/agent pod ErrImagePulls
at the next bring-up. Direct mode imports through each node's
containerd and actually fails when it fails.
openbao_groups.rs covers every row of the ADR's convergence table:
member reads / non-member denied, policy attach+detach bind existing
batch tokens at request time, group membership re-reads only at login,
and entity disabled=true kills an in-flight batch token (the ADR-025-3
open item). Self-contained: the JWT mount validates a test RSA key, the
test mints its own device tokens, and drives the real grant backend
over a pod port-forward.
Both openbao tests now run Topology::ensure_ready before raw
Score::interpret — without it the k8s client is never initialized
(stack bring-up did this; the tests didn't).
Scheduling = members(allowedGroups) ∩ label selector; absent
allowedGroups stays ungated (and secretless). Membership comes from a
DeviceGroupSource snapshot polled every 60s (Zitadel role grants in
prod via ZITADEL_URL/PAT/PROJECT_ID, FLEET_DEVICE_GROUPS static map
for dev/e2e); a membership change re-evaluates only affected devices.
Grant sync flips from device-centric (entity policy lists, O(targeted
devices) writes per deployment) to deployment-centric (allowed groups,
O(1)): synced before the NATS desired-state publish so a device is
never told to run a deployment whose secrets it cannot read. Device
events no longer touch grants at all — grants bind deployments to
groups, not devices.
Operator chart sources the OpenBao/Zitadel env from the operator
Secret, all optional. Test fixtures rename the 'group' label to
'zone': labels are placement vocabulary, the authorization group is a
distinct admin-only concept.
Groups gate, labels place (ADR-025): DeploymentSpec.allowedGroups
declares which admin-managed device groups may view a deployment —
run it and read its secrets. Labels stay device-reported placement
vocabulary and can never widen access.
DeploymentSecretGrants (deployment → groups, absolute + idempotent)
replaces DeviceSecretAccess; DeviceGroupSource provides membership
(StaticDeviceGroups for dev/e2e, ZitadelDeviceGroups reading project
role grants — roles-as-groups until Zitadel groups GA). The
ZitadelSetupScore groups_claim_action provisions the Action that
flattens role keys into the groups array claim OpenBao binds against;
fleet_staging_install enables it and configures the fleet-device JWT
role (batch tokens, 1h TTL).
OpenBaoDeploymentSecretGrants replaces the per-device entity-policy
backend (ADR-025): one policy deployment-<name> attached to each
allowed group's external identity group — O(groups) writes per
deployment change, request-time binding for existing tokens, group +
JWT-mount alias created on first grant so grants precede first login.
OpenbaoJwtAuth grows groups_claim, token_type (fleet roles use batch:
zero storage writes per login, natural expiry is the membership sync)
and jwt_validation_pubkeys so e2e can validate against a static test
key with no live IdP; the PEM is single-quoted for the pod-exec shell.
Chart values disable the agent injector and authDelegator: both unused
and both install cluster-scoped RBAC that outlives namespace cleanup
and breaks the next install.
Rewrite ADR-025 around the group-based design: groups gate (admin-
managed identity), labels place, OpenBao enforces via external groups;
batch tokens with 1h TTL make expiry re-login the membership sync.
Iteration trail preserved under 025-device-secret-access/ (025-1 JWT
claim scoping, 025-2 entity-by-policy as built, 025-3 the scale and
security analysis with verified sources). Add the user-facing guide
and the implementation + branch-consolidation rollout plan.
The implementation superseded ADR-025's JWT-`deployments`-claim /
`groups_claim` scoping with OpenBao entity-by-policy (device keeps one
token; membership changes rewrite the entity's policy list — no re-auth).
Mark the ADR Accepted-revised with an Addendum recording the pivot and
what was consequently dropped (groups_claim, Zitadel Action, cached_scope,
agent refresh-check). Flag the handoff work-plan as superseded with an
as-built mapping. Coordinate final ADR wording with the original author.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The entity concept moved to identity.rs/OpenBaoDeviceSecretAccess; the
example still set the now-removed `entities` field (E0560).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
openbao_from_env() hardcoded zitadel_jwt_bearer/jwt_role/jwt_auth_mount to
None, so a generic ConfigClient consumer could never use the headless
Zitadel-machine → OpenBao login rung. Build a ZitadelJwtBearerConfig from
env when a machine keyfile (HARMONY_ZITADEL_KEY_PATH or _KEY_JSON) plus
HARMONY_ZITADEL_AUDIENCE are present; the issuer reuses the existing
HARMONY_SSO_URL (one source of truth for the Zitadel base). Also wire
OPENBAO_JWT_ROLE / OPENBAO_JWT_AUTH_MOUNT so the rung can complete
/auth/jwt/login. Absent any piece, the rung stays off and the auth ladder
falls through unchanged.
Re-export ZitadelJwtBearerConfig from harmony_secret so consumers can build
the options.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Consolidating the stack exposed a cycle the individual PRs didn't have:
master made harmony_zitadel_auth high-level (config.rs derives
harmony_config::Config → depends on harmony_config), while the JWT-bearer
branch made harmony_secret depend on harmony_zitadel_auth for the token
minter. Result: harmony_secret → harmony_zitadel_auth → harmony_config →
harmony_secret. Cargo refuses to build it.
The minter (jwt_bearer) is a self-contained, harmony-free leaf used by
harmony_secret + harmony-fleet-auth (and re-exported by
harmony_zitadel_auth). Move it into a new leaf crate harmony_zitadel_jwt;
harmony_zitadel_auth re-exports MachineKeyFile/ZitadelJwtBearer from it so
downstream consumers are unchanged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
