NationTech/harmony
3
0
Fork 2
Code Issues 38 Pull Requests 16 Actions Packages Projects Releases 1 Wiki Activity

feat(fleet): device secret access via headless Zitadel→OpenBao JWT-bearer (consolidates #319, #316, #332) #333

Closed
stremblay wants to merge 0 commits from feat/fleet-device-secret-access into master
pull from: feat/fleet-device-secret-access
merge into: NationTech:master
Conversation 0 Commits - Files Changed -
stremblay commented 2026-06-11 18:04:07 +00:00
Owner
Copy Link

Consolidates #319 (ADR-025 docs), #316 (auth next-URL redirect) and #332 (headless Zitadel→OpenBao JWT-bearer, entity-by-policy secret access) onto current master,
resolved as one branch. Commit authorship preserved. Please close #319/#316/#332 as superseded — don't merge them.

./build/check.sh green. Net: 34 files, +2460/−629.

👀 Needs your eyes

  • @#316 author — your next-URL redirect is intact (login.rs unchanged). I dropped your config_from_env()/HARMONY_SSO_* env plumbing because master moved the operator
    to typed ConfigClient. Confirm nothing else used those vars.
  • @JG (#332/ADR-025) — implementation supersedes ADR-025's scoping: as-built is entity-by-policy (OpenBaoDeviceSecretAccess), so groups_claim / the Zitadel Action /
    cached_scope / the agent refresh-check were never built. I marked ADR-025 Accepted–revised with an Addendum — please validate the wording. Membership SoT = K8s
    Deployment CRs, not Zitadel metadata.
  • Everyone — consolidation exposed a dependency cycle (harmony_secret → harmony_zitadel_auth → harmony_config → harmony_secret). Fixed by moving the JWT minter to a
    new leaf crate harmony_zitadel_jwt (re-exported by harmony_zitadel_auth, so consumers unchanged).

Also

  • harmony_config: JWT-bearer rung now wired from env (HARMONY_ZITADEL_KEY_PATH/_KEY_JSON + HARMONY_ZITADEL_AUDIENCE + OPENBAO_JWT_ROLE/_AUTH_MOUNT; issuer reuses
    HARMONY_SSO_URL). Off unless set — existing consumers unaffected.
  • Green-fixes: dropped a removed entities field in examples/openbao (E0560) + fmt/clippy.
  • Conflict picks: kept master's typed ConfigClient config + spawn_dashboard + device_status removal; kept #316's redirect; kept #332's entity-by-policy code.

▎ Not yet run against live infra: end-to-end JWT-bearer login + entity grant on staging (checklist in the ADR-025 Addendum).

Consolidates #319 (ADR-025 docs), #316 (auth next-URL redirect) and #332 (headless Zitadel→OpenBao JWT-bearer, entity-by-policy secret access) onto current master, resolved as one branch. Commit authorship preserved. Please close #319/#316/#332 as superseded — don't merge them. ./build/check.sh green. Net: 34 files, +2460/−629. 👀 Needs your eyes - @#316 author — your next-URL redirect is intact (login.rs unchanged). I dropped your config_from_env()/HARMONY_SSO_* env plumbing because master moved the operator to typed ConfigClient. Confirm nothing else used those vars. - @JG (#332/ADR-025) — implementation supersedes ADR-025's scoping: as-built is entity-by-policy (OpenBaoDeviceSecretAccess), so groups_claim / the Zitadel Action / cached_scope / the agent refresh-check were never built. I marked ADR-025 Accepted–revised with an Addendum — please validate the wording. Membership SoT = K8s Deployment CRs, not Zitadel metadata. - Everyone — consolidation exposed a dependency cycle (harmony_secret → harmony_zitadel_auth → harmony_config → harmony_secret). Fixed by moving the JWT minter to a new leaf crate harmony_zitadel_jwt (re-exported by harmony_zitadel_auth, so consumers unchanged). Also - harmony_config: JWT-bearer rung now wired from env (HARMONY_ZITADEL_KEY_PATH/_KEY_JSON + HARMONY_ZITADEL_AUDIENCE + OPENBAO_JWT_ROLE/_AUTH_MOUNT; issuer reuses HARMONY_SSO_URL). Off unless set — existing consumers unaffected. - Green-fixes: dropped a removed entities field in examples/openbao (E0560) + fmt/clippy. - Conflict picks: kept master's typed ConfigClient config + spawn_dashboard + device_status removal; kept #316's redirect; kept #332's entity-by-policy code. ▎ Not yet run against live infra: end-to-end JWT-bearer login + entity grant on staging (checklist in the ADR-025 Addendum).
stremblay added 16 commits 2026-06-11 18:04:08 +00:00
add files for feature
Some checks failed
Run Check Script / check (pull_request) Failing after 52s
Details
aacbc509b1
update env variable names a185ccc78a
remove env file
Some checks failed
Run Check Script / check (pull_request) Failing after 51s
Details
e220f2ff10
docs: ADR on how to handle securely fleet device secrets with openbao + zitadel SSO
All checks were successful
Run Check Script / check (pull_request) Successful in 2m28s
Details
c71fde4344
docs: fix openbao sso implementation plan to reuse harmony_config instead of writing a new implementation of a secret store
All checks were successful
Run Check Script / check (pull_request) Successful in 2m19s
Details
aecb08a7e9
docs: Simplify architecture for openbao sso via harmony config
All checks were successful
Run Check Script / check (pull_request) Successful in 2m16s
Details
3d01d7482f
stand alone module for jwt
Some checks failed
Run Check Script / check (pull_request) Failing after 50s
Details
788227b8c0
add files
All checks were successful
Run Check Script / check (pull_request) Successful in 2m17s
Details
57d9685c57
Merge branch 'feat/auth-add-next-url-redirect' into feat/jwt-bearer-openbao-auth 8d7ffeee91
resolve conflict, add files ob entity policy 649cfe84bf
add new files
Some checks failed
Run Check Script / check (pull_request) Failing after 53s
Details
d453d2c6be
Merge fleet device-secret-access (JWT-bearer OpenBao auth) into master 4ff7dcc9dd 
Consolidates the #319 (docs/ADR-025) + #316 (auth next-url redirect) +
#332 (JWT-bearer OpenBao auth, entity-by-policy device secret access)
stack onto current master.

Conflict resolutions:
- harmony_zitadel_auth config: keep master's typed ConfigClient approach
  (OperatorCookieKey + ZitadelAuthConfig via Config derive); drop the
  branch's superseded config_from_env/cookie_key_from_env/*_ENV path.
  Operator already loads via ConfigClient on master.
- harmony_zitadel_auth/lib.rs: master's config exports + branch's
  jwt_bearer exports (MachineKeyFile, ZitadelJwtBearer).
- fleet-auth/lib.rs: keep master's OPERATOR_CREDENTIALS_ENV_VAR export.
- operator/main.rs: keep master's in-process spawn_dashboard + branch's
  openbao secret-access wiring and extended run() signature; drop the
  removed device_status task.
- dev.sh: master's HARMONY_CONFIG_* typed env vars.

#316's next-url-redirect feature (login.rs valid_next/RawAuthCallbackQuery)
is preserved unchanged.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
fix: break harmony_secret↔harmony_zitadel_auth dependency cycle 62471e2edf 
Consolidating the stack exposed a cycle the individual PRs didn't have:
master made harmony_zitadel_auth high-level (config.rs derives
harmony_config::Config → depends on harmony_config), while the JWT-bearer
branch made harmony_secret depend on harmony_zitadel_auth for the token
minter. Result: harmony_secret → harmony_zitadel_auth → harmony_config →
harmony_secret. Cargo refuses to build it.

The minter (jwt_bearer) is a self-contained, harmony-free leaf used by
harmony_secret + harmony-fleet-auth (and re-exported by
harmony_zitadel_auth). Move it into a new leaf crate harmony_zitadel_jwt;
harmony_zitadel_auth re-exports MachineKeyFile/ZitadelJwtBearer from it so
downstream consumers are unchanged.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
feat(harmony_config): wire OpenBao JWT-bearer rung from env e2c5dcfab4 
openbao_from_env() hardcoded zitadel_jwt_bearer/jwt_role/jwt_auth_mount to
None, so a generic ConfigClient consumer could never use the headless
Zitadel-machine → OpenBao login rung. Build a ZitadelJwtBearerConfig from
env when a machine keyfile (HARMONY_ZITADEL_KEY_PATH or _KEY_JSON) plus
HARMONY_ZITADEL_AUDIENCE are present; the issuer reuses the existing
HARMONY_SSO_URL (one source of truth for the Zitadel base). Also wire
OPENBAO_JWT_ROLE / OPENBAO_JWT_AUTH_MOUNT so the rung can complete
/auth/jwt/login. Absent any piece, the rung stays off and the auth ladder
falls through unchanged.

Re-export ZitadelJwtBearerConfig from harmony_secret so consumers can build
the options.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
fix(examples/openbao): drop removed entities field from OpenbaoSetupScore 04f210fc33 
The entity concept moved to identity.rs/OpenBaoDeviceSecretAccess; the
example still set the now-removed `entities` field (E0560).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
docs(adr-025): reconcile to as-built entity-by-policy design
All checks were successful
Run Check Script / check (pull_request) Successful in 2m22s
Details
6515590bac 
The implementation superseded ADR-025's JWT-`deployments`-claim /
`groups_claim` scoping with OpenBao entity-by-policy (device keeps one
token; membership changes rewrite the entity's policy list — no re-auth).
Mark the ADR Accepted-revised with an Addendum recording the pivot and
what was consequently dropped (groups_claim, Zitadel Action, cached_scope,
agent refresh-check). Flag the handoff work-plan as superseded with an
as-built mapping. Coordinate final ADR wording with the original author.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
johnride added 1 commit 2026-06-11 20:36:58 +00:00
chore: Never specify Co-Authored-By in commit messages
All checks were successful
Run Check Script / check (pull_request) Successful in 2m33s
Details
659b159251
johnride added 6 commits 2026-06-12 04:44:56 +00:00
docs(adr-025): groups are the security boundary — rewrite, trail, guide 397f266d31 
Rewrite ADR-025 around the group-based design: groups gate (admin-
managed identity), labels place, OpenBao enforces via external groups;
batch tokens with 1h TTL make expiry re-login the membership sync.
Iteration trail preserved under 025-device-secret-access/ (025-1 JWT
claim scoping, 025-2 entity-by-policy as built, 025-3 the scale and
security analysis with verified sources). Add the user-facing guide
and the implementation + branch-consolidation rollout plan.
feat(openbao): group-grant backend + JWT role groups/batch/static-keys ebd65b0d09 
OpenBaoDeploymentSecretGrants replaces the per-device entity-policy
backend (ADR-025): one policy deployment-<name> attached to each
allowed group's external identity group — O(groups) writes per
deployment change, request-time binding for existing tokens, group +
JWT-mount alias created on first grant so grants precede first login.

OpenbaoJwtAuth grows groups_claim, token_type (fleet roles use batch:
zero storage writes per login, natural expiry is the membership sync)
and jwt_validation_pubkeys so e2e can validate against a static test
key with no live IdP; the PEM is single-quoted for the pod-exec shell.

Chart values disable the agent injector and authDelegator: both unused
and both install cluster-scoped RBAC that outlives namespace cleanup
and breaks the next install.
feat(fleet): allowedGroups on Deployment + device-group sources + groups Action 9910b6a414 
Groups gate, labels place (ADR-025): DeploymentSpec.allowedGroups
declares which admin-managed device groups may view a deployment —
run it and read its secrets. Labels stay device-reported placement
vocabulary and can never widen access.

DeploymentSecretGrants (deployment → groups, absolute + idempotent)
replaces DeviceSecretAccess; DeviceGroupSource provides membership
(StaticDeviceGroups for dev/e2e, ZitadelDeviceGroups reading project
role grants — roles-as-groups until Zitadel groups GA). The
ZitadelSetupScore groups_claim_action provisions the Action that
flattens role keys into the groups array claim OpenBao binds against;
fleet_staging_install enables it and configures the fleet-device JWT
role (batch tokens, 1h TTL).
feat(operator): gate scheduling by allowedGroups, sync group grants ca246a49e8 
Scheduling = members(allowedGroups) ∩ label selector; absent
allowedGroups stays ungated (and secretless). Membership comes from a
DeviceGroupSource snapshot polled every 60s (Zitadel role grants in
prod via ZITADEL_URL/PAT/PROJECT_ID, FLEET_DEVICE_GROUPS static map
for dev/e2e); a membership change re-evaluates only affected devices.

Grant sync flips from device-centric (entity policy lists, O(targeted
devices) writes per deployment) to deployment-centric (allowed groups,
O(1)): synced before the NATS desired-state publish so a device is
never told to run a deployment whose secrets it cannot read. Device
events no longer touch grants at all — grants bind deployments to
groups, not devices.

Operator chart sources the OpenBao/Zitadel env from the operator
Secret, all optional. Test fixtures rename the 'group' label to
'zone': labels are placement vocabulary, the authorization group is a
distinct admin-only concept.
test(fleet-e2e): prove ADR-025 group enforcement against real OpenBao 526d2ae3c9 
openbao_groups.rs covers every row of the ADR's convergence table:
member reads / non-member denied, policy attach+detach bind existing
batch tokens at request time, group membership re-reads only at login,
and entity disabled=true kills an in-flight batch token (the ADR-025-3
open item). Self-contained: the JWT mount validates a test RSA key, the
test mints its own device tokens, and drives the real grant backend
over a pod port-forward.

Both openbao tests now run Topology::ensure_ready before raw
Score::interpret — without it the k8s client is never initialized
(stack bring-up did this; the tests didn't).
fix(fleet-e2e): k3d image import --mode direct
All checks were successful
Run Check Script / check (pull_request) Successful in 2m22s
Details
3ad2873e44 
The default tools-node mode can fail to copy the image tar (docker:
'mkdirat var/run: file exists') while still exiting 0 — the image
never lands in containerd and every operator/agent pod ErrImagePulls
at the next bring-up. Direct mode imports through each node's
containerd and actually fails when it fails.
johnride added 1 commit 2026-06-12 05:01:56 +00:00
fix(fleet-e2e): stage release binaries from CARGO_TARGET_DIR when set
All checks were successful
Run Check Script / check (pull_request) Successful in 2m23s
Details
ef693dff72 
The harness's child cargo honors CARGO_TARGET_DIR but the staging copy
read <workspace>/target/release unconditionally — image builds fail
with 'No such file or directory' on shared-target setups (e.g. git
worktrees).
johnride added 1 commit 2026-06-12 05:17:11 +00:00
fix(harmony-k8s): typed apply takes field ownership on SSA conflict
All checks were successful
Run Check Script / check (pull_request) Successful in 2m33s
Details
e35614cf22 
What harmony applies is declared in code — the source of truth. The
non-forced server-side apply 409'd forever once any other field
manager had touched the resource (e.g. the operator helm chart and
install_crds both installing the fleet CRDs); taking ownership on
conflict is the documented SSA pattern for controllers, and the
dynamic-apply path already supports it.
johnride added 1 commit 2026-06-12 05:47:41 +00:00
docs(roadmap): tick groups rollout + branch consolidation as done
All checks were successful
Run Check Script / check (pull_request) Successful in 2m19s
Details
21149bd66b
johnride added 5 commits 2026-06-12 18:37:16 +00:00
feat(fleet): device-side secret injection from OpenBao (ADR-025) 23e4709f1f 
PodmanService gains secret_env (env name + secret name under
<kv>/<prefix>/<deployment>/); the agent resolves references at apply
time through harmony_secret's OpenbaoSecretStore, reusing the NATS
path's ZitadelJwtBearer — one machine identity for both consumers.
Resolution failure surfaces as Phase::Failed naming the denied fetch.

- OpenbaoSecretStore::with_zitadel_bearer: no token-file cache; login
  is lazy and get_raw retries once through refresh_auth on permission
  denied (batch tokens expire without renewal; re-login IS the sync)
- agent [openbao] config section (url + secret_prefix; mounts/role
  default to the staging shape)
- FleetDeviceSetupScore renders the [openbao] block when configured
- resolution binds at apply, never on the periodic tick, keeping the
  secret store off the 30s hot path (fleet-scale login load is the
  ADR-025 constraint)
fix(zitadel): pin helm chart to 9.27.1 ec03cba007 
Newer charts ship a login UI that expects a newer server; with
v4.12.x the login pod crashloops on Token.Invalid.
feat(fleet-e2e): callout-mode stack — Zitadel + NATS auth callout + OpenBao 99bdc0cb38 
AuthMode::Callout now deploys the production auth fabric from the
same Scores fleet_staging_install uses: Zitadel (stable namespace,
groups-claim Action, roles-as-groups) + FleetNatsScore::callout +
NatsAuthCalloutScore + OpenBao with JWT auth bound to the live
issuer. VM devices enroll as Zitadel machine users (username =
device id) and authenticate everywhere with that one identity.

Issuer plumbing: http://sso.fleet-e2e.local:30424 resolves from the
host (k3d port mapping + Host header), in-cluster (CoreDNS rewrite to
a zitadel-e2e NodePort Service whose port equals the nodePort), and
VMs (/etc/hosts → libvirt gateway). 8080 was rejected — dev hosts
commonly occupy it.

Also: Stack::connect_admin for per-test NATS connections (the shared
client's driver dies with the bring-up test's runtime), callout image
joins the generic image builder, harness pins
HARMONY_SECRET_STORE=file + a namespace so credential-persisting
Scores work unconfigured, port-mapping preflight with an actionable
delete-and-recreate error.
test(fleet-e2e): full-stack device secret access e2e cb9a649ffa 
Two HARMONY_FLEET_VM_E2E-gated tests on a real VM agent, nothing
mocked: (1) agent connects to NATS through the callout with its
Zitadel machine identity, fetches a referenced secret from OpenBao
via the groups claim (flattening Action verified live), and the
container env carries the value (SSH podman inspect ground truth);
(2) deleting the deployment policy denies the next fetch, surfaced
as Phase::Failed naming the denied secret.
docs(roadmap): agent secret access full-stack e2e entry
All checks were successful
Run Check Script / check (pull_request) Successful in 2m33s
Details
022e4c130a
johnride closed this pull request 2026-06-19 14:47:45 +00:00
All checks were successful
Run Check Script / check (pull_request) Successful in 2m33s
Details

Pull request closed

Please reopen this pull request to perform a merge.
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
Clear labels
Architecture Decision Record
Good first issue
RFC
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
johnride jvtrudel reda stremblay
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: NationTech/harmony#333
No description provided.
Delete Branch "feat/fleet-device-secret-access"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?