Fleet and Zitadel FAQ

This page describes the authentication path used by the public Fleet production composition.

What does Zitadel provision for Fleet?

ZitadelSetupScore creates a Fleet project, an API application for NATS, a web PKCE application for the production dashboard, the fleet-admin and device roles, and the operator machine identity. Device enrollment creates a separate machine user and JSON key for each device.

An application is an OIDC client configuration. Users belong to a Zitadel organization; project role grants associate a user with roles in the Fleet project. A project role is not a Zitadel administrator role.

How does a device authenticate?

The agent keeps the JSON machine key at /etc/fleet-agent/zitadel-key.json. On each NATS connection it:

signs a JWT assertion whose iss and sub are the machine user ID;
sends the assertion to <issuer>/oauth/v2/token with the RFC 7523 JWT-bearer grant;
requests openid, project roles, and the Fleet project audience;
takes id_token from the response and presents it as the NATS bearer token.

Production requires id_token, not access_token. Zitadel access tokens are opaque by default. The ID token is a JWT that the callout can verify through Zitadel's JWKS and that contains the audience, client_id, and role claims used by Fleet.

The assertion lasts 60 seconds. The ID token is cached in memory and replaced before its JWT exp; no refresh token is stored. The machine key remains valid until it expires or is removed in Zitadel.

How is a device ID derived?

Enrollment conventionally creates the username device-<device-id>. The callout reads the ID token's client_id claim and strips device-. It validates the result before inserting it into device-scoped NATS subjects.

What happens while a device is offline?

The machine key remains on disk. Once the network returns, a NATS connection attempt obtains a fresh ID token if the cached one is near expiry. The agent cannot mint while Zitadel is unreachable, so reconnect waits until both Zitadel and NATS are available.

Are machine keys equivalent to PATs?

Both permit impersonation if copied. A machine key signs short-lived assertions sent to Zitadel, while a PAT is itself a bearer credential. Zitadel allows more than one key per machine user, which permits installing a replacement before removing the old key. Fleet does not currently use TPM-backed key storage.

Does enrollment make OpenBao secrets available?

No. The same ID token can authenticate to an OpenBao JWT role, but enrollment must also configure the agent's OpenBao endpoint. Group membership, OpenBao policies, policy attachments, and secret values must exist separately.

The operator has optional group-source and OpenBao grant-sync support. The public production composition does not wire the required group source or OpenBao administrator token, so those behaviors are disabled there.

The dashboard uses an OIDC authorization-code flow with PKCE. It validates the returned ID token, stores it in an encrypted private cookie, and requires the fleet-admin role. Its Zitadel application is configured to include roles in the ID token. See Operator dashboard SSO.

Why is the IAM administrator PAT in a Kubernetes Secret?

ZitadelSetupScore uses the PAT for management API operations. Kubernetes Secrets are not encrypted at rest unless the cluster enables encryption at rest. Anyone who can read this Secret can act with its privileges. Moving this bootstrap credential to a stronger storage path remains production-hardening work.

Token reference

Material	Storage	Purpose
Machine JSON key	Device filesystem	Signs JWT-bearer assertions
JWT assertion	Memory and token request	Proves possession of the machine key; valid for 60 seconds
OIDC ID token	Agent memory and NATS connection	Verifiable bearer used by the NATS callout and OpenBao JWT login
Zitadel access token	Token response only	Opaque by default; not used for production Fleet bearer authentication
NATS user JWT	NATS authorization exchange	Carries the permissions issued by the auth callout

Code map

Topic	File
Fleet identity composition	`fleet/harmony-fleet-deploy/src/app.rs`
Machine-key token exchange	`harmony_zitadel_jwt/src/lib.rs`
Shared agent/operator credentials	`fleet/harmony-fleet-auth/src/credentials.rs`
Callout validation	`nats/callout/src/zitadel.rs`
Device enrollment	`fleet/harmony-fleet-deploy/src/device_setup.rs`
Manual diagnosis	Manual token mint

4.6 KiB Raw Permalink Blame History