harmony/docs/guides/fleet-device-secrets.md

Fleet enrollment and device secrets

Enrollment gives a Linux device its Fleet identity and installs the agent. It does not, by itself, authorize group placement or make application secrets available.

Enroll a device

Use the example_fleet_device_enroll binary as the current enrollment driver. It runs locally or targets a device over SSH:

cargo run -p example_fleet_device_enroll -- \
  --target ssh://pi@10.0.0.42 \
  --issuer-url https://sso.example.com \
  --audience <fleet-project-id> \
  --nats-url wss://nats.example.com \
  --admin-oidc-client-id <numeric-client-id> \
  --agent-binary ./build/fleet-agent-aarch64 \
  --device-id warehouse-042 \
  --labels site=warehouse-east,arch=aarch64

The operator must authenticate to Zitadel through the browser flow or provide HARMONY_ZITADEL_ADMIN_TOKEN. Enrollment then finds or creates the device machine user, grants the requested project role, and mints a new JSON machine key. Zitadel does not return an existing private key, so a repeated enrollment mints another key.

FleetDeviceSetupScore installs Podman, the unprivileged fleet-agent user, the root updater, systemd units, the agent binary, and its configuration. When a machine key is supplied or minted, the Score writes it to /etc/fleet-agent/zitadel-key.json with mode 0640. The device cannot obtain that long-lived key from NATS or OpenBao automatically.

The device ID must be a valid Kubernetes RFC 1123 subdomain. Labels are self-reported placement data. Do not treat labels such as group=production as authorization.

Verify the installation on the device and control plane:

systemctl status fleet-agent harmony-fleet-updater
journalctl -u fleet-agent -u harmony-fleet-updater
kubectl -n <namespace> get devices.fleet.nationtech.io

Identity and tokens

The device stores only its Zitadel machine key. On each NATS connection, the agent signs a 60-second JWT assertion and exchanges it at Zitadel's token endpoint. Production requires the returned id_token: it is a JWT that the NATS callout and OpenBao can verify against Zitadel's JWKS. Zitadel's default access_token is opaque and is not suitable for this path.

The id_token is cached in memory and renewed before expiry. No refresh token is stored. Rotating a machine key requires installing the replacement on the device and deleting the old key in Zitadel.

Group placement and OpenBao

The intended authorization rule is:

admin-managed group membership AND self-reported target labels

allowedGroups names the authorized groups; targetSelector narrows placement within those groups. An empty group list authorizes no device.

The operator supports a Zitadel-backed group source and OpenBao policy grant synchronization, but the public production composition does not configure either one today. Without an explicit group source, deployments match no devices. Without OpenBao administrator configuration, changing allowedGroups does not create or attach OpenBao policies.

OpenBao use therefore requires an operator to configure all of the following:

• the agent's OpenBao URL and secret prefix during device setup;
• Zitadel group roles and device grants;
• the operator group source;
• OpenBao JWT auth, deployment policies, and group-policy attachments;
• secret values under the configured deployment prefix.

Do not assume enrollment completed these steps. The E2E tests exercise explicit group and grant setup, not the default production composition.

Revocation

For a compromised device, remove or deactivate its Zitadel machine keys and revoke any active OpenBao identity or token. Rotate secrets the device could read. Group removal takes effect only after the device obtains a token carrying the new claims; policy detachment or secret rotation is the immediate control.

See ADR-025 for the intended group model. Treat its unwired portions as design until production composition adds and verifies the required configuration.