Harmony Fleet

Harmony Fleet deploys Podman workloads and runs bounded maintenance operations on Linux devices outside a Kubernetes cluster. A Kubernetes operator holds the desired state, while an agent on each device reconciles its local Podman runtime. This fits small edge installations where devices may disconnect and reconnect.

Fleet is not a general Kubernetes distribution. The current workload contract is PodmanV0, and the current maintenance task is an apt full upgrade for Debian and Raspberry Pi OS.

Data flow

Deployment and TaskRun CRs
          |
          v
Kubernetes operator -----> NATS JetStream KV -----> device agent -----> Podman
          ^                         ^                       |
          |                         |                       v
          +------ CR status <-------+------------- device status

The operator watches namespaced Deployment, Device, and TaskRun resources. It resolves targets, writes per-device intent to NATS, and aggregates reported state. Agents do not receive Kubernetes credentials and do not call the Kubernetes API.

Zitadel authenticates operators and devices. A device keeps a Zitadel machine key and exchanges a signed assertion for an OIDC id_token. The NATS auth callout validates that token and grants device-scoped subjects. OpenBao can use the same verifiable id_token for its JWT login. Zitadel access tokens are opaque by default and are not used as the production NATS or OpenBao bearer.

Trust boundaries

Kubernetes and the Fleet operator are the control plane. An operator compromise can change workload and maintenance intent.
NATS carries desired and reported state. The auth callout restricts a device identity to its subjects, but NATS remains in the control-plane trust domain.
The unprivileged agent controls workloads owned by the fleet-agent account.
A root updater exposes fixed agent-upgrade and system-upgrade operations over a Unix socket. It does not accept arbitrary commands, package names, or paths.
Device labels are self-reported placement data, not authorization data.
A copied machine key permits impersonation of that device until the key is revoked. Hardware-backed key storage is not implemented.

Current production composition

The production deploy composition installs PostgreSQL, Zitadel, NATS, the auth callout, OpenBao, the operator, and the dashboard. It does not currently pass a Zitadel group source or OpenBao administrator token to the operator. As a result, allowedGroups placement fails closed and deployment-to-group OpenBao grant synchronization is disabled unless a private deploy composition supplies the corresponding operator configuration.

Enrollment installs a machine key on the device only when the enrollment operator provides or mints one. It does not automatically make application secrets available. OpenBao endpoint configuration, group placement, policies, and secret data are separate setup steps.

Dashboard

Production exposes the operator's web dashboard through Zitadel SSO and requires the fleet-admin role. The live service reads Device and Deployment CRs and can:

show current device liveness and deployment aggregate state;
list device inventory and labels;
derive alerts for stale devices and failing deployments;
quarantine a device by adding the blacklist label;
send an agent command from a device page.

Alert acknowledgements are held in operator memory and disappear after restart. The displayed 24-hour trend is currently static presentation, not historical telemetry. The dashboard does not create deployments, show TaskRun history, or provide rollout controls.

For local UI work, serve-web --mock uses generated data and proves only the web surface, not Fleet connectivity.

Guide

Enroll a device.
Deploy Podman workloads.
Run tasks and upgrades.
Deploy and operate the control plane.
Use the Zitadel FAQ and manual token mint when diagnosing authentication.

Limits

The production operator is a single replica. Active-active reconciliation and tested control-plane failover are not provided.
There is no published or validated fleet-size target. Some controllers still poll complete device sets.
Deployment rollout supports one mode: immediate release, with an optional canary=true gate. There are no percentages, batches, approval steps, or workload rollback.
System upgrades are direct TaskRun resources. Recurring schedules are not implemented.
Agent upgrades can restore the previous agent executable if activation fails. Workload rollback and operating-system package rollback are not implemented.
A device is trusted to report its own labels, inventory, workload status, and task status.

4.9 KiB Raw Permalink Blame History