Jean-Gabriel Gill-Couture
5e3b0522e0
Some checks failed
Run Check Script / check (pull_request) Failing after 13s
105 lines
4.9 KiB
Markdown
105 lines
4.9 KiB
Markdown
# Harmony Fleet
|
|
|
|
Harmony Fleet deploys Podman workloads and runs bounded maintenance operations
|
|
on Linux devices outside a Kubernetes cluster. A Kubernetes operator holds the
|
|
desired state, while an agent on each device reconciles its local Podman runtime.
|
|
This fits small edge installations where devices may disconnect and reconnect.
|
|
|
|
Fleet is not a general Kubernetes distribution. The current workload contract is
|
|
`PodmanV0`, and the current maintenance task is an apt full upgrade for Debian and
|
|
Raspberry Pi OS.
|
|
|
|
## Data flow
|
|
|
|
```text
|
|
Deployment and TaskRun CRs
|
|
|
|
|
v
|
|
Kubernetes operator -----> NATS JetStream KV -----> device agent -----> Podman
|
|
^ ^ |
|
|
| | v
|
|
+------ CR status <-------+------------- device status
|
|
```
|
|
|
|
The operator watches namespaced `Deployment`, `Device`, and `TaskRun` resources.
|
|
It resolves targets, writes per-device intent to NATS, and aggregates reported
|
|
state. Agents do not receive Kubernetes credentials and do not call the
|
|
Kubernetes API.
|
|
|
|
Zitadel authenticates operators and devices. A device keeps a Zitadel machine
|
|
key and exchanges a signed assertion for an OIDC `id_token`. The NATS auth
|
|
callout validates that token and grants device-scoped subjects. OpenBao can use
|
|
the same verifiable `id_token` for its JWT login. Zitadel access tokens are
|
|
opaque by default and are not used as the production NATS or OpenBao bearer.
|
|
|
|
## Trust boundaries
|
|
|
|
- Kubernetes and the Fleet operator are the control plane. An operator
|
|
compromise can change workload and maintenance intent.
|
|
- NATS carries desired and reported state. The auth callout restricts a device
|
|
identity to its subjects, but NATS remains in the control-plane trust domain.
|
|
- The unprivileged agent controls workloads owned by the `fleet-agent` account.
|
|
- A root updater exposes fixed agent-upgrade and system-upgrade operations over a
|
|
Unix socket. It does not accept arbitrary commands, package names, or paths.
|
|
- Device labels are self-reported placement data, not authorization data.
|
|
- A copied machine key permits impersonation of that device until the key is
|
|
revoked. Hardware-backed key storage is not implemented.
|
|
|
|
## Current production composition
|
|
|
|
The production deploy composition installs PostgreSQL, Zitadel, NATS, the auth
|
|
callout, OpenBao, the operator, and the dashboard. It does **not** currently pass
|
|
a Zitadel group source or OpenBao administrator token to the operator. As a
|
|
result, `allowedGroups` placement fails closed and deployment-to-group OpenBao
|
|
grant synchronization is disabled unless a private deploy composition supplies
|
|
the corresponding operator configuration.
|
|
|
|
Enrollment installs a machine key on the device only when the enrollment
|
|
operator provides or mints one. It does not automatically make application
|
|
secrets available. OpenBao endpoint configuration, group placement, policies,
|
|
and secret data are separate setup steps.
|
|
|
|
## Dashboard
|
|
|
|
Production exposes the operator's web dashboard through Zitadel SSO and requires
|
|
the `fleet-admin` role. The live service reads Device and Deployment CRs and can:
|
|
|
|
- show current device liveness and deployment aggregate state;
|
|
- list device inventory and labels;
|
|
- derive alerts for stale devices and failing deployments;
|
|
- quarantine a device by adding the blacklist label;
|
|
- send an agent command from a device page.
|
|
|
|
Alert acknowledgements are held in operator memory and disappear after restart.
|
|
The displayed 24-hour trend is currently static presentation, not historical
|
|
telemetry. The dashboard does not create deployments, show `TaskRun` history, or
|
|
provide rollout controls.
|
|
|
|
For local UI work, `serve-web --mock` uses generated data and proves only the web
|
|
surface, not Fleet connectivity.
|
|
|
|
## Guide
|
|
|
|
1. [Enroll a device](./fleet-device-secrets.md).
|
|
2. [Deploy Podman workloads](./fleet-podman-deployments.md).
|
|
3. [Run tasks and upgrades](./fleet-tasks-upgrades.md).
|
|
4. [Deploy and operate the control plane](./fleet-staging-install.md).
|
|
5. Use the [Zitadel FAQ](./fleet-zitadel-faq.md) and [manual token
|
|
mint](./fleet-manual-token-mint.md) when diagnosing authentication.
|
|
|
|
## Limits
|
|
|
|
- The production operator is a single replica. Active-active reconciliation and
|
|
tested control-plane failover are not provided.
|
|
- There is no published or validated fleet-size target. Some controllers still
|
|
poll complete device sets.
|
|
- Deployment rollout supports one mode: immediate release, with an optional
|
|
`canary=true` gate. There are no percentages, batches, approval steps, or
|
|
workload rollback.
|
|
- System upgrades are direct `TaskRun` resources. Recurring schedules are not
|
|
implemented.
|
|
- Agent upgrades can restore the previous agent executable if activation fails.
|
|
Workload rollback and operating-system package rollback are not implemented.
|
|
- A device is trusted to report its own labels, inventory, workload status, and
|
|
task status.
|