11 KiB
Fleet agent upgrades
Scope
This design updates the Harmony agent on IoT devices that reconcile Podman workloads from NATS desired state. It covers one agent binary on one Linux device. Operating-system, Podman, workload, and future MDC/OKD upgrades are outside its scope.
Ownership
| Component | Responsibility |
|---|---|
| Fleet operator | Select the desired release and CAS-write the current per-device attempt |
| Active agent | Validate and forward intent, reconcile during preparation, drain on SIGTERM, publish status |
| Root updater | Prepare and probe the candidate, stop, switch, start, commit or roll back |
| systemd | Serialize service ownership and enforce startup and shutdown limits |
The active process holds an exclusive lock, so only one agent can mutate workloads. Preparation does not pause Podman reconciliation. Running containers continue under their restart policies during agent cutover.
The updater has no NATS, OpenBao, or operator client. Its Unix socket accepts
upgrade and read-only status requests. It derives every managed path and
accepts no shell command or caller-selected destination.
Trust boundary
The operator CAS-writes AgentUpgradeAttempt to the device's
agent-upgrade-intent key. CAS orders competing writers; it does not make the KV
key append-only. After local acceptance, the agent and updater bind an attempt
UUID to the digest of its complete content.
An attempt contains the device, target version, architecture, artifact URL, byte limit, SHA-256, and UUID. It has no source version or expiry: a device returning after days or months can upgrade directly to the current target. Any required data migration belongs in the binary and must handle the versions it supports.
Artifacts are either direct HTTPS URLs or anonymous oci:// references with an
explicit tag or manifest digest. OCI artifacts must declare the Harmony agent
artifact type and contain exactly one Harmony agent binary layer. HTTPS
authenticates transport; SHA-256 binds the downloaded bytes to the NATS attempt.
There is no artifact signature, second cutover authorization, device upgrade
key, or OpenBao dependency. NationTech administrators, the operator, and NATS
share one administrative trust domain, so a separate release-signing domain is
not required today.
The updater trusts callers admitted by the fleet-agent socket group to forward
NATS intent faithfully. Compromise of that account includes control of agent
upgrades. Installed binaries still run as fleet-agent, not root.
The auth callout validates the Zitadel JWT signature through Zitadel JWKS and
checks its registered claims. It then extracts device_id, rejects characters
unsafe for NATS subjects, and interpolates the value into device-scoped data
permissions. No unverified client-supplied device ID is used.
Upgrade flow
Intent
When desired and reported versions differ, the operator writes an attempt with a new UUID. The agent rejects an attempt for another device, the wrong architecture, or an invalid UUID. Reuse of a UUID with different content is rejected. Intent has no age or previous-version gate.
The per-device intent key can later be replaced by CAS. An unchanged release whose attempt has failed is reused rather than retried automatically. A new attempt currently requires changed release metadata or direct control-plane repair.
Preparation and probe
The agent publishes preparing and sends one upgrade request. The updater
writes preparing to its journal before it:
- downloads over HTTPS with redirects disabled;
- enforces the attempt limit and 100 MiB compiled ceiling;
- verifies SHA-256;
- installs or re-verifies
/usr/lib/harmony-fleet/fleet-agent-v<version>; - runs
<candidate> --self-testasfleet-agentand requires its compiled version to match the target.
The live agent continues reconciling during these steps.
Probe mode parses the real configuration, reaches Podman when enabled, authenticates to NATS, opens required buckets, reads per-device keys, creates a filtered desired-state consumer, checks the updater socket, and consumes a complete snapshot. It starts no loops, publishes no normal status or heartbeat, and mutates no workload. Creating the ephemeral consumer is a JetStream management operation. The probe does not prove publish permissions; active startup exercises its startup writes before readiness.
NATS connection attempts are limited to 15 seconds over a 3-minute retry window. The complete probe is limited to 4 minutes.
Stop, switch, start
After probe success, the updater writes activating and runs
systemctl stop fleet-agent.service. On SIGTERM, the old agent stops admitting
new runtime mutations and waits for the current mutation. It then attempts to
publish acknowledged stopping status with reason=updater, records drain
duration, flushes NATS, notifies systemd, and exits.
This drain is bounded by TimeoutStopSec=60s. systemd may kill an agent whose
current Podman mutation does not finish in time. The stopping status is
diagnostic; the updater does not wait for it. Cutover proceeds when systemd
reports the service inactive.
The updater atomically replaces /usr/local/bin/fleet-agent, syncs its parent
directory, and starts the service. It never deliberately runs old and new
reconcilers together.
Readiness and commit
The new process sends READY=1 only after it:
- acquires the process lock and parses configuration;
- reaches Podman when enabled;
- reconnects to NATS and opens required buckets;
- publishes device information;
- reads and validates the recovered updater transaction;
- verifies its compiled version against the activation target;
- consumes a complete desired-state snapshot;
- publishes transaction status when an upgrade is active.
The local transaction need not match the current NATS intent during recovery. Its stored digest, target, and exact previous binary path are authoritative after cutover starts. Rollback restores that path and does not depend on a source version declared by the operator.
Successful systemd readiness commits immediately. A network failure after a
successful probe can therefore cause rollback. After committed, later crashes
use the normal systemd restart policy; this protocol does not roll back a
candidate that was already ready.
Recovery
The updater journal stores attempt identity, previous and target paths, bounded errors, and up to 16 timed transitions.
| Durable phase | Recovery |
|---|---|
preparing |
Restore the previous link, record failed, start the previous service; restore or start failure enters rollback-failed |
activating |
Record rolling-back, restore the previous link, stop, then start the previous service |
committed |
Keep the target |
rolling-back |
Restore the previous link and repeat stop/start |
failed |
Keep the previous target; no automatic retry |
rollback-failed |
Refuse every new attempt until direct root repair |
Before the updater reports systemd readiness, interrupted pre-commit recovery has selected the previous symlink and persisted its recovery phase. Service stop/start completion continues asynchronously after updater readiness so the agent can query local status while starting. Corrupt journal data prevents the updater socket from starting; state is not inferred from filesystem contents.
Journal and binary writes use temporary files, file sync, atomic rename, and parent-directory sync. Symlink selection uses a temporary symlink, atomic rename, and parent-directory sync.
rollback-failed has no remote clear operation. Repair requires root access to
inspect the journal, active symlink, managed binaries, and both systemd units.
The updater must not be unquarantined until the previous binary is selected and
starts successfully.
Status
Public phases are preparing, stopping, starting, complete,
rolling-back, failed, and rollback-failed. The wire contract in
harmony-reconciler-contracts/src/upgrade.rs
is authoritative for fields and serialization.
Status includes attempt and target identity, current phase, bounded detail and error, attempt and transition timing, completed drain duration, typed reason, boot and systemd invocation IDs when available, one current journald unit reference, and up to 16 transitions. Log content remains in journald.
The updater never publishes NATS status. The active agent polls the local
transaction and reflects changes, including complete, into NATS and the Device
CR.
Limits
| Limit | Value |
|---|---|
| HTTPS connect / complete request | 15 seconds / 5 minutes |
| Artifact size | 100 MiB |
| NATS connection attempt / retry window | 15 seconds / 3 minutes |
| Candidate probe | 4 minutes |
| systemd stop / updater stop bound | 60 / 65 seconds |
| systemd start / updater start bound | 4 minutes / 4 minutes 15 seconds |
| Socket request-read / status response / upgrade response | 10 seconds / 15 seconds / 20 minutes |
| Socket request or response | 1 MiB |
| Concurrent socket handlers | 32 |
| Retained transitions | 16 |
| Detail or error | 1,024 characters |
Only one mutating socket request runs at a time; status remains available
during preparation and recovery. Versioned binaries are not garbage-collected.
Devices currently have many GB available, so collection is deferred until disk
pressure makes retention an operational concern.
Rollback changes only the executable. Configuration, local databases, Podman labels, and any persistent format touched before commit must remain readable by the previous release.
Bootstrap
FleetDeviceSetupScore installs the bootstrap binary, updater and agent units,
socket ownership, state directories, and active symlink. The root updater is not
self-updated. Repairing or replacing it requires another device setup operation.
Running device setup during an active transaction is unsupported.
Devices installed before OCI support require one device setup operation before
their first oci:// upgrade. This replaces the bootstrap updater; an older
updater cannot fetch an OCI artifact even when the active agent can accept the
intent.