229 lines
11 KiB
Markdown
229 lines
11 KiB
Markdown
# Fleet agent upgrades
|
|
|
|
## Scope
|
|
|
|
This design updates the Harmony agent on IoT devices that reconcile Podman
|
|
workloads from NATS desired state. It covers one agent binary on one Linux
|
|
device. Operating-system, Podman, workload, and future MDC/OKD upgrades are
|
|
outside its scope.
|
|
|
|
|
|
|
|
## Ownership
|
|
|
|
| Component | Responsibility |
|
|
|---|---|
|
|
| Fleet operator | Select the desired release and CAS-write the current per-device attempt |
|
|
| Active agent | Validate and forward intent, reconcile during preparation, drain on SIGTERM, publish status |
|
|
| Root updater | Prepare and probe the candidate, stop, switch, start, commit or roll back |
|
|
| systemd | Serialize service ownership and enforce startup and shutdown limits |
|
|
|
|
The active process holds an exclusive lock, so only one agent can mutate
|
|
workloads. Preparation does not pause Podman reconciliation. Running containers
|
|
continue under their restart policies during agent cutover.
|
|
|
|
The updater has no NATS, OpenBao, or operator client. Its Unix socket accepts
|
|
`upgrade` and read-only `status` requests. It derives every managed path and
|
|
accepts no shell command or caller-selected destination.
|
|
|
|
## Trust boundary
|
|
|
|
The operator CAS-writes `AgentUpgradeAttempt` to the device's
|
|
`agent-upgrade-intent` key. CAS orders competing writers; it does not make the KV
|
|
key append-only. After local acceptance, the agent and updater bind an attempt
|
|
UUID to the digest of its complete content.
|
|
|
|
An attempt contains the device, target version, architecture, artifact URL, byte
|
|
limit, SHA-256, and UUID. It has no source version or expiry: a device returning
|
|
after days or months can upgrade directly to the current target. Any required
|
|
data migration belongs in the binary and must handle the versions it supports.
|
|
|
|
Artifacts are either direct HTTPS URLs or anonymous `oci://` references with an
|
|
explicit tag or manifest digest. OCI artifacts must declare the Harmony agent
|
|
artifact type and contain exactly one Harmony agent binary layer. HTTPS
|
|
authenticates transport; SHA-256 binds the downloaded bytes to the NATS attempt.
|
|
There is no artifact signature, second cutover authorization, device upgrade
|
|
key, or OpenBao dependency. NationTech administrators, the operator, and NATS
|
|
share one administrative trust domain, so a separate release-signing domain is
|
|
not required today.
|
|
|
|
The updater trusts callers admitted by the `fleet-agent` socket group to forward
|
|
NATS intent faithfully. Compromise of that account includes control of agent
|
|
upgrades. Installed binaries still run as `fleet-agent`, not root.
|
|
|
|
The auth callout validates the Zitadel JWT signature through Zitadel JWKS and
|
|
checks its registered claims. It then extracts `device_id`, rejects characters
|
|
unsafe for NATS subjects, and interpolates the value into device-scoped data
|
|
permissions. No unverified client-supplied device ID is used.
|
|
|
|
## Upgrade flow
|
|
|
|
|
|
|
|
### Intent
|
|
|
|
When desired and reported versions differ, the operator writes an attempt with
|
|
a new UUID. The agent rejects an attempt for another device, the wrong
|
|
architecture, or an invalid UUID. Reuse of a UUID with different content is
|
|
rejected. Intent has no age or previous-version gate.
|
|
|
|
The per-device intent key can later be replaced by CAS. An unchanged release
|
|
whose attempt has failed is reused rather than retried automatically. A new
|
|
attempt currently requires changed release metadata or direct control-plane
|
|
repair.
|
|
|
|
### Preparation and probe
|
|
|
|
The agent publishes `preparing` and sends one `upgrade` request. The updater
|
|
writes `preparing` to its journal before it:
|
|
|
|
1. downloads over HTTPS with redirects disabled;
|
|
2. enforces the attempt limit and 100 MiB compiled ceiling;
|
|
3. verifies SHA-256;
|
|
4. installs or re-verifies `/usr/lib/harmony-fleet/fleet-agent-v<version>`;
|
|
5. runs `<candidate> --self-test` as `fleet-agent` and requires its compiled
|
|
version to match the target.
|
|
|
|
The live agent continues reconciling during these steps.
|
|
|
|
Probe mode parses the real configuration, reaches Podman when enabled,
|
|
authenticates to NATS, opens required buckets, reads per-device keys, creates a
|
|
filtered desired-state consumer, checks the updater socket, and consumes a
|
|
complete snapshot. It starts no loops, publishes no normal status or heartbeat,
|
|
and mutates no workload. Creating the ephemeral consumer is a JetStream
|
|
management operation. The probe does not prove publish permissions; active
|
|
startup exercises its startup writes before readiness.
|
|
|
|
NATS connection attempts are limited to 15 seconds over a 3-minute retry
|
|
window. The complete probe is limited to 4 minutes.
|
|
|
|
### Stop, switch, start
|
|
|
|
After probe success, the updater writes `activating` and runs
|
|
`systemctl stop fleet-agent.service`. On SIGTERM, the old agent stops admitting
|
|
new runtime mutations and waits for the current mutation. It then attempts to
|
|
publish acknowledged `stopping` status with `reason=updater`, records drain
|
|
duration, flushes NATS, notifies systemd, and exits.
|
|
|
|
This drain is bounded by `TimeoutStopSec=60s`. systemd may kill an agent whose
|
|
current Podman mutation does not finish in time. The `stopping` status is
|
|
diagnostic; the updater does not wait for it. Cutover proceeds when systemd
|
|
reports the service inactive.
|
|
|
|
The updater atomically replaces `/usr/local/bin/fleet-agent`, syncs its parent
|
|
directory, and starts the service. It never deliberately runs old and new
|
|
reconcilers together.
|
|
|
|
### Readiness and commit
|
|
|
|
The new process sends `READY=1` only after it:
|
|
|
|
- acquires the process lock and parses configuration;
|
|
- reaches Podman when enabled;
|
|
- reconnects to NATS and opens required buckets;
|
|
- publishes device information;
|
|
- reads and validates the recovered updater transaction;
|
|
- verifies its compiled version against the activation target;
|
|
- consumes a complete desired-state snapshot;
|
|
- publishes transaction status when an upgrade is active.
|
|
|
|
The local transaction need not match the current NATS intent during recovery.
|
|
Its stored digest, target, and exact previous binary path are authoritative after
|
|
cutover starts. Rollback restores that path and does not depend on a source
|
|
version declared by the operator.
|
|
|
|
Successful systemd readiness commits immediately. A network failure after a
|
|
successful probe can therefore cause rollback. After `committed`, later crashes
|
|
use the normal systemd restart policy; this protocol does not roll back a
|
|
candidate that was already ready.
|
|
|
|
## Recovery
|
|
|
|
|
|
|
|
The updater journal stores attempt identity, previous and target paths, bounded
|
|
errors, and up to 16 timed transitions.
|
|
|
|
| Durable phase | Recovery |
|
|
|---|---|
|
|
| `preparing` | Restore the previous link, record `failed`, start the previous service; restore or start failure enters `rollback-failed` |
|
|
| `activating` | Record `rolling-back`, restore the previous link, stop, then start the previous service |
|
|
| `committed` | Keep the target |
|
|
| `rolling-back` | Restore the previous link and repeat stop/start |
|
|
| `failed` | Keep the previous target; no automatic retry |
|
|
| `rollback-failed` | Refuse every new attempt until direct root repair |
|
|
|
|
Before the updater reports systemd readiness, interrupted pre-commit recovery
|
|
has selected the previous symlink and persisted its recovery phase. Service
|
|
stop/start completion continues asynchronously after updater readiness so the
|
|
agent can query local status while starting. Corrupt journal data prevents the
|
|
updater socket from starting; state is not inferred from filesystem contents.
|
|
|
|
Journal and binary writes use temporary files, file sync, atomic rename, and
|
|
parent-directory sync. Symlink selection uses a temporary symlink, atomic
|
|
rename, and parent-directory sync.
|
|
|
|
`rollback-failed` has no remote clear operation. Repair requires root access to
|
|
inspect the journal, active symlink, managed binaries, and both systemd units.
|
|
The updater must not be unquarantined until the previous binary is selected and
|
|
starts successfully.
|
|
|
|
## Status
|
|
|
|
Public phases are `preparing`, `stopping`, `starting`, `complete`,
|
|
`rolling-back`, `failed`, and `rollback-failed`. The wire contract in
|
|
[`harmony-reconciler-contracts/src/upgrade.rs`](https://git.nationtech.io/Nationtech/harmony/src/branch/master/harmony-reconciler-contracts/src/upgrade.rs)
|
|
is authoritative for fields and serialization.
|
|
|
|
Status includes attempt and target identity, current phase, bounded detail and
|
|
error, attempt and transition timing, completed drain duration, typed reason,
|
|
boot and systemd invocation IDs when available, one current journald unit
|
|
reference, and up to 16 transitions. Log content remains in journald.
|
|
|
|
The updater never publishes NATS status. The active agent polls the local
|
|
transaction and reflects changes, including `complete`, into NATS and the Device
|
|
CR.
|
|
|
|
## Limits
|
|
|
|
| Limit | Value |
|
|
|---|---:|
|
|
| HTTPS connect / complete request | 15 seconds / 5 minutes |
|
|
| Artifact size | 100 MiB |
|
|
| NATS connection attempt / retry window | 15 seconds / 3 minutes |
|
|
| Candidate probe | 4 minutes |
|
|
| systemd stop / updater stop bound | 60 / 65 seconds |
|
|
| systemd start / updater start bound | 4 minutes / 4 minutes 15 seconds |
|
|
| Socket request-read / status response / upgrade response | 10 seconds / 15 seconds / 20 minutes |
|
|
| Socket request or response | 1 MiB |
|
|
| Concurrent socket handlers | 32 |
|
|
| Retained transitions | 16 |
|
|
| Detail or error | 1,024 characters |
|
|
|
|
Only one mutating socket request runs at a time; `status` remains available
|
|
during preparation and recovery. Versioned binaries are not garbage-collected.
|
|
Devices currently have many GB available, so collection is deferred until disk
|
|
pressure makes retention an operational concern.
|
|
|
|
Rollback changes only the executable. Configuration, local databases, Podman
|
|
labels, and any persistent format touched before commit must remain readable by
|
|
the previous release.
|
|
|
|
## Bootstrap
|
|
|
|
`FleetDeviceSetupScore` installs the bootstrap binary, updater and agent units,
|
|
socket ownership, state directories, and active symlink. The root updater is not
|
|
self-updated. Repairing or replacing it requires another device setup operation.
|
|
Running device setup during an active transaction is unsupported.
|
|
|
|
Devices installed before OCI support require one device setup operation before
|
|
their first `oci://` upgrade. This replaces the bootstrap updater; an older
|
|
updater cannot fetch an OCI artifact even when the active agent can accept the
|
|
intent.
|
|
|
|
## Related decisions
|
|
|
|
- [ADR-016: Harmony agent and global mesh](../adr/016-Harmony-Agent-And-Global-Mesh-For-Decentralized-Workload-Management.md)
|
|
- [ADR-022: Fleet agent upgrade procedure](../adr/022-fleet-agent-upgrade.md)
|
|
- [ADR-023: Deploy architecture](../adr/023-deploy-architecture.md)
|