harmony/docs/adr/022-fleet-agent-upgrade.md

Architecture Decision Record: Fleet Agent Upgrade Procedure

Initial Author: Jean-Gabriel Gill-Couture

Initial Date: 2026-05-06

Last Updated Date: 2026-07-22

Status

Accepted. This revision replaces the dual-active design and the later signed switch-authorization design.

Context

Harmony Fleet currently manages IoT devices whose agents reconcile Podman workloads from NATS desired state. Agent replacement must preserve one workload owner across process or power loss.

NationTech fleet administrators, the operator, and the NATS control plane share one administrative trust domain. Device identity comes from a Zitadel JWT whose signature is validated through Zitadel JWKS and whose registered claims are checked before its device_id is interpolated into device-scoped NATS subjects. Separate artifact and cutover signatures are omitted because they would remain inside this trust domain while adding another key lifecycle.

Future MDC fleets and their OKD control plane are outside this decision.

Decision

The operator CAS-writes one per-device attempt. Once accepted, its UUID is bound to its complete content. The live agent forwards it to a root updater and keeps reconciling while the updater downloads, verifies, installs, and probes the candidate.

After probe success, the updater owns one local transaction:

stop old service
select candidate atomically
start candidate
commit on strict readiness, otherwise select and start the previous binary

The old agent attempts to drain its current mutation only after systemd sends SIGTERM. Shutdown is bounded by systemd, so an operation that exceeds the stop limit may be killed rather than drained.

Artifacts use HTTPS and a SHA-256 digest carried by NATS intent. The updater trusts its fleet-agent socket group, derives all managed paths, accepts no command, and executes candidates without root privileges. It has no NATS, OpenBao, or operator connection.

There is no artifact signature, second cutover authorization, device trust key, or extended probation.

The active service reports ready only after local dependencies, NATS, the complete desired-state snapshot, activation target-version check, transaction recovery, and required startup publications succeed. Strict readiness may reject a valid candidate during a coincident network outage. Retry and failure classification will be refined separately.

Every pre-commit activation failure biases executable selection toward the previous binary. rollback-failed blocks new attempts until direct repair. Successful readiness commits immediately. Failures after commit use systemd's normal restart policy and do not trigger protocol rollback.

Consequences

• One process owns workload mutation; there is a bounded interval with no agent during cutover.
• Preparation does not delay workload reconciliation.
• Rollback changes only the executable. Persistent changes made before commit must remain readable by the previous release.
• The first updater-capable release requires FleetDeviceSetupScore. The root updater remains outside automatic upgrades.
• NationTech administration, the fleet-agent socket, the Zitadel identity chain, the auth callout, and NATS are trusted control-plane components.

The fleet agent upgrade guide owns the current sequence, recovery behavior, limits, status, and repair constraints. Wire types remain authoritative in harmony-reconciler-contracts.