83 lines
3.4 KiB
Markdown
83 lines
3.4 KiB
Markdown
# Architecture Decision Record: Fleet Agent Upgrade Procedure
|
|
|
|
Initial Author: Jean-Gabriel Gill-Couture
|
|
|
|
Initial Date: 2026-05-06
|
|
|
|
Last Updated Date: 2026-07-22
|
|
|
|
## Status
|
|
|
|
Accepted. This revision replaces the dual-active design and the later signed
|
|
switch-authorization design.
|
|
|
|
## Context
|
|
|
|
Harmony Fleet currently manages IoT devices whose agents reconcile Podman
|
|
workloads from NATS desired state. Agent replacement must preserve one workload
|
|
owner across process or power loss.
|
|
|
|
NationTech fleet administrators, the operator, and the NATS control plane share
|
|
one administrative trust domain. Device identity comes from a Zitadel JWT whose
|
|
signature is validated through Zitadel JWKS and whose registered claims are
|
|
checked before its `device_id` is interpolated into device-scoped NATS subjects.
|
|
Separate artifact and cutover signatures are omitted because they would remain
|
|
inside this trust domain while adding another key lifecycle.
|
|
|
|
Future MDC fleets and their OKD control plane are outside this decision.
|
|
|
|
## Decision
|
|
|
|
The operator CAS-writes one per-device attempt. Once accepted, its UUID is bound
|
|
to its complete content. The live agent forwards it to a root updater and keeps
|
|
reconciling while the updater downloads, verifies, installs, and probes the
|
|
candidate.
|
|
|
|
After probe success, the updater owns one local transaction:
|
|
|
|
```text
|
|
stop old service
|
|
select candidate atomically
|
|
start candidate
|
|
commit on strict readiness, otherwise select and start the previous binary
|
|
```
|
|
|
|
The old agent attempts to drain its current mutation only after systemd sends
|
|
SIGTERM. Shutdown is bounded by systemd, so an operation that exceeds the stop
|
|
limit may be killed rather than drained.
|
|
|
|
Artifacts use HTTPS and a SHA-256 digest carried by NATS intent. The updater
|
|
trusts its `fleet-agent` socket group, derives all managed paths, accepts no
|
|
command, and executes candidates without root privileges. It has no NATS,
|
|
OpenBao, or operator connection.
|
|
|
|
There is no artifact signature, second cutover authorization, device trust key,
|
|
or extended probation.
|
|
|
|
The active service reports ready only after local dependencies, NATS, the
|
|
complete desired-state snapshot, activation target-version check, transaction
|
|
recovery, and required startup publications succeed. Strict readiness may reject
|
|
a valid candidate during a coincident network outage. Retry and failure
|
|
classification will be refined separately.
|
|
|
|
Every pre-commit activation failure biases executable selection toward the
|
|
previous binary. `rollback-failed` blocks new attempts until direct repair.
|
|
Successful readiness commits immediately. Failures after commit use systemd's
|
|
normal restart policy and do not trigger protocol rollback.
|
|
|
|
## Consequences
|
|
|
|
- One process owns workload mutation; there is a bounded interval with no agent
|
|
during cutover.
|
|
- Preparation does not delay workload reconciliation.
|
|
- Rollback changes only the executable. Persistent changes made before commit
|
|
must remain readable by the previous release.
|
|
- The first updater-capable release requires `FleetDeviceSetupScore`. The root
|
|
updater remains outside automatic upgrades.
|
|
- NationTech administration, the `fleet-agent` socket, the Zitadel identity
|
|
chain, the auth callout, and NATS are trusted control-plane components.
|
|
|
|
The [fleet agent upgrade guide](../design/fleet-agent-upgrades.md) owns the
|
|
current sequence, recovery behavior, limits, status, and repair constraints.
|
|
Wire types remain authoritative in `harmony-reconciler-contracts`.
|