harmony/docs/design/fleet-system-upgrades.md

Fleet system-upgrade executor

Summary

System upgrade is a privileged built-in Fleet task. TaskRun owns placement, the frozen target plan, deadlines, and aggregate status as described in Fleet tasks. This document covers the device executor.

The unprivileged agent requests one fixed operation from the existing root updater over its Unix socket. The updater accepts no shell command, package name, repository, path, unit, or reboot argument.

System and agent upgrades share the updater process and mutation lock. System upgrades have a separate protocol operation, journal, and state machine because package upgrades are not atomically reversible.

The first executor runs an apt full upgrade using the device's configured, signed repositories and then requires a reboot. It supports Debian and Raspberry Pi OS. It does not configure repositories, select packages, roll back packages, cancel an active apt/dpkg process, or retry a terminal failure.

Trust boundary

The updater socket admits only the fleet-agent Unix group. The agent can ask for AptFullUpgradeV1, but cannot alter what that operation does. A compromised agent can invoke the upgrade and is treated as a compromised device; adding a second key held by the same agent would not improve that boundary.

A system-upgrade request contains only:

attemptId
runUid
deviceId
expiresAt

The updater validates the identifiers and expiry before starting. Duplicate requests with the same content return the durable result. Reusing an attempt ID with different content fails. Expiry prevents an intent delayed in NATS from starting later; it does not interrupt apt/dpkg after mutation begins.

The updater publishes protocol and AptFullUpgradeV1 capability through device observation. Planning fails if a selected device lacks that capability. The privileged updater is installed out of band by FleetDeviceSetupScore and does not self-update.

The updater has no NATS credentials. The agent relays updater status. A compromised agent can therefore falsify status, which is consistent with the device-compromise trust model.

Privileged state machine

Agent and system upgrades share one durable mutation lock. System upgrades use one journal per attempt under /var/lib/harmony-fleet-updater/system-upgrades/.

Device-reported phases are:

blocked
preflight
applying
rebooting
verifying
complete
failed
repair-required

blocked is non-terminal and covers an active agent upgrade or a bounded wait for apt/dpkg locks. repair-required forbids automatic retry.

AptFullUpgradeV1 has one compiled transaction:

1. Confirm package state is healthy and no configured apt source disables repository authentication.
2. Run apt-get update.
3. Run noninteractive apt-get full-upgrade, preserving locally modified configuration files.
4. Audit dpkg state.
5. Record rebooting and the current boot ID, then request reboot.
6. After startup, require a changed boot ID and healthy dpkg state.

Durable recovery is:

Journal state	Recovery
preflight	Repeat non-mutating checks.
applying	Audit dpkg and enter repair-required; never rerun the interrupted full upgrade.
rebooting, old boot ID	Retry the fixed reboot request up to a bound, then report failed.
rebooting, changed boot ID	Persist verifying.
verifying	Audit package state, then complete or require repair.
complete, failed, repair-required	Return the same terminal result for duplicate intent.

The updater persists applying before spawning apt and rebooting before requesting reboot. Startup fails if more than one journal appears active or a journal is corrupt. Journal writes use a temporary file, fsync, and atomic rename. repair-required remains until an operator repairs the device.

Task result

Disconnect after rebooting is expected; request timeout is not a failure signal. A device succeeds only after:

• the updater observes a boot ID different from the pre-upgrade boot ID;
• post-boot dpkg state is healthy;
• the agent reconnects and relays complete;
• the agent records a post-completion heartbeat before the run deadline.

The operator owns the run deadline. A device that never returns fails the run. Already upgraded devices are not rolled back.

Invariants

• A device runs at most one privileged mutation at a time.
• An expired attempt cannot start.
• Restart does not repeat a completed attempt.
• Corrupt or incomplete package state fails closed.
• All apt, dpkg, and reboot arguments are compiled into the updater.

Remaining work

1. Prove the path on a disposable Debian VM, then Raspberry Pi OS.
2. Test updater and agent restart, package failure, reboot failure, expiry, and duplicate delivery against real systemd, apt, and dpkg.
3. Add UTC recurring schedules after the direct run passes those tests.