129 lines
4.8 KiB
Markdown
129 lines
4.8 KiB
Markdown
# Fleet system-upgrade executor
|
|
|
|
## Summary
|
|
|
|
System upgrade is a privileged built-in Fleet task. `TaskRun` owns placement,
|
|
the frozen target plan, deadlines, and aggregate status as described in
|
|
[Fleet tasks](./fleet-tasks.md). This document covers the device executor.
|
|
|
|
The unprivileged agent requests one fixed operation from the existing root
|
|
updater over its Unix socket. The updater accepts no shell command, package
|
|
name, repository, path, unit, or reboot argument.
|
|
|
|
System and agent upgrades share the updater process and mutation lock. System
|
|
upgrades have a separate protocol operation, journal, and state machine because
|
|
package upgrades are not atomically reversible.
|
|
|
|
The first executor runs an apt full upgrade using the device's configured,
|
|
signed repositories and then requires a reboot. It supports Debian and
|
|
Raspberry Pi OS. It does not configure repositories, select packages, roll back
|
|
packages, cancel an active apt/dpkg process, or retry a terminal failure.
|
|
|
|
## Trust boundary
|
|
|
|
The updater socket admits only the `fleet-agent` Unix group. The agent can ask
|
|
for `AptFullUpgradeV1`, but cannot alter what that operation does. A compromised
|
|
agent can invoke the upgrade and is treated as a compromised device; adding a
|
|
second key held by the same agent would not improve that boundary.
|
|
|
|
A system-upgrade request contains only:
|
|
|
|
```text
|
|
attemptId
|
|
runUid
|
|
deviceId
|
|
expiresAt
|
|
```
|
|
|
|
The updater validates the identifiers and expiry before starting. Duplicate
|
|
requests with the same content return the durable result. Reusing an attempt ID
|
|
with different content fails. Expiry prevents an intent delayed in NATS from
|
|
starting later; it does not interrupt apt/dpkg after mutation begins.
|
|
|
|
The updater publishes protocol and `AptFullUpgradeV1` capability through device
|
|
observation. Planning fails if a selected device lacks that capability. The
|
|
privileged updater is installed out of band by `FleetDeviceSetupScore` and does
|
|
not self-update.
|
|
|
|
The updater has no NATS credentials. The agent relays updater status. A
|
|
compromised agent can therefore falsify status, which is consistent with the
|
|
device-compromise trust model.
|
|
|
|
## Privileged state machine
|
|
|
|
Agent and system upgrades share one durable mutation lock. System upgrades use
|
|
one journal per attempt under
|
|
`/var/lib/harmony-fleet-updater/system-upgrades/`.
|
|
|
|
Device-reported phases are:
|
|
|
|
```text
|
|
blocked
|
|
preflight
|
|
applying
|
|
rebooting
|
|
verifying
|
|
complete
|
|
failed
|
|
repair-required
|
|
```
|
|
|
|
`blocked` is non-terminal and covers an active agent upgrade or a bounded wait
|
|
for apt/dpkg locks. `repair-required` forbids automatic retry.
|
|
|
|
`AptFullUpgradeV1` has one compiled transaction:
|
|
|
|
1. Confirm package state is healthy and no configured apt source disables
|
|
repository authentication.
|
|
2. Run `apt-get update`.
|
|
3. Run noninteractive `apt-get full-upgrade`, preserving locally modified
|
|
configuration files.
|
|
4. Audit dpkg state.
|
|
5. Record `rebooting` and the current boot ID, then request reboot.
|
|
6. After startup, require a changed boot ID and healthy dpkg state.
|
|
|
|
Durable recovery is:
|
|
|
|
| Journal state | Recovery |
|
|
|---|---|
|
|
| `preflight` | Repeat non-mutating checks. |
|
|
| `applying` | Audit dpkg and enter `repair-required`; never rerun the interrupted full upgrade. |
|
|
| `rebooting`, old boot ID | Retry the fixed reboot request up to a bound, then report `failed`. |
|
|
| `rebooting`, changed boot ID | Persist `verifying`. |
|
|
| `verifying` | Audit package state, then complete or require repair. |
|
|
| `complete`, `failed`, `repair-required` | Return the same terminal result for duplicate intent. |
|
|
|
|
The updater persists `applying` before spawning apt and `rebooting` before
|
|
requesting reboot. Startup fails if more than one journal appears active or a
|
|
journal is corrupt. Journal writes use a temporary
|
|
file, fsync, and atomic rename. `repair-required` remains until an operator
|
|
repairs the device.
|
|
|
|
## Task result
|
|
|
|
Disconnect after `rebooting` is expected; request timeout is not a failure
|
|
signal. A device succeeds only after:
|
|
|
|
- the updater observes a boot ID different from the pre-upgrade boot ID;
|
|
- post-boot dpkg state is healthy;
|
|
- the agent reconnects and relays `complete`;
|
|
- the agent records a post-completion heartbeat before the run deadline.
|
|
|
|
The operator owns the run deadline. A device that never returns fails the run.
|
|
Already upgraded devices are not rolled back.
|
|
|
|
## Invariants
|
|
|
|
- A device runs at most one privileged mutation at a time.
|
|
- An expired attempt cannot start.
|
|
- Restart does not repeat a completed attempt.
|
|
- Corrupt or incomplete package state fails closed.
|
|
- All apt, dpkg, and reboot arguments are compiled into the updater.
|
|
|
|
## Remaining work
|
|
|
|
1. Prove the path on a disposable Debian VM, then Raspberry Pi OS.
|
|
2. Test updater and agent restart, package failure, reboot failure, expiry, and
|
|
duplicate delivery against real systemd, apt, and dpkg.
|
|
3. Add UTC recurring schedules after the direct run passes those tests.
|