harmony/docs/guides/fleet-tasks-upgrades.md

Fleet tasks and upgrades

Fleet has two distinct upgrade paths. TaskRun invokes the fixed operating system upgrade executor. Setting Device.spec.agentUpgrade invokes the agent binary upgrade protocol.

System upgrade task

Create a namespaced TaskRun:

apiVersion: fleet.nationtech.io/v1alpha1
kind: TaskRun
metadata:
  name: upgrade-device-1
spec:
  allowedGroups: [production]
  targetSelector:
    matchLabels:
      device-id: device-1
  deadlineSeconds: 21600
  systemUpgradeV1: {}

Task placement uses the same group source as deployments. The public production composition does not configure that source, so a task has no targets until a private composition supplies group membership.

The operator freezes the matching target set. Each device must advertise the AptFullUpgradeV1 updater capability. The root updater then runs its compiled sequence: package preflight, apt-get update, noninteractive full upgrade, dpkg audit, reboot, and post-boot verification. Callers cannot supply package names, repositories, commands, or reboot arguments.

If a matched target has canary=true, only canaries are released first. Every canary must complete before the remaining frozen targets are released. A failed or timed-out canary stops further release. Completed upgrades are not rolled back.

The deadline prevents an expired intent from starting and determines when the operator fails a run. It does not interrupt apt or dpkg after the updater has accepted the attempt. An interrupted package transaction can enter repair-required; repair then requires direct device access.

The executor exists in the agent and updater, but validation on a disposable Debian VM and Raspberry Pi OS device remains listed as work in the current design. Treat it as unproven on physical production devices until that run is recorded.

Inspect progress with:

kubectl -n <namespace> get taskruns
kubectl -n <namespace> get taskrun <name> -o yaml
journalctl -u harmony-fleet-updater

Recurring schedules and dashboard task history are not implemented.

Agent upgrade

An agent upgrade target contains a version, architecture, artifact URL, maximum size, and SHA-256 digest. The operator writes a per-device attempt when desired and reported versions differ.

The root updater downloads and verifies the candidate, runs its self-test and probe, stops the old agent, switches the active symlink, and starts the new agent. Readiness commits the change. Failure before commit restores the previous agent executable; rollback-failed requires root repair.

This rollback covers only the executable. Configuration, local databases, and formats changed by the candidate must remain readable by the previous release. The root updater does not update itself. Devices installed before OCI artifact support need FleetDeviceSetupScore run once before their first oci:// upgrade.

See Fleet agent upgrades for the protocol, timeouts, journal states, and repair conditions.