Fleet tasks

Summary

Fleet tasks are finite jobs selected with the same group and label rules as Fleet deployments. A task freezes its targets, runs once on each device, and retains a terminal result.

The first release supports directly created, multi-device TaskRuns and the built-in system upgrade. Recurring schedules come after the direct path is proven on Debian and Raspberry Pi OS.

`TaskRun`

apiVersion: fleet.nationtech.io/v1alpha1
kind: TaskRun
metadata:
  name: upgrade-device-1
spec:
  allowedGroups: [production]
  targetSelector:
    matchLabels:
      device-id: device-1
  deadlineSeconds: 21600
  systemUpgradeV1: {}

systemUpgradeV1 is empty because apt, dpkg, repository, and reboot policy is compiled into the updater. The API cannot pass package names, repository paths, commands, or reboot arguments.

Admission rejects changes to the spec. The run starts immediately and has one phase:

Planning | Running | Complete | Failed

During Planning, the operator freezes every matching device that supports AptFullUpgradeV1. No match fails with NoTargets.

Status records target, success, and failure counts; start and completion times; a reason; and the latest bounded error. The CR never contains an unbounded per-device map.

Placement

Task placement uses the deployment aggregator's rule:

allowed group membership AND targetSelector

allowedGroups requires at least one group. matchLabels is a conjunction, and an empty selector matches every authorized device. matchExpressions fails closed until implemented.

The operator checks group authorization again before creating the intent. A device revoked before release fails the run with TargetRevoked; another device is never substituted into the frozen plan. A group-source error retries planning instead of becoming NoTargets.

The target plan is created atomically in JetStream before the run enters Running. Concurrent operator instances therefore load the same frozen plan. The run UID is also the attempt ID, so intent creation and updater execution are idempotent across restarts.

Execution

The operator writes the fixed system-upgrade attempt to:

system-upgrade-intent: <device-id>.<run-uid>

The agent publishes updater-owned status to:

system-upgrade-status: <device-id>.<run-uid>

The agent watches current and new intents, so reboot does not lose the active run. The operator accepts terminal status only when its attempt and run IDs match. For success, the agent records the first heartbeat after updater completion in the durable run status; that heartbeat must precede the deadline. A late result cannot reopen a failed run.

Intent and status buckets are file-backed, keep one value per key, and have byte and age limits. The operator removes a terminal intent; terminal status remains for bounded diagnostics.

The deadline is copied into the attempt as expiresAt. Expiry prevents a delayed intent from starting. It does not interrupt apt or dpkg after the updater has accepted the attempt.

Canary release

If any matched device has the canary=true label, the operator releases only those devices first. Every canary must complete successfully before the operator releases the remaining devices. A failed, missing, or timed-out canary stops the rollout. Completed devices are not rolled back.

When no matched device has canary=true, the operator releases the full frozen target set immediately. There is no percentage, batch-size, or concurrency setting.

The frozen target set must remain bounded outside the CR, while the CR retains aggregate counts and the latest error.

Recurring schedules

A later namespaced TaskSchedule creates immutable TaskRun resources. Daily, weekly, yearly, and five-field cron forms use UTC. Time-zone configuration is not supported.

One schedule has at most one active run. Ticks during an active run are skipped, and restart creates at most one catch-up run for the latest eligible tick. Suspension prevents new runs without stopping an active run.

Other workloads

Container and agent-upgrade tasks remain deferred. Add a workload discriminator only when a second workload is implemented. Agent upgrades continue to use their existing activation, health-check, and rollback state machine.

Invariants

A run UID identifies one immutable target plan and operation.
Canary and non-canary membership is frozen with the target plan.
Group authorization is current when an intent is released.
A device executes one run attempt at most once.
A terminal run never advances.
Restart does not repeat a completed device attempt.
No Kubernetes status map grows with fleet size.

Remaining work

Prove package upgrade, reboot, restart recovery, expiry, and duplicate intent handling on a disposable Debian VM and Raspberry Pi OS device.
Add UTC schedules.
Add dashboard run history and alerts.

4.8 KiB Raw Permalink Blame History