143 lines
4.8 KiB
Markdown
143 lines
4.8 KiB
Markdown
# Fleet tasks
|
|
|
|
## Summary
|
|
|
|
Fleet tasks are finite jobs selected with the same group and label rules as
|
|
Fleet deployments. A task freezes its targets, runs once on each device, and
|
|
retains a terminal result.
|
|
|
|
The first release supports directly created, multi-device `TaskRun`s and the
|
|
built-in system upgrade. Recurring schedules come after the direct path is
|
|
proven on Debian and Raspberry Pi OS.
|
|
|
|
## `TaskRun`
|
|
|
|
```yaml
|
|
apiVersion: fleet.nationtech.io/v1alpha1
|
|
kind: TaskRun
|
|
metadata:
|
|
name: upgrade-device-1
|
|
spec:
|
|
allowedGroups: [production]
|
|
targetSelector:
|
|
matchLabels:
|
|
device-id: device-1
|
|
deadlineSeconds: 21600
|
|
systemUpgradeV1: {}
|
|
```
|
|
|
|
`systemUpgradeV1` is empty because apt, dpkg, repository, and reboot policy is
|
|
compiled into the updater. The API cannot pass package names, repository paths,
|
|
commands, or reboot arguments.
|
|
|
|
Admission rejects changes to the spec. The run starts immediately and has one
|
|
phase:
|
|
|
|
```text
|
|
Planning | Running | Complete | Failed
|
|
```
|
|
|
|
During `Planning`, the operator freezes every matching device that supports
|
|
`AptFullUpgradeV1`. No match fails with `NoTargets`.
|
|
|
|
Status records target, success, and failure counts; start and completion times;
|
|
a reason; and the latest bounded error. The CR never contains an unbounded
|
|
per-device map.
|
|
|
|
## Placement
|
|
|
|
Task placement uses the deployment aggregator's rule:
|
|
|
|
```text
|
|
allowed group membership AND targetSelector
|
|
```
|
|
|
|
`allowedGroups` requires at least one group. `matchLabels` is a conjunction, and
|
|
an empty selector matches every authorized device.
|
|
`matchExpressions` fails closed until implemented.
|
|
|
|
The operator checks group authorization again before creating the intent. A
|
|
device revoked before release fails the run with `TargetRevoked`; another device
|
|
is never substituted into the frozen plan. A group-source error retries
|
|
planning instead of becoming `NoTargets`.
|
|
|
|
The target plan is created atomically in JetStream before the run enters
|
|
`Running`. Concurrent operator instances therefore load the same frozen plan.
|
|
The run UID is also the attempt ID, so intent creation and updater execution are
|
|
idempotent across restarts.
|
|
|
|
## Execution
|
|
|
|
The operator writes the fixed system-upgrade attempt to:
|
|
|
|
```text
|
|
system-upgrade-intent: <device-id>.<run-uid>
|
|
```
|
|
|
|
The agent publishes updater-owned status to:
|
|
|
|
```text
|
|
system-upgrade-status: <device-id>.<run-uid>
|
|
```
|
|
|
|
The agent watches current and new intents, so reboot does not lose the active
|
|
run. The operator accepts terminal status only when its attempt and run IDs
|
|
match. For success, the agent records the first heartbeat after updater
|
|
completion in the durable run status; that heartbeat must precede the deadline.
|
|
A late result cannot reopen a failed run.
|
|
|
|
Intent and status buckets are file-backed, keep one value per key, and have byte
|
|
and age limits. The operator removes a terminal intent; terminal status remains
|
|
for bounded diagnostics.
|
|
|
|
The deadline is copied into the attempt as `expiresAt`. Expiry prevents a
|
|
delayed intent from starting. It does not interrupt apt or dpkg after the
|
|
updater has accepted the attempt.
|
|
|
|
## Canary release
|
|
|
|
If any matched device has the `canary=true` label, the operator releases only
|
|
those devices first. Every canary must complete successfully before the
|
|
operator releases the remaining devices. A failed, missing, or timed-out canary
|
|
stops the rollout. Completed devices are not rolled back.
|
|
|
|
When no matched device has `canary=true`, the operator releases the full
|
|
frozen target set immediately. There is no percentage, batch-size, or
|
|
concurrency setting.
|
|
|
|
The frozen target set must remain bounded outside the CR, while the CR retains
|
|
aggregate counts and the latest error.
|
|
|
|
## Recurring schedules
|
|
|
|
A later namespaced `TaskSchedule` creates immutable `TaskRun` resources. Daily,
|
|
weekly, yearly, and five-field cron forms use UTC. Time-zone configuration is
|
|
not supported.
|
|
|
|
One schedule has at most one active run. Ticks during an active run are skipped,
|
|
and restart creates at most one catch-up run for the latest eligible tick.
|
|
Suspension prevents new runs without stopping an active run.
|
|
|
|
## Other workloads
|
|
|
|
Container and agent-upgrade tasks remain deferred. Add a workload discriminator
|
|
only when a second workload is implemented. Agent upgrades continue to use
|
|
their existing activation, health-check, and rollback state machine.
|
|
|
|
## Invariants
|
|
|
|
- A run UID identifies one immutable target plan and operation.
|
|
- Canary and non-canary membership is frozen with the target plan.
|
|
- Group authorization is current when an intent is released.
|
|
- A device executes one run attempt at most once.
|
|
- A terminal run never advances.
|
|
- Restart does not repeat a completed device attempt.
|
|
- No Kubernetes status map grows with fleet size.
|
|
|
|
## Remaining work
|
|
|
|
1. Prove package upgrade, reboot, restart recovery, expiry, and duplicate intent
|
|
handling on a disposable Debian VM and Raspberry Pi OS device.
|
|
2. Add UTC schedules.
|
|
3. Add dashboard run history and alerts.
|