harmony/fleet/agent-reconciliation-and-upgrade-plan.md

# Fleet agent reconciliation and upgrade plan

## Decisions

- One agent instance owns workload reconciliation for a device at a time.
- NATS desired state is authoritative. Podman labels are the durable observed
  state. The agent rebuilds its in-memory state from both after every restart.
- Desired entries carry their JetStream revision. Older watch or snapshot data
  never replaces a newer revision.
- KV watches accelerate convergence. A periodic full snapshot repairs missed
  events and drives orphan cleanup.
- One worker serializes runtime mutations. New events replace older intent
  before the next reconciliation pass.
- A deployment upgrade follows Kubernetes replacement semantics: validate and
  pull first, send the old container its normal stop signal, wait up to the
  agent's fixed 30-second termination grace period, force it only after that
  deadline, remove it, then start the replacement. A score-level override is
  deferred until a real deployment needs one.
- An existing container owned by another deployment or by a human is a
  conflict. The agent reports it and does not delete the container.
- Agent upgrade never runs two workload reconcilers. The candidate runs only a
  non-workload-mutating probe. A root-owned updater then switches the permanent
  systemd service and rolls back if the new agent does not become ready.

## Deployment reconciliation

### Inputs and reconstruction

The reconciler owns one map from `DeploymentName` to the latest desired value.
A value is a valid unresolved score or an invalid payload with its error. At
startup and every periodic resync it replaces that map from the current
`desired-state.<device>.>` KV snapshot. Live puts and deletes update the same
map and wake the worker. A failed or incomplete snapshot cannot authorize
deletion. Because KV key enumeration and value reads are not atomic, a managed
deployment must be absent from two consecutive complete snapshots before it is
removed as an orphan. Any newer watch revision resets that absence count.
An explicit revisioned delete schedules removal immediately; the two-snapshot
rule applies only when reconstructing an absent key.

No local deployment database is needed. After a reboot or power loss, current
KV values reconstruct intent and Harmony-managed Podman labels reconstruct
observed state.

### Reconciliation pass

For one complete desired snapshot:

1. Validate every deployment. A duplicate service name or explicit host port
   fails the conflicting deployments but does not block unrelated deployments.
2. Resolve secrets for the current score revision.
3. Ask the runtime to converge each valid deployment.
4. List Harmony-managed runtime deployments and remove deployments absent from
   two complete snapshots. Cleanup runs before retrying a desired deployment
   that needs the same service name.
5. Publish coarse `Phase` plus bounded `lastError`. Only an acknowledged phase
   and error pair is deduplicated; failed status writes remain dirty.

The runtime convergence operation preflights the full score before stopping
anything. It validates ports and service names, pulls missing images, and checks
container ownership. Preflight prevents known destructive failures; it cannot
make Podman operations transactional. For each changed service the runtime
performs stop, remove, create, start, and inspect in that order. Successfully
changed services remain changed if a later service fails, and the next pass
resumes from observed labels. Unchanged running services are untouched. Missing
or stopped desired services are recreated or restarted. Services no longer in
the score are removed with the same graceful-stop behavior.

A malformed current payload remains present for orphan accounting, reports
`Failed`, and causes no workload mutation. The last valid workload is preserved
but no longer reported healthy until the payload is corrected or deleted.
If a newer revision arrives during a runtime call, that call may finish but its
result is not published. The worker immediately reconciles the latest revision;
it never interrupts one Podman operation halfway.

### Edge cases

The first implementation and its tests cover:

- new deployment;
- unchanged deployment as a no-op;
- upgrade with stop-before-replace ordering;
- preflight failure without stopping the old version;
- exited desired container reported as failed with its exit details; Podman
  restart count is included for diagnosis but no time-window crashloop
  classifier is introduced;
- online deletion;
- offline deletion followed by orphan cleanup when the device returns;
- agent restart and host reboot;
- power loss before stop, after stop, after remove, after create, and after
  start, each repaired by the next full pass;
- service-name conflict between deployments;
- conflict with an unmanaged or differently owned container;
- desired container manually stopped or deleted;
- malformed desired payload replacing, rather than preserving, old intent;
- partial failure in a multi-service deployment;
- Podman unavailability and retry;
- failed state publication and retry;
- watch interruption or missed events repaired by full resync;
- failed and incomplete full snapshots;
- puts and deletes before, during, and after snapshot enumeration;
- invalid desired keys and failed state deletion;
- a newer desired revision arriving during reconciliation.

Secret values remain cached for one desired revision. Secret-only rotation is
out of scope until the secret source exposes a revision or watch contract.

## Agent upgrade

The signed authorization and probation design was replaced by stop-switch-start.
The maintained protocol is
[`docs/design/fleet-agent-upgrades.md`](../docs/design/fleet-agent-upgrades.md);
the original implementation plan remains as a link-preserving pointer in
[`agent-upgrade-stop-switch-start-plan.md`](agent-upgrade-stop-switch-start-plan.md).

Deferred follow-up:

- retry and classify activation readiness failures before rollback or quarantine;
- add an isolated VM agent-upgrade E2E covering success and failure recovery;
- rerun `FleetDeviceSetupScore` to refresh the non-self-updating bootstrap updater,
  then verify an anonymous pull from a digest-pinned OCI artifact and a
  target/binary version mismatch. The mismatch must fail during candidate
  self-test without stopping the active agent; updater child processes must not
  emit stray systemd notifications or expected broken-pipe warnings;
- replace the operator's two-second full-device scan before fleet scale;
- add binary garbage collection when disk pressure makes retention relevant.

## Device commands

The dashboard sends bounded one-shot exec requests over Core NATS request/reply.
Exec is an administrative capability, not a sandbox: it runs as the
`fleet-agent` Unix account and can access that account's credentials and rootless
Podman runtime. Only a verified dashboard session with the exact `fleet-admin`
project role may use it.
Manual QA must cover stdout, stderr and non-zero exit status, timeout cleanup,
output truncation, an offline device, and denial for a dashboard user without the
`fleet-admin` role.

## Delivery checkpoints

1. Rework deployment reconciliation around a complete snapshot and a fakeable
   runtime boundary. Add the full unit matrix and retain the existing VM E2E.
2. Update Podman convergence for ownership checks, preflight, graceful
   stop-before-replace, inspection, and orphan inventory.
3. Amend ADR-022 and add attempt-scoped upgrade contracts.
4. Add the updater, probe mode, exclusive agent lock, durable transaction, and
   operator coordination.
5. Run unit tests, compile environment-gated E2E, run focused local VM tests if
   available without production credentials, and complete independent reviews.
6. Production QA is manual and stepwise: deployment recovery, bounded device
   commands, bootstrap-updater diagnostics, then one agent upgrade attempt with
   rollback rehearsed before any canary rollout.