harmony/fleet/agent-reconciliation-and-upgrade-plan.md

Fleet agent reconciliation and upgrade plan

Decisions

• One agent instance owns workload reconciliation for a device at a time.
• NATS desired state is authoritative. Podman labels are the durable observed state. The agent rebuilds its in-memory state from both after every restart.
• Desired entries carry their JetStream revision. Older watch or snapshot data never replaces a newer revision.
• KV watches accelerate convergence. A periodic full snapshot repairs missed events and drives orphan cleanup.
• One worker serializes runtime mutations. New events replace older intent before the next reconciliation pass.
• A deployment upgrade follows Kubernetes replacement semantics: validate and pull first, send the old container its normal stop signal, wait up to the agent's fixed 30-second termination grace period, force it only after that deadline, remove it, then start the replacement. A score-level override is deferred until a real deployment needs one.
• An existing container owned by another deployment or by a human is a conflict. The agent reports it and does not delete the container.
• Agent upgrade never runs two workload reconcilers. The candidate runs only a non-workload-mutating probe. A root-owned updater then switches the permanent systemd service and rolls back if the new agent does not become ready.

Deployment reconciliation

Inputs and reconstruction

The reconciler owns one map from DeploymentName to the latest desired value. A value is a valid unresolved score or an invalid payload with its error. At startup and every periodic resync it replaces that map from the current desired-state.<device>.> KV snapshot. Live puts and deletes update the same map and wake the worker. A failed or incomplete snapshot cannot authorize deletion. Because KV key enumeration and value reads are not atomic, a managed deployment must be absent from two consecutive complete snapshots before it is removed as an orphan. Any newer watch revision resets that absence count. An explicit revisioned delete schedules removal immediately; the two-snapshot rule applies only when reconstructing an absent key.

No local deployment database is needed. After a reboot or power loss, current KV values reconstruct intent and Harmony-managed Podman labels reconstruct observed state.

Reconciliation pass

For one complete desired snapshot:

1. Validate every deployment. A duplicate service name or explicit host port fails the conflicting deployments but does not block unrelated deployments.
2. Resolve secrets for the current score revision.
3. Ask the runtime to converge each valid deployment.
4. List Harmony-managed runtime deployments and remove deployments absent from two complete snapshots. Cleanup runs before retrying a desired deployment that needs the same service name.
5. Publish coarse Phase plus bounded lastError. Only an acknowledged phase and error pair is deduplicated; failed status writes remain dirty.

The runtime convergence operation preflights the full score before stopping anything. It validates ports and service names, pulls missing images, and checks container ownership. Preflight prevents known destructive failures; it cannot make Podman operations transactional. For each changed service the runtime performs stop, remove, create, start, and inspect in that order. Successfully changed services remain changed if a later service fails, and the next pass resumes from observed labels. Unchanged running services are untouched. Missing or stopped desired services are recreated or restarted. Services no longer in the score are removed with the same graceful-stop behavior.

A malformed current payload remains present for orphan accounting, reports Failed, and causes no workload mutation. The last valid workload is preserved but no longer reported healthy until the payload is corrected or deleted. If a newer revision arrives during a runtime call, that call may finish but its result is not published. The worker immediately reconciles the latest revision; it never interrupts one Podman operation halfway.

Edge cases

The first implementation and its tests cover:

• new deployment;
• unchanged deployment as a no-op;
• upgrade with stop-before-replace ordering;
• preflight failure without stopping the old version;
• exited desired container reported as failed with its exit details; Podman restart count is included for diagnosis but no time-window crashloop classifier is introduced;
• online deletion;
• offline deletion followed by orphan cleanup when the device returns;
• agent restart and host reboot;
• power loss before stop, after stop, after remove, after create, and after start, each repaired by the next full pass;
• service-name conflict between deployments;
• conflict with an unmanaged or differently owned container;
• desired container manually stopped or deleted;
• malformed desired payload replacing, rather than preserving, old intent;
• partial failure in a multi-service deployment;
• Podman unavailability and retry;
• failed state publication and retry;
• watch interruption or missed events repaired by full resync;
• failed and incomplete full snapshots;
• puts and deletes before, during, and after snapshot enumeration;
• invalid desired keys and failed state deletion;
• a newer desired revision arriving during reconciliation.

Secret values remain cached for one desired revision. Secret-only rotation is out of scope until the secret source exposes a revision or watch contract.

Agent upgrade

The signed authorization and probation design was replaced by stop-switch-start. The maintained protocol is docs/design/fleet-agent-upgrades.md; the original implementation plan remains as a link-preserving pointer in agent-upgrade-stop-switch-start-plan.md.

Deferred follow-up:

• retry and classify activation readiness failures before rollback or quarantine;
• add an isolated VM agent-upgrade E2E covering success and failure recovery;
• rerun FleetDeviceSetupScore to refresh the non-self-updating bootstrap updater, then verify an anonymous pull from a digest-pinned OCI artifact and a target/binary version mismatch. The mismatch must fail during candidate self-test without stopping the active agent; updater child processes must not emit stray systemd notifications or expected broken-pipe warnings;
• replace the operator's two-second full-device scan before fleet scale;
• add binary garbage collection when disk pressure makes retention relevant.

Device commands

The dashboard sends bounded one-shot exec requests over Core NATS request/reply. Exec is an administrative capability, not a sandbox: it runs as the fleet-agent Unix account and can access that account's credentials and rootless Podman runtime. Only a verified dashboard session with the exact fleet-admin project role may use it. Manual QA must cover stdout, stderr and non-zero exit status, timeout cleanup, output truncation, an offline device, and denial for a dashboard user without the fleet-admin role.

Delivery checkpoints

1. Rework deployment reconciliation around a complete snapshot and a fakeable runtime boundary. Add the full unit matrix and retain the existing VM E2E.
2. Update Podman convergence for ownership checks, preflight, graceful stop-before-replace, inspection, and orphan inventory.
3. Amend ADR-022 and add attempt-scoped upgrade contracts.
4. Add the updater, probe mode, exclusive agent lock, durable transaction, and operator coordination.
5. Run unit tests, compile environment-gated E2E, run focused local VM tests if available without production credentials, and complete independent reviews.
6. Production QA is manual and stepwise: deployment recovery, bounded device commands, bootstrap-updater diagnostics, then one agent upgrade attempt with rollback rehearsed before any canary rollout.