feat(fleet-operator): aggregator recovery signal + orphan GC + recovery e2e (Ch2) #328

Open

johnride wants to merge 3 commits from feat/fleet-ch2-operator-recovery into feat/fleet-device-exec-logs

Author	SHA1	Message	Date
Jean-Gabriel Gill-Couture	086d905586	fix(fleet): clarify recovery tests + add missing scenario 4 + dedup test Some checks failed Run Check Script / check (pull_request) Failing after 52s Details - Rewrite e2e tests with explicit SETUP/ACTION/ASSERT structure per scenario. Each test header documents devices, deployments, and expected end state. - Add scenario 4 test: device offline during restart counts as pending (2 devices, 1 deployment, only 1 reports → succeeded=1, pending=1). - Add apply_state_rejects_older_timestamp unit test (previously claimed in docs but missing). - Fix split_once → rsplit_once in parse_state_key and seed_owned_targets (device IDs with dots would silently drop entries).	2026-06-09 16:49:52 -04:00
Jean-Gabriel Gill-Couture	81b0f79f55	docs(fleet): rewrite operator recovery scenarios with diagrams ASCII diagrams per scenario replace dense paragraphs. Architecture overview shows the three watchers, FleetState, and convergence latches. Cold-start sequence and key invariants in scannable tables.	2026-06-09 16:45:35 -04:00
Jean-Gabriel Gill-Couture	56602b505c	feat(fleet-operator): aggregator recovery signal + orphan GC + recovery e2e (Ch2) All checks were successful Run Check Script / check (pull_request) Successful in 2m35s Details Operator restart + aggregator recovery (v0.3 plan Ch2). The aggregator already cold-rebuilds from NATS KV + CR watches; this makes recovery observable, closes an orphan gap, and pins each failure shape with a regression test. - OperatorLiveness: a shared in-process latch (Recovering → Converged) the aggregator sets once all three cold-start sources replay (Deployment/Device watcher InitDone, device-state KV seen_current; empty-bucket short-circuit). The in-process dashboard reads it and shows a self-clearing banner via an HTMX self-poll (/__recovery), so the customer sees progress, not a blank. - gc_orphaned_desired_state: at convergence, purge desired-state whose Deployment CR no longer exists (force-deleted while the operator was down, finalizer bypassed). Belt-and-suspenders with the controller finalizer. - run() now owns its watchers in a JoinSet, so cancelling the aggregator aborts its children — no orphan tasks outliving a restart (matters for the restart-simulation tests and clean process teardown). Also made run() Send (hoisted a .await out of a tracing macro) so it can be spawned. - docs/fleet-operator-recovery-scenarios.md enumerates the failure shapes and maps each to its test. - harmony-fleet-e2e/tests/operator_recovery.rs: regression test per scenario (cold restart converges from KV; orphan GC; two operators write identical bytes; chaos kill under write load converges <30s) + AdminKv::put_device_state. Writes stay idempotent + byte-deterministic, so two operators racing agree without leader election (operator HA = D3, deferred).	2026-06-05 15:26:00 -04:00

Author

SHA1

Message

Date

Jean-Gabriel Gill-Couture

086d905586

fix(fleet): clarify recovery tests + add missing scenario 4 + dedup test

Run Check Script / check (pull_request) Failing after 52s

Details

- Rewrite e2e tests with explicit SETUP/ACTION/ASSERT structure per
  scenario. Each test header documents devices, deployments, and
  expected end state.
- Add scenario 4 test: device offline during restart counts as pending
  (2 devices, 1 deployment, only 1 reports → succeeded=1, pending=1).
- Add apply_state_rejects_older_timestamp unit test (previously claimed
  in docs but missing).
- Fix split_once → rsplit_once in parse_state_key and seed_owned_targets
  (device IDs with dots would silently drop entries).

2026-06-09 16:49:52 -04:00

Jean-Gabriel Gill-Couture

81b0f79f55

docs(fleet): rewrite operator recovery scenarios with diagrams

ASCII diagrams per scenario replace dense paragraphs. Architecture
overview shows the three watchers, FleetState, and convergence latches.
Cold-start sequence and key invariants in scannable tables.

2026-06-09 16:45:35 -04:00

Jean-Gabriel Gill-Couture

56602b505c

feat(fleet-operator): aggregator recovery signal + orphan GC + recovery e2e (Ch2)

Run Check Script / check (pull_request) Successful in 2m35s

Details

Operator restart + aggregator recovery (v0.3 plan Ch2). The aggregator already
cold-rebuilds from NATS KV + CR watches; this makes recovery observable, closes
an orphan gap, and pins each failure shape with a regression test.

- OperatorLiveness: a shared in-process latch (Recovering → Converged) the
  aggregator sets once all three cold-start sources replay (Deployment/Device
  watcher InitDone, device-state KV seen_current; empty-bucket short-circuit).
  The in-process dashboard reads it and shows a self-clearing banner via an
  HTMX self-poll (/__recovery), so the customer sees progress, not a blank.
- gc_orphaned_desired_state: at convergence, purge desired-state whose
  Deployment CR no longer exists (force-deleted while the operator was down,
  finalizer bypassed). Belt-and-suspenders with the controller finalizer.
- run() now owns its watchers in a JoinSet, so cancelling the aggregator
  aborts its children — no orphan tasks outliving a restart (matters for the
  restart-simulation tests and clean process teardown). Also made run() Send
  (hoisted a .await out of a tracing macro) so it can be spawned.
- docs/fleet-operator-recovery-scenarios.md enumerates the failure shapes and
  maps each to its test.
- harmony-fleet-e2e/tests/operator_recovery.rs: regression test per scenario
  (cold restart converges from KV; orphan GC; two operators write identical
  bytes; chaos kill under write load converges <30s) + AdminKv::put_device_state.

Writes stay idempotent + byte-deterministic, so two operators racing agree
without leader election (operator HA = D3, deferred).

2026-06-05 15:26:00 -04:00

feat(fleet-operator): aggregator recovery signal + orphan GC + recovery e2e (Ch2) #328

3 Commits