harmony/docs/adr/022-agent-desired-state-alternatives.md

# ADR-022: Agent Desired-State — Alternatives and Recommendation

**Status:** Proposed
**Date:** 2026-04-09
**Supersedes (candidate):** ADR-021 shell-executor proposal

## Context

ADR-021 drafted a first-pass "desired-state convergence" mechanism for the Harmony Agent (ADR-016) in the form of a shell-command executor. On review, that shape raised serious concerns (see ADR-021 §"Open Questions and Concerns"): it is incoherent with Harmony's Score-Topology-Interpret pattern, it is not idempotent, it has no resource model, no typed status, no lifecycle, and it weakens the agent's security posture.

Separately, the team has been converging on a **"mini-kubelet" framing** for the IoT agent:

- The agent owns a small, fixed set of *reconcilers*, one per resource type it can manage (systemd unit, container, file, network interface, overlay config...).
- The desired state is a *typed manifest* — a bag of resources with identities, generations, and typed status.
- The agent runs reconcile loops similar to kubelet's Pod Lifecycle Event Generator (PLEG): for each managed resource, observe actual, compare to desired, apply the minimum delta, update typed status.
- Failure and drift are first-class. "I tried, it failed, here is why" is a valid steady state.

ADR-017-3 already borrows Kubernetes vocabulary (staleness, fencing, promotion) on purpose. Doubling down on the kubelet metaphor at the desired-state layer is the natural continuation, not a tangent.

This ADR enumerates the candidate designs, argues their tradeoffs honestly, and recommends a path.

## Alternatives

### Alternative A — Shell Command Executor (ADR-021 as-is)

**Shape:** `DesiredState { command: String, generation: u64 }`, agent does `sh -c $command`, pipes stdout/stderr/exit into `ActualState`.

**Pros:**
- Trivial to implement. ~200 LOC, already on this branch.
- Works for any task that can be expressed as a shell pipeline — maximum flexibility at v1.
- Zero new abstractions: reuses existing NATS KV watch + CAS patterns.
- End-to-end demo-able in an afternoon.

**Cons:**
- **Wrong abstraction level.** Harmony's entire thesis is "no more stringly-typed YAML/shell mud pits". This design ships that mud pit *to the edge*.
- **Not idempotent.** The burden of idempotency falls on whoever writes the command string. `systemctl start foo` run twice is fine; `apt install foo && echo "done" >> /etc/state` run twice is broken. We cannot enforce correctness.
- **No resource model.** No concept of "this manifest owns X". No diffing, no GC, no drift detection, no "what does this agent currently run?".
- **No typed status.** stdout/stderr/exit_code does not tell a fleet dashboard "container nginx is running, restarted 3 times, last healthy 2s ago". It tells it "this bash ran and exited 0 once, three minutes ago".
- **No lifecycle.** Fire-and-forget; post-exit the agent has no notion of whether the resource is still healthy.
- **Security.** Even with NATS ACLs, the API's *shape* invites abuse. Any bug in the control plane that lets a user influence a desired-state write equals RCE on every Pi.
- **Incoherent with ADR-017-3 and Score-Topology-Interpret.** Introduces a parallel concept that has nothing to do with the rest of Harmony.

**Verdict:** Acceptable only as a *named escape hatch* inside a richer design (a `ShellJob` resource variant, explicitly labeled as such and audited). Not acceptable as the whole design.

---

### Alternative B — Mini-Kubelet with Typed Resource Manifests

**Shape:** The agent owns a fixed set of `Resource` variants and one reconciler per variant.

```rust
/// The unit of desired state shipped to an agent.
/// Serialized to JSON, pushed via NATS KV to `desired-state.<agent-id>`.
struct AgentManifest {
    generation: u64,               // monotonic, control-plane assigned
    resources: Vec<ManagedResource>,
}

struct ManagedResource {
    /// Stable, manifest-unique identity. Used for diffing across generations.
    id: ResourceId,
    spec: ResourceSpec,
}

enum ResourceSpec {
    SystemdUnit(SystemdUnitSpec),     // ensure unit exists, enabled, active
    Container(ContainerSpec),          // podman/docker run with image, env, volumes
    File(FileSpec),                    // path, mode, owner, content (hash or inline)
    NetworkConfig(NetworkConfigSpec),  // interface, addresses, routes
    ShellJob(ShellJobSpec),            // explicit escape hatch, audited separately
    // ...extend carefully
}

/// What the agent reports back.
struct AgentStatus {
    manifest_generation: u64,          // which desired-state gen this reflects
    observed_generation: u64,          // highest gen the agent has *processed*
    resources: HashMap<ResourceId, ResourceStatus>,
    conditions: Vec<AgentCondition>,   // Ready, Degraded, Reconciling, ...
}

enum ResourceStatus {
    Pending,
    Reconciling { since: Timestamp },
    Ready { since: Timestamp, details: ResourceReadyDetails },
    Failed { since: Timestamp, error: String, retry_after: Option<Timestamp> },
}
```

Each reconciler implements a small trait:

```rust
trait Reconciler {
    type Spec;
    type Status;
    async fn observe(&self, id: &ResourceId) -> Result<Self::Status>;
    async fn reconcile(&self, id: &ResourceId, spec: &Self::Spec) -> Result<Self::Status>;
    async fn delete(&self, id: &ResourceId) -> Result<()>;
}
```

The agent loop becomes:

1. Watch `desired-state.<agent-id>` for the latest `AgentManifest`.
2. On change, compute diff vs. observed set: additions, updates, deletions.
3. Dispatch each resource to its reconciler. Reconcilers are idempotent by contract.
4. Aggregate per-resource status into `AgentStatus`, write to `actual-state.<agent-id>` via CAS.
5. Re-run periodically to detect drift even when desired state has not changed (PLEG-equivalent).

**Pros:**
- **Declarative and idempotent by construction.** Reconcilers are required to be idempotent; the contract is enforced in Rust traits, not in docs.
- **Typed status.** Dashboards, alerts, and the control plane get structured data.
- **Drift detection.** Periodic re-observation catches "someone SSH'd in and stopped the service".
- **Lifecycle.** Each resource has a clear state machine; health is a first-class concept.
- **Coherent with ADR-017-3.** The kubelet framing becomes literal, not metaphorical.
- **Narrow attack surface.** The agent only knows how to do a handful of well-audited things. Adding a new capability is an explicit code change, not a new shell string.
- **Composable with Harmony's existing philosophy.** `ManagedResource` is to the agent what a Score is to a Topology, at a smaller scale.

**Cons:**
- More upfront design. Each reconciler needs to be written and tested.
- Requires us to *commit* to a resource type set and its status schema. Adding a new kind is a versioned change to the wire format.
- Duplicates, at the edge, some of the vocabulary already present in Harmony's Score layer (e.g., `FileDeployment`, container deployments). Risk of two parallel abstractions evolving in tension.
- Harder to demo in a single afternoon.

**Verdict:** Strong candidate. Matches the team's mini-kubelet intuition directly.

---

### Alternative C — Embedded Score Interpreter on the Agent

**Shape:** The desired state *is* a Harmony Score (or a set of Scores), serialized and pushed via NATS. The agent hosts a local `PiTopology` that exposes a small, carefully chosen set of capabilities (`SystemdHost`, `ContainerRuntime`, `FileSystemHost`, `NetworkConfigurator`, ...). The agent runs the Score's `interpret` against that local topology.

```rust
// On the control plane:
let score = SystemdServiceScore { ... };
let wire: SerializedScore = score.to_wire()?;
nats.put(format!("desired-state.{agent_id}"), wire).await?;

// On the agent:
let score = SerializedScore::decode(payload)?;
let topology = PiTopology::new();
let outcome = score.interpret(&inventory, &topology).await?;
// Outcome is already a typed Harmony result (SUCCESS/NOOP/FAILURE/RUNNING/...).
```

**Pros:**
- **Zero new abstractions.** The agent becomes "a Harmony executor that happens to run on a Pi". Everything we already know how to do in Harmony works, for free.
- **Maximum coherence.** There is exactly one way to describe desired state in the whole system: a Score. The type system enforces that a score requesting `K8sclient` cannot be shipped to a Pi topology that does not offer it — at compile time on the control plane, at deserialization time on the agent.
- **Composability.** Higher-order topologies (ADR-015) work unchanged: `FailoverTopology<PiTopology>` gets you HA at the edge for free.
- **Single mental model for the whole team.** "Write a Score" is already the Harmony primitive; no one needs to learn a second one.

**Cons:**
- **Serializability.** This is the hard one. Harmony Scores today hold trait objects, references to live topology state, and embedded closures in places. Making them uniformly serde-serializable is a non-trivial refactor that touches dozens of modules. We would be gating the IoT MVP on a cross-cutting refactor.
- **Agent binary size.** If "the agent can run any Score", it links every module. On a Pi Zero 2 W, that matters. We can mitigate with feature flags, but then we are back to "which scores does *this* agent support?" — i.e., we have reinvented resource-type registration, just spelled differently.
- **Capability scoping is subtle.** We have to be extremely careful about which capabilities `PiTopology` exposes. "A Pi can run containers" is true; "a Pi can run arbitrary k8s clusters" is not. Getting that boundary wrong opens the same attack surface as Alternative A, just hidden behind a Score.
- **Control-plane UX.** The central platform now needs to instantiate Scores for specific Pis, handle their inventories, and ship them. That is heavier than "push a JSON blob".

**Verdict:** The principled end state, almost certainly where we want to be in 18 months. Not shippable for the IoT MVP.

---

### Alternative D — Hybrid: Typed Manifests Now, Scores Later

**Shape:** Ship Alternative B (typed `AgentManifest` with a fixed set of reconcilers). Keep the Score ambition (Alternative C) as an explicit roadmap item. When Scores become uniformly wire-serializable and `PiTopology` is mature, migrate by adding a `ResourceSpec::Score(SerializedScore)` variant. Eventually that variant may subsume the others.

**Pros:**
- **Shippable soon.** Alternative B is the implementable core; we can have a fleet demo in weeks, not months.
- **On a path to the ideal.** We do not dead-end. The `ResourceSpec` enum becomes the migration seam.
- **De-risks the Score serialization refactor.** We learn what resource types we *actually* need on the edge before we refactor the Score layer.
- **Lets us delete Alternative A cleanly.** The shell executor either disappears or survives as a narrow, explicitly-audited `ResourceSpec::ShellJob` variant that documents itself as an escape hatch.

**Cons:**
- Temporarily maintains two vocabularies (`ResourceSpec` at the edge, `Score` in the core). There is a risk they drift before they reconverge.
- Requires team discipline to actually do the C migration and not leave B as the permanent design.

**Verdict:** Recommended.

---

## Recommendation

**Adopt Alternative D (Hybrid: typed manifests now, Scores later).**

Reasoning:

1. **Speed to IoT MVP** is real. Alternative C is a 3-6 month refactor of the Score layer before we can deploy anything; Alternative B can ship within the current iteration.
2. **Long-term coherence with Harmony's design philosophy** is preserved because D has an explicit migration seam to C. We do not paint ourselves into a corner.
3. **The mini-kubelet framing is directly satisfied by B.** Typed resources, reconciler loops, observed-generation pattern, PLEG-style drift detection. This is exactly what the team has been describing.
4. **Capability-trait discipline carries over cleanly.** `Reconciler` is the agent-side analog of a capability trait (`DnsServer`, `K8sclient`, etc.). The rule "capabilities are industry concepts, not tools" applies to `ResourceSpec` too: we name it `Container`, not `Podman`; `SystemdUnit`, not `Systemctl`.
5. **The shell executor is not wasted work.** It proved the NATS KV watch + typed CAS write pattern that Alternative B will also need. It becomes either `ResourceSpec::ShellJob` (audited escape hatch) or gets deleted.
6. **Security posture improves immediately.** A fixed resource-type allowlist is dramatically tighter than "run any shell", even before we add signing or sandboxing.
7. **The IoT product use case actually is "deploy simple workloads to Pi fleets".** Containers, systemd services, config files, network config. That is a short list, and it maps to four or five resource types. We do not need the full expressive power of a Score layer to hit the product milestone.

## Specific Findings on the Current Implementation

`harmony_agent/src/desired_state.rs` (≈250 lines, implemented on this branch):

- **Keep as scaffolding**, do not wire into user tooling.
- The NATS KV watch loop, the `ActualState` CAS write, and the generation-tracking skeleton are all reusable by Alternative B. They are the only parts worth keeping.
- The `execute_command` function (shelling out via `Command::new("sh").arg("-c")`) is the part that bakes in the wrong abstraction. It should be:
  1. **Moved behind a `ResourceSpec::ShellJob` reconciler** if we decide to keep shell as an explicit, audited escape hatch, **or**
  2. **Deleted** when the first two real reconcilers (Container, SystemdUnit) land.
- The `DesiredStateConfig` / `ActualState` types in `harmony_agent/src/agent/config.rs` are too narrow. They should be replaced by `AgentManifest` / `AgentStatus` as sketched above. `generation: u64` at the manifest level stays; per-resource status is added.
- The existing tests (`executes_command_and_reports_result`, `reports_failure_for_bad_command`) are testing the shell executor specifically; they will be deleted or repurposed when the resource model lands.

## Open Questions (to resolve before implementing B)

1. **What is the minimum viable resource type set for the IoT MVP?** Proposal: `Container`, `SystemdUnit`, `File`. Defer `NetworkConfig`, `ShellJob` until a concrete use case appears.
2. **Where does `AgentManifest` live in the crate graph?** It is consumed by both the control plane and the agent. Likely `harmony_agent_types` (new) or an existing shared types crate.
3. **How are images, files, and secrets referenced?** By content hash + asset store URL (ADR: `harmony_assets`)? By inline payload under a size cap?
4. **What is the reconcile cadence?** On NATS KV change + periodic drift check every N seconds? What is N on a Pi?
5. **How does `AgentStatus` interact with the heartbeat loop?** Is the status written on every reconcile, or aggregated into the heartbeat payload? The heartbeat cares about liveness; the status cares about workload health. They are probably separate KV keys, coupled by generation.
6. **How do we handle partial failures and retry?** Exponential backoff per resource? Global pause on repeated failures? Surface to the control plane via `conditions`?
7. **Can the agent refuse a manifest it does not understand?** (Forward compatibility: new `ResourceSpec` variant rolled out before the agent upgrade.) Proposal: fail loudly and report a typed `UnknownResource` status so the control plane can detect version skew.

## Decision

**None yet.** This ADR is explicitly a proposal to adopt **Alternative D**, pending team review. If approved, a follow-up ADR-023 will specify the concrete `AgentManifest` / `AgentStatus` schema and the initial reconciler set.