harmony/fleet/README.md

Harmony Fleet

IoT / decentralized-edge orchestration for harmony. A fleet stack is:

Component	Crate	Role
Operator	harmony-fleet-operator	Watches Deployment CRs, writes desired state into NATS JetStream KV, aggregates device state back into CR status. Runtime binary; no harmony dep.
Agent	harmony-fleet-agent	One per device. Watches the desired-state KV, drives the local runtime (podman today), publishes heartbeats + per-deployment state, answers device-commands.* request/reply.
Auth	harmony-fleet-auth	Shared NATS credential plumbing — TomlShared (dev) and ZitadelJwt (prod with auth-callout).
Deploy	harmony-fleet-deploy	The canonical deploy crate. It composes independent component Scores through typed references; there is no Fleet-wide aggregate Score. See ADR-023 and ADR-028.
E2E harness	harmony-fleet-e2e	Brings the stack up in a fresh k3d namespace and runs integration tests against it.

The on-the-wire types both ends agree on (KV bucket names, key formats, command-protocol payloads) live in ../harmony-reconciler-contracts.

Architecture in one line

FleetOperatorScore, FleetAgentScore, etc. are real Rust types with capability-bound Topology parameters. Production deploys, the e2e harness, and any future control-plane tool all compose the same Scores; the only thing that changes is the Topology instance. No handrolled YAML or imperative manifest factories anywhere. Read ADR-023 before adding deploy logic.

Local Fleet

The local context creates or reuses the harmony k3d cluster, builds and imports the Fleet images, and deploys PostgreSQL, Zitadel, NATS with auth callout, OpenBao, and the Fleet operator:

cargo run -p harmony-fleet-deploy -- ship --context local

The command returns only after the operator has authenticated through Zitadel and initialized its NATS JetStream state. The credentials Score creates an immutable local Kubernetes Secret on the first run and reuses it thereafter.

---

Quickstart — run the e2e ping test

The fastest path to a green fleet stack on your laptop. Requires podman, kubectl, and helm on $PATH; everything else (k3d, the NATS chart, all images) is fetched / built on demand.

HARMONY_FLEET_E2E=1 cargo test -p harmony-fleet-e2e --test ping -- --nocapture

What it does, in order:

1. Ensures a fleet-e2e k3d cluster exists (creates one if not). NodePort 30423 on the host forwards to NATS inside the cluster.
2. Builds harmony-fleet-agent in release mode, packages it into localhost/harmony-fleet-agent:e2e, and sideloads the image into the k3d cluster's containerd store.
3. Mints a per-bring-up namespace e2e-<uuid8> and prunes any leftover e2e-* namespaces from prior runs (NodePort 30423 is cluster-scoped, so a stuck Terminating namespace would block the new bring-up — the prune waits up to 90 s for full cleanup before proceeding).
4. Deploys NATS via NatsScore (helm chart, JetStream on, static admin/device users, NodePort Service).
5. Waits for NATS to be reachable from the host on nats://localhost:30423 (admin/e2e-admin).
6. Deploys one FleetAgentScore { target: Pod } — runs with runtime_enabled = false so it skips podman and only runs the command-server + heartbeat loop.
7. FleetAgentScore waits for the agent Deployment to be Ready.
8. The test publishes device-commands.<device_id>.ping via FleetCommandsClient::ping and asserts the agent replies with { device_id, agent_version, uptime_s }.

Cold first run: ~80 s (release build of the agent dominates). Warm: ~25 s.

Useful env knobs

Var	Effect
HARMONY_FLEET_E2E=1	Required. Without it the test is skipped — keeps cargo test --workspace cheap on machines without k3d.
FLEET_E2E_KEEP=1	Skip namespace teardown on Drop. Lets you kubectl -n e2e-<…> logs deploy/… after a failure. The next run prunes it.
RUST_LOG=info	Or debug for the per-message command dispatch traces inside harmony-fleet-agent::command_server.

Connecting to NATS while the stack is up

# Host-side, via the NodePort
nats://localhost:30423           # user=admin pass=e2e-admin (full access)
nats://localhost:30423           # user=device pass=e2e-device (device permissions)

# In-cluster, from any Pod in the same namespace
nats://fleet-nats.e2e-<uuid8>.svc.cluster.local:4222

FLEET_E2E_KEEP=1 + the harness's stdout line [e2e] NATS: nats://127.0.0.1:30423 … is the path most tests will take — leave the harness running, point a NATS client at that URL.

Inspecting the agent

# Find your namespace
kubectl get ns -l harmony.io/managed-by=fleet-e2e

# Tail the agent
kubectl -n e2e-<uuid8> logs deploy/fleet-agent-<device-id> -f

# Tail NATS (StatefulSet, not Deployment)
kubectl -n e2e-<uuid8> logs sts/fleet-nats -c nats -f

# Send a ping by hand (requires the `nats` CLI:
#   https://github.com/nats-io/natscli/releases)
nats --server nats://localhost:30423 --user admin --password e2e-admin \
     request "device-commands.vm-device-00-<uuid8>.ping" ""

Or if you don't want to install the nats binary :

alias natsbox='podman run --network=host --rm docker.io/natsio/nats-box:latest nats --server nats://localhost:30423 --user admin --password e2e-admin'

You should see something like {"device_id":"vm-device-00-<uuid8>","agent_version":"0.1.0","uptime_s":12}.

Cleaning up

The shared OnceCell in harmony-fleet-e2e lives for the test binary's lifetime, so namespaces survive a cargo test exit (the static is never explicitly dropped). The next cargo test invocation prunes them. To force a manual cleanup:

kubectl delete ns -l harmony.io/managed-by=fleet-e2e
# wipe the whole cluster:
k3d cluster delete fleet-e2e

---

Production deploys

Production contexts live in the customer's private repository. They provide a registry, project, domain, image-pull Secret, and OpenBao-brokered kubeconfig. The cluster must also contain the immutable fleet-system/fleet-callout-credentials Secret with issuer-nkey-seed, issuer-public-key, and nats-auth-pass keys. The deploy validates this Secret before configuring NATS.

cargo run -p harmony-fleet-deploy -- \
  ship --context customer-prod \
  --config path/to/contexts.toml

See deployment-process.md for the clickable CD workflow and the in-cluster runner bootstrap.

Connecting to the operator

The operator runs as a single-replica Deployment in --namespace (default fleet-system).

# Tail logs
kubectl -n fleet-system logs deploy/harmony-fleet-operator -f

# Port-forward the embedded web dashboard (web-frontend feature)
kubectl -n fleet-system port-forward deploy/harmony-fleet-operator 18080:18080

# Or run the dashboard standalone with seeded fake data — no NATS, no cluster
cargo run -p harmony-fleet-operator --features web-frontend -- serve-web --mock
# browse http://127.0.0.1:18080

---

VM rehearsal

examples/fleet_e2e_demo is a small executable over harmony-fleet-e2e::VmStack. It runs the same callout, OpenBao, operator, and device enrollment Scores as the maintained VM tests:

cargo run -p example-fleet-e2e-demo -- --num-devices 2