Component	Crate	Role
Operator	`harmony-fleet-operator`	Watches `Deployment` CRs, writes desired state into NATS JetStream KV, aggregates device state back into CR status. Runtime binary; no `harmony` dep.
Agent	`harmony-fleet-agent`	One per device. Watches the desired-state KV, drives the local runtime (podman today), publishes heartbeats + per-deployment state, answers `device-commands.*` request/reply.
Auth	`harmony-fleet-auth`	Shared NATS credential plumbing — `TomlShared` (dev) and `ZitadelJwt` (prod with auth-callout).
Deploy	`harmony-fleet-deploy`	The canonical deploy crate. It composes independent component Scores through typed references; there is no Fleet-wide aggregate Score. See ADR-023 and ADR-028.
E2E harness	`harmony-fleet-e2e`	Brings the stack up in a fresh k3d namespace and runs integration tests against it.

The on-the-wire types both ends agree on (KV bucket names, key formats, command-protocol payloads) live in ../harmony-reconciler-contracts.

Architecture in one line

FleetOperatorScore, FleetAgentScore, etc. are real Rust types with capability-bound Topology parameters. Production deploys, the e2e harness, and any future control-plane tool all compose the same Scores; the only thing that changes is the Topology instance. No handrolled YAML or imperative manifest factories anywhere. Read ADR-023 before adding deploy logic.

Local Fleet

The local context creates or reuses the harmony k3d cluster, builds and imports the Fleet images, and deploys PostgreSQL, Zitadel, NATS with auth callout, OpenBao, and the Fleet operator:

cargo run -p harmony-fleet-deploy --bin harmony-fleet-crds-deploy -- \
  deploy --context local
cargo run -p harmony-fleet-deploy -- ship --context local

The first command manages the cluster-wide Fleet CRD definitions. The tenant deployment has a separate lifecycle and puts PostgreSQL, Zitadel, NATS, OpenBao, the callout, and the operator in the namespace compiled into its context. Both Deployment and Device resources are namespaced, and each operator watches only its tenant namespace.

The command returns only after the operator has authenticated through Zitadel and initialized its NATS JetStream state. The credentials Score creates an immutable local Kubernetes Secret on the first run and reuses it thereafter.

Quickstart — run the e2e ping test

The fastest path to a green fleet stack on your laptop. Requires podman, kubectl, and helm on $PATH; everything else (k3d, the NATS chart, all images) is fetched / built on demand.

HARMONY_FLEET_E2E=1 cargo test -p harmony-fleet-e2e --test ping -- --nocapture

What it does, in order:

Ensures a fleet-e2e k3d cluster exists (creates one if not). NodePort 30423 on the host forwards to NATS inside the cluster.
Builds harmony-fleet-agent in release mode, packages it into localhost/harmony-fleet-agent:e2e, and sideloads the image into the k3d cluster's containerd store.
Mints a per-bring-up namespace e2e-<uuid8> and prunes any leftover e2e-* namespaces from prior runs (NodePort 30423 is cluster-scoped, so a stuck Terminating namespace would block the new bring-up — the prune waits up to 90 s for full cleanup before proceeding).
Deploys NATS via NatsScore (helm chart, JetStream on, static admin/device users, NodePort Service).
Waits for NATS to be reachable from the host on nats://localhost:30423 (admin/e2e-admin).
Deploys one FleetAgentScore { target: Pod } — runs with runtime_enabled = false so it skips podman and only runs the command-server + heartbeat loop.
FleetAgentScore waits for the agent Deployment to be Ready.
The test publishes device-commands.<device_id>.ping via FleetCommandsClient::ping and asserts the agent replies with { device_id, agent_version, uptime_s }.

Cold first run: ~80 s (release build of the agent dominates). Warm: ~25 s.

Useful env knobs

Var	Effect
`HARMONY_FLEET_E2E=1`	Required. Without it the test is skipped — keeps `cargo test --workspace` cheap on machines without k3d.
`FLEET_E2E_KEEP=1`	Skip namespace teardown on Drop. Lets you `kubectl -n e2e-<…> logs deploy/…` after a failure. The next run prunes it.
`RUST_LOG=info`	Or `debug` for the per-message `command dispatch` traces inside `harmony-fleet-agent::command_server`.

Connecting to NATS while the stack is up

# Host-side, via the NodePort
nats://localhost:30423           # user=admin pass=e2e-admin (full access)
nats://localhost:30423           # user=device pass=e2e-device (device permissions)

# In-cluster, from any Pod in the same namespace
nats://fleet-nats.e2e-<uuid8>.svc.cluster.local:4222

FLEET_E2E_KEEP=1 + the harness's stdout line [e2e] NATS: nats://127.0.0.1:30423 … is the path most tests will take — leave the harness running, point a NATS client at that URL.

Inspecting the agent

# Find your namespace
kubectl get ns -l harmony.io/managed-by=fleet-e2e

# Tail the agent
kubectl -n e2e-<uuid8> logs deploy/fleet-agent-<device-id> -f

# Tail NATS (StatefulSet, not Deployment)
kubectl -n e2e-<uuid8> logs sts/fleet-nats -c nats -f

# Send a ping by hand (requires the `nats` CLI:
#   https://github.com/nats-io/natscli/releases)
nats --server nats://localhost:30423 --user admin --password e2e-admin \
     request "device-commands.vm-device-00-<uuid8>.ping" ""

Or if you don't want to install the nats binary :

alias natsbox='podman run --network=host --rm docker.io/natsio/nats-box:latest nats --server nats://localhost:30423 --user admin --password e2e-admin'

You should see something like {"device_id":"vm-device-00-<uuid8>","agent_version":"0.1.0","uptime_s":12}.

Cleaning up

The shared OnceCell in harmony-fleet-e2e lives for the test binary's lifetime, so namespaces survive a cargo test exit (the static is never explicitly dropped). The next cargo test invocation prunes them. To force a manual cleanup:

kubectl delete ns -l harmony.io/managed-by=fleet-e2e
# wipe the whole cluster:
k3d cluster delete fleet-e2e

Production deploys

Production contexts live in the customer's private repository. They provide a tenant namespace, registry, project, domain, optional image-pull Secret, and OpenBao-brokered kubeconfig. The kubeconfig's current context selects the cluster. The cluster must also contain the immutable <context namespace>/fleet-callout-credentials Secret with issuer-nkey-seed, issuer-public-key, and nats-auth-pass keys. The deploy validates this Secret before configuring NATS.

cargo run -- ship --context customer-prod

Run this from the private deploy repository whose binary calls deploy_fleet_with_context with its compiled context.

See deployment-process.md for the clickable CD workflow and the in-cluster runner bootstrap.

Releases

scripts/release.sh publishes immutable releases. Control-plane and agent versions are independent; the operator and NATS callout share one control-plane version.

# Publish either release independently.
fleet/scripts/release.sh --control-version 0.4.0 --publish-only
fleet/scripts/release.sh --agent-version 0.2.0 --publish-only

# Publish both, deploy through a private deploy crate, and upgrade one Device.
fleet/scripts/release.sh \
  --control-version 0.4.0 \
  --agent-version 0.2.0 \
  --deploy-manifest /path/to/Cargo.toml \
  --deploy-bin private-deploy \
  --context production \
  --namespace fleet \
  --device device-1

Control-plane images publish to hub.nationtech.io/harmony. The agent is a raw, single-layer OCI artifact and Devices receive a manifest-digest reference. Registry pulls are anonymous, while publication uses REGISTRY_USER and REGISTRY_TOKEN. Run the script with --help for its tool and deployment inputs.

Before the first OCI-based upgrade, rerun FleetDeviceSetupScore for each device to install an updater that understands oci:// references. The updater does not update itself. Direct https:// agent artifacts remain supported.

Connecting to the operator

The operator runs as a single-replica Deployment in the context namespace.

export FLEET_NAMESPACE=customer-fleet
# Tail logs
kubectl -n "$FLEET_NAMESPACE" logs deploy/harmony-fleet-operator -f

# Port-forward the embedded web dashboard (web-frontend feature)
kubectl -n "$FLEET_NAMESPACE" port-forward deploy/harmony-fleet-operator 18080:18080

# Or run the dashboard standalone with seeded fake data — no NATS, no cluster
cargo run -p harmony-fleet-operator --features web-frontend -- serve-web --mock
# browse http://127.0.0.1:18080

VM rehearsal

examples/fleet_e2e_demo is a small executable over harmony-fleet-e2e::VmStack. It runs the same callout, OpenBao, operator, and device enrollment Scores as the maintained VM tests:

cargo run -p example-fleet-e2e-demo -- --num-devices 2