205 lines
9.2 KiB
Markdown
205 lines
9.2 KiB
Markdown
# Harmony Fleet
|
|
|
|
IoT / decentralized-edge orchestration for harmony. A fleet stack is:
|
|
|
|
| Component | Crate | Role |
|
|
|---|---|---|
|
|
| **Operator** | [`harmony-fleet-operator`](harmony-fleet-operator/) | Watches `Deployment` CRs, writes desired state into NATS JetStream KV, aggregates device state back into CR status. Runtime binary; no `harmony` dep. |
|
|
| **Agent** | [`harmony-fleet-agent`](harmony-fleet-agent/) | One per device. Watches the desired-state KV, drives the local runtime (podman today), publishes heartbeats + per-deployment state, answers `device-commands.*` request/reply. |
|
|
| **Auth** | [`harmony-fleet-auth`](harmony-fleet-auth/) | Shared NATS credential plumbing — `TomlShared` (dev) and `ZitadelJwt` (prod with auth-callout). |
|
|
| **Deploy** | [`harmony-fleet-deploy`](harmony-fleet-deploy/) | The canonical deploy crate. It composes independent component Scores through typed references; there is no Fleet-wide aggregate Score. See [ADR-023](../docs/adr/023-deploy-architecture.md) and [ADR-028](../docs/adr/028-typed-score-references.md). |
|
|
| **E2E harness** | [`harmony-fleet-e2e`](harmony-fleet-e2e/) | Brings the stack up in a fresh k3d namespace and runs integration tests against it. |
|
|
|
|
The on-the-wire types both ends agree on (KV bucket names, key formats, command-protocol payloads) live in [`../harmony-reconciler-contracts`](../harmony-reconciler-contracts/).
|
|
|
|
## Architecture in one line
|
|
|
|
`FleetOperatorScore`, `FleetAgentScore`, etc. are real Rust types with capability-bound `Topology` parameters. Production deploys, the e2e harness, and any future control-plane tool all compose the **same** Scores; the only thing that changes is the `Topology` instance. **No handrolled YAML or imperative manifest factories anywhere.** Read [ADR-023](../docs/adr/023-deploy-architecture.md) before adding deploy logic.
|
|
|
|
## Local Fleet
|
|
|
|
The local context creates or reuses the `harmony` k3d cluster, builds and imports
|
|
the Fleet images, and deploys PostgreSQL, Zitadel, NATS with auth callout,
|
|
OpenBao, and the Fleet operator:
|
|
|
|
```bash
|
|
cargo run -p harmony-fleet-deploy --bin harmony-fleet-crds-deploy -- \
|
|
deploy --context local
|
|
cargo run -p harmony-fleet-deploy -- ship --context local
|
|
```
|
|
|
|
The first command manages the cluster-wide Fleet CRD definitions. The tenant
|
|
deployment has a separate lifecycle and puts PostgreSQL, Zitadel, NATS,
|
|
OpenBao, the callout, and the operator in the namespace compiled into its
|
|
context. Both `Deployment` and `Device` resources are namespaced, and each
|
|
operator watches only its tenant namespace.
|
|
|
|
The command returns only after the operator has authenticated through Zitadel
|
|
and initialized its NATS JetStream state. The credentials Score creates an
|
|
immutable local Kubernetes Secret on the first run and reuses it thereafter.
|
|
|
|
---
|
|
|
|
## Quickstart — run the e2e ping test
|
|
|
|
The fastest path to a green fleet stack on your laptop. Requires `podman`, `kubectl`, and `helm` on `$PATH`; everything else (`k3d`, the NATS chart, all images) is fetched / built on demand.
|
|
|
|
```bash
|
|
HARMONY_FLEET_E2E=1 cargo test -p harmony-fleet-e2e --test ping -- --nocapture
|
|
```
|
|
|
|
What it does, in order:
|
|
|
|
1. Ensures a `fleet-e2e` k3d cluster exists (creates one if not). NodePort `30423` on the host forwards to NATS inside the cluster.
|
|
2. Builds `harmony-fleet-agent` in release mode, packages it into `localhost/harmony-fleet-agent:e2e`, and sideloads the image into the k3d cluster's containerd store.
|
|
3. Mints a per-bring-up namespace `e2e-<uuid8>` and prunes any leftover `e2e-*` namespaces from prior runs (NodePort `30423` is cluster-scoped, so a stuck `Terminating` namespace would block the new bring-up — the prune waits up to 90 s for full cleanup before proceeding).
|
|
4. Deploys NATS via `NatsScore` (helm chart, JetStream on, static admin/device users, NodePort Service).
|
|
5. Waits for NATS to be reachable from the host on `nats://localhost:30423` (admin/e2e-admin).
|
|
6. Deploys one `FleetAgentScore { target: Pod }` — runs with `runtime_enabled = false` so it skips podman and only runs the command-server + heartbeat loop.
|
|
7. `FleetAgentScore` waits for the agent Deployment to be Ready.
|
|
8. The test publishes `device-commands.<device_id>.ping` via `FleetCommandsClient::ping` and asserts the agent replies with `{ device_id, agent_version, uptime_s }`.
|
|
|
|
Cold first run: ~80 s (release build of the agent dominates). Warm: ~25 s.
|
|
|
|
### Useful env knobs
|
|
|
|
| Var | Effect |
|
|
|---|---|
|
|
| `HARMONY_FLEET_E2E=1` | Required. Without it the test is skipped — keeps `cargo test --workspace` cheap on machines without k3d. |
|
|
| `FLEET_E2E_KEEP=1` | Skip namespace teardown on Drop. Lets you `kubectl -n e2e-<…> logs deploy/…` after a failure. The next run prunes it. |
|
|
| `RUST_LOG=info` | Or `debug` for the per-message `command dispatch` traces inside `harmony-fleet-agent::command_server`. |
|
|
|
|
### Connecting to NATS while the stack is up
|
|
|
|
```bash
|
|
# Host-side, via the NodePort
|
|
nats://localhost:30423 # user=admin pass=e2e-admin (full access)
|
|
nats://localhost:30423 # user=device pass=e2e-device (device permissions)
|
|
```
|
|
|
|
```bash
|
|
# In-cluster, from any Pod in the same namespace
|
|
nats://fleet-nats.e2e-<uuid8>.svc.cluster.local:4222
|
|
```
|
|
|
|
`FLEET_E2E_KEEP=1` + the harness's stdout line `[e2e] NATS: nats://127.0.0.1:30423 …` is the path most tests will take — leave the harness running, point a NATS client at that URL.
|
|
|
|
### Inspecting the agent
|
|
|
|
```bash
|
|
# Find your namespace
|
|
kubectl get ns -l harmony.io/managed-by=fleet-e2e
|
|
|
|
# Tail the agent
|
|
kubectl -n e2e-<uuid8> logs deploy/fleet-agent-<device-id> -f
|
|
|
|
# Tail NATS (StatefulSet, not Deployment)
|
|
kubectl -n e2e-<uuid8> logs sts/fleet-nats -c nats -f
|
|
|
|
# Send a ping by hand (requires the `nats` CLI:
|
|
# https://github.com/nats-io/natscli/releases)
|
|
nats --server nats://localhost:30423 --user admin --password e2e-admin \
|
|
request "device-commands.vm-device-00-<uuid8>.ping" ""
|
|
```
|
|
|
|
Or if you don't want to install the nats binary :
|
|
|
|
```
|
|
alias natsbox='podman run --network=host --rm docker.io/natsio/nats-box:latest nats --server nats://localhost:30423 --user admin --password e2e-admin'
|
|
```
|
|
|
|
You should see something like `{"device_id":"vm-device-00-<uuid8>","agent_version":"0.1.0","uptime_s":12}`.
|
|
|
|
### Cleaning up
|
|
|
|
The shared `OnceCell` in `harmony-fleet-e2e` lives for the test binary's lifetime, so namespaces survive a `cargo test` exit (the static is never explicitly dropped). The next `cargo test` invocation prunes them. To force a manual cleanup:
|
|
|
|
```bash
|
|
kubectl delete ns -l harmony.io/managed-by=fleet-e2e
|
|
# wipe the whole cluster:
|
|
k3d cluster delete fleet-e2e
|
|
```
|
|
|
|
---
|
|
|
|
## Production deploys
|
|
|
|
Production contexts live in the customer's private repository. They provide a
|
|
tenant namespace, registry, project, domain, optional image-pull Secret, and
|
|
OpenBao-brokered kubeconfig. The kubeconfig's current context selects the
|
|
cluster.
|
|
The cluster must also contain the immutable
|
|
`<context namespace>/fleet-callout-credentials` Secret with
|
|
`issuer-nkey-seed`, `issuer-public-key`, and `nats-auth-pass` keys. The deploy
|
|
validates this Secret before configuring NATS.
|
|
|
|
```bash
|
|
cargo run -- ship --context customer-prod
|
|
```
|
|
|
|
Run this from the private deploy repository whose binary calls
|
|
`deploy_fleet_with_context` with its compiled context.
|
|
|
|
See [`deployment-process.md`](deployment-process.md) for the clickable CD workflow and the in-cluster runner bootstrap.
|
|
|
|
### Releases
|
|
|
|
`scripts/release.sh` publishes immutable releases. Control-plane and agent
|
|
versions are independent; the operator and NATS callout share one control-plane
|
|
version.
|
|
|
|
```bash
|
|
# Publish either release independently.
|
|
fleet/scripts/release.sh --control-version 0.4.0 --publish-only
|
|
fleet/scripts/release.sh --agent-version 0.2.0 --publish-only
|
|
|
|
# Publish both, deploy through a private deploy crate, and upgrade one Device.
|
|
fleet/scripts/release.sh \
|
|
--control-version 0.4.0 \
|
|
--agent-version 0.2.0 \
|
|
--deploy-manifest /path/to/Cargo.toml \
|
|
--deploy-bin private-deploy \
|
|
--context production \
|
|
--namespace fleet \
|
|
--device device-1
|
|
```
|
|
|
|
Control-plane images publish to `hub.nationtech.io/harmony`. The agent is a
|
|
raw, single-layer OCI artifact and Devices receive a manifest-digest reference.
|
|
Registry pulls are anonymous, while publication uses `REGISTRY_USER` and
|
|
`REGISTRY_TOKEN`. Run the script with `--help` for its tool and deployment
|
|
inputs.
|
|
|
|
Before the first OCI-based upgrade, rerun `FleetDeviceSetupScore` for each
|
|
device to install an updater that understands `oci://` references. The updater
|
|
does not update itself. Direct `https://` agent artifacts remain supported.
|
|
|
|
### Connecting to the operator
|
|
|
|
The operator runs as a single-replica Deployment in the context namespace.
|
|
|
|
```bash
|
|
export FLEET_NAMESPACE=customer-fleet
|
|
# Tail logs
|
|
kubectl -n "$FLEET_NAMESPACE" logs deploy/harmony-fleet-operator -f
|
|
|
|
# Port-forward the embedded web dashboard (web-frontend feature)
|
|
kubectl -n "$FLEET_NAMESPACE" port-forward deploy/harmony-fleet-operator 18080:18080
|
|
|
|
# Or run the dashboard standalone with seeded fake data — no NATS, no cluster
|
|
cargo run -p harmony-fleet-operator --features web-frontend -- serve-web --mock
|
|
# browse http://127.0.0.1:18080
|
|
```
|
|
|
|
---
|
|
|
|
## VM rehearsal
|
|
|
|
`examples/fleet_e2e_demo` is a small executable over
|
|
`harmony-fleet-e2e::VmStack`. It runs the same callout, OpenBao, operator, and
|
|
device enrollment Scores as the maintained VM tests:
|
|
|
|
```bash
|
|
cargo run -p example-fleet-e2e-demo -- --num-devices 2
|
|
```
|