Large merge: 209 commits, +47k/-5k across 428 files. Five clusters of work, plus a deliberate behavior change and a known-deferred backlog spelled out below. 1. IoT walking skeleton — harmony-reconciler-contracts crate, harmony-fleet-operator (Deployment CRD → NATS KV), harmony-fleet- agent reconciling PodmanV0Score into containers, K8sBareTopology, maud+htmx operator frontend, request/reply commands over NATS. 2. NATS auth callout + from-scratch nats/jwt crate (algorithm, claims, builders, xkey). nats/callout maps Zitadel roles to NATS permissions; harmony-fleet-auth holds the shared credential plumbing. 3. Zitadel deepening — setup.rs rewritten (cross-org admin, persisted admin password, device-code OIDC, OKD-compat values). New harmony_zitadel_auth crate (axum login flow, JWKS, sessions). Agent gets a JWT-bearer credential source. 4. Deploy-architecture refactor per ADR-023 — Scores everywhere, new harmony-fleet-deploy crate, harmony-fleet-e2e harness covering Pod target plus VM target (aarch64 production + x86_64 fast iteration). First Score companion lands: AgentObservation (ADR-023 P7). 5. Device enrollment via Zitadel SSO + aarch64 KVM support (AAVMF firmware, per-VM NVRAM, TCG perf overrides), IotDeviceSetupScore /FleetDeviceSetupScore SSH apply, fleet-sso-login example. Behavior change worth flagging in this merge: harmony_secret now panics on an unknown SECRET_STORE value instead of silently defaulting to Infisical. No in-tree caller hits this; a typo in the env var is now loud rather than mysterious. Deferred (not merge blockers — file as issues post-merge): - CI is red on master because hub.nationtech.io/harmony/ harmony_composer:latest is missing libvirt-dev + pkg-config, which `cargo check --all-features` requires once harmony's `kvm` feature is unified. Local `build/check.sh` is green. Fix is a two-line Dockerfile edit + rebuild of the composer image; top priority post-merge so master goes green again. - Smoke-test contract (ADR-023 P4) — the principle is locked but the trait/struct shape is open. Each Score implements its own readiness today; harmony-fleet-e2e has `wait_until_ready` as a per-test stand-in. - Rust equivalents for smoke-a1.sh / smoke-a4.sh (smoke-a3* are superseded by the new VM e2e tests). - Five #[ignore]'d tests in examples/fleet_e2e_demo and nats/ integration-test-callout — waiting on a CI runner with libvirt + k3d + podman. - ADR-024 (capability decomposition) — kept as draft under docs/adr/drafts/024- pending more conviction.
Examples
This directory contains runnable examples demonstrating Harmony's capabilities. Each example is a self-contained program that can be run with cargo run -p example-<name>.
Quick Reference
| Example | Description | Local K3D | Existing Cluster | Hardware Needed |
|---|---|---|---|---|
postgresql |
Deploy a PostgreSQL cluster | ✅ | ✅ | — |
ntfy |
Deploy ntfy notification server | ✅ | ✅ | — |
tenant |
Create a multi-tenant namespace | ✅ | ✅ | — |
cert_manager |
Provision TLS certificates | ✅ | ✅ | — |
node_health |
Check Kubernetes node health | ✅ | ✅ | — |
monitoring |
Deploy Prometheus alerting | ✅ | ✅ | — |
monitoring_with_tenant |
Monitoring + tenant isolation | ✅ | ✅ | — |
operatorhub_catalog |
Install OperatorHub catalog | ✅ | ✅ | — |
validate_ceph_cluster_health |
Verify Ceph cluster health | — | ✅ | Rook/Ceph |
remove_rook_osd |
Remove a Rook OSD | — | ✅ | Rook/Ceph |
brocade_snmp_server |
Configure Brocade switch SNMP | — | ✅ | Brocade switch |
opnsense_node_exporter |
Node exporter on OPNsense | — | ✅ | OPNsense firewall |
opnsense_vm_integration |
Full OPNsense firewall automation (11 Scores) | ✅ | — | KVM/libvirt |
opnsense_pair_integration |
OPNsense HA pair with CARP failover | ✅ | — | KVM/libvirt |
okd_pxe |
PXE boot configuration for OKD | — | — | ✅ |
okd_installation |
Full OKD bare-metal install | — | — | ✅ |
okd_cluster_alerts |
OKD cluster monitoring alerts | — | ✅ | OKD cluster |
multisite_postgres |
Multi-site PostgreSQL failover | — | ✅ | Multi-cluster |
nats |
Deploy NATS messaging | — | ✅ | Multi-cluster |
nats-supercluster |
NATS supercluster across sites | — | ✅ | Multi-cluster |
lamp |
LAMP stack deployment | ✅ | ✅ | — |
openbao |
Deploy OpenBao vault | ✅ | ✅ | — |
zitadel |
Deploy Zitadel identity provider | ✅ | ✅ | — |
try_rust_webapp |
Rust webapp with packaging | ✅ | ✅ | Submodule |
rust |
Rust webapp with full monitoring | ✅ | ✅ | — |
rhob_application_monitoring |
RHOB monitoring setup | ✅ | ✅ | — |
sttest |
Full OKD stack test | — | — | ✅ |
application_monitoring_with_tenant |
App monitoring + tenant | — | ✅ | OKD cluster |
kube-rs |
Direct kube-rs client usage | ✅ | ✅ | — |
k8s_drain_node |
Drain a Kubernetes node | ✅ | ✅ | — |
k8s_write_file_on_node |
Write files to K8s nodes | ✅ | ✅ | — |
harmony_inventory_builder |
Discover hosts via subnet scan | ✅ | — | — |
cli |
CLI tool with inventory discovery | ✅ | — | — |
tui |
Terminal UI demonstration | ✅ | — | — |
Status Legend
| Symbol | Meaning |
|---|---|
| ✅ | Works out-of-the-box |
| — | Not applicable or requires specific setup |
By Category
Data Services
postgresql— Deploy a PostgreSQL cluster via CloudNativePGmultisite_postgres— Multi-site PostgreSQL with failoverpublic_postgres— Public-facing PostgreSQL (⚠️ uses NationTech DNS)
Kubernetes Utilities
node_health— Check node health in a clusterk8s_drain_node— Drain and reboot a nodek8s_write_file_on_node— Write files to nodesvalidate_ceph_cluster_health— Verify Ceph/Rook cluster healthremove_rook_osd— Remove an OSD from Rook/Cephkube-rs— Direct Kubernetes client usage demo
Monitoring & Alerting
monitoring— Deploy Prometheus alerting with Discord webhooksmonitoring_with_tenant— Monitoring with tenant isolationntfy— Deploy ntfy notification serverokd_cluster_alerts— OKD-specific cluster alerts
Application Deployment
try_rust_webapp— Deploy a Rust webapp with packaging (⚠️ requirestryrust.orgsubmodule)rust— Rust webapp with full monitoring featuresrhob_application_monitoring— Red Hat Observability Stack monitoringlamp— LAMP stack deployment (⚠️ uses NationTech DNS)application_monitoring_with_tenant— App monitoring with tenant isolation
Infrastructure & Bare Metal
opnsense_vm_integration— Recommended demo. Boot an OPNsense VM and configure it with 11 Scores (load balancer, DHCP, TFTP, VLANs, firewall rules, NAT, VIPs, LAGG). Fully automated, requires only KVM. See the detailed guide.opnsense_pair_integration— Boot two OPNsense VMs and configure a CARP HA firewall pair withFirewallPairTopologyandCarpVipScore. Demonstrates NIC link control for sequential bootstrap.okd_installation— Full OKD cluster from scratchokd_pxe— PXE boot configuration for OKDsttest— Full OKD stack test with specific hardwarebrocade_snmp_server— Configure Brocade switch via SNMPopnsense_node_exporter— Node exporter on OPNsense firewall
Multi-Cluster
nats— NATS deployment on a clusternats-supercluster— NATS supercluster across multiple sitesmultisite_postgres— PostgreSQL with multi-site failover
Identity & Secrets
openbao— Deploy OpenBao vault (⚠️ uses NationTech DNS)zitadel— Deploy Zitadel identity provider (⚠️ uses NationTech DNS)
Cluster Services
cert_manager— Provision TLS certificatestenant— Create a multi-tenant namespaceoperatorhub_catalog— Install OperatorHub catalog sources
Development & Testing
cli— CLI tool with inventory discoverytui— Terminal UI demonstrationharmony_inventory_builder— Host discovery via subnet scan
Running Examples
# Build first
cargo build --release
# Run any example
cargo run -p example-postgresql
cargo run -p example-ntfy
cargo run -p example-tenant
For examples that need an existing Kubernetes cluster:
export KUBECONFIG=/path/to/your/kubeconfig
export HARMONY_USE_LOCAL_K3D=false
export HARMONY_AUTOINSTALL=false
cargo run -p example-monitoring
Notes on Private Infrastructure
Some examples use NationTech-hosted infrastructure by default (DNS domains like *.nationtech.io, *.harmony.mcd). These are not suitable for public use without modification. See the Getting Started Guide for the recommended public examples.