All checks were successful
Run Check Script / check (pull_request) Successful in 2m35s
Full ADR-022 protocol end to end. The state-machine brain and the operator's
commit decision are exhaustively unit-tested; OS side-effects sit behind a seam
so they're faked in tests and real on-device.
Contracts (harmony-reconciler-contracts):
- agent-upgrade marker + status KV buckets, AgentUpgradePhase, agent_version on
the heartbeat, Verb::UpgradeStop on the command protocol.
Shared (new crate harmony_downloadable_asset):
- download + SHA-256 verify, lifted from k3d's pub(crate) copy; k3d now depends
on it (DRY — second consumer is the agent). Tested with httptest.
Agent (harmony-fleet-agent):
- `drive`: Staging -> Verifying -> CutoverReady -> wait-for-operator-stop, with
heartbeat-timeout revert. 6 unit tests incl. every failure/rollback path.
- UpgradeExecutor seam + real SystemdUpgradeExecutor: download+verify,
`--self-test`, atomic symlink swap, systemd-run transient unit, revert. The
executor self-heals the on-disk layout so first-upgrade rollback is safe even
before M1 (preserves the running binary at its versioned path).
- `--self-test` flag; Verb::UpgradeStop handling gated by an armed
UpgradeStopSignal so only the cutover-waiting old agent acts (both agents are
subscribed). The agent never self-stops.
Operator (harmony-fleet-operator):
- upgrade_coordinator: sends the stop ONLY after independently observing the new
version's heartbeat (single source of truth); reflects currentVersion + the
upgrade phase onto the Device CR. 2 unit tests on the commit decision.
- FleetCommandsClient::upgrade_stop; Device.status.{currentVersion, upgrade}.
Deviations + flagged follow-ups (M1 clean install, libvirt vX->vX+1 e2e) in
ROADMAP/fleet_platform/ch4-agent-upgrade-status.md. Marker/status ride NATS KV
(survive operator restart, per Ch2).
Examples
This directory contains runnable examples demonstrating Harmony's capabilities. Each example is a self-contained program that can be run with cargo run -p example-<name>.
Quick Reference
| Example | Description | Local K3D | Existing Cluster | Hardware Needed |
|---|---|---|---|---|
postgresql |
Deploy a PostgreSQL cluster | ✅ | ✅ | — |
ntfy |
Deploy ntfy notification server | ✅ | ✅ | — |
tenant |
Create a multi-tenant namespace | ✅ | ✅ | — |
cert_manager |
Provision TLS certificates | ✅ | ✅ | — |
node_health |
Check Kubernetes node health | ✅ | ✅ | — |
monitoring |
Deploy Prometheus alerting | ✅ | ✅ | — |
monitoring_with_tenant |
Monitoring + tenant isolation | ✅ | ✅ | — |
operatorhub_catalog |
Install OperatorHub catalog | ✅ | ✅ | — |
validate_ceph_cluster_health |
Verify Ceph cluster health | — | ✅ | Rook/Ceph |
remove_rook_osd |
Remove a Rook OSD | — | ✅ | Rook/Ceph |
brocade_snmp_server |
Configure Brocade switch SNMP | — | ✅ | Brocade switch |
opnsense_node_exporter |
Node exporter on OPNsense | — | ✅ | OPNsense firewall |
opnsense_vm_integration |
Full OPNsense firewall automation (11 Scores) | ✅ | — | KVM/libvirt |
opnsense_pair_integration |
OPNsense HA pair with CARP failover | ✅ | — | KVM/libvirt |
okd_pxe |
PXE boot configuration for OKD | — | — | ✅ |
okd_installation |
Full OKD bare-metal install | — | — | ✅ |
okd_cluster_alerts |
OKD cluster monitoring alerts | — | ✅ | OKD cluster |
multisite_postgres |
Multi-site PostgreSQL failover | — | ✅ | Multi-cluster |
nats |
Deploy NATS messaging | — | ✅ | Multi-cluster |
nats-supercluster |
NATS supercluster across sites | — | ✅ | Multi-cluster |
lamp |
LAMP stack deployment | ✅ | ✅ | — |
openbao |
Deploy OpenBao vault | ✅ | ✅ | — |
zitadel |
Deploy Zitadel identity provider | ✅ | ✅ | — |
try_rust_webapp |
Rust webapp with packaging | ✅ | ✅ | Submodule |
rust |
Rust webapp with full monitoring | ✅ | ✅ | — |
rhob_application_monitoring |
RHOB monitoring setup | ✅ | ✅ | — |
sttest |
Full OKD stack test | — | — | ✅ |
application_monitoring_with_tenant |
App monitoring + tenant | — | ✅ | OKD cluster |
kube-rs |
Direct kube-rs client usage | ✅ | ✅ | — |
k8s_drain_node |
Drain a Kubernetes node | ✅ | ✅ | — |
k8s_write_file_on_node |
Write files to K8s nodes | ✅ | ✅ | — |
harmony_inventory_builder |
Discover hosts via subnet scan | ✅ | — | — |
cli |
CLI tool with inventory discovery | ✅ | — | — |
tui |
Terminal UI demonstration | ✅ | — | — |
Status Legend
| Symbol | Meaning |
|---|---|
| ✅ | Works out-of-the-box |
| — | Not applicable or requires specific setup |
By Category
Data Services
postgresql— Deploy a PostgreSQL cluster via CloudNativePGmultisite_postgres— Multi-site PostgreSQL with failoverpublic_postgres— Public-facing PostgreSQL (⚠️ uses NationTech DNS)
Kubernetes Utilities
node_health— Check node health in a clusterk8s_drain_node— Drain and reboot a nodek8s_write_file_on_node— Write files to nodesvalidate_ceph_cluster_health— Verify Ceph/Rook cluster healthremove_rook_osd— Remove an OSD from Rook/Cephkube-rs— Direct Kubernetes client usage demo
Monitoring & Alerting
monitoring— Deploy Prometheus alerting with Discord webhooksmonitoring_with_tenant— Monitoring with tenant isolationntfy— Deploy ntfy notification serverokd_cluster_alerts— OKD-specific cluster alerts
Application Deployment
try_rust_webapp— Deploy a Rust webapp with packaging (⚠️ requirestryrust.orgsubmodule)rust— Rust webapp with full monitoring featuresrhob_application_monitoring— Red Hat Observability Stack monitoringlamp— LAMP stack deployment (⚠️ uses NationTech DNS)application_monitoring_with_tenant— App monitoring with tenant isolation
Infrastructure & Bare Metal
opnsense_vm_integration— Recommended demo. Boot an OPNsense VM and configure it with 11 Scores (load balancer, DHCP, TFTP, VLANs, firewall rules, NAT, VIPs, LAGG). Fully automated, requires only KVM. See the detailed guide.opnsense_pair_integration— Boot two OPNsense VMs and configure a CARP HA firewall pair withFirewallPairTopologyandCarpVipScore. Demonstrates NIC link control for sequential bootstrap.okd_installation— Full OKD cluster from scratchokd_pxe— PXE boot configuration for OKDsttest— Full OKD stack test with specific hardwarebrocade_snmp_server— Configure Brocade switch via SNMPopnsense_node_exporter— Node exporter on OPNsense firewall
Multi-Cluster
nats— NATS deployment on a clusternats-supercluster— NATS supercluster across multiple sitesmultisite_postgres— PostgreSQL with multi-site failover
Identity & Secrets
openbao— Deploy OpenBao vault (⚠️ uses NationTech DNS)zitadel— Deploy Zitadel identity provider (⚠️ uses NationTech DNS)
Cluster Services
cert_manager— Provision TLS certificatestenant— Create a multi-tenant namespaceoperatorhub_catalog— Install OperatorHub catalog sources
Development & Testing
cli— CLI tool with inventory discoverytui— Terminal UI demonstrationharmony_inventory_builder— Host discovery via subnet scan
Running Examples
# Build first
cargo build --release
# Run any example
cargo run -p example-postgresql
cargo run -p example-ntfy
cargo run -p example-tenant
For examples that need an existing Kubernetes cluster:
export KUBECONFIG=/path/to/your/kubeconfig
export HARMONY_USE_LOCAL_K3D=false
export HARMONY_AUTOINSTALL=false
cargo run -p example-monitoring
Notes on Private Infrastructure
Some examples use NationTech-hosted infrastructure by default (DNS domains like *.nationtech.io, *.harmony.mcd). These are not suitable for public use without modification. See the Getting Started Guide for the recommended public examples.