Customer apps frequently need a one-shot setup step (DB migration,
config render, cache warm-up) to succeed before the long-running
service starts. Without init containers each customer either inlines
the step into the service entrypoint (slow, racy, no failure surface)
or bolts on a sidecar that the platform can't introspect. This change
adds k8s-style init containers at the score layer so the contract is
the same one the customer already knows.
Score:
- New `InitContainer { name, image, args, env, volumes, timeout }`
in `harmony::modules::podman`.
- `PodmanV0Score.init_containers: Vec<InitContainer>` with
`#[serde(default)]` — pre-init-container wire payloads parse as an
empty vec and behave unchanged.
- `DEFAULT_INIT_CONTAINER_TIMEOUT = 300s`; timeout serializes as
whole seconds for operator readability.
- Idempotency is the customer's contract — documented at module
level: init containers re-run on every reconcile that needs a
fresh main container set.
Runtime contract:
- `ContainerRuntime::run_to_completion(spec, timeout) -> RunOutcome`
added to the domain trait. `RunOutcome::Exited { exit_code }`
vs `TimedOut { waited }` — distinct arms because the caller's
failure path is different (operator gets the exit code for
actionable diagnosis).
- Init containers are NOT surfaced via `list_managed_services`;
they're removed after they exit so the host's managed-container
surface stays bounded to long-running services.
PodmanTopology implementation:
- Pre-remove any prior container with the same name (retry-safe).
- Restart policy forced to `No` — a retrying init defeats the
run-to-completion contract.
- `tokio::time::timeout` around `podman wait`; force-remove + return
`TimedOut` on deadline.
- Single 200ms retry on inspect for the libpod race where state can
briefly read `running` between `wait` returning and conmon writing
the exit code.
- `INIT_CONTAINER_LABEL` on every init container so operators can
`podman ps -a --filter label=...` to spot init failures.
Interpret:
- Init containers run sequentially before any service. Non-zero exit
or timeout fails the deployment with a typed `InterpretError`
carrying the container name + cause.
- Success message reports both counts.
Tests (in tree):
- 3 new wire-format tests in `podman::score`: roundtrip, default
timeout hydration, ordering preservation.
- All 10 existing podman::score tests still pass; legacy roundtrip
test now also asserts `init_containers.is_empty()` as a wire-compat
canary.
Call-site updates (5 sites) — all existing constructors of
`PodmanV0Score` add `init_containers: vec![]`: harmony_apply_deployment
example, fleet_load_test example, operator e2e, vm_deploy_lifecycle
e2e, vm_isolation e2e.
Deferred: per-version "run-once" semantics (customer can build with a
marker file today); the agent-side handler for surfacing init logs to
the operator dashboard (covered by the logs companion PR's deferred
work).
Examples
This directory contains runnable examples demonstrating Harmony's capabilities. Each example is a self-contained program that can be run with cargo run -p example-<name>.
Quick Reference
| Example | Description | Local K3D | Existing Cluster | Hardware Needed |
|---|---|---|---|---|
postgresql |
Deploy a PostgreSQL cluster | ✅ | ✅ | — |
ntfy |
Deploy ntfy notification server | ✅ | ✅ | — |
tenant |
Create a multi-tenant namespace | ✅ | ✅ | — |
cert_manager |
Provision TLS certificates | ✅ | ✅ | — |
node_health |
Check Kubernetes node health | ✅ | ✅ | — |
monitoring |
Deploy Prometheus alerting | ✅ | ✅ | — |
monitoring_with_tenant |
Monitoring + tenant isolation | ✅ | ✅ | — |
operatorhub_catalog |
Install OperatorHub catalog | ✅ | ✅ | — |
validate_ceph_cluster_health |
Verify Ceph cluster health | — | ✅ | Rook/Ceph |
remove_rook_osd |
Remove a Rook OSD | — | ✅ | Rook/Ceph |
brocade_snmp_server |
Configure Brocade switch SNMP | — | ✅ | Brocade switch |
opnsense_node_exporter |
Node exporter on OPNsense | — | ✅ | OPNsense firewall |
opnsense_vm_integration |
Full OPNsense firewall automation (11 Scores) | ✅ | — | KVM/libvirt |
opnsense_pair_integration |
OPNsense HA pair with CARP failover | ✅ | — | KVM/libvirt |
okd_pxe |
PXE boot configuration for OKD | — | — | ✅ |
okd_installation |
Full OKD bare-metal install | — | — | ✅ |
okd_cluster_alerts |
OKD cluster monitoring alerts | — | ✅ | OKD cluster |
multisite_postgres |
Multi-site PostgreSQL failover | — | ✅ | Multi-cluster |
nats |
Deploy NATS messaging | — | ✅ | Multi-cluster |
nats-supercluster |
NATS supercluster across sites | — | ✅ | Multi-cluster |
lamp |
LAMP stack deployment | ✅ | ✅ | — |
openbao |
Deploy OpenBao vault | ✅ | ✅ | — |
zitadel |
Deploy Zitadel identity provider | ✅ | ✅ | — |
try_rust_webapp |
Rust webapp with packaging | ✅ | ✅ | Submodule |
rust |
Rust webapp with full monitoring | ✅ | ✅ | — |
rhob_application_monitoring |
RHOB monitoring setup | ✅ | ✅ | — |
sttest |
Full OKD stack test | — | — | ✅ |
application_monitoring_with_tenant |
App monitoring + tenant | — | ✅ | OKD cluster |
kube-rs |
Direct kube-rs client usage | ✅ | ✅ | — |
k8s_drain_node |
Drain a Kubernetes node | ✅ | ✅ | — |
k8s_write_file_on_node |
Write files to K8s nodes | ✅ | ✅ | — |
harmony_inventory_builder |
Discover hosts via subnet scan | ✅ | — | — |
cli |
CLI tool with inventory discovery | ✅ | — | — |
tui |
Terminal UI demonstration | ✅ | — | — |
Status Legend
| Symbol | Meaning |
|---|---|
| ✅ | Works out-of-the-box |
| — | Not applicable or requires specific setup |
By Category
Data Services
postgresql— Deploy a PostgreSQL cluster via CloudNativePGmultisite_postgres— Multi-site PostgreSQL with failoverpublic_postgres— Public-facing PostgreSQL (⚠️ uses NationTech DNS)
Kubernetes Utilities
node_health— Check node health in a clusterk8s_drain_node— Drain and reboot a nodek8s_write_file_on_node— Write files to nodesvalidate_ceph_cluster_health— Verify Ceph/Rook cluster healthremove_rook_osd— Remove an OSD from Rook/Cephkube-rs— Direct Kubernetes client usage demo
Monitoring & Alerting
monitoring— Deploy Prometheus alerting with Discord webhooksmonitoring_with_tenant— Monitoring with tenant isolationntfy— Deploy ntfy notification serverokd_cluster_alerts— OKD-specific cluster alerts
Application Deployment
try_rust_webapp— Deploy a Rust webapp with packaging (⚠️ requirestryrust.orgsubmodule)rust— Rust webapp with full monitoring featuresrhob_application_monitoring— Red Hat Observability Stack monitoringlamp— LAMP stack deployment (⚠️ uses NationTech DNS)application_monitoring_with_tenant— App monitoring with tenant isolation
Infrastructure & Bare Metal
opnsense_vm_integration— Recommended demo. Boot an OPNsense VM and configure it with 11 Scores (load balancer, DHCP, TFTP, VLANs, firewall rules, NAT, VIPs, LAGG). Fully automated, requires only KVM. See the detailed guide.opnsense_pair_integration— Boot two OPNsense VMs and configure a CARP HA firewall pair withFirewallPairTopologyandCarpVipScore. Demonstrates NIC link control for sequential bootstrap.okd_installation— Full OKD cluster from scratchokd_pxe— PXE boot configuration for OKDsttest— Full OKD stack test with specific hardwarebrocade_snmp_server— Configure Brocade switch via SNMPopnsense_node_exporter— Node exporter on OPNsense firewall
Multi-Cluster
nats— NATS deployment on a clusternats-supercluster— NATS supercluster across multiple sitesmultisite_postgres— PostgreSQL with multi-site failover
Identity & Secrets
openbao— Deploy OpenBao vault (⚠️ uses NationTech DNS)zitadel— Deploy Zitadel identity provider (⚠️ uses NationTech DNS)
Cluster Services
cert_manager— Provision TLS certificatestenant— Create a multi-tenant namespaceoperatorhub_catalog— Install OperatorHub catalog sources
Development & Testing
cli— CLI tool with inventory discoverytui— Terminal UI demonstrationharmony_inventory_builder— Host discovery via subnet scan
Running Examples
# Build first
cargo build --release
# Run any example
cargo run -p example-postgresql
cargo run -p example-ntfy
cargo run -p example-tenant
For examples that need an existing Kubernetes cluster:
export KUBECONFIG=/path/to/your/kubeconfig
export HARMONY_USE_LOCAL_K3D=false
export HARMONY_AUTOINSTALL=false
cargo run -p example-monitoring
Notes on Private Infrastructure
Some examples use NationTech-hosted infrastructure by default (DNS domains like *.nationtech.io, *.harmony.mcd). These are not suitable for public use without modification. See the Getting Started Guide for the recommended public examples.