- Add nodes_to_backend_server() function to include both control plane and worker nodes
- Update public services (ports 80, 443) to use worker-inclusive backend pool
- Add comprehensive tests covering all backend configurations
- Add documentation with OKD reference link and usage examples
- Updated harmony-k8s doc tests to import from harmony_k8s instead of harmony
- Changed CloudNativePgOperatorScore::default() to default_openshift()
This ensures doc tests work correctly after moving K8sClient to the harmony-k8s crate.
Update Zitadel deployment to use correct username format (admin@zitadel.{host}), generate secure passwords with required complexity (uppercase, lowercase, digit, symbol), configure edge TLS termination for OpenShift, and add cert-manager annotations. Also refactor password generation to ensure all complexity requirements are met.
First use case is creating a privileged pod allowing writing to nodes on
openshift family clusters. This requires creating the clusterrolebinding
and pod and other resources.
Refactors the `HelmChartInterpret` to move away from the `helm-wrapper-rs` crate in favor of a custom command builder pattern. This allows the `HelmCommand` trait to provide topology-specific configurations, such as `kubeconfig` and `kube-context`, directly to the `helm` CLI.
- Implements `get_helm_command` for `K8sAnywhereTopology` to inject configuration flags.
- Replaces `DefaultHelmExecutor` with a manual `Command` construction in `run_helm_command`.
- Updates `HelmChartInterpret` to pass the topology through to repository and installation logic.
- Cleans up unused imports and removes the temporary `HelmCommand` implementation for `LocalhostTopology`.
These types of Topologies will orchestrate behavior in regular Topologies.
For example, a FailoverTopology is a Higher Order, it will orchestrate its capabilities between a primary and a replica topology. A great use case for this is a database deployment. The FailoverTopology will deploy both instances, connect them, and the able to execute the appropriate actions to promote de replica to primary and revert back to original state.
Other use cases are ShardedTopology, DecentralizedTopology, etc.
The NetworkManager logic was implemented directly into the `HaClusterTopology`, which wasn't directly its concern and prevented us from being able to reuse that NetworkManaager implementations in the future for a different Topology.
* Extract a `NetworkManager` trait
* Implement a `OpenShiftNmStateNetworkManager` for `NetworkManager`
* Dynamically instantiate the NetworkManager in the Topology to delegate calls to it
Reviewed-on: #183
Reviewed-by: johnride <jg@nationtech.io>
In order to query the current network state `NodeNetworkState` and to apply a `NodeNetworkConfigurationPolicy` for a given node, we first needed to find its hostname. As all we had was the UUID of a node.
We had different options available (e.g. updating the Harmony Inventory Agent to retrieve it, store it in the OKD installation pipeline on assignation, etc.). But for the sake of simplicity and for better flexibility (e.g. being able to run this score on a cluster that wasn't setup with Harmony), the `hostname` was retrieved directly in the cluster by running the equivalent of `kubectl get nodes -o yaml` and matching the nodes with the system UUID.
### Other changes
* Find the next available bond id for a node
* Apply a network config policy for a node (configuring a bond in our case)
* Adjust the CRDs for NMState
Note: to see a quick demo, watch the recording in #183
Reviewed-on: #182
Reviewed-by: johnride <jg@nationtech.io>
With this work, no need to initialize a full HAClusterTopology to run
opnsense scores.
Also added an example showing how to use it and perform basic
operations.
Made a video out of it, might publish it at some point!
## Description
* Replace the CatalogSource approach to install the OperatorHub.io catalog by a more simple & straightforward way to install NMState
* Improve logging
* Add report summarizing the host network configuration that was applied (which host, bonds, port-channels)
* Fix command to find next available port channel id
## Extra info
Using the `apply_url` approach to install the NMState operator isn't the best approach: it's harder to maintain and upgrade. But it helps us achieve waht we wanted for now: install the NMState Operator to configure bonds on a host.
The preferred approach, installing an operator from the OperatorHub.io catalog, didn't work for now. We had a timeout error with DeadlineExceeded probably caused by an insufficient CPU/Memory allocation to query such a big catalog, even though we tweaked the RAM allocation (we couldn't find a way to do it for CPU).
Spent too much time on this so we stopped these efforts for now. It would be good to get back to it when we need to install something else from a custom catalog.
Reviewed-on: #175
k8s returns None rather than zero when checking deployment for replicas
exec_app requires commands 's' and '-c' to run correctly
Reviewed-on: #120
Co-authored-by: Willem <wrolleman@nationtech.io>
Co-committed-by: Willem <wrolleman@nationtech.io>
The previous implementation blindly added HAProxy components without checking for existing configurations on the same port, which caused duplicate entries and errors when a service was updated.
This commit refactors the logic to a robust "remove-then-add" strategy. The configure_service method now finds and removes any existing frontend and its dependent components (backend, servers, health check) before adding the new, complete service definition.
This change makes the process fully idempotent, preventing configuration drift and ensuring a predictable state.
Co-authored-by: Ian Letourneau <letourneau.ian@gmail.com>
Reviewed-on: #129
* Expose a high-level `brocade::init()` function to connect to a Brocade switch and automatically pick the best implementation based on its OS and version
* Implement a client for Brocade switches running on Network Operating System (NOS)
* Implement a client for older Brocade switches running on FastIron (partial implementation)
The architecture for the library is based on 3 layers:
1. The `BrocadeClient` trait to describe the available capabilities to
interact with a Brocade switch. It is partly opinionated in order to
offer higher level features to group multiple commands into a single
function (e.g. create a port channel). Its implementations are
basically just the commands to run on the switch and the functions to
parse the output.
2. The `BrocadeShell` struct to make it easier to authenticate, send commands, and interact with the switch.
3. The `ssh` module to actually connect to the switch over SSH and execute the commands.
With time, we will add support for more Brocade switches and their various OS/versions. If needed, shared behavior could be extracted into a separate module to make it easier to add new implementations.
* it was named `hurl!` instead of just `url!` because it was clashing with the crate `url` so we would have been forced to use it with `harmony_macros::url!` which is less sexy
Reviewed-on: #135
* Okd needs to use the cluster observability operator in order to deploy namespaced prometheuses and alertmanagers
* allow namespaced deployments of alertmanager and prometheuses as well as its associated rules, etc.
Co-authored-by: Ian Letourneau <ian@noma.to>
Reviewed-on: #134
Co-authored-by: Willem <wrolleman@nationtech.io>
Co-committed-by: Willem <wrolleman@nationtech.io>
## Fully automated inventory gathering now works!
Boot up harmony_inventory_agent with `cargo run -p harmony_inventory_agent`
Launch the DiscoverInventoryAgentScore , currently available this way :
`RUST_LOG=info cargo run -p example-cli -- -f Discover -y`
And you will have automatically all hosts saved to the database. Run `cargo sqlx setup` if you have not done it yet.
Co-authored-by: Ian Letourneau <ian@noma.to>
Reviewed-on: #127
Co-authored-by: Jean-Gabriel Gill-Couture <jg@nationtech.io>
Co-committed-by: Jean-Gabriel Gill-Couture <jg@nationtech.io>
The process will setup DHCP dnsmasq on opnsense to boot the correct ipxe file depending on the architecture
Then ipxe will chainload to either a mac-specific ipxe boot file or the fallback inventory boot file
Then a kickstart pre script will setup the cluster ssh key to allow ssh connections to the machine and also setup and start harmony_inventory_agent to allow being scraped
Note: there is a bug with the inventory agent currently, it cannot find lsmod on centos stream 9, will fix this soon
Reviewed-on: #111
This pull request introduces a comprehensive and ergonomic secret management system via a new harmony-secret crate.
What's Done
New harmony-secret Crate:
A new crate dedicated to secret management, providing a clean, static API: SecretManager::get::<MySecret>() and SecretManager::set(&my_secret).
A #[derive(Secret)] procedural macro that automatically uses the struct's name as the secret key, simplifying usage.
An async SecretStore trait to support various backend implementations.
Two Secret Store Implementations:
LocalFileSecretStore: A simple file-based store that saves secrets as JSON in the user's data directory. Ideal for local development and testing.
InfisicalSecretStore: A production-ready implementation that integrates with Infisical for centralized, secure secret management.
Configuration via Environment Variables:
The secret store is selected at runtime via the HARMONY_SECRET_STORE environment variable (file or infisical).
Infisical integration is configured through HARMONY_SECRET_INFISICAL_* variables.
What's Not Done (Future Work)
Automated Infisical Setup: The initial configuration for the Infisical backend is currently manual. Developers must create a project and a Universal Auth identity in Infisical and set the corresponding environment variables to run tests or use the backend. The new test_harmony_secret_infisical.sh script serves as a clear example of the required variables.
This new secrets module provides a solid and secure foundation for managing credentials for components like OPNsense, Kubernetes, and other infrastructure services going forward. Even with the manual first-time setup for Infisical, this architecture is robust enough to serve our needs for the foreseeable future.
* define Ntfy ingress (naive implementation) based on current target
* use patched Ntfy Helm Chart
* create Ntfy main user only if needed
* add info logs
* better error bubbling
* instrument feature installations
* upgrade prometheus alerting charts if already installed
* harmony_composer params to control deployment `target` and `profile`
Co-authored-by: Ian Letourneau <letourneau.ian@gmail.com>
Co-authored-by: Jean-Gabriel Gill-Couture <jg@nationtech.io>
Reviewed-on: #107
The multiprogress wasn't used properly and leading to conflicting progress bars (within our own progress bars, as well as the log wrapper).
This PR introduce a layer on top of `indicatif::MultiProgress` to properly handle sections of progress bars, where we can dynamically add/update/remove progress bars from any sections.
We can see in the demo that new sections + progress bars are added on the fly and that extra logs (e.g. info logs) are appended on top of the progress bars.
Progress are also grouped together based on their parent score.
Co-authored-by: Ian Letourneau <letourneau.ian@gmail.com>
Co-authored-by: johnride <jg@nationtech.io>
Reviewed-on: #101
The CI pipeline (`./check.sh`) was failing because of test errors, which was caused by the instrumentation framework complaining that no subscribers/listeners were registered.
Instead of setting up all tests to run with a dummy subscriber, move the implementation of the instrumentation behind a feature flag so that it runs only for tests.
There's a catch though: the `#[cfg(test)]` directive works only when directly testing the crate. If a crate `A` depends on another crate `B`, `B` will be compiled as usual (aka not in test mode) which will not trigger the `test` flag.
So we need to introduce our own `testing` feature flag for `harmony` core and import it with that flag (only during dev/test).
More info: https://github.com/rust-lang/rust/issues/59168
Co-authored-by: Ian Letourneau <letourneau.ian@gmail.com>
Reviewed-on: https://git.nationtech.io/NationTech/harmony/pulls/102
# Harmony : Open-source infrastructure orchestration that treats your platform like first-class code
# Harmony
**Infrastructure orchestration that treats your platform like first-class code.**
Harmony is an open-source framework that brings the rigor of software engineering to infrastructure management. Write Rust code to define what you want, and Harmony handles the rest — from local development to production clusters.
Modern infrastructure is messy. Your Kubernetes cluster needs monitoring. Your bare-metal servers need provisioning. Your applications need deployments. Each comes with its own tooling, its own configuration format, and its own failure modes.
### Deploy anywhere
**What if you could describe your entire platform in one consistent language?**
From a **developer laptop** to a **global production cluster**, a single **source of truth** drives the **full software lifecycle.**
That's Harmony. It unifies project scaffolding, infrastructure provisioning, application deployment, and day-2 operations into a single strongly-typed Rust codebase.
---
## 1 · The Harmony Philosophy
## Three Principles That Make the Difference
Infrastructure is essential, but it shouldn’t be your core business. Harmony is built on three guiding principles that make modern platforms reliable, repeatable, and easy to reason about.
| **Infrastructure as Resilient Code** | Replace sprawling YAML and bash scripts with type-safe Rust. Test, refactor, and version your platform just like application code. |
| **Prove It Works — Before You Deploy** | Harmony uses the compiler to verify that your application’s needs match the target environment’s capabilities at **compile-time**, eliminating an entire class of runtime outages. |
| **One Unified Model** | Software and infrastructure are a single system. Harmony models them together, enabling deep automation—from bare-metal servers to Kubernetes workloads—with zero context switching. |
These principles surface as simple, ergonomic Rust APIs that let teams focus on their product while trusting the platform underneath.
| Principle | What It Means |
|-----------|---------------|
| **Infrastructure as Resilient Code** | Stop fighting with YAML and bash. Write type-safe Rust that you can test, version, and refactor like any other code. |
| **Prove It Works Before You Deploy** | Harmony verifies at _compile time_ that your application can actually run on your target infrastructure. No more "the config looks right but it doesn't work" surprises. |
| **One Unified Model**| Software and infrastructure are one system. Deploy from laptop to production cluster without switching contexts or tools. |
---
## 2 · Quick Start
## How It Works: The Core Concepts
The snippet below spins up a complete **production-grade LAMP stack** with monitoring. Swap it for your own scores to deploy anything from microservices to machine-learning pipelines.
Harmony is built around three concepts that work together:
### Score — "What You Want"
A `Score` is a declarative description of desired state. Think of it as a "recipe" that says _what_ you want without specifying _how_ to get there.
```rust
// "I want a PostgreSQL cluster running with default settings"
A `Topology` represents your infrastructure environment and its capabilities. It answers the question: "What can this environment actually do?"
```rust
// Deploy to a local K3D cluster, or any Kubernetes cluster via environment variables
K8sAnywhereTopology::from_env()
```
### Interpret — "How It Happens"
An `Interpret` is the execution logic that connects your `Score` to your `Topology`. It translates "what you want" into "what the infrastructure does."
**The Compile-Time Check:** Before your code ever runs, Harmony verifies that your `Score` is compatible with your `Topology`. If your application needs a feature your infrastructure doesn't provide, you get a compile error — not a runtime failure.
| `HARMONY_K8S_CONTEXT` | _none_ | Use a specific kubeconfig context |
| `HARMONY_PUBLIC_DOMAIN` | _none_ | Public domain for ingress endpoints |
- **Community** – discussions and roadmap live in [GitLab issues](https://git.nationtech.io/nationtech/harmony/-/issues). PRs, ideas, and feedback are welcome!
To connect to an existing Kubernetes cluster instead of provisioning K3D:
```bash
# Point to your kubeconfig
exportKUBECONFIG=/path/to/your/kubeconfig
exportHARMONY_USE_LOCAL_K3D=false
exportHARMONY_AUTOINSTALL=false
# Then run
cargo run -p example-postgresql
```
---
## 6 · License
## Documentation
| I want to... | Start here |
|--------------|------------|
| Understand the core concepts | [Core Concepts](./docs/concepts.md) |
| Deploy my first application | [Getting Started Guide](./docs/guides/getting-started.md) |
> We choose a strong copyleft license to ensure the project—and every improvement to it—remains open and benefits the entire community. Fork it, enhance it, even out-innovate us; just keep it open.
> We choose a strong copyleft license to ensure the project—and every improvement to it—remains open and benefits the entire community.
See [LICENSE](LICENSE) for the full text.
---
_Made with ❤️ & 🦀 by the NationTech and the Harmony community_
_Made with ❤️ & 🦀 by NationTech and the Harmony community_
- **Visual:** Clean and simple. Your company logo (NationTech) and the Harmony logo.
---
#### **Slide 2: The YAML Labyrinth**
**Goal:** Get every head in the room nodding in agreement. Start with their world, not yours.
- **Visual:**
- Option A: "The Pull Request from Hell". A screenshot of a GitHub pull request for a seemingly minor change that touches dozens of YAML files across multiple directories. A sea of red and green diffs that is visually overwhelming.
- Option B: A complex flowchart connecting dozens of logos: Terraform, Ansible, K8s, Helm, etc.
- **Narration:**\
[...ADD SOMETHING FOR INTRODUCTION...]\
"We love the power that tools like Kubernetes and the CNCF landscape have given us. But let's be honest... when did our infrastructure code start looking like _this_?"\
"We have GitOps, which is great. But it often means we're managing this fragile cathedral of YAML, Helm charts, and brittle scripts. We spend more time debugging indentation and tracing variables than we do building truly resilient systems."
---
#### **Slide 3: The Real Cost of Infrastructure**
- **Visual:** "The Jenga Tower of Tools". A tall, precarious Jenga tower where each block is the logo of a different tool (Terraform, K8s, Helm, Ansible, Prometheus, ArgoCD, etc.). One block near the bottom is being nervously pulled out.
- **Narration:**
"The real cost isn't just complexity; it's the constant need to choose, learn, integrate, and operate a dozen different tools, each with its own syntax and failure modes. It's the nagging fear that a tiny typo in a config file could bring everything down. Click-ops isn't the answer, but the current state of IaC feels like we've traded one problem for another."
---
#### **Slide 4: The Broken Promise of "Code"**
**Goal:** Introduce the core idea before introducing the product. This makes the solution feel inevitable.
- **(Initial Visual):** A two-panel slide.
- **Left Panel Title: "The Plan"** - A terminal showing a green, successful `terraform plan` output.
- **Right Panel Title: "The Reality"** - The _next_ screen in the terminal, showing the `terraform apply` failing with a cascade of red error text.
- **Narration:**
"We call our discipline **Infrastructure as Code**. And we've all been here. Our 'compiler' is a `terraform plan` that says everything looks perfect. We get the green light."
(Pause for a beat)
"And then we `apply`, and reality hits. It fails halfway through, at runtime, when it's most expensive and painful to fix."
**(Click to transition the slide)**
- **(New Visual):** The entire slide is replaced by a clean screenshot of a code editor (like nvim 😉) showing Harmony's Rust DSL. A red squiggly line is under a config line. The error message is clear in the "Problems" panel: `error: Incompatible deployment. Production target 'gcp-prod-cluster' requires a StorageClass with 'snapshots' capability, but 'standard-sc' does not provide it.`
- **Narration (continued):**
"In software development, we solved these problems years ago. We don't accept 'it compiled, but crashed on startup'. We have real tools, type systems, compilers, test frameworks, and IDEs that catch our mistakes before they ever reach production. **So, what if we could treat our entire infrastructure... like a modern, compiled application?**"
"What if your infrastructure code could get compile-time checks, straight into the editor... instead of runtime panics and failures at 3 AM in production?"
---
#### **Slide 5: Introducing Harmony**
**Goal:** Introduce Harmony as the answer to the "What If?" question.
- **Visual:** The Harmony logo, large and centered.
- **Tagline:** `Infrastructure in type-safe Rust. No YAML required.`
- **Narration:**
"This is Harmony. It's an open-source orchestrator that lets you define your entire stack — from a dev laptop to a multi-site bare-metal cluster—in a single, type-safe Rust codebase."
---
#### **Slide 6: Before & After**
- **Visual:** A side-by-side comparison. Left side: A screen full of complex, nested YAML. Right side: 10-15 lines of clean, readable Harmony Rust DSL that accomplishes the same thing.
- **Narration:**
"This is the difference. On the left, the fragile world of strings and templates. On the right, a portable, verifiable program that describes your apps, your infra, and your operations. We unify scaffolding, provisioning, and Day-2 ops, all verified by the Rust compiler. But enough slides... let's see it in action."
---
#### **Slide 7: Live Demo: Zero to Monitored App**
**Goal:** Show, don't just tell. Make it look effortless. This is where you build the "dream."
- **Visual:** Your terminal/IDE, ready to go.
- **Narration Guide:**
"Okay, for this demo, we're going to take a standard web app from GitHub. Nothing special about it."
_(Show the repo)_
"Now, let's bring it into Harmony. This is the entire definition we need to describe the application and its needs."
_(Show the Rust DSL)_
"First, let's run it locally on k3d. The exact same definition for dev as for prod."
_(Deploy locally, show it works)_
"Cool. But a real app needs monitoring. In Harmony, that's just adding a feature to our code."
_(Uncomment one line: `.with_feature(Monitoring)` and redeploy)_
"And just like that, we have a fully configured Prometheus and Grafana stack, scraping our app. No YAML, no extra config."
"Finally, let's push this to our production staging cluster. We just change the target and specify our multi-site Ceph storage."
_(Deploy to the remote cluster)_
"And there it is. We've gone from a simple web app to a monitored, enterprise-grade service in minutes."
---
#### **Slide 8: Live Demo: Embracing Chaos**
**Goal:** Prove the "predictable" and "resilient" claims in the most dramatic way possible.
- **Visual:** A slide showing a map or diagram of your distributed infrastructure (the different data centers). Then switch back to your terminal.
- **Narration Guide:**
"This is great when things are sunny. But production is chaos. So... let's break things. On purpose."
"First, a network failure." _(Kill a switch/link, show app is still up)_
"Now, let's power off a storage server." _(Force off a server, show Ceph healing and the app is unaffected)_
"How about a control plane node?" _(Force off a k8s control plane, show the cluster is still running)_
"Okay, for the grand finale. What if we have a cascading failure? I'm going to kill _another_ storage server. This should cause a total failure in this data center."
_(Force off the second server, narrate what's happening)_
"And there it is... Ceph has lost quorum in this site... and Harmony has automatically failed everything over to our other datacenter. The app is still running."
---
#### **Slide 9: The New Reality**
**Goal:** Summarize the dream and tell the audience what you want them to do.
- **Visual:** The clean, simple Harmony Rust DSL code from Slide 6. A summary of what was just accomplished is listed next to it: `✓ GitHub to Prod in minutes`, `✓ Type-Safe Validation`, `✓ Built-in Monitoring`, `✓ Automated Multi-Site Failover`.
- **Narration:**
"So, in just a few minutes, we went from a simple web app to a multi-site, monitored, and chaos-proof production deployment. We did it with a small amount of code that is easy to read, easy to verify, and completely portable. This is our vision: to offload the complexity, and make infrastructure simple, predictable, and even fun again."
---
#### **Slide 10: Join Us**
- **Visual:** A clean, final slide with QR codes and links.
- GitHub Repo (`github.com/nation-tech/harmony`)
- Website (`harmony.sh` or similar)
- Your contact info (`jg@nation.tech` / LinkedIn / Twitter)
- **Narration:**
"Harmony is open-source, AGPLv3. We believe this is the future, but we're just getting started. We know this crowd has great infrastructure minds out there, and we need your feedback. Please, check out the project on GitHub. Star it if you like what you see. Tell us what's missing. Let's build this future together. Thank you."
Not much here yet, see the `adr` folder for now. More to come in time!
# Harmony Documentation Hub
Welcome to the Harmony documentation. This is the main entry point for learning everything from core concepts to building your own Score, Topologies, and Capabilities.
## 1. Getting Started
If you're new to Harmony, start here:
- [**Getting Started Guide**](./guides/getting-started.md): A step-by-step tutorial that takes you from an empty project to deploying your first application.
- [**Core Concepts**](./concepts.md): A high-level overview of the key concepts in Harmony: `Score`, `Topology`, `Capability`, `Inventory`, `Interpret`, ...
## 2. Use Cases & Examples
See how to use Harmony to solve real-world problems.
- [**PostgreSQL on Local K3D**](./use-cases/postgresql-on-local-k3d.md): Deploy a production-grade PostgreSQL cluster on a local K3D cluster. The fastest way to get started.
- [**OKD on Bare Metal**](./use-cases/okd-on-bare-metal.md): A detailed walkthrough of bootstrapping a high-availability OKD cluster from physical hardware.
## 3. Component Catalogs
Discover existing, reusable components you can use in your Harmony projects.
- [**Scores Catalog**](./catalogs/scores.md): A categorized list of all available `Scores` (the "what").
- [**Topologies Catalog**](./catalogs/topologies.md): A list of all available `Topologies` (the "where").
- [**Capabilities Catalog**](./catalogs/capabilities.md): A list of all available `Capabilities` (the "how").
## 4. Developer Guides
Ready to build your own components? These guides show you how.
- [**Writing a Score**](./guides/writing-a-score.md): Learn how to create your own `Score` and `Interpret` logic to define a new desired state.
- [**Writing a Topology**](./guides/writing-a-topology.md): Learn how to model a new environment (like AWS, GCP, or custom hardware) as a `Topology`.
- [**Adding Capabilities**](./guides/adding-capabilities.md): See how to add a `Capability` to your custom `Topology`.
## 5. Architecture Decision Records
Harmony's design is documented through Architecture Decision Records (ADRs). See the [ADR Overview](./adr/README.md) for a complete index of all decisions.
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.