adr: Higher order topologies

These types of Topologies will orchestrate behavior in regular Topologies.

For example, a FailoverTopology is a Higher Order, it will orchestrate its capabilities between a primary and a replica topology. A great use case for this is a database deployment. The FailoverTopology will deploy both instances, connect them, and the able to execute the appropriate actions to promote de replica to primary and revert back to original state.

Other use cases are ShardedTopology, DecentralizedTopology, etc.

adr/015-higher-order-topologies.md

@@ -0,0 +1,114 @@
+# Architecture Decision Record: Higher-Order Topologies
+
+**Initial Author:** Jean-Gabriel Gill-Couture  
+**Initial Date:** 2025-12-08  
+**Last Updated Date:** 2025-12-08  
+
+## Status
+
+Implemented
+
+## Context
+
+Harmony models infrastructure as **Topologies** (deployment targets like `K8sAnywhereTopology`, `LinuxHostTopology`) implementing **Capabilities** (tech traits like `PostgreSQL`, `Docker`).
+
+**Higher-Order Topologies** (e.g., `FailoverTopology<T>`) compose/orchestrate capabilities *across* multiple underlying topologies (e.g., primary+replica `T`).
+
+Naive design requires manual `impl Capability for HigherOrderTopology<T>` *per T per capability*, causing:
+- **Impl explosion**: N topologies × M capabilities = N×M boilerplate.
+- **ISP violation**: Topologies forced to impl unrelated capabilities.
+- **Maintenance hell**: New topology needs impls for *all* orchestrated capabilities; new capability needs impls for *all* topologies/higher-order.
+- **Barrier to extension**: Users can't easily add topologies without todos/panics.
+
+This makes scaling Harmony impractical as ecosystem grows.
+
+## Decision
+
+Use **blanket trait impls** on higher-order topologies to *automatically* derive orchestration:
+
+````rust
+/// Higher-Order Topology: Orchestrates capabilities across sub-topologies.
+pub struct FailoverTopology<T> {
+    /// Primary sub-topology.
+    primary: T,
+    /// Replica sub-topology.
+    replica: T,
+}
+
+/// Automatically provides PostgreSQL failover for *any* `T: PostgreSQL`.
+/// Delegates to primary for queries; orchestrates deploy across both.
+#[async_trait]
+impl<T: PostgreSQL> PostgreSQL for FailoverTopology<T> {
+    async fn deploy(&self, config: &PostgreSQLConfig) -> Result<String, String> {
+        // Deploy primary; extract certs/endpoint;
+        // deploy replica with pg_basebackup + TLS passthrough.
+        // (Full impl logged/elaborated.)
+    }
+
+    // Delegate queries to primary.
+    async fn get_replication_certs(&self, cluster_name: &str) -> Result<ReplicationCerts, String> {
+        self.primary.get_replication_certs(cluster_name).await
+    }
+    // ...
+}
+
+/// Similarly for other capabilities.
+#[async_trait]
+impl<T: Docker> Docker for FailoverTopology<T> {
+    // Failover Docker orchestration.
+}
+````
+
+**Key properties:**
+- **Auto-derivation**: `Failover<K8sAnywhere>` gets `PostgreSQL` iff `K8sAnywhere: PostgreSQL`.
+- **No boilerplate**: One blanket impl per capability *per higher-order type*.
+
+## Rationale
+
+- **Composition via generics**: Rust trait solver auto-selects impls; zero runtime cost.
+- **Compile-time safety**: Missing `T: Capability` → compile error (no panics).
+- **Scalable**: O(capabilities) impls per higher-order; new `T` auto-works.
+- **ISP-respecting**: Capabilities only surface if sub-topology provides.
+- **Centralized logic**: Orchestration (e.g., cert propagation) in one place.
+
+**Example usage:**
+````rust
+// ✅ Works: K8sAnywhere: PostgreSQL → Failover provides failover PG
+let pg_failover: FailoverTopology<K8sAnywhereTopology> = ...;
+pg_failover.deploy_pg(config).await;
+
+// ✅ Works: LinuxHost: Docker → Failover provides failover Docker
+let docker_failover: FailoverTopology<LinuxHostTopology> = ...;
+docker_failover.deploy_docker(...).await;
+
+// ❌ Compile fail: K8sAnywhere !: Docker
+let invalid: FailoverTopology<K8sAnywhereTopology>;
+invalid.deploy_docker(...); // `T: Docker` bound unsatisfied
+````
+
+## Consequences
+
+**Pros:**
+- **Extensible**: New topology `AWSTopology: PostgreSQL` → instant `Failover<AWSTopology>: PostgreSQL`.
+- **Lean**: No useless impls (e.g., no `K8sAnywhere: Docker`).
+- **Observable**: Logs trace every step.
+
+**Cons:**
+- **Monomorphization**: Generics generate code per T (mitigated: few Ts).
+- **Delegation opacity**: Relies on rustdoc/logs for internals.
+
+## Alternatives considered
+
+| Approach | Pros | Cons |
+|----------|------|------|
+| **Manual per-T impls**<br>`impl PG for Failover<K8s> {..}`<br>`impl PG for Failover<Linux> {..}` | Explicit control | N×M explosion; violates ISP; hard to extend. |
+| **Dynamic trait objects**<br>`Box<dyn AnyCapability>` | Runtime flex | Perf hit; type erasure; error-prone dispatch. |
+| **Mega-topology trait**<br>All-in-one `OrchestratedTopology` | Simple wiring | Monolithic; poor composition. |
+| **Registry dispatch**<br>Runtime capability lookup | Decoupled | Complex; no compile safety; perf/debug overhead. |
+
+**Selected**: Blanket impls leverage Rust generics for safe, zero-cost composition.
+
+## Additional Notes
+
+- Applies to `MultisiteTopology<T>`, `ShardedTopology<T>`, etc.
+- `FailoverTopology` in `failover.rs` is first implementation.

adr/015-higher-order-topologies/example.rs

@@ -0,0 +1,153 @@
+//! Example of Higher-Order Topologies in Harmony.
+//! Demonstrates how `FailoverTopology<T>` automatically provides failover for *any* capability
+//! supported by a sub-topology `T` via blanket trait impls.
+//! 
+//! Key insight: No manual impls per T or capability -- scales effortlessly.
+//! Users can:
+//! - Write new `Topology` (impl capabilities on a struct).
+//! - Compose with `FailoverTopology` (gets capabilities if T has them).
+//! - Compile fails if capability missing (safety).
+
+use async_trait::async_trait;
+use tokio;
+
+/// Capability trait: Deploy and manage PostgreSQL.
+#[async_trait]
+pub trait PostgreSQL {
+    async fn deploy(&self, config: &PostgreSQLConfig) -> Result<String, String>;
+    async fn get_replication_certs(&self, cluster_name: &str) -> Result<ReplicationCerts, String>;
+}
+
+/// Capability trait: Deploy Docker.
+#[async_trait]
+pub trait Docker {
+    async fn deploy_docker(&self) -> Result<String, String>;
+}
+
+/// Configuration for PostgreSQL deployments.
+#[derive(Clone)]
+pub struct PostgreSQLConfig;
+
+/// Replication certificates.
+#[derive(Clone)]
+pub struct ReplicationCerts;
+
+/// Concrete topology: Kubernetes Anywhere (supports PostgreSQL).
+#[derive(Clone)]
+pub struct K8sAnywhereTopology;
+
+#[async_trait]
+impl PostgreSQL for K8sAnywhereTopology {
+    async fn deploy(&self, _config: &PostgreSQLConfig) -> Result<String, String> {
+        // Real impl: Use k8s helm chart, operator, etc.
+        Ok("K8sAnywhere PostgreSQL deployed".to_string())
+    }
+
+    async fn get_replication_certs(&self, _cluster_name: &str) -> Result<ReplicationCerts, String> {
+        Ok(ReplicationCerts)
+    }
+}
+
+/// Concrete topology: Linux Host (supports Docker).
+#[derive(Clone)]
+pub struct LinuxHostTopology;
+
+#[async_trait]
+impl Docker for LinuxHostTopology {
+    async fn deploy_docker(&self) -> Result<String, String> {
+        // Real impl: Install/configure Docker on host.
+        Ok("LinuxHost Docker deployed".to_string())
+    }
+}
+
+/// Higher-Order Topology: Composes multiple sub-topologies (primary + replica).
+/// Automatically derives *all* capabilities of `T` with failover orchestration.
+/// 
+/// - If `T: PostgreSQL`, then `FailoverTopology<T>: PostgreSQL` (blanket impl).
+/// - Same for `Docker`, etc. No boilerplate!
+/// - Compile-time safe: Missing `T: Capability` → error.
+#[derive(Clone)]
+pub struct FailoverTopology<T> {
+    /// Primary sub-topology.
+    pub primary: T,
+    /// Replica sub-topology.
+    pub replica: T,
+}
+
+/// Blanket impl: Failover PostgreSQL if T provides PostgreSQL.
+/// Delegates reads to primary; deploys to both.
+#[async_trait]
+impl<T: PostgreSQL + Send + Sync + Clone> PostgreSQL for FailoverTopology<T> {
+    async fn deploy(&self, config: &PostgreSQLConfig) -> Result<String, String> {
+        // Orchestrate: Deploy primary first, then replica (e.g., via pg_basebackup).
+        let primary_result = self.primary.deploy(config).await?;
+        let replica_result = self.replica.deploy(config).await?;
+        Ok(format!("Failover PG deployed: {} | {}", primary_result, replica_result))
+    }
+
+    async fn get_replication_certs(&self, cluster_name: &str) -> Result<ReplicationCerts, String> {
+        // Delegate to primary (replica follows).
+        self.primary.get_replication_certs(cluster_name).await
+    }
+}
+
+/// Blanket impl: Failover Docker if T provides Docker.
+#[async_trait]
+impl<T: Docker + Send + Sync + Clone> Docker for FailoverTopology<T> {
+    async fn deploy_docker(&self) -> Result<String, String> {
+        // Orchestrate across primary + replica.
+        let primary_result = self.primary.deploy_docker().await?;
+        let replica_result = self.replica.deploy_docker().await?;
+        Ok(format!("Failover Docker deployed: {} | {}", primary_result, replica_result))
+    }
+}
+
+#[tokio::main]
+async fn main() {
+    let config = PostgreSQLConfig;
+
+    println!("=== ✅ PostgreSQL Failover (K8sAnywhere supports PG) ===");
+    let pg_failover = FailoverTopology {
+        primary: K8sAnywhereTopology,
+        replica: K8sAnywhereTopology,
+    };
+    let result = pg_failover.deploy(&config).await.unwrap();
+    println!("Result: {}", result);
+
+    println!("\n=== ✅ Docker Failover (LinuxHost supports Docker) ===");
+    let docker_failover = FailoverTopology {
+        primary: LinuxHostTopology,
+        replica: LinuxHostTopology,
+    };
+    let result = docker_failover.deploy_docker().await.unwrap();
+    println!("Result: {}", result);
+
+    println!("\n=== ❌ Would fail to compile (K8sAnywhere !: Docker) ===");
+    // let invalid = FailoverTopology {
+    //     primary: K8sAnywhereTopology,
+    //     replica: K8sAnywhereTopology,
+    // };
+    // invalid.deploy_docker().await.unwrap(); // Error: `K8sAnywhereTopology: Docker` not satisfied!
+    // Very clear error message :
+    // error[E0599]: the method `deploy_docker` exists for struct `FailoverTopology<K8sAnywhereTopology>`, but its trait bounds were not satisfied
+    //   --> src/main.rs:90:9
+    //    |
+    //  4 | pub struct FailoverTopology<T> {
+    //    | ------------------------------ method `deploy_docker` not found for this struct because it doesn't satisfy `FailoverTopology<K8sAnywhereTopology>: Docker`
+    // ...
+    // 37 | struct K8sAnywhereTopology;
+    //    | -------------------------- doesn't satisfy `K8sAnywhereTopology: Docker`
+    // ...
+    // 90 | invalid.deploy_docker(); // `T: Docker` bound unsatisfied
+    //    |         ^^^^^^^^^^^^^ method cannot be called on `FailoverTopology<K8sAnywhereTopology>` due to unsatisfied trait bounds
+    //    |
+    // note: trait bound `K8sAnywhereTopology: Docker` was not satisfied
+    //   --> src/main.rs:61:9
+    //    |
+    // 61 | impl<T: Docker + Send + Sync> Docker for FailoverTopology<T> {
+    //    |         ^^^^^^                ------     -------------------
+    //    |         |
+    //    |         unsatisfied trait bound introduced here
+    // note: the trait `Docker` must be implemented
+}
+

brocade/src/fast_iron.rs

@@ -1,8 +1,7 @@
 use super::BrocadeClient;
 use crate::{
     BrocadeInfo, Error, ExecutionMode, InterSwitchLink, InterfaceInfo, MacAddressEntry,
-    PortChannelId, PortOperatingMode, SecurityLevel, parse_brocade_mac_address,
+    PortChannelId, PortOperatingMode, parse_brocade_mac_address, shell::BrocadeShell,
-    shell::BrocadeShell,
 };
 
 use async_trait::async_trait;
@@ -210,20 +209,4 @@ impl BrocadeClient for FastIronClient {
         info!("[Brocade] Port-channel '{channel_name}' cleared.");
         Ok(())
     }
-
-    async fn enable_snmp(&self, user_name: &str, auth: &str, des: &str) -> Result<(), Error> {
-        let commands = vec![
-            "configure terminal".into(),
-            "snmp-server view ALL 1 included".into(),
-            "snmp-server group public v3 priv read ALL".into(),
-            format!(
-                "snmp-server user {user_name} groupname public auth md5 auth-password {auth} priv des priv-password {des}"
-            ),
-            "exit".into(),
-        ];
-        self.shell
-            .run_commands(commands, ExecutionMode::Regular)
-            .await?;
-        Ok(())
-    }
 }

brocade/src/lib.rs

@@ -237,15 +237,6 @@ pub trait BrocadeClient: std::fmt::Debug {
         ports: &[PortLocation],
     ) -> Result<(), Error>;
 
-    /// Enables Simple Network Management Protocol (SNMP) server for switch
-    ///
-    /// # Parameters
-    ///
-    ///  * `user_name`: The user name for the snmp server
-    ///  * `auth`: The password for authentication process for verifying the identity of a device
-    ///  * `des`: The Data Encryption Standard algorithm key
-    async fn enable_snmp(&self, user_name: &str, auth: &str, des: &str) -> Result<(), Error>;
-
     /// Removes all configuration associated with the specified Port-Channel name.
     ///
     /// This operation should be idempotent; attempting to clear a non-existent
@@ -309,11 +300,6 @@ fn parse_brocade_mac_address(value: &str) -> Result<MacAddress, String> {
     Ok(MacAddress(bytes))
 }
 
-#[derive(Debug)]
-pub enum SecurityLevel {
-    AuthPriv(String),
-}
-
 #[derive(Debug)]
 pub enum Error {
     NetworkError(String),

brocade/src/network_operating_system.rs

@@ -8,7 +8,7 @@ use regex::Regex;
 use crate::{
     BrocadeClient, BrocadeInfo, Error, ExecutionMode, InterSwitchLink, InterfaceInfo,
     InterfaceStatus, InterfaceType, MacAddressEntry, PortChannelId, PortOperatingMode,
-    SecurityLevel, parse_brocade_mac_address, shell::BrocadeShell,
+    parse_brocade_mac_address, shell::BrocadeShell,
 };
 
 #[derive(Debug)]
@@ -330,20 +330,4 @@ impl BrocadeClient for NetworkOperatingSystemClient {
         info!("[Brocade] Port-channel '{channel_name}' cleared.");
         Ok(())
     }
-
-    async fn enable_snmp(&self, user_name: &str, auth: &str, des: &str) -> Result<(), Error> {
-        let commands = vec![
-            "configure terminal".into(),
-            "snmp-server view ALL 1 included".into(),
-            "snmp-server group public v3 priv read ALL".into(),
-            format!(
-                "snmp-server user {user_name} groupname public auth md5 auth-password {auth} priv des priv-password {des}"
-            ),
-            "exit".into(),
-        ];
-        self.shell
-            .run_commands(commands, ExecutionMode::Regular)
-            .await?;
-        Ok(())
-    }
 }

examples/brocade_snmp_server/Cargo.toml

@@ -1,20 +0,0 @@
-[package]
-name = "brocade-snmp-server"
-edition = "2024"
-version.workspace = true
-readme.workspace = true
-license.workspace = true
-
-[dependencies]
-harmony = { path = "../../harmony" }
-brocade = { path = "../../brocade" }
-harmony_secret = { path = "../../harmony_secret" }
-harmony_cli = { path = "../../harmony_cli" }
-harmony_types = { path = "../../harmony_types" }
-harmony_macros = { path = "../../harmony_macros" }
-tokio = { workspace = true }
-log = { workspace = true }
-env_logger = { workspace = true }
-url = { workspace = true }
-base64.workspace = true
-serde.workspace = true

examples/brocade_snmp_server/src/main.rs

@@ -1,22 +0,0 @@
-use std::net::{IpAddr, Ipv4Addr};
-
-use harmony::{
-    inventory::Inventory, modules::brocade::BrocadeEnableSnmpScore, topology::K8sAnywhereTopology,
-};
-
-#[tokio::main]
-async fn main() {
-    let brocade_snmp_server = BrocadeEnableSnmpScore {
-        server_ips: vec![IpAddr::V4(Ipv4Addr::new(192, 168, 1, 111))],
-        dry_run: true,
-    };
-
-    harmony_cli::run(
-        Inventory::autoload(),
-        K8sAnywhereTopology::from_env(),
-        vec![Box::new(brocade_snmp_server)],
-        None,
-    )
-    .await
-    .unwrap();
-}

harmony/src/infra/brocade.rs

@@ -121,7 +121,7 @@ mod tests {
     use async_trait::async_trait;
     use brocade::{
         BrocadeClient, BrocadeInfo, Error, InterSwitchLink, InterfaceInfo, InterfaceStatus,
-        InterfaceType, MacAddressEntry, PortChannelId, PortOperatingMode, SecurityLevel,
+        InterfaceType, MacAddressEntry, PortChannelId, PortOperatingMode,
     };
     use harmony_types::switch::PortLocation;
 
@@ -279,10 +279,6 @@ mod tests {
         async fn clear_port_channel(&self, _channel_name: &str) -> Result<(), Error> {
             todo!()
         }
-
-        async fn enable_snmp(&self, user_name: &str, auth: &str, des: &str) -> Result<(), Error> {
-            todo!()
-        }
     }
 
     impl FakeBrocadeClient {

harmony/src/infra/network_manager.rs

@@ -135,8 +135,6 @@ impl OpenShiftNmStateNetworkManager {
                 description: Some(format!("Member of bond {bond_name}")),
                 r#type: nmstate::InterfaceType::Ethernet,
                 state: "up".to_string(),
-                mtu: Some(switch_port.interface.mtu),
-                mac_address: Some(switch_port.interface.mac_address.to_string()),
                 ipv4: Some(nmstate::IpStackSpec {
                     enabled: Some(false),
                     ..Default::default()
@@ -162,7 +160,7 @@ impl OpenShiftNmStateNetworkManager {
 
         interfaces.push(nmstate::Interface {
             name: bond_name.to_string(),
-            description: Some(format!("Network bond for host {host}")),
+            description: Some(format!("HARMONY - Network bond for host {host}")),
             r#type: nmstate::InterfaceType::Bond,
             state: "up".to_string(),
             copy_mac_from,

harmony/src/modules/brocade.rs

@@ -1,117 +0,0 @@
-use std::net::{IpAddr, Ipv4Addr};
-
-use async_trait::async_trait;
-use brocade::BrocadeOptions;
-use harmony_secret::{Secret, SecretManager};
-use harmony_types::id::Id;
-use serde::{Deserialize, Serialize};
-
-use crate::{
-    data::Version,
-    interpret::{Interpret, InterpretError, InterpretName, InterpretStatus, Outcome},
-    inventory::Inventory,
-    score::Score,
-    topology::Topology,
-};
-
-#[derive(Debug, Clone, Serialize)]
-pub struct BrocadeEnableSnmpScore {
-    pub server_ips: Vec<IpAddr>,
-    pub dry_run: bool,
-}
-
-impl<T: Topology> Score<T> for BrocadeEnableSnmpScore {
-    fn name(&self) -> String {
-        "BrocadeEnableSnmpScore".to_string()
-    }
-
-    fn create_interpret(&self) -> Box<dyn Interpret<T>> {
-        Box::new(BrocadeEnableSnmpInterpret {
-            score: self.clone(),
-        })
-    }
-}
-
-#[derive(Debug, Clone, Serialize)]
-pub struct BrocadeEnableSnmpInterpret {
-    score: BrocadeEnableSnmpScore,
-}
-
-#[derive(Secret, Clone, Debug, Serialize, Deserialize)]
-struct BrocadeSwitchAuth {
-    username: String,
-    password: String,
-}
-
-#[derive(Secret, Clone, Debug, Serialize, Deserialize)]
-struct BrocadeSnmpAuth {
-    username: String,
-    auth_password: String,
-    des_password: String,
-}
-
-#[async_trait]
-impl<T: Topology> Interpret<T> for BrocadeEnableSnmpInterpret {
-    async fn execute(
-        &self,
-        _inventory: &Inventory,
-        _topology: &T,
-    ) -> Result<Outcome, InterpretError> {
-        let switch_addresses = &self.score.server_ips;
-
-        let snmp_auth = SecretManager::get_or_prompt::<BrocadeSnmpAuth>()
-            .await
-            .unwrap();
-
-        let config = SecretManager::get_or_prompt::<BrocadeSwitchAuth>()
-            .await
-            .unwrap();
-
-        let brocade = brocade::init(
-            &switch_addresses,
-            22,
-            &config.username,
-            &config.password,
-            Some(BrocadeOptions {
-                dry_run: self.score.dry_run,
-                ..Default::default()
-            }),
-        )
-        .await
-        .expect("Brocade client failed to connect");
-
-        brocade
-            .enable_snmp(
-                &snmp_auth.username,
-                &snmp_auth.auth_password,
-                &snmp_auth.des_password,
-            )
-            .await
-            .map_err(|e| InterpretError::new(e.to_string()))?;
-
-        Ok(Outcome::success(format!(
-            "Activated snmp server for Brocade at {}",
-            switch_addresses
-                .iter()
-                .map(|s| s.to_string())
-                .collect::<Vec<_>>()
-                .join(", ")
-        )))
-    }
-
-    fn get_name(&self) -> InterpretName {
-        InterpretName::Custom("BrocadeEnableSnmpInterpret")
-    }
-
-    fn get_version(&self) -> Version {
-        todo!()
-    }
-
-    fn get_status(&self) -> InterpretStatus {
-        todo!()
-    }
-
-    fn get_children(&self) -> Vec<Id> {
-        todo!()
-    }
-}

harmony/src/modules/mod.rs

@@ -1,5 +1,4 @@
 pub mod application;
-pub mod brocade;
 pub mod cert_manager;
 pub mod dhcp;
 pub mod dns;

harmony/src/modules/okd/crd/nmstate.rs

@@ -417,6 +417,7 @@ pub struct EthernetSpec {
 #[serde(rename_all = "kebab-case")]
 pub struct BondSpec {
     pub mode: String,
+    #[serde(alias = "port")]
     pub ports: Vec<String>,
     #[serde(skip_serializing_if = "Option::is_none")]
     pub options: Option<BTreeMap<String, Value>>,

harmony_types/src/net.rs

@@ -1,6 +1,6 @@
 use serde::{Deserialize, Serialize};
 
-#[derive(Copy, Clone, Debug, PartialEq, Eq, Hash, Serialize, Deserialize, PartialOrd, Ord)]
+#[derive(Copy, Clone, PartialEq, Eq, Hash, Serialize, Deserialize, PartialOrd, Ord)]
 pub struct MacAddress(pub [u8; 6]);
 
 impl MacAddress {
@@ -19,6 +19,14 @@ impl From<&MacAddress> for String {
     }
 }
 
+impl std::fmt::Debug for MacAddress {
+    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+        f.debug_tuple("MacAddress")
+            .field(&String::from(self))
+            .finish()
+    }
+}
+
 impl std::fmt::Display for MacAddress {
     fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
         f.write_str(&String::from(self))