feat/opnsense-codegen #256
Reference in New Issue
Block a user
No description provided.
Delete Branch "feat/opnsense-codegen"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Generate typed API models for HAProxy, Caddy, Firewall, VLAN, LAGG, WireGuard (client/server/general), and regenerate Dnsmasq. All core modules validated against a live OPNsense 26.1.2 instance. Codegen improvements: - Add --module-name and --api-key CLI flags for controlling output filenames and API response envelope keys - Fix enum variant names starting with digits (prefix with V) - Use value="" XML attribute for wire values instead of element names - Handle unknown *Field types as opn_string (select widget safe) - Forgiving enum deserialization (warn instead of error on unknown) - Handle empty arrays in opn_string deserializer Add per-module examples (list_haproxy, list_caddy, list_vlan, etc.) and utility examples (raw_get, check_package, install_and_wait). Extract shared client setup into examples/common/mod.rs. Fix post_typed sending empty JSON body ({}) instead of no body, which was causing 400 errors on firmware endpoints. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>- Pin vendor/core submodule to 26.1.5 tag (matches running firewall) - Regenerate dnsmasq from model v1.0.9 (migrated during firmware upgrade) - Handle array-style select widgets in enum deserialization: OPNsense sometimes returns [{value, selected}, ...] instead of {key: {value, selected}} - Add firmware_upgrade and reboot examples for managing OPNsense updates - All 7 modules validated against live OPNsense 26.1.5: dnsmasq, haproxy, caddy, vlan, lagg, wireguard, firewall Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>The hand-written HaproxyGetResponse structs used HashMap which fails when OPNsense returns [] for empty collections. The generated types in opnsense-api handle this via opn_map, but opnsense-config had duplicated structs without that fix. Replace all hand-written HAProxy response types with serde_json::Value traversal. This avoids the duplication and handles the []/{} duality. Also fix integration example: - Use high ports (16443, 18443) to avoid conflicting with web UI on 443 - Skip package install if already installed - Use harmony_cli::cli_logger::init() instead of env_logger (safe to call multiple times) - Increase verification timeout to 60s Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>Move vendor-neutral firewall and network types (FirewallAction, Direction, IpProtocol, NetworkProtocol, VipMode, LaggProtocol) from harmony Score modules to harmony_types::firewall as industry-standard IaC types. Display impls use human-readable names (IPv4, CARP, LACP) — not wire format. OPNsense-specific wire translations live in opnsense-api::wire via the ToOPNsenseValue trait ("inet", "carp", "lacp"). Dependency chain: harmony_types → opnsense-api → opnsense-config → harmony. Users import types from harmony_types, translations happen transparently in the infrastructure layer. Includes 6 new tests verifying all wire value translations.Replace all Command::new("kubectl") calls with harmony-k8s K8sClient methods: - wait_for_pod_ready() instead of kubectl get pod jsonpath - exec_pod_capture_output() for OpenBao init/unseal/configure - delete_resource<MutatingWebhookConfiguration>() for webhook cleanup - port_forward() instead of kubectl port-forward subprocess Thread K3d and K8sClient through all functions instead of reconstructing context strings. Consolidate path helpers into harmony_data_dir(). Add Zitadel deployment via ZitadelScore with retry logic for CNPG CRD registration race and PostgreSQL cluster readiness timing. Add CLI flags: --demo, --sso-demo, --skip-zitadel, --cleanup. Add --demo mode: ConfigManager with EnvSource + StoreSource<OpenbaoSecretStore>. Configure OpenBao with harmony-dev policy, userpass auth, and JWT auth.Fix the core SSO authentication flow: instead of storing the Zitadel access_token as the OpenBao token (which OpenBao doesn't recognize), exchange the id_token with OpenBao's JWT auth method via POST /v1/auth/{mount}/login to get a real OpenBao client token. Changes: - ZitadelOidcAuth: add openbao_url, jwt_auth_mount, jwt_role fields - New exchange_jwt_for_openbao_token() method using reqwest (vaultrs 0.7.4 has no JWT auth module) - process_token_response() now exchanges id_token when openbao_url is set, falls back to access_token for backward compat - OpenbaoSecretStore::new() accepts optional jwt_role + jwt_auth_mount - All callers updated (lib.rs, openbao_chain example, harmony_sso) This implements ADR 020-1 Step 6 (OpenBao JWT exchange).Many significant improvements worth doing, overall great progress and most stuff works nicely. At the moement this is a lot of new modules that are separated, share a common "style" but lack the magic binding them all together. Some are pretty high level like the HA OPNSense KVM demo tying most of the new stuff together but then the openbao / zitadel modules are not integrated with the rest yet.
@@ -0,0 +1,140 @@#!/bin/bashThis should be a score that leverages the phased topology approach : LinuxHostTopology -> KvmHostTopology
This does not exist yet and is a major architectural feature missing, likely not that hard to implement but requires careful type safe mechanisms.
@@ -0,0 +176,4 @@}async fn list_static_mappings(&self) -> Vec<(MacAddress, IpAddress)> {// Return primary's view — both should be identicalThis is weak, we should compare both and warn or error if any mismatch
@@ -33,0 +39,4 @@////// Default implementation is a no-op for topologies that don't manage/// firewall rules (e.g., cloud environments with security groups).async fn ensure_wan_access(&self, _port: u16) -> Result<(), ExecutorError> {Should not be a no-op default impl here.
@@ -90,2 +113,2 @@None => return vec![],};fn haproxy_service_to_harmony(svc: &HaproxyService) -> Option<LoadBalancerService> {let listening_port = svc.bind.parse().unwrap_or_else(|_| {This should not panic, just return a clear error. This will require some refactoring of this module.
@@ -0,0 +10,4 @@/// Device model (e.g. "virtio")pub model: String,/// MAC addresspub mac: String,We have better types for mac address in harmony_types
@@ -0,0 +19,4 @@/// Local hypervisor via UNIX socket. Equivalent to `qemu:///system`.Local,/// Remote hypervisor over SSH. Equivalent to `qemu+ssh://user@host/system`.RemoteSsh { host: String, username: String },Ideally this would be a specific type for KvmHost with the relevant validation logic and a macro to make it easy to hardcode it type-safely.
@@ -60,6 +66,11 @@ impl<T: Topology + LoadBalancer> Interpret<T> for LoadBalancerInterpret {load_balancer.ensure_initialized().await?);for port in &self.score.wan_firewall_ports {This is a bit too opinionated at this level, we should let the score caller decide if he wants to expose them publicly.
@@ -332,0 +351,4 @@inventory: &Inventory,) -> Result<(), InterpretError> {info!("[Stage 02/Bootstrap] Waiting for bootstrap to complete...");info!("[Stage 02/Bootstrap] Running: openshift-install wait-for bootstrap-complete");Almost every time we ran this there was a longer delay than the default wait-for timeout (30 or 45 minutes iirc). We should take that into account.
@@ -0,0 +143,4 @@fn keys_dir() -> PathBuf {directories::BaseDirs::new().map(|dirs| dirs.data_dir().join("harmony").join("openbao"))We should have a clean module to handle directories, not hardcode harmony everywhere.
@@ -0,0 +199,4 @@Err(e) => !e.contains("not initialized"),};if is_initialized {Here this is a practical but very naive way to do that. We should be using harmony_config/secret but we have a chicken-and-egg problem where we want to use openbao as the secret store but it needs to be initialized first.
This will require some more thinking, the obvious solution is to use the harmony_config or secret crates to load this, and they have their own config to tell which backend to use. We could very well have openbao storing seal keys for another openbao downstream. Upstream eventually it is an administrator storing them somewhere, we should explore the industry standards here, ideally not relying on a third party. A local file vault with a password is relatively weak, a totp based vault somehwere, a bank / notary vault... There are many solutions, none is perfect.
@@ -0,0 +358,4 @@self.bao(k8s,root_token,&[This feels a bit fragile, but I'm not very familiar with openbao. Is there an api we can call using the root token to provision the first user? Is there a rust crate for vault/openbao that would allow doing this type-safely?
@@ -0,0 +390,4 @@None => return Ok(()),};let _ = selfSame here, is there an api or a crate to do that?
Rewrite api_codegen to generate proper envelope-wrapping methods that accept model structs directly. Callers no longer need to manually construct RuleBody wrappers or extract UUIDs from raw JSON. Key changes: - Generated API clients wrap request bodies internally via serde rename (e.g., add_rule(&my_rule) serializes as {"rule": {...}}) - Add shared SearchRow type to response.rs with label() and is_enabled() helpers, eliminating per-module RuleSearchRow type conflicts - Extract body_key from PHP controller addBase/setBase calls - Rewrite dnat.rs and firewall.rs to use the typed API end-to-end: search returns SearchResponse<SearchRow>, add returns UuidResponse, set/del return StatusResponse — zero raw JSON in production code - Add EnsureApi trait in firewall.rs for generic find-or-create pattern The only remaining json!() calls in dnat.rs and firewall.rs are in test mock responses, which is expected.- Make UuidResponse.uuid default to empty string so validation failures ({"result": "failed", "validations": {...}}) don't cause deserialization errors. Add is_failed() helper method. - Fix HAProxy healthcheck construction: map check_type string to HealthcheckType enum (was sending empty string, OPNsense rejected it) - Fix HAProxy server construction: set mode (ServerMode) and type (ServerType) enum fields (were defaulting to empty, OPNsense rejected) Discovered by running E2E tests against real OPNsense VM — the typed structs with ..Default::default() sent empty strings for required enum fields, which OPNsense rejected as validation errors. Still needed: HAProxy backend mode/algorithm and frontend mode/ connectionBehaviour enums, and fixing search API pagination for filter/snat/vip verification counts.The E2E test revealed that OPNsense validation failures were being silently swallowed: add/set operations returned {"result": "failed", "validations": {...}} but the code treated them as success. Critical fixes: - add_item/set_item now return Error::Validation on failure instead of silently returning empty/failed responses - VLAN: set pcp (PriorityCodePoint) — required in OPNsense 26.1 - Firewall filter: set sequence and statetype (KeepState) - SNAT: set sequence - BINAT: set sequence and destination_net ("any") - DNAT: set sequence - VIP: default advbase=1 and advskew=0 (required even for IP aliases) - HAProxy backend: set mode, algorithm, persistence_cookiemode enums - HAProxy frontend: set mode, connectionBehaviour enums E2E test now passes: all 11 Scores run successfully against a real OPNsense VM, and the idempotency test (run twice, verify counts unchanged) confirms zero duplicates.