harmony/docs/use-cases/opnsense-vm-integration.md

Use Case: OPNsense VM Integration

Boot a real OPNsense firewall in a local KVM virtual machine and configure it entirely through Harmony — load balancer, DHCP, TFTP, VLANs, firewall rules, NAT, VIPs, and link aggregation. Fully automated, zero manual steps, CI-friendly.

This is the best way to discover Harmony: you'll see 11 different Scores configure a production firewall through type-safe Rust code and the OPNsense REST API.

What you'll have at the end

A local OPNsense VM fully configured by Harmony with:

• HAProxy load balancer with health-checked backends
• DHCP server with static host bindings and PXE boot options
• TFTP server serving boot files
• Prometheus node exporter enabled
• 2 VLANs on the LAN interface
• Firewall filter rules, outbound NAT, and bidirectional NAT
• Virtual IPs (IP aliases)
• Port forwarding (DNAT) rules
• LAGG interface (link aggregation)

All applied idempotently through the OPNsense REST API — the same Scores used in production bare-metal deployments.

Prerequisites

• Linux with KVM support (Intel VT-x/AMD-V enabled in BIOS)
• libvirt + QEMU installed and running (libvirtd service active)
• ~10 GB free disk space
• ~15 minutes for the first run (image download + OPNsense firmware update)
• Docker running (if installed — the setup handles compatibility)

Supported distributions: Arch, Manjaro, Fedora, Ubuntu, Debian.

Quick start (single command)

# One-time: install libvirt and configure permissions
./examples/opnsense_vm_integration/setup-libvirt.sh
newgrp libvirt

# Verify
cargo run -p opnsense-vm-integration -- --check

# Boot + bootstrap + run all 11 Scores (fully unattended)
cargo run -p opnsense-vm-integration -- --full

That's it. No browser clicks, no manual SSH configuration, no wizard interaction.

What happens step by step

Phase 1: Boot the VM

Downloads the OPNsense 26.1 nano image (~350 MB, cached after first run), injects a config.xml with virtio NIC assignments, creates a 4 GiB qcow2 disk, and boots the VM with 4 NICs:

vtnet0 = LAN  (192.168.1.1/24)    -- management
vtnet1 = WAN  (DHCP)               -- internet access
vtnet2 = LAGG member 1             -- for aggregation test
vtnet3 = LAGG member 2             -- for aggregation test

Phase 2: Automated bootstrap

Once the web UI responds (~20 seconds after boot), OPNsenseBootstrap takes over:

1. Logs in to the web UI (root/opnsense) with automatic CSRF token handling
2. Aborts the initial setup wizard via the OPNsense API
3. Enables SSH with root login and password authentication
4. Changes the web GUI port to 9443 (prevents HAProxy conflicts on standard ports)
5. Restarts lighttpd via SSH to apply the port change

No browser, no Playwright, no expect scripts — just HTTP requests with session cookies and SSH commands.

Phase 3: Run 11 Scores

Creates an API key via SSH, then configures the entire firewall:

#	Score	What it configures
1	LoadBalancerScore	HAProxy with 2 frontends (ports 16443 and 18443), backends with health checks
2	DhcpScore	DHCP range, 2 static host bindings (MAC-to-IP), PXE boot options
3	TftpScore	TFTP server serving PXE boot files
4	NodeExporterScore	Prometheus node exporter on OPNsense
5	VlanScore	2 test VLANs (tags 100 and 200) on vtnet0
6	FirewallRuleScore	Firewall filter rules (allow/block with logging)
7	OutboundNatScore	Source NAT rule for outbound traffic
8	BinatScore	Bidirectional 1:1 NAT
9	VipScore	Virtual IPs (IP aliases for CARP/HA)
10	DnatScore	Port forwarding rules
11	LaggScore	Link aggregation group (failover on vtnet2+vtnet3)

Each Score reports its status:

[LoadBalancerScore] SUCCESS in 2.2s -- Load balancer configured 2 services
[DhcpScore]         SUCCESS in 1.4s -- Dhcp Interpret execution successful
[VlanScore]         SUCCESS in 0.2s -- Configured 2 VLANs
...
PASSED -- All OPNsense integration tests successful

Phase 4: Verify

After all Scores run, the integration test verifies each configuration via the REST API:

• HAProxy has 2+ frontends
• Dnsmasq has 2+ static hosts and a DHCP range
• TFTP is enabled
• Node exporter is enabled
• 2+ VLANs exist
• Firewall filter rules are present
• VIPs, DNAT, BINAT, SNAT rules are configured
• LAGG interface exists

Explore in the web UI

After the test completes, open https://192.168.1.1:9443 (login: root/opnsense) and explore:

• Services > HAProxy > Settings -- frontends, backends, servers with health checks
• Services > Dnsmasq DNS > Settings -- host overrides (static DHCP entries)
• Services > TFTP -- enabled with uploaded files
• Interfaces > Other Types > VLAN -- two tagged VLANs
• Firewall > Automation > Filter -- filter rules created by Harmony
• Firewall > NAT > Port Forward -- DNAT rules
• Firewall > NAT > Outbound -- SNAT rules
• Firewall > NAT > One-to-One -- BINAT rules
• Interfaces > Virtual IPs > Settings -- IP aliases
• Interfaces > Other Types > LAGG -- link aggregation group

Clean up

cargo run -p opnsense-vm-integration -- --clean

Destroys the VM and virtual networks. The cached OPNsense image is kept for next time.

How it works

Architecture

Your workstation                          OPNsense VM (KVM)
+--------------------+                   +---------------------+
|  Harmony           |                   |  OPNsense 26.1      |
|  +---------------+ |   REST API        |  +---------------+  |
|  | OPNsense      |----(HTTPS:9443)---->|  | API + Plugins |  |
|  | Scores        | |                   |  +---------------+  |
|  +---------------+ |   SSH             |  +---------------+  |
|  +---------------+ |----(port 22)----->|  | FreeBSD Shell |  |
|  | OPNsense-     | |                   |  +---------------+  |
|  | Bootstrap     | |   HTTP session    |                     |
|  +---------------+ |----(HTTPS:443)--->|  (first-boot only)  |
|  +---------------+ |                   |                     |
|  | opnsense-     | |                   |  LAN: 192.168.1.1   |
|  | config        | |                   |  WAN: DHCP           |
|  +---------------+ |                   +---------------------+
+--------------------+

The stack has four layers:

1. opnsense-api -- auto-generated typed Rust client from OPNsense XML model files
2. opnsense-config -- high-level configuration modules (DHCP, firewall, load balancer, etc.)
3. OPNsenseBootstrap -- first-boot automation via HTTP session auth (login, wizard, SSH, webgui port)
4. Harmony Scores -- declarative desired-state descriptions that make the firewall match

The Score pattern

// 1. Declare desired state
let score = VlanScore {
    vlans: vec![
        VlanDef { parent: "vtnet0", tag: 100, description: "management" },
        VlanDef { parent: "vtnet0", tag: 200, description: "storage" },
    ],
};

// 2. Execute against topology -- queries current state, applies diff
score.interpret(&inventory, &topology).await?;
// Output: [VlanScore] SUCCESS in 0.9s -- Created 2 VLANs

Scores are idempotent: running the same Score twice produces the same result.

Network architecture

Host (192.168.1.10) --- virbr-opn bridge --- OPNsense LAN (192.168.1.1)
                         192.168.1.0/24        vtnet0
                         NAT to internet

                    --- virbr0 (default) --- OPNsense WAN (DHCP)
                         192.168.122.0/24      vtnet1
                         NAT to internet

Available commands

Command	Description
--check	Verify prerequisites (libvirtd, virsh, qemu-img)
--download	Download the OPNsense image (cached)
--boot	Create VM + automated bootstrap
(default)	Run integration test (assumes VM is bootstrapped)
--full	Boot + bootstrap + integration test (CI mode)
--status	Show VM state, ports, and connectivity
--clean	Destroy VM and networks

Environment variables

Variable	Default	Description
RUST_LOG	(unset)	Log level: info, debug, trace
HARMONY_KVM_URI	qemu:///system	Libvirt connection URI
HARMONY_KVM_IMAGE_DIR	~/.local/share/harmony/kvm/images	Cached disk images

Troubleshooting

VM won't start / permission denied Ensure your user is in the libvirt group and that the image directory is traversable by the qemu user. Run setup-libvirt.sh to fix.

192.168.1.0/24 conflict If your host network already uses this subnet, the VM will be unreachable. Edit the constants in src/main.rs to use a different subnet.

Web GUI didn't come up after bootstrap The bootstrap runs diagnose_via_ssh() automatically when the web UI doesn't respond. Check the diagnostic output for lighttpd status and listening ports. You can also access the serial console: virsh -c qemu:///system console opn-integration

HAProxy install fails OPNsense may need a firmware update. The integration test handles this automatically but it may take a few minutes for the update + reboot cycle.

What's next

• OPNsense Firewall Pair -- boot two VMs, configure CARP HA failover with FirewallPairTopology and CarpVipScore. Uses NIC link control to bootstrap both VMs sequentially despite sharing the same default IP.
• OKD on Bare Metal -- the full 7-stage OKD installation pipeline using OPNsense as the infrastructure backbone
• PostgreSQL on Local K3D -- a simpler starting point using Kubernetes