harmony/docs/use-cases/opnsense-vm-integration.md

# Use Case: OPNsense VM Integration

Boot a real OPNsense firewall in a local KVM virtual machine and configure it entirely through Harmony — load balancer, DHCP, TFTP, VLANs, firewall rules, NAT, VIPs, and link aggregation. Fully automated, zero manual steps, CI-friendly.

This is the best way to discover Harmony: you'll see 11 different Scores configure a production firewall through type-safe Rust code and the OPNsense REST API.

## What you'll have at the end

A local OPNsense VM fully configured by Harmony with:
- HAProxy load balancer with health-checked backends
- DHCP server with static host bindings and PXE boot options
- TFTP server serving boot files
- Prometheus node exporter enabled
- 2 VLANs on the LAN interface
- Firewall filter rules, outbound NAT, and bidirectional NAT
- Virtual IPs (IP aliases)
- Port forwarding (DNAT) rules
- LAGG interface (link aggregation)

All applied idempotently through the OPNsense REST API — the same Scores used in production bare-metal deployments.

## Prerequisites

- **Linux** with KVM support (Intel VT-x/AMD-V enabled in BIOS)
- **libvirt + QEMU** installed and running (`libvirtd` service active)
- **~10 GB** free disk space
- **~15 minutes** for the first run (image download + OPNsense firmware update)
- Docker running (if installed — the setup handles compatibility)

Supported distributions: Arch, Manjaro, Fedora, Ubuntu, Debian.

## Quick start (single command)

```bash
# One-time: install libvirt and configure permissions
./examples/opnsense_vm_integration/setup-libvirt.sh
newgrp libvirt

# Verify
cargo run -p opnsense-vm-integration -- --check

# Boot + bootstrap + run all 11 Scores (fully unattended)
cargo run -p opnsense-vm-integration -- --full
```

That's it. No browser clicks, no manual SSH configuration, no wizard interaction.

## What happens step by step

### Phase 1: Boot the VM

Downloads the OPNsense 26.1 nano image (~350 MB, cached after first run), injects a `config.xml` with virtio NIC assignments, creates a 4 GiB qcow2 disk, and boots the VM with 4 NICs:

```
vtnet0 = LAN  (192.168.1.1/24)    -- management
vtnet1 = WAN  (DHCP)               -- internet access
vtnet2 = LAGG member 1             -- for aggregation test
vtnet3 = LAGG member 2             -- for aggregation test
```

### Phase 2: Automated bootstrap

Once the web UI responds (~20 seconds after boot), `OPNsenseBootstrap` takes over:

1. **Logs in** to the web UI (root/opnsense) with automatic CSRF token handling
2. **Aborts the initial setup wizard** via the OPNsense API
3. **Enables SSH** with root login and password authentication
4. **Changes the web GUI port** to 9443 (prevents HAProxy conflicts on standard ports)
5. **Restarts lighttpd** via SSH to apply the port change

No browser, no Playwright, no expect scripts — just HTTP requests with session cookies and SSH commands.

### Phase 3: Run 11 Scores

Creates an API key via SSH, then configures the entire firewall:

| # | Score | What it configures |
|---|-------|--------------------|
| 1 | `LoadBalancerScore` | HAProxy with 2 frontends (ports 16443 and 18443), backends with health checks |
| 2 | `DhcpScore` | DHCP range, 2 static host bindings (MAC-to-IP), PXE boot options |
| 3 | `TftpScore` | TFTP server serving PXE boot files |
| 4 | `NodeExporterScore` | Prometheus node exporter on OPNsense |
| 5 | `VlanScore` | 2 test VLANs (tags 100 and 200) on vtnet0 |
| 6 | `FirewallRuleScore` | Firewall filter rules (allow/block with logging) |
| 7 | `OutboundNatScore` | Source NAT rule for outbound traffic |
| 8 | `BinatScore` | Bidirectional 1:1 NAT |
| 9 | `VipScore` | Virtual IPs (IP aliases for CARP/HA) |
| 10 | `DnatScore` | Port forwarding rules |
| 11 | `LaggScore` | Link aggregation group (failover on vtnet2+vtnet3) |

Each Score reports its status:

```
[LoadBalancerScore] SUCCESS in 2.2s -- Load balancer configured 2 services
[DhcpScore]         SUCCESS in 1.4s -- Dhcp Interpret execution successful
[VlanScore]         SUCCESS in 0.2s -- Configured 2 VLANs
...
PASSED -- All OPNsense integration tests successful
```

### Phase 4: Verify

After all Scores run, the integration test verifies each configuration via the REST API:
- HAProxy has 2+ frontends
- Dnsmasq has 2+ static hosts and a DHCP range
- TFTP is enabled
- Node exporter is enabled
- 2+ VLANs exist
- Firewall filter rules are present
- VIPs, DNAT, BINAT, SNAT rules are configured
- LAGG interface exists

## Explore in the web UI

After the test completes, open https://192.168.1.1:9443 (login: root/opnsense) and explore:

- **Services > HAProxy > Settings** -- frontends, backends, servers with health checks
- **Services > Dnsmasq DNS > Settings** -- host overrides (static DHCP entries)
- **Services > TFTP** -- enabled with uploaded files
- **Interfaces > Other Types > VLAN** -- two tagged VLANs
- **Firewall > Automation > Filter** -- filter rules created by Harmony
- **Firewall > NAT > Port Forward** -- DNAT rules
- **Firewall > NAT > Outbound** -- SNAT rules
- **Firewall > NAT > One-to-One** -- BINAT rules
- **Interfaces > Virtual IPs > Settings** -- IP aliases
- **Interfaces > Other Types > LAGG** -- link aggregation group

## Clean up

```bash
cargo run -p opnsense-vm-integration -- --clean
```

Destroys the VM and virtual networks. The cached OPNsense image is kept for next time.

## How it works

### Architecture

```
Your workstation                          OPNsense VM (KVM)
+--------------------+                   +---------------------+
|  Harmony           |                   |  OPNsense 26.1      |
|  +---------------+ |   REST API        |  +---------------+  |
|  | OPNsense      |----(HTTPS:9443)---->|  | API + Plugins |  |
|  | Scores        | |                   |  +---------------+  |
|  +---------------+ |   SSH             |  +---------------+  |
|  +---------------+ |----(port 22)----->|  | FreeBSD Shell |  |
|  | OPNsense-     | |                   |  +---------------+  |
|  | Bootstrap     | |   HTTP session    |                     |
|  +---------------+ |----(HTTPS:443)--->|  (first-boot only)  |
|  +---------------+ |                   |                     |
|  | opnsense-     | |                   |  LAN: 192.168.1.1   |
|  | config        | |                   |  WAN: DHCP           |
|  +---------------+ |                   +---------------------+
+--------------------+
```

The stack has four layers:

1. **`opnsense-api`** -- auto-generated typed Rust client from OPNsense XML model files
2. **`opnsense-config`** -- high-level configuration modules (DHCP, firewall, load balancer, etc.)
3. **`OPNsenseBootstrap`** -- first-boot automation via HTTP session auth (login, wizard, SSH, webgui port)
4. **Harmony Scores** -- declarative desired-state descriptions that make the firewall match

### The Score pattern

```rust
// 1. Declare desired state
let score = VlanScore {
    vlans: vec![
        VlanDef { parent: "vtnet0", tag: 100, description: "management" },
        VlanDef { parent: "vtnet0", tag: 200, description: "storage" },
    ],
};

// 2. Execute against topology -- queries current state, applies diff
score.interpret(&inventory, &topology).await?;
// Output: [VlanScore] SUCCESS in 0.9s -- Created 2 VLANs
```

Scores are idempotent: running the same Score twice produces the same result.

## Network architecture

```
Host (192.168.1.10) --- virbr-opn bridge --- OPNsense LAN (192.168.1.1)
                         192.168.1.0/24        vtnet0
                         NAT to internet

                    --- virbr0 (default) --- OPNsense WAN (DHCP)
                         192.168.122.0/24      vtnet1
                         NAT to internet
```

## Available commands

| Command | Description |
|---------|-------------|
| `--check` | Verify prerequisites (libvirtd, virsh, qemu-img) |
| `--download` | Download the OPNsense image (cached) |
| `--boot` | Create VM + automated bootstrap |
| (default) | Run integration test (assumes VM is bootstrapped) |
| `--full` | Boot + bootstrap + integration test (CI mode) |
| `--status` | Show VM state, ports, and connectivity |
| `--clean` | Destroy VM and networks |

## Environment variables

| Variable | Default | Description |
|----------|---------|-------------|
| `RUST_LOG` | (unset) | Log level: `info`, `debug`, `trace` |
| `HARMONY_KVM_URI` | `qemu:///system` | Libvirt connection URI |
| `HARMONY_KVM_IMAGE_DIR` | `~/.local/share/harmony/kvm/images` | Cached disk images |

## Troubleshooting

**VM won't start / permission denied**
Ensure your user is in the `libvirt` group and that the image directory is traversable by the qemu user. Run `setup-libvirt.sh` to fix.

**192.168.1.0/24 conflict**
If your host network already uses this subnet, the VM will be unreachable. Edit the constants in `src/main.rs` to use a different subnet.

**Web GUI didn't come up after bootstrap**
The bootstrap runs `diagnose_via_ssh()` automatically when the web UI doesn't respond. Check the diagnostic output for lighttpd status and listening ports. You can also access the serial console: `virsh -c qemu:///system console opn-integration`

**HAProxy install fails**
OPNsense may need a firmware update. The integration test handles this automatically but it may take a few minutes for the update + reboot cycle.

## What's next

- **[OPNsense Firewall Pair](../../examples/opnsense_pair_integration/README.md)** -- boot two VMs, configure CARP HA failover with `FirewallPairTopology` and `CarpVipScore`. Uses NIC link control to bootstrap both VMs sequentially despite sharing the same default IP.
- [OKD on Bare Metal](./okd-on-bare-metal.md) -- the full 7-stage OKD installation pipeline using OPNsense as the infrastructure backbone
- [PostgreSQL on Local K3D](./postgresql-on-local-k3d.md) -- a simpler starting point using Kubernetes