harmony/examples/opnsense_vm_integration/README.md

# OPNsense VM Integration Example

Fully automated end-to-end integration test: boots an OPNsense VM via KVM, bootstraps SSH and API access without any manual browser interaction, installs packages, runs 11 Harmony Scores, and verifies idempotency (runs all Scores twice, asserts zero duplicates). CI-friendly.

## Quick start

```bash
# 1. One-time setup (libvirt, Docker compatibility)
./examples/opnsense_vm_integration/setup-libvirt.sh

# 2. Verify prerequisites
cargo run -p opnsense-vm-integration -- --check

# 3. Boot + bootstrap + integration test (fully unattended)
cargo run -p opnsense-vm-integration -- --full

# 4. Clean up
cargo run -p opnsense-vm-integration -- --clean
```

Or use the build script:

```bash
./build/opnsense-e2e.sh              # check + boot + test
./build/opnsense-e2e.sh --download   # download image first
./build/opnsense-e2e.sh --clean      # tear down
```

That's it. No browser clicks, no manual SSH setup, no wizard interaction.

## What happens during `--full`

1. Downloads OPNsense 26.1 nano image (~350MB, cached after first download)
2. Injects `config.xml` with virtio interface assignments (vtnet0=LAN, vtnet1=WAN)
3. Creates a 4 GiB qcow2 disk and boots via KVM (1 vCPU, 1GB RAM, 4 NICs)
4. Waits for web UI to respond (~20s)
5. **Automated bootstrap** via `OPNsenseBootstrap`:
   - Logs in (root/opnsense) with CSRF token handling
   - Aborts the initial setup wizard
   - Enables SSH with root login and password auth
   - Changes web GUI port to 9443 (avoids HAProxy conflicts)
   - Restarts lighttpd via SSH to apply the port change
6. Creates OPNsense API key via SSH (PHP script)
7. Installs `os-haproxy` via firmware API
8. Runs 11 Scores configuring the entire firewall
9. Verifies all configurations via REST API assertions
10. **Idempotency test**: runs all 11 Scores again, asserts entity counts are unchanged

## Step-by-step mode

If you prefer to separate boot and test:

```bash
# Boot + bootstrap (creates VM, enables SSH, sets port)
cargo run -p opnsense-vm-integration -- --boot

# Run integration test (assumes VM is bootstrapped)
cargo run -p opnsense-vm-integration

# Check VM status at any time
cargo run -p opnsense-vm-integration -- --status
```

## Prerequisites

### System requirements

- **Linux** with KVM support (Intel VT-x/AMD-V)
- **~10 GB** free disk space
- **~15 minutes** for first run (image download + firmware update)
- Subsequent runs: ~2 minutes

### Required packages

**Arch/Manjaro:**
```bash
sudo pacman -S libvirt qemu-full dnsmasq
```

**Fedora:**
```bash
sudo dnf install libvirt qemu-kvm dnsmasq
```

**Ubuntu/Debian:**
```bash
sudo apt install libvirt-daemon-system qemu-kvm dnsmasq
```

### Automated setup

```bash
./examples/opnsense_vm_integration/setup-libvirt.sh
```

This handles: user group membership, libvirtd startup, default storage pool, Docker FORWARD policy conflict.

After running setup, apply group membership:
```bash
newgrp libvirt
```

### Docker + libvirt compatibility

Docker sets the iptables FORWARD policy to DROP, which blocks libvirt's NAT networking. The setup script detects this and switches libvirt to the iptables firewall backend so both coexist.

## Scores applied

| # | Score | What it configures |
|---|-------|--------------------|
| 1 | `LoadBalancerScore` | HAProxy with 2 frontends, backends with TCP health checks |
| 2 | `DhcpScore` | DHCP range, 2 static host bindings, PXE boot options |
| 3 | `TftpScore` | TFTP server serving boot files |
| 4 | `NodeExporterScore` | Prometheus node exporter |
| 5 | `VlanScore` | 2 VLANs (tags 100, 200) on vtnet0 |
| 6 | `FirewallRuleScore` | Firewall filter rules with logging |
| 7 | `OutboundNatScore` | Source NAT for outbound traffic |
| 8 | `BinatScore` | Bidirectional 1:1 NAT |
| 9 | `VipScore` | Virtual IPs (IP aliases) |
| 10 | `DnatScore` | Port forwarding rules |
| 11 | `LaggScore` | Link aggregation (vtnet2+vtnet3) |

All Scores are idempotent: the test runs them twice and asserts entity counts are unchanged. This catches duplicate creation bugs that mock tests cannot detect.

## Network architecture

```
Host (192.168.1.10) --- virbr-opn bridge --- OPNsense LAN (192.168.1.1)
                         192.168.1.0/24        vtnet0
                         NAT to internet

                    --- virbr0 (default) --- OPNsense WAN (DHCP)
                         192.168.122.0/24      vtnet1
                         NAT to internet
```

## Environment variables

| Variable | Default | Description |
|----------|---------|-------------|
| `RUST_LOG` | (unset) | Log level: `info`, `debug`, `trace` |
| `HARMONY_KVM_URI` | `qemu:///system` | Libvirt connection URI |
| `HARMONY_KVM_IMAGE_DIR` | `~/.local/share/harmony/kvm/images` | Cached disk images |

## Troubleshooting

**VM won't start / permission denied**
Ensure your user is in the `libvirt` group and that the image directory is traversable by the qemu user. The setup script handles this.

**192.168.1.0/24 conflict**
If your host network already uses this subnet, the VM will be unreachable. Edit the constants in `src/main.rs` to use a different subnet.

**HAProxy install fails**
OPNsense may need a firmware update first. The integration test attempts this automatically. If it fails, connect to the web UI at https://192.168.1.1:9443 and update manually.

**Serial console access**
```bash
virsh -c qemu:///system console opn-integration
# Press Ctrl+] to exit
```