In enterprise networking, financial infrastructure, and mission-critical self-hosted datacenters, the perimeter firewall is the ultimate single point of failure (SPOF). If a physical hardware appliance suffers a power supply failure, memory panic, or motherboard short, an entire enterprise loses external WAN connectivity. For organizations deploying open-source security appliances, OPNsense High Availability (HA) provides enterprise-grade resilience utilizing CARP (Common Address Redundancy Protocol), pfsync (Packet Filter State Synchronization), and XMLRPC configuration synchronization.

However, engineering a bulletproof dual-node OPNsense firewall cluster is fraught with subtle Layer 2 and Layer 3 pitfalls. System administrators frequently encounter devastating failure modes: CARP “split-brain” conditions where both the primary and secondary nodes claim the master VIP simultaneously, asymmetric routing where return traffic is dropped by out-of-sync state tables, or XMLRPC sync loops that corrupt firewall alias configurations. Solving CARP state synchronization requires mastering Layer 2 multicast routing, skew timers, and dedicated physical heartbeat networks.

Why Does OPNsense State Synchronization (pfsync) Fail During Failover?

To understand state synchronization, one must understand how stateful packet inspection operates. When a client initiates an outbound HTTPS session, OPNsense creates a dynamic entry in its state table tracking the source IP, destination IP, sequence numbers, and TCP window flags. As long as incoming packets match an established state, they bypass firewall rule evaluation.

In a High-Availability cluster, CARP provides the Virtual IP (VIP) that acts as the default gateway for clients. When Node 1 fails, Node 2 detects missed CARP heartbeats and claims the VIP. However, if pfsync was not continuously replicating the active state table in real time, Node 2’s state table is completely empty. When thousands of active enterprise TCP streams arrive at Node 2, the firewall evaluates them as unsolicited out-of-state packets (TCP flags ACK without SYN) and immediately issues TCP RST packets, dropping all VoIP calls, SSH terminals, and database connections.

OPNsense CARP High-Availability Architecture Matrix

The following engineering matrix outlines the technical parameters separating stable OPNsense HA clusters from misconfigured deployments across multicast routing, synchronization interfaces, and failover timings:

The Split-Brain Disaster: Understanding CARP Multicast Mechanics

CARP operates by broadcasting advertisement packets over IP protocol 112 to the multicast address 224.0.0.18. Each node advertises its availability using an advertisement interval (advbase, typically 1 second) and an advertisement skew (advskew). The node with the lowest skew becomes the active MASTER, while nodes with higher skews remain in BACKUP status.

A “split-brain” condition occurs when the backup node stops receiving CARP advertisement packets from the master, despite the master node remaining fully powered and operational. Common root causes include:

1. IGMP Snooping Filtering: Enterprise managed switches (such as Cisco, UniFi, or Aruba) frequently enable IGMP Snooping by default. Because CARP multicast heartbeats do not send standard IGMP join reports, the switch’s snooping filter assumes no clients want the multicast group and silently drops the packets after a 5-minute timeout. The backup node assumes the master is dead and transitions to MASTER status, creating duplicate IP conflicts across the network.
2. Virtual MAC Address Filtering: In virtualized environments (such as Proxmox VE, VMware ESXi, or Hyper-V), the hypervisor virtual switch incorporates MAC spoofing protection. Because CARP dynamically generates a virtual MAC address (e.g., 00:00:5e:00:01:vhid), the hypervisor blocks the virtual MAC by default unless Promiscuous Mode and MAC Address Changes are explicitly allowed on the virtual bridge.

Dedicated Physical Sync Interfaces: Security and Bandwidth Isolation

A critical architectural vulnerability in amateur OPNsense HA deployments is running pfsync state synchronization over the standard LAN subnet. In a high-traffic enterprise network, state synchronization generates substantial continuous traffic—often exceeding 200 Mbps during heavy database queries or multi-gigabit file transfers.

More dangerously, pfsync transmits raw state tables in cleartext. Anyone with packet capture access on the shared LAN subnet can reconstruct active TCP sequence numbers, inspect remote client IP mappings, and execute state-injection attacks. High-availability clusters must always dedicate a direct, isolated physical Ethernet interface (e.g., igb3 or a 10GbE SFP+ Direct Attach Copper cable) connecting the two nodes directly without any intervening network switch. Assign this private link a dedicated non-routable subnet (such as 172.16.254.0/30) and configure pfsync to bind exclusively to this interface.

XMLRPC Synchronization Guardrails: Preventing Configuration Erasure

OPNsense incorporates os-xmlrpc to automatically replicate firewall rules, aliases, NAT port forwards, and DHCP static reservations from the primary node to the secondary node. To prevent catastrophic configuration loops:

1. Configure XMLRPC Strictly One-Way: Under System > High Availability > Settings on the Primary node, enter the Secondary node’s sync IP, administrator credentials, and check the specific synchronization checkboxes (Firewall Rules, Aliases, NAT, Virtual IPs).
2. Leave Secondary HA Settings 100% Blank: Never configure XMLRPC synchronization on the Secondary node. If both nodes attempt to sync configurations to each other, a race condition will occur where older backup rules overwrite newly edited primary rules, or create infinite synchronization loops that peg CPU cores at 100%.
3. Ensure Identical Interface Names: Both physical firewalls must have identical network interface assignments (e.g., WAN must be vtnet0 on both, LAN must be vtnet1). If Node 1 uses Intel NICs (igb0) and Node 2 uses Realtek NICs (re0), XMLRPC synchronization will push incompatible interface bindings, breaking network connectivity on the secondary node.

The 4:00 PM Actionable OPNsense HA Deployment Playbook

To configure and verify a flawless dual-node OPNsense CARP cluster today, execute this four-step engineering sequence:

For more on firewall hardware benchmarking and bare-metal throughput optimization, review our guide on OPNsense vs. pfSense CE: SecOps Guide.