Enterprise High Availability Architecture Takeaways at a Glance:

  • CARP Heartbeat Frequency & Advertisement Skew: The Common Address Redundancy Protocol (CARP) operates via multicast heartbeats on IP protocol 112 (vrrp). The primary node advertises with advbase 1 and advskew 0 (sending heartbeats precisely every 1,000ms), while the secondary standby node operates at advbase 1 and advskew 100. If the primary drops three consecutive heartbeats (3,000ms), the secondary transitions from BACKUP to MASTER state in less than 250 milliseconds.
  • pfsync Firewall State Table Replication: Redundant routing is useless if stateful connections drop during failover. pfsync synchronizes FreeBSD pf kernel state tables across a dedicated point-to-point Layer 2 link using IP protocol 240. Every TCP 3-way handshake, UDP translation table, and active NAT state is streamed in real time, preventing active SSH, VoIP, and database sessions from terminating when node election triggers.
  • Dedicated Interconnect Bandwidth Math: For networks processing 100,000 concurrent states with a 15,000 states/sec churn rate, pfsync generates approximately 42 Mbps to 65 Mbps of raw synchronization traffic. Running pfsync across production LAN or WAN interfaces introduces broadcast storm risks and MTU fragmentation; production high availability demands a dedicated physical 2.5GbE or 10GbE point-to-point DAC cable between nodes.
  • XMLRPC Configuration Synchronization: OPNsense synchronizes firewall rules, alias objects, DHCP reservations, and certificate authorities via secure XMLRPC calls over port 443. Excluding node-specific parameters (such as physical interface bindings, root credentials, and CARP virtual IP skews) is mandatory to prevent split-brain synchronization loops.

What Is OPNsense CARP and How Does Stateful Failover Function?

Position 1 Direct Answer: OPNsense CARP (Common Address Redundancy Protocol) provides automated hardware failover by sharing virtual IP addresses (VIPs) between two or more physical firewall appliances. Combined with pfsync (Packet Filter State Synchronization) over a dedicated interconnect and XMLRPC configuration mirroring, CARP delivers sub-second, hitless failover where active TCP sessions, VPN tunnels, and NAT translations remain uninterrupted when the primary node experiences hardware, power, or link failure.

Enterprise perimeter engineering demands zero tolerance for single points of failure (SPOF). While high-end proprietary firewalls charge tens of thousands of dollars in annual licensing fees for active/passive clustering, open-source OPNsense on bare-metal hardware delivers carrier-grade high availability using native FreeBSD kernel subsystems. Achieving deterministic failover requires deep comprehension of the underlying synchronization protocols: CARP for virtual gateway election, pfsync for state table replication, and XMLRPC for administrative configuration parity.

When deploying dual OPNsense nodes, each physical interface (WAN, LAN, DMZ, Management) requires its own dedicated physical IP address, supplemented by a shared Virtual IP (VIP) that serves as the default gateway for client subnets and the public-facing endpoint for upstream transit providers. The master node actively claims the VIP’s MAC address (in the format 00:00:5e:00:01:vhid), processing all inbound and outbound frames. The backup node passively listens to CARP advertisements. Should the primary node experience a kernel panic, sudden power loss, or severed upstream fiber, the backup immediately broadcasts gratuitous ARP (GARP) frames to upstream and downstream switches, redirecting frame delivery within milliseconds.

Clustering Subsystem Protocol / Port Primary Node Role Standby Node Role Bandwidth & Latency Impact
CARP Virtual IP (VIP) IP Proto 112 (VRRP), Multicast 224.0.0.18 MASTER (advskew 0, broadcasts every 1s) BACKUP (advskew 100, listens silently) < 5 Kbps; 250ms failover trigger
pfsync State Sync IP Proto 240 (pfsync), Unicast / Multicast Transmits active pf state updates Ingests and updates local kernel state table 20–100+ Mbps depending on connection churn
XMLRPC Config Sync HTTPS (TCP 443), XML Payload Pushes rule / alias changes on commit Applies configuration changes into config.xml Negligible (bursts only during admin commits)
Interface Tracking Kernel linkstate devd daemon Demotes CARP VIPs if WAN link drops Promotes backup to master across all VIPs Sub-100ms hardware link state detection

pfsync Traffic Engineering & Dedicated Synchronization Links

The most catastrophic architectural error in deploying OPNsense high availability is transmitting pfsync packets across production LAN subnets. Because pfsync transmits state insertions, updates, and deletions for every single socket established across the firewall, it generates high-frequency packet bursts. In high-throughput environments—such as container clusters running Kafka, Redis, or microservices—state churn can exceed 25,000 states per second.

Consider the mathematical formula for calculating pfsync network overhead:

Bandwidth (bps) = (State Insertions/sec * Avg Update Size in Bytes * 8) * Overhead Multiplier
Where Avg Update Size ≈ 240 bytes (Ethernet + IP Proto 240 + pf_state struct) and Overhead Multiplier ≈ 1.35 (pfsync headers + ACKs)
Example: 20,000 states/sec churn = (20,000 * 240 * 8) * 1.35 = 51,840,000 bps ≈ 51.84 Mbps

Transmitting 52 Mbps of unencrypted, raw state data over a shared production bridge consumes switch buffer depth and exposes sensitive socket telemetry (including internal IP addresses, ephemeral ports, and NAT mappings) to anyone monitoring the local broadcast domain. For this reason, production CARP deployments must mandate a dedicated, direct physical patch cable (or isolated VLAN on redundant switching fabrics) connecting the pfsync interfaces of Node 1 and Node 2. Utilizing direct point-to-point 10GbE SFP+ DAC cables guarantees sub-millisecond serialization latency and eliminates MTU fragmentation risks.

XMLRPC Configuration Synchronization & Preventing Split-Brain

While CARP coordinates routing and pfsync coordinates state tables, OPNsense uses XMLRPC to ensure firewall rules, aliases, NAT port forwards, static routes, and intrusion detection signatures remain perfectly mirrored across both appliances. If an administrator creates a new firewall rule on the primary node, OPNsense connects to the secondary node’s administrative IP address via HTTPS, authenticates with a dedicated sync account, and commits the updated XML elements into the secondary’s config.xml.

However, specific configuration elements must NEVER be synchronized via XMLRPC:

  • Interface Assignments: If Node 1 and Node 2 possess differing PCIe NIC enumerations (e.g., igb0 vs em0 or dual-port Mellanox mlx5_core0), synchronizing interface bindings will instantly corrupt the secondary node’s networking stack.
  • CARP Virtual IP Settings: Both nodes must have CARP configured, but the secondary node’s advertising skew must remain higher (e.g., advskew 100) than the primary’s (advskew 0). Syncing raw CARP objects directly can overwrite the skew, resulting in simultaneous master claims and severe packet flapping.
  • Local Hostname & Static Management IPs: Synchronizing local system identifiers causes IP collision on the management subnet.
Companion SecOps Engineering Teardown:

Review our companion architecture blueprint on OPNsense vs pfSense CE: Bare-Metal Firewall Benchmarks to size CPU cores and throughput requirements for high-availability hardware clusters.

Step-by-Step OPNsense CARP Deployment & Verification Checklist

  1. Assign Static Physical IPs: Allocate dedicated IPs on WAN, LAN, and DMZ interfaces for both firewalls (e.g., Node 1 LAN: 192.168.1.2/24, Node 2 LAN: 192.168.1.3/24).
  2. Provision Dedicated Sync Interconnect: Configure a dedicated physical interface (e.g., opt1) on both nodes with private link-local addressing (e.g., 10.99.99.1/30 and 10.99.99.2/30). Enable Jumbo Frames (MTU 9000) if supported.
  3. Configure pfsync: Under System > High Availability > Settings, check “Enable pfsync”, select the dedicated sync interface, and specify the peer node’s unicast IP address to avoid multicast flooding.
  4. Create CARP Virtual IPs: Navigate to Interfaces > Virtual IPs > Settings. Add a new Virtual IP of type CARP for each subnet (e.g., LAN VIP: 192.168.1.1/24). Set VHID to 1, Password to a secure shared secret, advbase to 1, and advskew to 0 on Node 1. On Node 2, mirror the configuration but assign advskew to 100.
  5. Enable Preemption & Link Demotion: Under System > Settings > Tunables, ensure net.inet.carp.preempt=1 is enabled. This ensures that if Node 1 loses link state on the WAN interface, it immediately demotes all its CARP VIPs simultaneously, forcing a clean, fleet-wide failover to Node 2 rather than creating asymmetric routing loops.
  6. Validate State Failover: Initiate a continuous ping and an active, long-running SSH session through the CARP gateway. Disconnect the WAN cable on Node 1. Verify via tcpdump -i any -n vrrp that Node 2 claims master state within 250ms and that the SSH session remains responsive without packet drop.