It is 2:15 AM on a Sunday when a power supply unit dies inside your primary edge firewall. In a standard single-node firewall architecture, this single point of failure instantly triggers an outage cascade: active VPN tunnels collapse, VoIP calls drop mid-sentence, outbound multi-WAN sessions break, and your on-call team faces hours of emergency routing reconfiguration and physical cable swapping in a cold datacenter. If your business depends on zero-downtime operations, relying on a single firewall chassis is an operational gamble you will eventually lose.

High availability (HA) in OPNsense eliminates this failure mode entirely. By combining the Common Address Redundancy Protocol (CARP), state table synchronization (`pfsync`), and configuration replication (XML-RPC), OPNsense allows two identical firewall nodes to operate as a single logical gateway cluster. When properly engineered, a total hardware failure on your primary master firewall causes the secondary backup firewall to assume master status in under two seconds—without dropping active TCP connections, breaking IPsec/OpenVPN tunnels, or requiring manual administrative intervention.

Direct Practical Verdict:

OPNsense CARP High Availability delivers seamless multi-WAN failover by pairing CARP Virtual IPs for gateway redundancy with pfsync state replication over a dedicated physical link. When the primary node fails, the secondary node assumes identical master Virtual IPs and existing TCP, UDP, and VPN session states instantaneously, ensuring zero drop in active enterprise network connections.

How Does OPNsense CARP Architecture Keep Multi-WAN Sessions Alive?

Building a resilient, stateful high-availability cluster requires understanding three distinct protocol layers that work in concert across your OPNsense nodes: CARP, pfsync, and XML-RPC.

1. CARP (Common Address Redundancy Protocol)
CARP operates at Layer 3/4 to provide IP address redundancy. Instead of assigning static gateway IP addresses directly to LAN and WAN network interfaces on client devices, you assign a shared Virtual IP (VIP) to a designated Virtual Host ID (VHID). The master node broadcasts periodic CARP heartbeat packets via multicast (`224.0.0.18`, IP protocol 112) across all CARP-enabled interfaces. These heartbeats notify the backup node that the master is healthy.

The timing of these heartbeats is governed by two parameters: advbase (advertisement base, measured in seconds, typically set to 1) and advskew (advertisement skew, measured in 1/256ths of a second). On the Master node, advskew is set to 0, meaning heartbeats transmit immediately. On the Backup node, advskew is set higher (typically 100 or 110). If the backup node fails to hear heartbeats from the master within (3 * advbase) + (advskew / 256) seconds, it transitions its own CARP interfaces from Backup to Master state and begins advertising with `advskew = 0`.

2. pfsync (State Table Synchronization)
CARP alone only moves IP addresses; it does not preserve active firewall connections. Without state synchronization, a CARP failover forces every active TCP connection to reset, dropping web sessions, SSH terminals, and voice traffic. This is where pfsync (IP protocol 240) becomes essential. pfsync broadcasts packet state updates—such as state creation, state deletion, and NAT translations—across a dedicated interface between the primary and secondary nodes. When a failover occurs, the secondary node’s state table is already identical to the master’s, allowing traffic passing through the newly promoted secondary firewall to hit existing state matches without interruption.

3. XML-RPC (Configuration Replication)
To maintain identical security postures, OPNsense uses XML-RPC over HTTPS to synchronize system configurations from the primary node to the secondary node. When you add a firewall rule, modify an outbound NAT policy, create an IPsec tunnel, or update an alias on the primary node, XML-RPC automatically pushes those changes to the secondary firewall, saving administrative overhead and eliminating manual configuration drift.

                 +-----------------------------------+
                 |     ISP 1 (WAN1) / ISP 2 (WAN2)    |
                 +-----------------------------------+
                                   |
                     +-------------+-------------+
                     |  Upstream WAN Switch / VLAN |
                     +-------------+-------------+
                                   |
         +-------------------------+-------------------------+
         |                                                   |
   [WAN1: 192.0.2.2]                                   [WAN1: 192.0.2.3]
   [WAN2: 198.51.100.2]                                [WAN2: 198.51.100.3]
+-----------------------+                           +-----------------------+
|  PRIMARY NODE (FW01)  |                           |  BACKUP NODE (FW02)   |
|  Master for VIPs      |=== Dedicated SYNC Link ===|  Backup for VIPs       |
|  advskew: 0           |    (pfsync & XML-RPC)     |  advskew: 100         |
+-----------------------+                           +-----------------------+
   [LAN: 192.168.1.2]                                  [LAN: 192.168.1.3]
         |                                                   |
         +-------------------------+-------------------------+
                                   |
                     +-------------+-------------+
                     |   Downstream LAN Switch   |
                     +-------------+-------------+
                                   |
                     [LAN VIP Gateway: 192.168.1.1]
                     [WAN1 VIP: 192.0.2.1        ]
                     [WAN2 VIP: 198.51.100.1     ]

What Is the Network Topology and IP Addressing Requirement for HA?

Designing a zero-downtime OPNsense CARP cluster requires dedicated, non-overlapping IP addressing across three operational planes: Management/Node IPs, Virtual IPs (VIPs), and a Dedicated Sync Link.

Every interface participating in CARP requires three static IP addresses within the same subnet: one real static IP for the Primary node, one real static IP for the Backup node, and one shared CARP Virtual IP that floats between them. Never attempt to configure CARP using dynamic (DHCP) WAN interfaces; CARP requires static IP allocations from your Internet Service Providers.

  • WAN1 Subnet (e.g., Primary ISP /29 Block):
    • Primary Node Physical WAN1: 192.0.2.2/29
    • Backup Node Physical WAN1: 192.0.2.3/29
    • WAN1 CARP VIP (VHID 1): 192.0.2.1/29
    • Upstream ISP Gateway: 192.0.2.6
  • WAN2 Subnet (e.g., Secondary ISP /29 Block):
    • Primary Node Physical WAN2: 198.51.100.2/29
    • Backup Node Physical WAN2: 198.51.100.3/29
    • WAN2 CARP VIP (VHID 2): 198.51.100.1/29
    • Upstream ISP Gateway: 198.51.100.6
  • LAN Subnet (Private /24):
    • Primary Node Physical LAN: 192.168.1.2/24
    • Backup Node Physical LAN: 192.168.1.3/24
    • LAN CARP VIP (VHID 3): 192.168.1.1/24 (Configured as default gateway for clients/switches)
  • Dedicated High Availability SYNC Link:
    • Primary Node SYNC Interface: 10.99.99.1/30
    • Backup Node SYNC Interface: 10.99.99.2/30
    • Note: Direct crossover cable or isolated dedicated VLAN. No CARP VIP is configured on the SYNC link.

Why Do Most OPNsense High Availability Deployments Suffer Split-Brain and Dropped State?

A surprising number of network administrators configure CARP, run a quick manual failover test, and assume their cluster is production-ready. However, critical structural misconfigurations frequently surface during real outages, leading to split-brain states, asymmetric routing, or complete connection teardowns.

A split-brain condition occurs when both the Primary and Backup nodes believe they are the Master simultaneously. When this happens, both nodes claim ownership of the exact same CARP Virtual IP addresses and attempt to process traffic. This triggers IP conflict alarms on connected switches, corrupts Layer 2 ARP tables, corrupts state tables, and drops 100% of network traffic.

Failure Mode Root Cause Architecture Symptom in Production Architectural Fix
Multicast Blocked / CARP Split-Brain Switch IGMP snooping, port security, or cloud vSwitch dropping CARP multicast (`224.0.0.18`). Both nodes show “MASTER” status for all VIPs; duplicate IP warnings; massive packet loss. Disable IGMP snooping on firewall VLANs; allow MAC spoofing & promiscuous mode on virtual switches.
State Drops on Failover `pfsync` bound to shared LAN/WAN interface or missing explicit sync interface setting. VIP moves smoothly, but all TCP sessions (SSH, VoIP, VPN, HTTPS) instantly drop and disconnect. Isolate `pfsync` to a dedicated physical interface or isolated VLAN (`10.99.99.0/30`).
Outbound NAT Traffic Reset Outbound NAT rules configured to use “Interface Address” instead of CARP Virtual IPs. Outbound traffic matches secondary node physical IP during failover, breaking established states. Convert NAT to Hybrid/Manual Outbound NAT mode and map source translations explicitly to CARP VIPs.
XML-RPC Synchronization Loop XML-RPC sync configured on both Primary and Secondary nodes bi-directionally. Config overwritten constantly, parameters oscillating between master and backup skew settings. Configure XML-RPC target settings ONLY on the Primary node. Secondary node settings remain empty.
Switch Port Security Lockdown Managed switch limits port MAC address count to 1 MAC per physical switch port. Switch disables port when CARP virtual MAC (`00:00:5e:00:01:XX`) appears alongside physical MAC. Set switch port maximum MAC address count to a minimum of 3 (Physical MAC, CARP Virtual MAC, IPv6 MACs).

The Root Mechanics of CARP Multicast and MAC Address Spoofing

Understanding Layer 2 mechanics is vital for preventing split-brain conditions. When you instantiate a CARP VIP with VHID 1, CARP automatically generates a virtual MAC address formatted as 00:00:5e:00:01:01 (where the last byte matches the VHID). When the Primary node is active, it sends Ethernet frames originating from this Virtual MAC, and responds to ARP queries with this MAC.

If your intermediate enterprise switches enforce Port Security (limiting MAC addresses per port) or utilize strict IGMP Snooping without an active IGMP querier, the switch will drop CARP heartbeat packets (`224.0.0.18`). The Backup node stops receiving heartbeats and assumes the Primary node has died. Both nodes immediately claim the Virtual MAC address, leading to complete network collapse. To prevent this, ensure your Layer 2 switches explicitly allow multiple MAC addresses on firewall facing ports and correctly forward multicast frames without dropping unregistered groups.

How Do You Build an Enterprise-Grade OPNsense CARP Multi-WAN Cluster Step-by-Step?

Follow this detailed playbook to build, configure, and validate an active/passive high-availability OPNsense multi-WAN firewall cluster from scratch.

Step 1: Physical Link and Interface Provisioning

  1. Connect dedicated ethernet ports between Node 1 (FW01) and Node 2 (FW02) using an isolated crossover cable or a dedicated, untagged sync VLAN on your switches.
  2. In OPNsense, navigate to Interfaces -> Assignments on both nodes. Assign this interface as HA_SYNC.
  3. Navigate to Interfaces -> [HA_SYNC]:
    • Enable interface: Checked
    • IPv4 Configuration Type: Static IPv4
    • IP Address on Primary (FW01): 10.99.99.1/30
    • IP Address on Backup (FW02): 10.99.99.2/30
  4. Save and apply changes on both firewalls. Verify connectivity by opening System -> Diagnostics -> Ping on FW01 and pinging 10.99.99.2.

Step 2: Configure High Availability Sync Settings (XML-RPC & pfsync)

Perform this step on the Primary Node (FW01) ONLY first, then mirror the basic sync interface setting on the Backup Node.

  1. On Primary (FW01), navigate to System -> High Availability -> Settings:
    • Disable Synchronize States: Unchecked (State sync enabled).
    • Synchronize Interface: Select HA_SYNC.
    • pfsync Synchronize Peer IP: Enter 10.99.99.2 (Backup node IP).
    • Synchronize Config to IP: Enter 10.99.99.2.
    • Remote System Username: Enter root (or dedicated sync user).
    • Remote System Password: Enter the admin password of FW02.
    • Sync Options: Check all boxes (Firewall Rules, Aliases, NAT, Virtual IPs, DHCP, IPsec, OpenVPN, System, etc.) to ensure full replication.
  2. Click Save.
  3. On Backup (FW02), navigate to System -> High Availability -> Settings:
    • Synchronize Interface: Select HA_SYNC.
    • pfsync Synchronize Peer IP: Enter 10.99.99.1 (Primary node IP).
    • Do NOT fill in Synchronize Config to IP, Remote Username, or Sync Options on the Backup node. XML-RPC replication is unidirectional (Primary to Backup).
  4. Click Save.

Step 3: Create CARP Virtual IPs for LAN and Multi-WAN Interfaces

Because XML-RPC is now active, creating CARP VIPs on the Primary node automatically pushes them to the Backup node with correct advertisement skews if preemption parameters are aligned.

  1. On Primary (FW01), navigate to Interfaces -> Virtual IPs -> Settings.
  2. Click + Add to create the LAN Virtual IP:
    • Mode: CARP
    • Interface: LAN
    • IP Address: 192.168.1.1/24
    • Virtual Host ID (VHID): 3
    • Advertising Base (advbase): 1
    • Advertising Skew (advskew): 0
    • Password: Enter a strong shared cluster password (must match across nodes).
  3. Click + Add to create the WAN1 Virtual IP:
    • Mode: CARP
    • Interface: WAN1
    • IP Address: 192.0.2.1/29
    • Virtual Host ID (VHID): 1
    • advbase: 1 | advskew: 0
    • Password: Shared cluster password.
  4. Click + Add to create the WAN2 Virtual IP:
    • Mode: CARP
    • Interface: WAN2
    • IP Address: 198.51.100.1/29
    • Virtual Host ID (VHID): 2
    • advbase: 1 | advskew: 0
    • Password: Shared cluster password.
  5. Click Save and Apply Changes.
  6. Verify on FW02 under Interfaces -> Virtual IPs -> Status. You should see all three Virtual IPs populated automatically, showing status BACKUP with an automatically adjusted advskew of 100. FW01 should show status MASTER.

Step 4: Reconfigure Outbound NAT for CARP VIP Alignment

By default, OPNsense uses Automatic Outbound NAT, which translates outgoing LAN traffic to the physical IP address of the egress interface. In an HA deployment, this causes connections to drop when failover switches nodes. You must bind outbound NAT to your Virtual IPs.

  1. On Primary (FW01), navigate to Firewall -> NAT -> Outbound.
  2. Select Hybrid Outbound NAT rule generation (or Manual Outbound NAT rule generation) and click Save.
  3. Click + Add to create outbound translation rules for WAN1 and WAN2:
    • Interface: WAN1
    • Source: 192.168.1.0/24 (Your LAN Subnet)
    • Translation / target: Select WAN1_VIP (192.0.2.1) instead of Interface Address.
  4. Add a secondary rule for WAN2:
    • Interface: WAN2
    • Source: 192.168.1.0/24
    • Translation / target: Select WAN2_VIP (198.51.100.1).
  5. Click Save and Apply Changes. XML-RPC will instantly push these NAT rules to FW02.

Step 5: Multi-WAN Gateway Group Setup across Dual CARP VIPs

To enable active failover between WAN1 and WAN2 across both physical firewalls, construct gateway groups using upstream ISP endpoints linked to CARP VIP bindings.

  1. Navigate to System -> Gateways -> Configuration.
  2. Ensure WAN1 Gateway (e.g., 192.0.2.6) and WAN2 Gateway (e.g., 198.51.100.6) are configured. Set monitor IPs (such as 1.1.1.1 and 8.8.8.8) for accurate ping loss monitoring.
  3. Navigate to System -> Gateways -> Group and click + Add:
    • Group Name: MultiWAN_Failover
    • WAN1 Gateway: Tier 1
    • WAN2 Gateway: Tier 2 (or Tier 1 for load balancing)
    • Trigger Level: Packet Loss
      Common Address Redundancy & Firewall Failover

      Routing site-to-site encrypted tunnels across redundant enterprise appliances ensures uninterrupted WAN connectivity. For OPNsense CARP cluster math, read WireGuard Site-to-Site on OPNsense: Routing & MTU.