You know that sinking feeling. It’s 2:14 AM on a Tuesday, your phone is screaming from the nightstand, and the monitoring dashboard is bleeding crimson. The primary OPNsense gateway has just suffered a catastrophic kernel panic, taking down the office network, the VPN tunnels feeding remote engineers, and three active VoIP trunks supporting customer service. You scramble to the rack, physically power down the smoking chassis, and watch the secondary firewall assume the virtual IP addresses. But when the dust settles, every single remote user is disconnected, active file transfers are dead, and your CEO’s VoIP call was dropped mid-sentence. You didn’t just experience a failover; you experienced an outage disguised as High Availability. Let’s grab a cup of coffee and unpack why this happens and how to build an enterprise-grade OPNsense Multi-WAN cluster that actually protects your uptime. ## Why Did My Active Sessions Drop Even Though CARP Switched Over? When people first spin up an OPNsense High Availability cluster using the Common Address Redundancy Protocol (CARP), they assume that moving the Virtual IP (VIP) addresses from Node A to Node B is enough to keep traffic flowing. It is not. CARP only handles layer 3/2 identity—it moves the IP and MAC addresses so that hosts on your network continue sending packets to the new active firewall. However, a modern enterprise firewall is a stateful beast. It doesn’t just look at a destination IP and forward a packet; it tracks every single TCP handshake, UDP flow, and ICMP request in a massive state table. When a packet arrives at Node B after a failover, Node B checks its state table. If Node B’s state table is empty because states weren’t synchronized from Node A, the packet hits the firewall rule engine, gets evaluated as an untracked, unsolicited packet, and is dropped immediately with a `TCP Flags: S` rejection or a default block rule. To prevent this, you must run `pfsync`. This protocol broadcasts state table modifications across a dedicated interface to the backup firewall in real time. But here is the trap: if your pfsync traffic shares a congested switch fabric or a general-purpose LAN trunk, network jitter or bandwidth saturation will cause state drops. When Node A dies, Node B will have missed the last ten seconds of state updates, resulting in dropped sessions. Furthermore, when you introduce multi-WAN redundancy—say, a primary fiber circuit and a secondary business cable line—the complexity multiplies. If your gateway groups aren’t configured with precise tiering and monitor IPs, a failure on WAN1 won’t trigger a clean transition to WAN2 on the backup node, leading to asymmetrical routing nightmares where return traffic hits a firewall that has no context for the connection. ## What Are the Hidden Pitfalls of OPNsense Multi-WAN Clustering? Most network administrators treat OPNsense clustering as a simple checkbox exercise. They assign a few CARP Virtual IPs, turn on XMLRPC configuration sync, and call it a day. Then production hits, and the edge collapses. Let’s look at what separates a resilient, production-ready edge from a fragile lab setup that fails under pressure. Notice the Spanning Tree Protocol (STP) trap. This is a silent killer in many data centers. When Node A fails or reboots, its network interfaces go down. If your upstream or downstream switches run standard STP without Portfast (or RSTP edge port configuration), the switch port connecting to Node B may sit in a “Listening” or “Learning” state for 30 to 50 seconds before moving to “Forwarding.” During this window, CARP advertisements cannot pass between the firewalls. Node B, assuming it is isolated, promotes itself to Master. Node A recovers its link simultaneously, and boom—you have a split-brain scenario where both firewalls assert ownership of the CARP VIPs, leading to ARP flapping, packet blackholing, and total network paralysis. ## How Do I Configure Dedicated pfsync and CARP Parameters Safely? To eliminate split-brain events and ensure lightning-fast state synchronization, you need to configure your interfaces with surgical precision. Never rely on default CARP advertisement intervals if you are running resource-heavy services; default values can occasionally miss packets under CPU load, triggering false master elections. Let’s walk through the proper architectural baseline for your cluster nodes. First, establish strict interface separation. On both Node A and Node B, you should have physical ports mapped as follows: – `em0` / `igb0`: WAN 1 (Primary Fiber) – `em1` / `igb1`: WAN 2 (Secondary Cable/Coax) – `em2` / `igb2`: LAN (Internal network trunk) – `em3` / `igb3`: OPT1 / pfsync (Dedicated direct crossover cable between Node A and Node B) Using a direct physical crossover cable for pfsync—or dedicated access ports on an isolated, non-routed VLAN if using switches—ensures that state updates saturate an uncontended medium. Next, navigate to `System: High Availability` in your OPNsense UI. Here is where you define the synchronization target. 1. Enable Synchronize to CARP backup. 2. Enter the local synchronization password. 3. Set the Synchronize user to a dedicated account with proper privileges (never use `root` in production environments). 4. Select your dedicated sync interface (`igb3`). 5. Designate the Peer IP address (the local IP address assigned to `igb3` on the opposite firewall node). When setting up your CARP Virtual IPs under `Interfaces: Virtual IPs: Settings`, you must stagger your Virtual Host ID (VHID) advertisement skews. Node A (your primary) should have an `advskew` of `0` (the default, which broadcasts advertisements every second), while Node B (your backup) must have an `advskew` of `100`. This ensures that Node B always hears Node A’s heartbeats and gracefully steps back, only taking over if Node A stops talking for three consecutive intervals. ## How Do I Build Redundant Multi-WAN Gateway Groups with Sticky Connections? Handling multiple WAN uplinks in an HA cluster requires careful design. If you simply load-balance traffic across WAN1 and WAN2 without “sticky” states, users accessing secure banking portals or SaaS platforms will experience session termination every time their egress IP shifts between requests due to round-robin balancing. To solve this, we use Gateway Groups combined with firewall rules. 1. Navigate to `System: Gateways: Group`. 2. Create a group named `WAN_FAILOVER_GROUP`. 3. Add WAN1 and assign it Tier 1. 4. Add WAN2 and assign it Tier 2. 5. Set the Trigger level to `Packet Loss or High RTT`. Now, configure your LAN firewall rules to use this Gateway Group. But to protect stateful applications, ensure your rules take advantage of OPNsense’s sticky connection features. Under `System: Settings: Advanced`, ensure that “Sticky Connections” is enabled for load balancing. This ensures that once a client establishes a TCP connection through WAN1, subsequent packets for that connection hash to the same WAN gateway, preventing session resets. Furthermore, do not rely on your ISP’s gateway IP for DPinger health checks. ISPs frequently rate-limit ICMP traffic to their gateway interfaces, causing DPinger to falsely flag a healthy circuit as down, which triggers an unnecessary failover. Instead, configure custom monitor IPs in `System: Gateways: Configuration`. Point WAN1’s monitor to a robust public resolver like `1.1.1.1` and WAN2’s monitor to `8.8.8.8`. This measures true end-to-end path availability rather than the health of the ISP’s edge router interface. ## How Do I Troubleshoot pfsync and CARP Failover Issues in Real Time? When you finish configuring your cluster, you must test it. Never assume it works until you pull the power plug on the primary node while running live packet captures and active data streams. If failover fails or states drop, open an SSH session to both nodes and use the native diagnostic tools to interrogate the kernel. Check your CARP status using the command line: “`bash ifconfig carp0 “` Look for the `carp: MASTER` or `carp: BACKUP` status. If both nodes show `MASTER`, your switches are blocking CARP multicast traffic (VRRP protocol 112). Check your switch configurations to ensure IGMP snooping is properly configured or disabled for the CARP VLAN. Next, verify that pfsync states are actively replicating between nodes by running: “`bash pfsync -i igb3 -v “` You should see a continuous stream of state insertions, updates, and deletions scrolling across the screen as traffic passes through the primary firewall. If the screen is static while clients are actively browsing the web, your pfsync interface bindings are misconfigured, or firewall rules on the sync interface are blocking protocol 240 (`pfsync`). Finally, inspect your system logs (`Log Files: System: General` or via grep in `/var/log/system.log`) for XMLRPC sync errors. If configuration changes made on Node A fail to propagate to Node B, check your API credentials and ensure that HTTPS certificate validation isn’t failing due to mismatched common names or expired internal self-signed certificates. By treating your OPNsense cluster not as a set of static configuration boxes, but as a synchronized, state-aware distributed system, you transform a fragile edge into an unshakeable fortress. Take your time, verify your physical cabling, test your failover paths during a maintenance window, and sleep soundly when the unexpected happens. ### Frequently Asked Questions

Why do my active VPN tunnels drop during an OPNsense CARP failover?

VPN tunnels (such as OpenVPN or WireGuard) maintain cryptographic state tables. If `pfsync` is not correctly synchronizing state tables across your dedicated sync interface, the secondary firewall will not possess the decryption keys or sequence numbers for active tunnels when it assumes the CARP virtual IPs, forcing clients to completely re-handshake and re-authenticate.

How can I prevent split-brain syndrome in a two-node OPNsense cluster?

Split-brain occurs when both nodes believe they are the Master. Prevent this by ensuring proper CARP advertisement skews (advskew 0 for primary, 100 for backup), enabling rapid link-state transition features (like Portfast/Edge ports) on all switch ports connected to the cluster, and using a dedicated, isolated physical crossover link for pfsync heartbeats.

Should I use XMLRPC configuration synchronization for all settings in a production cluster?

No. While syncing firewall rules, aliases, and NAT settings is useful, syncing global system settings (such as static management IP addresses, hostnames, and SSH keys) can cause administrative lockouts and network divergence. Selectively sync only what is required for edge packet processing.

Why is DPinger marking my Multi-WAN gateway as down even though the internet works?

ISPs frequently rate-limit ICMP echo requests sent directly to their edge gateway IP addresses. When DPinger pings the gateway IP, it experiences packet loss or latency spikes due to rate-limiting, triggering a false failover. Fix this by assigning reliable public monitor IPs (like 1.1.1.1 or 8.8.8.8) to your gateway configurations.