This is probably the most common deployment scenario in which CARP is involved. There are two common deployment scenarios you could employ for a two firewall setup in which one firewall is designated as the failover. Both of them involve a single virtual IP for the LAN interface, separate actual IPs for each LAN interface, and the LAN interface on each firewall connecting to a switch on the LAN side. Allocating additional IP addresses for the LAN is not a problem, since these are private addresses. The WAN side poses a problem. The WAN interface of a firewall would normally be connected directly to our modem (or whatever other internet connectivity device we have) and the WAN interface would be assigned a public IP address. In many cases, especially in a home or small office/home office scenario, we have a single public IP address allocated by our ISP. Both WAN interfaces need an IP address, and we need a virtual IP address to cover both of them, but we cannot just freely create public IP addresses. We therefore have two broad alternatives:

• Acquire two additional public IP addresses from our ISP. This is the most foolproof solution, but it is potentially expensive, depending on how much our ISP charges for public IP addresses.
• Place a router between the modem and the two WAN interfaces. This neatly solves the problem of needing additional public IP addresses. With the router in place, both WAN interfaces may have private IP addresses and a single virtual IP address that is also a private address. These addresses can be routed to the modem and the single public IP address we have. This is a less expensive solution, but it also adds a single point of failure to our setup. If the router fails, we have no internet connectivity, completely undermining the redundancy we were trying to incorporate into the network. (A simple router, however, is likely easier to replace and configure than a pfSense firewall.)

Fortunately, configuration for each setup is largely the same, with the only difference being that one uses public IP addresses for the WAN interfaces, and the other uses private IP addresses. Therefore, you should be able to follow along with this configuration regardless of which option we choose. As always, a diagram sometimes helps in clarifying where each component fits into the overall setup.

In order to create a firewall failover group, we need to synchronize data between the two firewalls. Synchronization does not need to be done via a dedicated interface - for example, it could be done via the LAN - but synchronization is generally more reliable if it is done via a dedicated interface. Thus, our example setup will have three interface: WAN, LAN, and a dedicated pfsync interface connected with a crossover cable. Setting up a CARP failover group involves several steps:

1. Configuration of virtual IP addresses
2. Configuration of a dedicated pfsync interface
3. Enabling pfsync and XML-RPC sync
4. Manual outbound NAT configuration
5. DHCP server configuration
6. Secondary firewall configuration

Once you have the concept of what your setup will look like and you have the required hardware, we can begin by determining what the virtual IP addresses will be. Keep in mind that each interface will require a virtual IP address, and the physical interface on each of the N firewalls will have its own IP address, for a total of N + 1 IP addresses for each interface. Thus, in the case of a two firewall CARP group, each interface will have three IP addresses. It is generally a good idea to have a consistent convention for IP address assignment. One way to do this is to assign the first address on an interface's subnet to the virtual IP address for that interface, with subsequent addresses being used for the interfaces on each firewall. In our example setup, we have a LAN subnet of 192.168.1.0. Thus, 192.168.1.1 will be the LAN interface's virtual IP, while the LAN interface's IP on first firewall will be 192.168.1.2 and the LAN interface's IP on second firewall will be 192.168.1.3.

To begin configuration, we will make sure that everything is functioning on the primary firewall. Additional firewalls should be offline until this point. Configuring virtual IPs is the first step, so navigate to Firewall | Virtual IPs. Then click on the Add button below the Virtual IP table on the right to add a new virtual IP entry.

On the Virtual IP configuration page, set Type to CARP. Set the interface in the Interface drop-down box to whichever interface you wish to configure. You are going to have to create virtual IPs for both the WAN and LAN interfaces, so you will likely be starting with one of these two. In the Address(es) edit box, enter the virtual IP address. The mask specified in the adjacent drop-down box should be the network's subnet mask.

In the Virtual IP Password edit boxes, you need to enter a Virtual Host ID (VHID) group password, which can be whatever you want. You won't have to enter this on the other firewalls, as it will automatically propagate to them via the pfsync interface. You do need to enter the password a second time for confirmation, however.

In the VHID Group drop-down box, you need to select the VHID group for this interface. The only requirement is that the VHID must not already be in use, and  the VHID must be unique on the entire interface's subnet. If there is no CARP or VHID traffic on your network, you can set this to 1; otherwise set it to the next available VHID.

The Advertising frequency drop-down boxes allow you to set the frequency with which the machine will advertise its presence. The lowest combination of both values (Base and Skew) determines which firewall is the master. For this reason, you will want to set Base to 1 and Skew to 0 on the master firewall (the lowest allowed values for these fields). Keep in mind that the XML-RPC process will automatically add 100 to each skew when syncing virtual IPs to secondary nodes. Thus, if x is the skew on the master firewall, the first slave will have a skew of x + 100; the second slave will have a skew of x + 200, and so on. In addition, the skew has a maximum value of 255, and XMLPRC will keep adding 100 to each skew when syncing VIPs to secondary nodes even if it results in values over 255, which it will if there are three or more slaves. In such cases, you should uncheck the Virtual IPs checkbox in the Select options to sync subsection of XMLPRC sync (which can be found under High Availability Sync). In the Description field, you may enter a brief non-parsed description. When you are done making changes, click on Save at the bottom of the page.

From the main Virtual IPs page, click on the Add button again and repeat the process for each interface that requires a virtual IP address. At a minimum, you will have to create virtual IPs for both WAN and LAN, but you may have multiple other interfaces on your firewall that will have virtual IPs. Just change Interface option to the interface you want to configure, and enter the virtual IP address for the new interface. The VHID group password will be for a different VHID group, so it can be different than the VHID password you entered previously. The VHID Group should be set to the next available VHID; this is likely the previously entered value +1. Finally, since this system is the master, in Advertising Frequency, Base should be set to 1 and Skew to 0. Enter a brief description and click on the Save button. When you have entered all the required virtual IPs, click on the Apply Changes button on the main Virtual IPs page.

The next step is to set up a dedicated pfsync interface. Setting up a dedicated interface for synchronization is not strictly required, but there are benefits to  such a configuration:

• Increased security: In a typical two interface (WAN and LAN) configuration, the alternative is to transmit sync data to between the primary and secondary firewalls via the LAN. Although it is less likely to happen in the era of switched ports, this traffic is potentially vulnerable to interception. Isolating pfsync traffic onto a separate network to which only other firewalls are connected eliminates this vulnerability.
• Improved resource utilization: The secondary firewalls must sync to the primary firewalls often to ensure their configuration is up to date, and the traffic generated can be considerable. Placing this traffic on a separate network thus reduces congestion on the LAN.

To set up an interface for pfsync, navigate to Interfaces | (assign), select an unused interface in the Available network ports drop-down box, and click on the Add button. Then click on interface name and begin configuration. There isn't much required for the pfsync interface, except that all the pfsync interfaces should be on a shared subnet. Give the interface an appropriate name (for example, SYNC). Click on the Save button at the bottom of the page when you are done configuring the pfsync interface and, when the page reloads, click on Apply Changes.

Next, you must create a firewall rule for the pfsync interface. Navigate to Firewall | Rules, and click on the tab for the newly created interface. Click on the Add button to add an allow pfsync interface to any rule. The Action column should be set to Pass and the Interface should be kept set to the pfsync interface. The protocol in the Protocol drop-down box should be set to PFSYNC. The Source field should be set to the pfsync network and Destination should be set to any. Enter a brief description in the Description field, and click on the Save button. When the main Firewall page reloads, click on the Apply Changes button.

You may also want to create a rule to allow ICMP traffic on the pfsync interface; this will allow you to ping the interface and send pings from the interface, which is always useful in verifying connectivity. If so, click on the Add button again to create a new rule. Change the protocol to ICMP; you can leave the ICMP Subtypes list box set to any, but if you opt for a more restrictive rule, be sure to select both Echo reply and Echo request, or ping won't work. A more restrictive rule would also set Destination to PFSYNC address. Add an appropriate description (for example Allow ICMP ping traffic) and click on the Save button.

The next step is enabling pfsync and XML-RPC. Navigate to System | High Availability Sync. In the State Synchronization Settings section, make sure the Synchronize states checkbox is checked. In the Synchronize Interface drop-down box, select the pfsync interface. If you set up a dedicated interface, select it as the interface; otherwise, select whichever interface you are using to synchronize the firewalls (the most likely alternative is LAN). By default, pfsync data is sent via multicast, but by specifying an IP address in the pfsync Synchronize Peer IP edit box, you can force pfsync to synchronize its state table to the specified IP address. If we are setting up the primary firewall, this field should be set to the secondary firewall's IP address. For example, if we have a primary firewall on which the dedicated pfsync interface has an IP address of 192.168.2.1 and the secondary firewall's pfsync interface is 192.168.2.2, this field should be set to 192.168.2.2 on the primary firewall (to sync the secondary firewall) and 192.168.2.1 on the secondary firewall.

Next, we must enable configuration synchronization for XML-RPC. On the same configuration page, scroll down to the Configuration Synchronization Settings (XML-RPC Sync) section. In the Synchronize Config to IP edit box, enter the IP address of the firewall to which the firewall should be synchronized. This should match the IP address entered in pfsync Synchronize Peer IP. The Remote System Username edit box should be set to admin (other usernames will not work) and then enter the password in the Remote System Password edit box (it should match on all firewalls). In the Select options to sync subsection, you should check all the checkboxes for items you want to synchronize. By default, only Firewall schedules and Virtual IPs are selected, but if you want to have truly redundant secondary firewalls, all checkboxes except DHCP Server settings (we handle the DHCP synchronization separately) should be checked (the Toggle All button at the bottom of this section lets you select everything). When you are done making changes, click on the Save button at the bottom of the page.

The next step is configuration of manual outbound NAT. Navigate to Firewall | NAT and click on the Outbound tab. If you haven't already, set Mode to either Hybrid Outbound NAT rule generation or Manual Outbound NAT rule generation and click on the Save button. In the Mappings table, find the autocreated LAN to WAN rule, and click on the edit icon for that rule. If you did not have any virtual IPs configured, then in the Translation section, Address is likely set to Interface Address. Change this setting to the WAN virtual IP that was added earlier for CARP. Then click on the Save button; click on Apply Changes on the main NAT page. If you configured everything correctly, connections leaving the WAN interface should be translated to the new WAN CARP IP. If you want to confirm this, one possible method is to access a website that displays the IP address from which the site is being accessed. Also, if you look at the Mappings table, the address in the NAT address column should be the virtual IP of the WAN interface. You should repeat this process for all local interfaces (non-WAN/non-pfsync) you have on the firewall.

Now, you need to update the DHCP settings. Navigate to Services | DHCP Server (these options are not available for DHCPv6) and scroll down to Servers. In the DNS servers subsection, set the first DNS server to the LAN virtual IP created earlier on in the process. In the Other Options section, set the Gateway field to the LAN virtual IP. Finally, enter the actual LAN IP address of the secondary firewall in the Failover peer IP edit box. This will allow the two firewalls to maintain a common set of DHCP leases. Then click on Save at the bottom of the page. Repeat this process for as many local interfaces you have on the firewall that are using DHCP.

This completes configuration of the primary firewall; we can now begin configuration of the secondary one. For the first phase of the secondary firewall configuration, keep the firewall offline. Assuming that pfSense is already installed and is working, you need to go through the interface assignment process. Set the LAN IP to whatever you previously designated as the backup LAN IP (for example, if we set the primary firewall's LAN IP to 192.168.1.1, we might set the backup LAN IP to 192.168.1.2). You need to assign an interface to pfsync, and set an IP address for that as well. The DHCP settings should be the same as they are on the primary firewall. Note that all of these configuration steps do not require use of the web GUI and can be done from the pfSense console. Once you have completed these steps, you should be able to put the secondary firewall online.

You can now log in to the secondary firewall via the web interface. If you haven't done so already, configure the WAN IP address (again, set it to the IP address you previously designated as the WAN IP). The admin password on the backup firewall must be set to the same value as that of the primary firewall, or synchronization will not work.

You need to create a firewall rule for the pfsync interface of the secondary firewall. This rule has two purposes. First, it will allow the initial synchronization data to pass from the pfsync interface to the primary firewall. Second, it will (hopefully) be overwritten by firewall rules from the primary firewall during the synchronization process, and thus will confirm that synchronization was successful. With this in mind, navigate to Firewall | Rules and create an allow any to any rule (Source should be set to any and Destination should be set to any). It may be helpful to use the Description field to flag this rule as one that will be overwritten. If you want to ping the second firewall, you may want to add a firewall rule to allow ICMP traffic as outlined in the configuration for the first firewall.

It will be necessary to set the CARP LAN virtual IP address as the gateway, and to set the primary firewall's LAN IP address as the failover peer. As with the primary firewall, this process must be repeated for each secondary firewall.

Now, you can bring the secondary firewall online. After you have done this, navigate to System | High Availability Sync on the secondary firewall, and make sure the synchronize states is enabled. The Synchronize Interface should be set to PFSYNC, and you likely want to enter the primary firewall's IP address in the pfSync Synchronize Peer IP edit box. The settings in the Configuration Synchronization Settings (XMLRPC Sync) section, however, should not be changed.

Now, all that is left to do is to activate CARP. Navigate to Status | CARP (failover) on both the primary and secondary firewall. If there is a button called Enable CARP on either firewall, click on it. Also, verify that the virtual IP for each interface is correct and that they show the correct status. On the primary firewall, the status should be MASTER. On the secondary firewall, the status should be BACKUP.

There are two additional buttons on this page that are useful, especially when troubleshooting. The Temporarily Disable CARP button disables CARP on the firewall on which it is invoked by removing the CARP virtual IPs from the system. If the firewall's status is MASTER, then the next available firewall whose status is BACKUP will take over.

This setting is not remembered through a reboot, and if you reboot a firewall that had been master before CARP was temporarily disabled, the firewall will regain its status as MASTER. The button toggles to Enable CARP when you press it, and pressing Enable CARP also enables a temporarily disabled master firewall to become master again.

Sometimes, you may need to keep CARP disabled through a reboot. This is where the other button, Enter Persistent CARP Maintenance Mode, comes in. If pressed, this button will disable CARP on the firewall, and this condition persists through a reboot, thus preventing the firewall from prematurely regaining the MASTER status. The button toggles to Leave Persistent CARP Maintenance Mode, and pressing this button is the only way to exit maintenance mode from within the web GUI.

If you have followed these steps, synchronization should take place. If not, you may want to navigate to System | High Avail. Sync on the primary firewall and confirm that the Synchronize States checkbox is checked. If not, you should check this box and save the settings. You may also want to navigate to Status | System Logs and check the logs to see if synchronization occurred. Successful synchronization will generate a series of log entries like these (as indicated in the log, pfsync screenshot shown below takes about 30 seconds to complete):