As I tested the two-firewall CARP configuration I began to wonder. Every demonstration of CARP on pfSense I have ever read about or seen involves two firewalls: a primary firewall (the master) and the failover (the backup). Is it possible to have more than one system as a failover? Some of the documentation refers to a failover cluster, so at the very least, it seems to have been contemplated. But I could not find anything in the documentation on the official pfSense site or in the existing literature that describes how to do this.
Nevertheless, I was able to successfully implement, in a virtualized environment, a setup that incorporated three firewalls. I call this setup CARP with N firewalls, because, at least in theory, it should be possible to add as many failover firewalls as you find practical. Implementing this setup requires the following:
- N pfSense firewalls (obviously)
- Either N + 1 public IP addresses (one for each firewall + one for the virtual IP) or a router to go between your modem and WAN interfaces with at least N ports
- Since we are going to have at least three pfsync interfaces, we can no longer use a crossover cable, and we will need a dedicated switch for the pfsync network
There were two distinct problems that had to be solved in order for this setup to work. First, the order of priority must be clear so that when the master goes offline, pfSense knows which backup system gets promoted to master. Second, the synchronization data must propagate from the master to all backups. If both of these do not work, then the failover is not going to go smoothly. The following steps should ensure that the process works:
- While system is offline, configure interfaces on the pfSense system you want to add as a backup
- Configure virtual IPs on this system
- Configure firewall rules
- Bring the system online and finish configuring high availability sync
As with configuration of the secondary firewall in first example, you will want to begin configuration of the new firewall offline at first. At the very least, you will want to configure the WAN, LAN, and PFSYNC interfaces. A good convention to follow is to add one to the last octet of the IP address of the last previously configured backup firewall; for example, if the last backup firewall had a LAN IP address of 192.168.1.3, set it to 192.168.1.4 on the new firewall.
Next, you must configure the virtual IP addresses on the new firewall. Set the WAN and LAN virtual IPs to the same IP as was set on the other firewalls. In addition, you should set the Advertising frequency for each virtual IP so that is different than the Advertising frequency for the same virtual IP on the other firewalls. This parameter sets the priority within each VHID group (higher totals for Base and Skew result in lower priorities), and you should set each firewall's virtual IPs to different totals. Otherwise, when the master firewall goes offline, two or more backups may become promoted to master at the same time. One possible convention is to set the Skew for the first backup firewall to 10, and then increase the Skew for each successive firewall by 10.
The next step is to configure the firewall rules. This means, at the very least, adding a firewall rule to the PFSYNC interface to pass all traffic (this rule will be overwritten when the firewall data is synchronized, so you probably want to put something like Temporary rule in the Description). You may also want to add a rule to pass ICMP traffic so you can ping the PFSYNC interface on this firewall if it doesn't work.
Next, you need to address the issue of how to synchronize firewall data. The Synchronize Config to IP setting, according to the notes on the System | High Availability Sync configuration page, should not be set on backup cluster members. But the setting only allows for a single IP address, which does not seem to make it possible to propagate synchronization data from the master to more than one backup. In order to get this setup to work, navigate to System | High Availability Sync on the last backup system added to the failover group, and set Synchronize Config to IP to the PFSYNC interface of the new firewall. In addition, set the Remote System Username and Remote System Password on this system. For Select options to sync, select everything you want to synchronize; I recommend disabling synchronization of DHCP Server settings and Virtual IPs (disabling synchronizing virtual IPs guarantees that the Advertising frequency settings we made previously will not be overwritten). I also recommend leaving pfsync Synchronize Peer IP blank (to cause the state table to be synchronized via directed multicast). Also check all other firewalls in the failover group, including the master, to make sure Virtual IP synchronization is disabled and pfsync Synchronize Peer IP is left blank.
Now, you can bring the new firewall online and complete the last step of configuration. Navigate to System | High Availability Sync and enable Synchronize states. Change the Synchronize Interface to PFSYNC, and leave the Synchronize Config to IP blank. All other settings on this page can be kept at their default values. Click on the Save button to save the settings.
The way we have set up this failover group, changes will propagate from the primary firewall to the secondary firewall, from the secondary firewall to the tertiary firewall, and so on to the Nth firewall. To confirm that it worked, first navigate to Firewall | Rules, click on the PFSYNC tab, and ensure that the temporary firewall rule previously created has been overwritten. To further confirm that this setup works as it should, take the master firewall offline and ensure that the backup firewall with the lowest Advertising frequency setting for its virtual IPs has now been promoted to master, and that all other backup firewalls remain as backups. You can do this by navigating to Status | CARP (failover) and checking the CARP status. Repeat this process, each time taking the master firewall offline, and checking to make sure there is only one new master firewall, until only one firewall in the failover group remains. Then, start bringing firewalls back online in reverse order and make sure the current master is demoted back to backup, and that each time there is only one master firewall, until all firewalls are back online.
One major weakness of this configuration is that synchronization data does not propagate from the primary to all the backups; instead, it propagates from the primary to the secondary, from the secondary to the tertiary and so on. Therefore, if a backup firewall goes down and it is not the last firewall in the chain, at least one firewall will not be receiving updates. If the firewall is only offline temporarily, this shouldn't be a problem; once it comes back online, synchronization will resume. If the system is going to be offline for more than a brief time, however, you'll probably want to update Synchronize Config to IP on the firewall that was sending synchronization data to the offline firewall. Instead, set this parameter to the PFSYNC IP of the firewall that was receiving synchronization data from the offline firewall. This will ensure that firewall settings propagate to all firewalls in the failover group. If and when you bring the offline firewall back online, of course, you should revert the back to the previous setting.
Another problem with this setup is that the way synchronization works within the failover group may cause unexpected results. If changes are made to a backup firewall, those changes can propagate to other backups, resulting in multiple firewalls being out of sync with the master firewall. This will persist until either a configuration change is made on the master firewall, which triggers synchronization, or synchronization is forced from the master firewall (this can be done by navigating to Status | Filter Reload and clicking on the Force Config Sync button). These problems can be avoided by taking care when making changes to the configuration of backup firewalls.