In the previous chapter, we covered an example network in which we implemented both load balancing (for an FTP server) and a CARP failover group. In this section, we will add multi-WAN capabilities to this network. We don't want our second connection to go unused even when the first connection is up, so we want to make it part of a load balancing group. However, our second connection is a DSL connection with 1.544 Mbps up and 1.544 Mbps down, whereas our primary connection is a standard broadband connection. We will want to take this into account when setting up our gateway group and make sure the primary connection handles the lion's share of internet-bound traffic. Adding multi-WAN capabilities to a CARP setup is not particularly difficult, but it does involved several steps:

1. Add and configure the OPT_WAN interfaces.
2. Configure DNS settings for the OPT_WAN interfaces.
3. Add virtual IP addresses for each of the OPT_WAN interfaces.
4. Add outbound NAT rules for each of the OPT_WAN interfaces for each of the internal interfaces using them.
5. Add gateways for each OPT_WAN interface. 
6. Create a gateway group containing the original WAN interface and every OPT_WAN interface we want in the multi-WAN setup.
7. Add firewall rules for the gateway group.

Now that all the OPT_WAN interfaces are installed and configured, we can move straight ahead and configure the DNS server settings for WAN and OPT_WAN. We navigate to System | General Setup and scroll down to DNS Server Settings. Since it is good practice to use different DNS servers for each WAN-type interface, we enter 208.67.220.220 (one of OpenDNS's DNS servers) in the DNS server 1 listbox and then select WAN in the corresponding drop-down box. We enter 8.8.8.8 (one of Google's DNS servers) in the DNS server 2 listbox and then select WAN in the corresponding drop-down box. We click on the Save button to save the settings. 

Since this is a CARP failover group and there is an OPT_WAN interface on each firewall, we need to add virtual IPs for OPT_WAN. Navigate to Firewall | Virtual IPs and click on the Add button. On the Edit page, for the Type we select CARP. We select OPT_WAN as the Interface. Address type can be kept at the default of Single address. In the Address(es) edit box, we enter 10.2.1.1 – the virtual IP for OPT_WAN. We enter a Virtual IP Password; VHID Group automatically increments, so we do not have to change this setting. We enter a brief Description (for example "OPT_WAN virtual IP") and click on the Save button and then click the Apply Changes button. We repeat the process on the other firewall, remembering to make sure we adjust the Skew setting on the secondary firewall so it is higher than the Skew setting on the primary firewall.

As was the case with a single-WAN CARP setup, we need to create NAT rules for OPT_WAN, and make sure that these rules refer to the virtual IP and not the physical interface for OPT_WAN. This is because the NAT settings will be copied to the secondary firewall, and since the physical interface on the secondary firewall will be different, if we use the physical interface for these rules, NAT won't work on the secondary firewall. Since our initial CARP setup involved switching to manual outbound NAT rule generation, we have to manually create these rules. Our task will be easier, however, if we copy the WAN NAT rules. Thus, we find the LAN to WAN virtual IP rule, click on the copy icon, change the Interface setting to OPT_WAN and change the Address to 10.2.1.1 (OPT_WAN virtual IP address). We then click on the Save button and the Apply Changes button. We click on the copy icon for the DMZ to WAN virtual IP rule and repeat the process, changing the Interface to OPT_WAN and Address to 10.2.1.1. 

Now we can move on to gateway configuration. We navigate to System | Routing, and on the Gateway tab, we click on the Add button. We enter a Name for the gateway (for example, GATEWAY2) and the Gateway IP address, taking care to make sure we use the physical interface's IP address and not the virtual IP address. In the Monitor IP edit box, we enter 8.8.8.8 (the same as the IP address for the DNS server). We also enter a brief Description (for example, Gateway for OPT_WAN) and click Save, then click Apply Changes.

Since our second connection is a DSL connection, we want to make sure the first gateway has a much higher weight than the new gateway. Therefore, we find the WAN gateway and click on the edit icon. On the Edit page for this gateway, we scroll down and click on the Advanced button. The first option in the Advanced section is Weight. Since we want the DSL connection to handle about 3% of the overall WAN-bound traffic, we select 30 in the drop-down box (higher numbers ensure that a gateway will receive a greater share of the traffic; with the WAN gateway set to 30 and the OPT_WAN left at its default of 1, we will ensure that the WAN gateway gets most of the traffic). We click Save when we are finished and then click Apply Changes.

Next, we click on the Gateway Groups tab and click on the Add button. We give the gateway group an appropriate name (for example, MULTIWAN), and under Gateway Priority, we assign both WAN and OPT_WAN to Tier 1 so they will be active. The Trigger Level for this group will be set to Member down. We enter a brief Description (for example, Multi-WAN gateway group) and click on Save.

It is good practice to create failover groups for each of the gateways, so we click on the Add button and add a group called FAILOVER1 which has WAN on Tier 1 and OPT_WAN on Tier 2. Thus, OPT_WAN only becomes invoked if WAN fails. Again, Trigger Level is set to Member down. We click on Save and then create another failover group called FAILOVER2 which has OPT_WAN on Tier 1 and WAN on Tier2. When we are finished configuring all the groups, we click on Apply Changes on the main Routing page.

Next, we must create firewall rules to pass traffic from the local interfaces to the gateway groups. In the section on load balancing in the previous chapter, I described a floating rule which covered all local interfaces. Because of the configuration of the network, we cannot use floating rules here. The LAN network has access to all local networks and the internet, but the DMZ, as its name suggests, only has access to the internet and has no access to local networks. Therefore, we cannot create a universal rule. Fortunately, it is not too difficult to modify the existing firewall rules for our needs.

First, we navigate to Firewall | Rules and click on the LAN tab. There should be a Default allow LAN to any rule. We click on the Edit option for this rule and, keeping all other settings the same, we scroll down to Extra Options, click on the Show Advanced button, and then scroll down to Gateway under Advanced options. In the Gateway drop-down box, we select the MULTIWAN gateway we created earlier. We change the description to reflect the change, and then click on the Save button.

We also need to create rules for each of the failover groups, so we click on the copy icon for the rule we just modified to create a new rule based on the old one. Again, keeping other settings the same, we scroll down to the Gateway drop-down box and set the gateway to FAILOVER1. We click on Save and repeat the process, only instead of FAILOVER1, FAILOVER2 will be the gateway. Once we have created the rules, we look at the table to make sure the rule which set MULTIWAN as the gateway is closer to the top of the table than the other two rules, since rules are evaluated from the top down and we want traffic to be directed to the MULTIWAN group if it is functional. Once we have confirmed that the rules are correct and are in the right order, we click the Apply Changes button.

Now we have to create rules for the DMZ, so we click on the DMZ tab. There won't be an Allow DMZ to any rule, but if whoever set up the system set it up correctly, there should be an Allow DMZ to WAN rule. When we find this rule, we click on the Edit icon for the rule and repeat the process outlined earlier for the LAN interface – namely, we keep all other settings the same, but set the gateway in the Gateway drop-down box to MULTIWAN and alter the Description field accordingly. Then we click on Save and repeat the process, clicking on the Copy icon for the modified rule and create rules with FAILOVER1 and FAILOVER2, making sure that the rule setting MULTIWAN is closer to the top of the table than the other rules. When we are done making changes, we click on the Apply Changes button.

Keep in mind that the firewall rules we created only cover the standard use case. If you want to alter or add to the rules to add policy-based routing to your setup, it is fairly easy to do so. When editing a firewall rule, choose the criteria for the traffic you want to match (for example, protocol or maybe only traffic that originates from certain IP addresses); then under Advanced Options, scroll down to Gateway and select the gateway to which you want to direct traffic. Save the rule, make sure the order of the firewall rules is correct, and then apply the changes. The Gateway option can be found in both per-interface firewall rules and floating rules, so this form of policy-based routing can be applied to the entire network or just a particular subnet quite easily.

Now that we have created the firewall rules, our gateway load balancing setup is complete. We may want to reset states in order to force all outbound traffic to use the newly configured groups (to do so, we would navigate to Diagnostics | States, click on the Reset States tab, and click on the Reset button), but otherwise, all we need to do is navigate to Status | Gateways and confirm that the gateways are up and running and that the gateway groups are working.

Thus, we have demonstrated that a multi-WAN setup can be added to a CARP failover group. The main considerations in such cases is that we add virtual IPs for each OPT_WAN interface we add to the group. If there are any special use cases within your setup, you will have to add a virtual IP for that as well. For example, you may have a mail server or FTP server for internal use that uses 1:1 NAT. In such a case, you will need a virtual IP so that the NAT rules work on both the primary and secondary firewalls. You will also have to create a NAT rule for each OPT_WAN interface and each internal interface you want to pass through the OPT_WAN interface. Finally, we must set up firewall rules that direct traffic to the gateway group.