Which VPN protocol you choose will likely be based on a number of factors. Interoperability is one factor to consider. If you need a VPN solution that is interoperable with another firewall or router, especially one from another vendor, then IPsec may be the ideal protocol to use, since it is included with every VPN-capable device. Using IPsec will also prevent you from being locked into a particular product or vendor, is easier to configure than L2TP, and is about as easy to configure as OpenVPN.  If interoperability is your main concern, you might consider OpenVPN, which is rapidly gaining traction, although it is not as ubiquitous as IPsec.

Another consideration is what type of authentication the protocol uses. IPsec allows you to use a pre-shared key or certificates, as well as username/password combinations. L2TP does not provide for any authentication, while OpenVPN supports pre-shared keys and certificates.

Ease of configuration is another consideration. All of the VPN protocol options available under the current version of pfSense (IPsec, L2TP, and OpenVPN) are fairly easy to configure, but some are easier than others. OpenVPN requires the use of certificates for remote access in many environments but is otherwise relatively easy to configure. IPsec, on the other hand, can be somewhat difficult for the uninitiated, although IPsec may be preferable because of its near-universal acceptance.

More often than not, your choice will be dictated by what operating systems you will be supporting and what clients are available for these operating systems. If your network is Windows-centric, you may consider using IPsec. Support for IPsec is built directly into Windows and has been since Windows Vista. As a result, connecting to a VPN with IPsec under Windows can be as easy as navigating to Settings | Control Panel, clicking on Network and Sharing Center, clicking on Set up a new connection or network, and then using the wizard to set up an IPsec/L2TP connection. You can also use third-party VPN clients, such as the Shrew Soft VPN Client.

On the other hand, most Linux distributions do not have built-in VPN support. Ubuntu and distributions that are derivative of Ubuntu (for example, Linux Mint) have built-in support for PPTP, a protocol that is no longer supported by pfSense. Third-party clients are available, and in some cases, can be downloaded from repositories, and these clients involve varying degrees of configuration. If your network is Linux-centric, you should be able to support IPsec, although OpenVPN is probably a better option in such cases.

macOS X has had IPsec support for years, and now even has a user-friendly interface for IPsec. OS X 10.6 (Snow Leopard) and later has a built-in Cisco IPsec VPN client that provides an easy-to-use graphical interface for connecting to a network that supports IPsec. Earlier versions of macOS X do not have the Cisco built-in VPN client, but you can install the Cisco Remote Access IPsec client on them. You can also use the Cisco AnyConnect Secure Mobility Client on earlier versions, although you should be aware that support for macOS X 10.5 (Leopard) was dropped with version 3.1 of AnyConnect.

For a network that is likely to have a variety of platforms, L2TP is a good choice. Because of the inherent lack of encryption and confidentiality in L2TP, it is usually implemented in conjunction with IPsec. Still, there are several clients on different platforms that support L2TP without IPsec. Beginning with Windows Vista, Windows has built-in support for L2TP without IPsec. One of the utilities provided for L2TP configuration is a Microsoft Management Console (MMC) snap-in called Windows Firewall with Advanced Security (WFwAS) and it can be found in Control Panel | Administrative Tools. The other is a command-line tool called netsh advfirewall.

Support for L2TP is not built into Linux, but there are third-party clients available. They are available for most popular distributions such as Arch Linux and Ubuntu, and configuration for most of these clients is relatively easy.

The Cisco IPsec client for macOS X supports L2TP, but it appears that it supports only L2TP over IPsec. At the time of writing, there does not appear to be a third-party client for macOS that supports native L2TP without IPsec. Thus L2TP is a poor choice if your network must support computers running macOS.

OpenVPN has been ported to several operating systems. Windows does not have built-in support for OpenVPN, but there are several third-party clients for Windows. In fact, the OpenVPN project has a client for Windows that works on XP or later, and it is easy to install and configure.

Linux not only supports OpenVPN, but OpenVPN support is built into many popular Linux distributions. OpenVPN configuration through the Network Connections applet in Ubuntu and its variants is rather easy, and it supports authentication with both certificates and with a pre-shared key. This makes OpenVPN an excellent choice if you are mainly supporting Linux clients.

macOS X does not have built-in support for OpenVPN. The OpenVPN project does not provide a macOS version of their client, and, to my knowledge, no one has successfully compiled the source code of the client under macOS. There is an open source project called Tunnelblick, which provides the necessary drivers for implementing OpenVPN under OS X. It has a graphical interface that provides a way to control either server or client connections. It can be used on its own or in conjunction with commercial software such as Viscosity. For more information, see the Tunnelblick website at http://tunnelblick.net.

If your network setup is fairly complex, your choice of protocol may be dictated at least in part by how well the protocol works behind multiple firewalls. Some of these firewalls may be beyond your control, and their configurations and capabilities may differ substantially.

IPsec uses both UDP port 500 (for IKE) and the ESP protocol. Not all firewalls handle ESP traffic well when NAT is used, because the ESP protocol does not have port numbers that make it easily trackable by NAT devices. IPsec clients behind firewalls may require NAT-T to function, which encapsulates ESP traffic over port 4500 using the UDP protocol. Versions 2.0 and later of pfSense support NAT-T, so you should be able to utilize NAT traversal with IPsec if necessary.

OpenVPN is generally more firewall friendly than IPsec. It uses TCP or UDP and thus is not affected by NAT behavior such as the rewriting of source ports. As a result, it is rare that a firewall won't work with OpenVPN. One possible issue is that the protocol and port may be blocked. OpenVPN uses port 1194 by default; if that port is blocked, you may want to switch to a port commonly used for something else to evade egress filtering. For example, ports 80 and 443 are assigned for HTTP and HTTPS respectively, but any TCP traffic should pass through these ports, so you could use them.

Since L2TP uses UDP, it shouldn't create any especially challenging issues with firewalls. It is often used with IPsec, however, so all of the issues related to IPsec come into play when you are using L2TP/IPsec.

One of the justifications for using VPNs is cryptographic security, so this is another factor to consider. Point-to-Point Tunneling Protocol (PPTP), which has been removed from the current version of pfSense, has numerous security vulnerabilities, and thus became a poor choice for the security-conscious administrator long ago. L2TP has no encryption capability; if you want encryption, you'll have to use it in combination with another protocol (usually IPsec). Therefore, the choice essentially comes down to either IPsec or OpenVPN.

OpenVPN uses the SSL encryption library, which provides a number of different ciphers. To find out what ciphers the version of OpenVPN installed with pfSense supports, execute the following command, either at the pfSense console's Command Prompt or from Diagnostics | Command Prompt:

openvpn –-show-ciphers

As you can see from the screenshot, there are quite a few options. OpenVPN's default encryption algorithm is BF-CBC, or Blowfish, block cipher, with a 128-bit (variable) key size. While this is not a terrible cipher, it may be beneficial to choose a stronger cipher, such as AES-256-CBC.

OpenVPN also offers a number of different digests for message authentication, including many of the digests supported by IPsec (for example, SHA512). To see a list of digests supported by OpenVPN, use the following command:

openvpn --show-digests

One factor working against OpenVPN is that it seems that OpenVPN developers have generally given priority to backward compatibility over security. This and the fact that IPsec operates at Layer 3 of the OSI model and therefore provides encryption on the IP level would seem to give IPsec a slight advantage over OpenVPN in cryptographic security.

It might prove useful to provide a summary of some of the features of each VPN protocol currently supported by pfSense, so with that in mind, here it is: