Troubleshooting VPN connections can be challenging, because the process of establishing a VPN tunnel involves multiple steps, and a failure at any one of these steps will result in a failure to establish a tunnel. For that reason, it might be helpful to adopt a step-by-step approach in troubleshooting:

1. Is the remote client (or peer) able to connect to pfSense (acting either as a server or peer)? If not, then it's possible that either the VPN service is not running at the pfSense end, or, more likely, ports may be blocked that need to be unblocked. IPsec's port requirements differ depending on your configuration; consult the table included in the section of this chapter on IPsec configuration for more information. OpenVPN normally accepts connections on port 1194; you can change this port, but if you are using OpenVPN, whichever port you use must be unblocked. If you are using IPsec, port 500 must be open for ISAKMP key exchange traffic, and port 50 or 51 must be kept open depending on your configuration (but not both); if NAT traversal is required, port 4500 must be open. L2TP uses UDP on port 1701, so if you are using L2TP, make sure that port is open. Keep in mind that autogenerated rules created by pfSense are floating rules, so check the Floating tab to confirm that the necessary rules exist. If you are using OpenVPN, keep in mind that the autogenerated firewall rule for OpenVPN only allows UDP traffic. Therefore, if you are using OpenVPN over TCP, you may have to modify the rule to allow OpenVPN traffic on the WAN interface to allow TCP traffic.

2. If the remote client is able to make an initial connection to the server, but the connection ultimately fails, then often the best source for information as to why the connection failed is the log file. You can find the logs by navigating to Status | System Logs. Click on the IPsec tab to find IPsec log entries and click on the OpenVPN tab to find OpenVPN log entries. Very often, the source of the problem is a mismatch between the settings required by the server and the settings specified by the client in the client configuration.
3. For example, assume that there has been a failure of a mobile client to establish an IPsec connection, and you see the following log entry:

Charon: 13[IKE] Aggressive Mode PSK disabled for security reasons

Such an entry is a good sign that there was a mismatch in the negotiation mode between the client and server. The Aggressive Mode PSK disabled message indicates that the negotiation mode on the server is set to main. Sure enough, we discover that the client has set Negotiation mode to Aggressive, which is why the connection failed. Changing the setting on the client to main does not guarantee it will work, but at least we know a negotiation mode mismatch won't be the problem.

Some log entries are a little more cryptic. For example, a pre-shared key (PSK) mismatch may appear in the logs as follows:

charon: 09[ENC] invalid HASH_V1 payload length, decryption failed?

This doesn't exactly give us a clear indication of a PSK mismatch; still, if you see such a message, it's a good idea to check the PSK entry.

In addition, many of the IPsec log entries indicate whether the failure was at Phase 1 of the connection or Phase 2. Such an indication can help clue us in as to where we should be focusing our troubleshooting efforts. When the connection fails at Phase 2, often the problem is mismatched encryption methods.

Very often, the client side has Auto as an option for some settings – in other words, autonegotiate settings. For example, the ShrewSoft VPN Client has an Auto setting for Cipher Algorithm and Hash Algorithm. Sometimes a setting of Auto will work; however, I have generally found that the connection is more likely to succeed if all settings are set to match the settings on the other end of the tunnel. For example, I was unable to connect to an IPsec tunnel with Auto settings for Cipher Algorithm and Hash Algorithm, but by setting these to AES-256 and SHA1 (the IPsec server settings for these two parameters), I was able to successfully establish an IPsec tunnel.

There may be some cases in which even though the parameters match on both ends of the tunnel, you are unable to establish a tunnel. In many cases, this is because the client only supports one method of connecting. For example, the ShrewSoft VPN Client seems to require IKEv1 and Negotiation mode of Aggressive, while the Windows built-in VPN Client requires IKEv2. In such cases, consulting the client software documentation may help, and looking online for additional information is always a good idea.

One thing that is worth noting is that if you switch from IKEv1 to IKEv2 and then back to IKEv1, Negotiation mode will be set to main, even if you originally set it to Aggressive, so be sure you have the correct setting for Negotiation mode before saving your settings.