There's no doubt that troubleshooting traffic shaping issues can be challenging. Even if we set up the queues and rules the way we think they should be set up, it may not work the way we think it should, and in many cases, we might not get it to work right the first time. Often we must fall back on the tried and true troubleshooting steps: diagnosing the problem, forming a hypothesis, testing it, implementing a solution, verifying that the system works, and documenting the solution. But there are some common issues that we should also cover.
We may have problems keeping P2P traffic in the P2P queue. The main reason for this is that for the most part, pfSense relies on ports as matching criteria for the traffic shaping rules. Many P2P applications, including BitTorrent, rely on non-standard or random ports. There are several possible solutions to this. You could try to use Layer 7 DPI, which does not rely on ports but instead looks at the packets. However, Layer 7 traffic shaping is not available with the base pfSense installation; it is only available through third-party packages such as Snort.
If you don't want to use Layer 7 traffic shaping, another possibility is to make the default queue for all interfaces a low-priority queue. This approach, however, would require you to create rules for every type of traffic you don't want to be relegated to this queue.
If traffic is not being shaped correctly, there are several tools available to help you diagnose the problem. To get an overview of traffic on all queues, navigate to Status | Queues. This will reveal how much traffic is on each queue, both graphically (each queue has its own bar graph), and numerically in the form of PPS (packets per second), bandwidth, borrows (indicating whether bandwidth has been borrowed from a parent queue), suspends (indicating how many times a queue has been suspended), and drops (dropped packets). Also, length indicates how many packets are currently in the queue as well as the queue length (the default queue length is 50 slots, although the queue length can be changed). At the very least, this page will indicate what traffic if any is in the queues, which might be all you need to know to diagnose the problem.
The Queues page only shows traffic interactively, so what you see is a snapshot of the current traffic, rather than a summary of all the traffic in the queues. To see a cumulative summary of all traffic on the queues, navigate to Diagnostics | pfTop and select queue in the View drop-down box. This will provide similar information to that provided on the Queues page, only they are expressed in raw totals.
If you are using limiters, you can find information about their configuration by navigating to Diagnostics | Limiter Info. This will show configuration information and statistics for all limiters and any child queues they may have.
Finally, if none of these diagnostic tools are adequate, navigate to Status | System Logs. The Firewall tab is most likely to yield relevant results, so you'll want to click on that. Once you do, you can filter the results by clicking on the + button on the Advanced Log Filter bar. This will display the advanced filtering options. You can filter the logs by port, which can be helpful in finding out whether certain traffic is being passed into queues. If necessary, you can enable logging on the relevant rule. Generally, we try not to log every rule we create, since logging consumes disk space, but if you think enabling logging might help with troubleshooting, temporarily enable logging is always an option.
If your system is not deployed in a production environment and you can experiment with different settings, you might want to try different configurations and see what results you get. For example, it is generally recommended that you not alter the queue size settings (this is why in the examples, we left Queue Limit blank, which leaves the queue size at its default of 50). Some users have reported better results with larger queues, although too large of a queue will likely lead to bufferbloat. But, often the only way to find out is to test it on your router. Therefore, if you think that altering a setting might help optimize performance, and you can do it without breaking your network, feel free to make the changes and document the results.