Let's take a look at the various nodes that the NetScaler operates in. First, we'll look at two different ways in which the NetScaler behavior is influenced, based on how your Virtual IPs are configured.
Let's clarify upfront that this isn't a knob you will find on the NetScaler and, as such, it doesn't pertain to the set of options you can select under System | Settings | Configure Modes. Instead, it refers to how the NetScaler handles the TCP connection when features such as Compression, TCP Buffering, SSL Offload, and AppFw (which require the packet to be written) are in play.
As a consequence of this rewriting, when operating in the Endpoint mode, the NetScaler takes ownership of reliable TCP delivery, managing window sizes and MTU independently for both legs of the conversation. So, if in your troubleshooting, you see that NetScaler is ACKing a packet before the server has sent in its ACK, it is down to the NetScaler acting in the Endpoint mode.
Now, with this understanding out of the way, let's look at some of the modes that are not enabled by default and what their impact on your environment will be.
You need to consider what level of protocol processing is in play. NetScaler has the most to offer in performance when it's running a protocol natively (as in, you select a VIP type that matches the type of traffic), for example, HTTP. This is also where it can offer the most security as it is able to inspect pretty much everything in the flow.
When troubleshooting, however, you might occasionally see a situation where you will need to dumb down the processing for a moment so you can understand whether the optimizations that are being offered are causing the failure that you are troubleshooting. A technique I sometimes find myself using is to move momentarily from a VIP configured as the native protocol type (HTTP) to TCP. This will still offer TCP optimizations but will help to understand whether, for example, one of the packet rewriting features is causing the App to fail.
Very rarely have I had to consider going a level further down for troubleshooting purposes and reconfigure the same HTTP VIP as type ANY, such as when deliberately bypassing the inbuilt window-based protection mechanisms to prove raw throughput in a POC. By setting up the VIP as type ANY, we pretty much turned the NetScaler into a Layer 3 Load balancer.