The peculiarities of choosing training and testing sets are especially important in both systematic investing and algorithmic trading. The main way to test trading algorithms is a process called backtesting.
Backtesting means we train the algorithm on data from a certain time period and then test its performance on older data. For example, we could train on data from a date range of 2015 to 2018 and then test on data from 1990 to 2015. By doing this, not only is the model's accuracy tested, but the backtested algorithm executes virtual trades so its profitability can be evaluated. Backtesting is done because there is plenty of past data available.
With all that being said, backtesting does suffer from several biases. Let's take a look at four of the most important biases that we need to be aware of:
- Look-ahead bias: This is introduced if future data is accidentally included at a point in the simulation where that data would not have been available yet. This can be caused by a technical bug in the simulator, but it can also stem from a parameter calculation. If a strategy makes use of the correlation between two securities, for example, and the correlation is calculated once for all time, a look-ahead bias is introduced. The same goes for the calculation of maxima or minima.
- Survivorship bias: This is introduced if only stocks that still exist at the time of testing are included in the simulation. Consider, for example, the 2008 financial crisis in which many firms went bankrupt. Leaving the stocks of these firms out when building a simulator in 2018 would introduce survivorship bias. After all, the algorithm could have invested in those stocks in 2008.
- Psychological tolerance bias: What looks good in a backtest might not be good in real life. Consider an algorithm that loses money for four months in a row before making it all back in a backtest. We might feel satisfied with this algorithm. However, if the algorithm loses money for four months in a row in real life and we don't know whether it will make that amount back, then will we sit tight or pull the plug? In the backtest, we know the final result, but in real life, we do not.
- Overfitting: This is a problem for all machine learning algorithms, but in backtesting, overfitting is a persistent and insidious problem. Not only does the algorithm potentially overfit, but the designer of the algorithm might also use knowledge about the past and build an algorithm that overfits to it. It is easy to pick stocks in hindsight, and knowledge can be incorporated into models that then look great in backtests. While it might be subtle, such as relying on certain correlations that held up well in the past, but it is easy to build bias into models that are evaluated in backtesting.
Building good testing regimes is a core activity of any quantitative investment firm or anyone working intensively with forecasting. One popular strategy for testing algorithms, other than backtesting, testing models on data that is statistically similar to stock data but differs because it's generated. We might build a generator for data that looks like real stock data but is not real, thus avoiding knowledge about real market events creeping into our models.
Another option is to deploy models silently and test them in the future. The algorithm runs but executes only virtual trades so that if things go wrong, no money will be lost. This approach makes use of future data instead of past data. However, the downside to this method is that we have to wait for quite a while before the algorithm can be used.
In practice, a combination regime is used. Statisticians carefully design regimes to see how an algorithm responds to different simulations. In our web traffic forecasting model, we will simply validate on different pages and then test on future data in the end.