In the previous chapter we addressed optimisation tests which are the essential tool for robustness testing. We spoke about the optimisation tests primarily in relation with the sophisticated software platform TradeStation. Another tool which I use very often in this platform and which I perceive as the platform´s major competitive advantage is the Cluster Walk Forward optimisation (CWFO). Before I explain the principle and the greatest advantages of CWFO in some of the next chapters, it is necessary that you fully understand the principle of testing and validation on Out-of-Sample data. It is because that the Out-of-Sample testing is in fact the fundamental basis of CWFO. First you have to understand the difference between In-Sample and Out-of-Sample historical data. When you backtest on historical data, it is always good to set apart a certain period for testing the results´ validity. So if we have a 10-year data set of historical data, we split the data in a certain ratio. Everyone can choose the ratio at their individual discretion. There are many traders who use, for example, the ratio of 70% for In-Sample data and 30% for Out-of-Sample data. Thus if we have historical data covering 10 years, we split it into 2 parts:
70% = 7 years of In-Sample historical data
30% = 3 years of Out-of-Sample historical data
In Fig. 1 you can see the graphical division to In-Sample and Out-of-Sample data.
Fig. 1: In-Sample and Out-of-Sample testing
We have set the ratio of our In-Sample and Out-of-Sample data and now we backtest our trading system on the In-Sample data (the first seven years of data). If the backtesting on the In-Sample data with a certain parameter setting (we explained the parameter values in the previous chapter) gives stable and profitable results, we perform the same backtesting with the same parameter values on the Out-of-Sample data (the remaining three years). In case the trading system is stable and profitable on the Out-of-Sample data as well, we have validated its functionality on unknown data. In other words, the division of data into two samples is a very important step in evaluation of an automated trading system´s (ATS) potential profitability because it allows us to test the system settings on an unknown data sample (Out-of-Sample) that was not used in building of the system. This is called the ATS optimisation. One of the testing criteria can be, for example, the condition that the Out-of-Sample (OOS) results are equally profitable as the In-Sample (IS) results. Given that the In-Sample and Out-of-Sample data always have different percentages (e.g. 70% IS and 30% OOS, 60% IS and 40% OOS, or 80% IS and 20% OOS), we have to standardise the ATS´s efficiency by calculating the profitability per day. Otherwise, for example, the 70% IS results will obviously be always better than the 30% OOS results. The result then show a certain efficiency percentage of the Out-of-Sample results (0% and more). If an OOS data efficiency is 55%, we can say that the ATS has a 55% performance on the OOS data in comparison with the IS data. In TradeStation you can find this percentage ratio under the Robustness Index value. It the Robustness Index reaches 120%, it means that the ATS is more efficient on the Out-of-Sample data. The exact calculation procedure of Robustness Index is not the subject of this chapter. You can find the calculation procedure in Chapter 24.
Let´s add another tool to the In-Sample and Out-of-Sample testing – the ATS optimisation.
In the previous chapter we have explained the optimisation on the example of an ATS based on crossing of two moving averages. Yet there are never enough examples. This time, I am going combine the issue of parameter value optimisation with the In-Sample and Out-of-Sample data division.
We will call our trading system HIGHLOWCLOSE CONDITIONS. To make the example more illustrate, we will focus on the entry conditions only and we leave all the exit conditions aside. The entry condition for the Long side is: If the current today´s Low is higher than the previous day’s Close, then enter the market at the current High by the MARKET BUY order. Now consider a situation that we changed the previous-day-Close condition to a Close two days back, Close three days back, or Close 4 days back. Why should we be interested in trying such an optimisation? Obviously, our goal is always to find the most efficient ATS, i.e. the highest Robustness Index, because our premise is that these systems have the greatest profit potential for the future live trading. The point is that Out-of-Sample data simulate live trading, because it is an unknown data for our trading system. In order to be able to perform a trading system optimisation by In-Sample and Out-of-Sample data, we first have to set some testing criteria and methodology by which we will choose the best In-Sample parameter settings to be validated on Out-of-Sample data. This methodology for evaluation and selection of ATS parameter settings is called the Fitness Function. It may be, for example, the highest Profit Factor, Net Profit, Net Profit/Max Drawdown, etc. It is virtually impossible for discretionary traders to optimise their systems on In-Sample data because it is an incredibly time-consuming process. Yet for a trader using automated trading systems such an optimisation is definitely not an insolvable problem. The trader simply sets the Close variable to X days back and then test various values of X (from 1 to any number of days). If the trader sets the variable X values from 1 to 10, TradeStation automatically performs all optimisation tests (i.e. 10 tests in this example). Optimisation tests showed that the best results of backtesting In-Sample data for the selected Fitness Function (in this example the Net Profit/Max Drawdown) has the setting «Close two days back» because it showed the highest profitability and the lowest drawdown of the 10 tests. Now we apply the HIGHLOWCLOSE CONDITIONS trading strategy with the Close two days back condition on the Out-of-Sample data. Out-of-Sample data substitutes the future data and gives us important information about whether the trading is likely to generate profits in live trading in the future. The metrics for this assessment is the already mentioned Robustness Index. In other words, if a trading system shows good results in terms of Fitness Function in time within In-Sample data backtesting and Out-of-Sample testing confirms the system´s stable profitability, the trader gets a very good signal that the trading system may be robust. You can add the term robust to your trading glossary because you are going to meet with it very often. Robust trading system is a system that is able to adapt to changing market conditions and that has a potential of stable long-term profits. But be careful, robustness does not mean that the system may not have losing periods! The point is that losing periods do not exceed the level predicted and defined on the basis of backtesting. The combination of In-Sample and Out-of-Sample tests gives us a high probability of a successful identification of robust trading systems. Yet it would be foolish to think that all we need is a common IS and OOS data sample. We should definitely employ also another very complex and sophisticated tool available in the TradeStation platform – Cluster Walk Forward Optimisation (CWFO). CWFO performs a huge amount of In-Sample and Out-of-Sample tests with different In-Sample/Out-of-Sample data ratios (e.g. 80% IS and 20% OOS, or 70% IS and 30% OOS) for various time periods. CWFA is such a complex instrument that we will address it in detail in other chapters
In conclusion, let´s have a look at two equity curves divided into In-Sample and Out-of-Sample data. In Fig. 2 you can see a promising Out-of-Sample test with potentially high Robustness Index and in Fig 3 you can see an Out-of-Sample test which clearly indicates that the ATS is not profitable on unknown data and therefore has a low Robustness Index.
Fig. 2: In-Sample and Out-of-Sample test
The In-sample test (blue curve) of the trading strategy shows promising results because the equity curve tends to reach new and new peaks. In this case, the Out-of-Sample test (green curve) indicates that the ATS has the potential to generate profits on future data in live trading because on unknown data in the form of Out-of-Sample test it showed stability and correlation (similarity of distribution of profits and losses) with In-Sample data. Of course, we need to carry out further tests via the Cluster Walk Forward Analysis.
Fig. 3: In-Sample and Out-of-Sample test
The In-sample test (blue curve) of the ATS shows promising results because the equity curve tends to reach new and new peaks. However, the unsatisfactory Out-of-Sample test (red curve) has clearly indicated that the trading system has not the potential to generate profits on future data in live trading.
We explained the principle of In-Sample and Out-of-Sample testing combined with optimisation tests and the basic concepts relating to this topic, such as test criteria and Fitness Function. It is necessary that you get acquainted with these concepts as much as possible because we are going to use them within the Cluster Walk Forward optimisation in some of the following chapters.
Petr
(c) AOStrading.cz
Previous: 20. Importance of Trading System Parameter Optimisation in TradeStation
Next: 22. Walk-Forward Analysis in TradeStation – Basic Tool for ATS Robustness Testing
