This chapter´s content follows the previous chapter in which we explained what the Cluster Walk Forward analysis (CWFA) is. Let´s recall that in the ideal case we are looking for a CWFA where all 30 Walk Forward analyses (WFA) pass the defined test criteria. In such case, the overall results for the given WFA include a box saying “PASS” (see Fig. 1). If the test criteria are not met, the box for this WFA says “FAILED”.
Fig. 1: Cluster Walk Forward Analysis comprising 30 Walk Forward Analyses all of which passed the demanding test
In this chapter we will introduce the various test criteria that must be met within a WFA for the different number of runs and the Out-of-Sample (OOS) percentage in order to obtain the “PASS” result (box). There are five CWFA test criteria in TradeStation:
Test criterion no. 1: Overall Profitability
The default setting of CWFA is that the overall profitability of all OOS runs must be higher than 0. In Fig. 2 you can see that in total the selected ATS is profitable on OOS.
Fig. 2: Test criterion – overall profitability
Test criterion no. 2: Walk Forward Robustness
Walk Forward Efficiency is based on the “Robustness Index” which represents and compares the Out-of-Sample (OOS) and In-Sample (IS) results. In essence, it is an indicator of the trading system´s quality. Robustness Index is calculated as follows:
Robustness Index = (OOS profit * 365 / number of OOS days) / (IS profit * 365) / number of IS days)
The construction of Robustness Index shows that it compares the efficiency of OOS profits against IS profits. Since the time periods of IS and OOS differ (OOSs are shorter in CWFA), it is necessary to standardise the generated profits and therefore we recalculate them to per-day values. The test criteria include the requirement that Robustness Index for each Walk Forward analyse was at least 50%. This is the default setting of CWFA in TradeStation. Beginners may naturally ask the question: “Only 50%? Isn´t it too little?” In my opinion it is not. After including commissions and slippage into the ATS´s performance calculation I consider 50% for a WFA a realistic estimate. We must keep in mind that we search for the best input parameter setting within the IS period according to the selected Fitness Function (FF). Sometimes, a WFA´s Robustness Index may exceed 100%. In such case we know that the OOS run was more profitable than the IS run in the same IS/OOS-days ratio. In Fig. 3 you can see the result of Cluster Walk Forward Analysis for 30 WFAs. You can see that the “Higher Average Cluster” chose the WFA 9 period in the 3 x 3 format with the highest average Robustness index´s value of 88.4%. This is a very high value which implies a really highly robust ATS. Matrix Average then has the value of 78.2% which represents an average Robustness Index value for all 30 WFAs. Again, 78.2% is a very optimistic value. This CWFA was applied to the Intraday Breakout ATS which shows a high robustness on all markets with equity indices. Advanced traders will maybe find it interesting that the system had stable and symmetrical results both for Long and Short side.
Fig. 3: Results of Walk Forward Efficiency within Cluster Walk Forward Analysis
Test criterion no. 3: Consistency of Profits
The test criterion is that each Walk Forward analysis must show the consistency of profits higher than 50%. In other words, we want more than half of the runs to be profitable. When looking at Fig. 3, all Walk Forward analyses easily complied with this requirement. You can see 100% in the selected blue box which means that all 10 OOS runs were profitable.
Test criterion no. 4: Distribution of Profits
The default setting of this criterion in TradeStation is that a single run may not contribute by more than 50% to the overall profit of all OOS runs. The logic behind this is clear. Of course, we do not want to use trading systems that do not bring stable profits throughout the testing period. The fact that one run contributes by 50% to the overall profitability is actually negative since such a run may be only a coincidence and may not repeat again. In Fig. 4 you can see that the Out-of-Sample run no. 4 (OOS4) brought the profit of $10,229 while the total profit from all runs was $17,338. In Fig. 5 you can see the Walk Forward Summary, which is one of the tabs that can be displayed in TradeStation´s CWFA. In this tab you can find, among other important information, the exact values of profits/losses from individual WFA runs.
Fig. 4: CLuster Walk Forward Analysis and the Distribution of Profits testing criterion
Fig. 5: Walk Forward Summary (Out-of-Sample)
Test criterion no. 5: Maximum Drawdown in %
The last test criterion is the maximum tolerable drawdown. There are actually three possible settings (see Fig. 6).
Fig. 6: Maximum Drawdown test criterion testing
- The first possibility is that the maximum drawdown is exceeded in any one of the runs by 60% of the initial capital. The initial capital value is set in the test criteria settings. Thus, if we had a capital of $20,000, the maximum drawdown for each WFA should not exceed $12,000.
- The second possibility is that the maximum drawdown exceeds the actual cumulative profit in any of the runs by 60%. For example, the tested system generated a net profit of $40,000 before the Out-of-Sample run no. 8. The maximum drawdown for the selected WFA should not therefore exceed $24,000.
- The third possibility is that the maximum drawdown exceeds the cumulative profit before the beginning of the drawdown by 60%. For example, the net profit in the middle of the 8th Out-of-Sample run was $80,000. Thus the maximum drawdown would be $48,000.
In order to have a quality CWFA we need to perform at least 5 WFA runs so that we avoid the risk of accidental and statistically invalid results. On the other hand, if the number of runs is absurdly high, we face the risk that we will have only few trades in individual Out-of-Sample and thus we get statistically invalid results. All these aspects must be taken into account when evaluating the CWFA results.
To sum it up, in this chapter you learned what testing criteria must a trading system meet within CWFA so that we can consider it robust for the selected number of runs and Out-of-Sample %.