In the previous chapter we explained what the Walk Forward Analysis (WFA) is. Now we are going to add a much more complex analysis – the Cluster Walk Forward Analysis (CWFA). CWFA is a very sophisticated tool that helps us determine whether the automated trading system (ATS) is really robust, i.e. able to generate profits in live trading. CWFA contains a set of many WFAs with different Out-of-Sample periods expressed as percentages and as different numbers of runs. In Fig. 1 you can see a classic example of WFA performed in TradeStation with default settings. This CWFA includes 30 WFAs with different Out-of-Sample periods (rows) — specifically 10%, 15%, 20%, 25%, and 30% — and with different numbers of runs — specifically 5, 10, 15, 20, 25, and 30 (columns). In Fig. 1 you can see that I chose an example in which all 30 WFAs have successfully passed the analysis. In other words, the system met the test criteria (which we will explain later within our series on CWFA). In this case, the overall results for the given WFA include a box saying «PASS». Yet in practice we will rather meet with the variant that most of the 30 WFAs will not meet some of the test criteria and in such case we will see a box saying «FAILED» in the WFA´s results – see fig. 2. This is because CWFA is a very demanding robustness test and only few ATSs are robust enough to meet the test criteria. In Fig. 2 you can see a CWFA the results of which clearly show that the ATS is not sufficiently robust for the selected market and timeframe. In contrast, Fig. 1 shows a perfect example of an ATS that is potentially very robust for the selected market and timeframe. It must be said that this is a fairly rare phenomenon. Searching for a truly robust ATOS by CWFA can often be likened to searching for a needle in a haystack (if it was easy then the algorithmic trading could be performed by anyone).
What is the threshold for a robust ATS according to CWFA results? I’m leaving that as an open topic which I will address on my portal AOStrading.cz in the near future.
Fig. 1: Cluster Walk Forward Analysis comprising 30 Walk Forward Analyses all of which passed the demanding test criteria (PASS)
Fig. 2: Cluster Walk Forward Analysis comprising 30 Walk Forward Analyses 27 of which did not pass the demanding test criteria (FAILED)
Since we have introduced many new terms, we should graphically show (see Fig. 3) the individual links between Walk Forward tests, Walk Forward analysis, and Cluster Walk Forward analysis. You can see that CWFA includes many Walk Forward tests that are the basis of WFA.
Fig. 3: The link between Walk Forward Tests, Walk Forward Analysis and Cluster Walk Forward Analysis
While one WFA can provide a preliminary and a very limited indication of an ATS´s robustness (one successful WFA may be a coincidence), CWFA is able to prove or disprove (in great detail) the ATS´s validity and robustness with a much higher degree of certainty (30 successful WFAs are not a coincidence but a valid statistical sample). CWFA, which is available in the TradeStation platform, is unique in its ability to execute a set of WFAs. It helps us to avoid the «Curve-Fitting» of results of the input parameter optimisation. Curve-Fitting is a phenomenon of overoptimisation and overvaluation of results. The problem is that after trying many different combinations we find a combination of input parameter values (Inputs) which captures the past trends very well and do not execute some crucial losing trades which would be executed with another input values. This combination of parameter values generates wonderful and unique results (such as very high net profit with the minimum possible maximum drawdown). Unfortunately, such results are often totally accidental. This means that in live trading the system will most likely have very different results (often literally catastrophic). In order to eliminate Curve Fitting as much as possible we use CWFA which helps us to confirm that the good performance results for a selected In-Sample input parameter setting are not a coincidence. The ATS that succeeds in CWFA according to predetermined test criteria will most likely withstand the constantly changing market conditions thanks to its natural qualities and stability. In Fig. 4 you can see the individual test criteria that we use in CWFA in TradeStation.
Fig. 4: Cluster Walk Forward Analysis Test Criteria in TradeStation
We have explained what CWFA is and showed examples of ATS that passed (PASS) and failed (FAILED) the demanding test criteria of individual WFAs. We have also explained the connection of Walk Forward tests, WFA, and CWFA. In the next chapter we will focus on the individual test criteria that must be met in order for the ATS to pass the individual WFA robustness tests within CWFA.
Petr
(c) AOStrading.cz
Previous: 22. Walk-Forward Analysis in TradeStation – Basic Tool for ATS Robustness Testing
Next: 24. Testing Criteria of Cluster Walk Forward Analysis in TradeStation
