Futures trading systems and commodity trading bear a high degree of risk. People can and do lose money. Hypothetical results have many inherent limitations. Past performance does not guarantee future results. Hypothetical results have many inherent limitations. Please read the disclosures & disclaimers page.
TradingVisions Systems, Inc. | Sedona, AZ | 928-554-4052
By allowing different parameters for different markets, WFO has the additional advantage of thereby decreasing correlation between results for those markets. With lower correlations, these markets then become stronger candidates for inclusion in a portfolio.
WFO also allows us to make in-flight corrections of our system parameters. In the past I have set an unswerving hoped-for course from hypothetical results to actual profits by carefully constructing a system and then not deviating from it. This straight-line approach ignores the fact that markets continually and of necessity change nature: Markets are moving targets. WFO allows a system to adapt to market changes.
This is both its inherent advantage, and its Achilles heel.
The advantage is that when there is a significant change in a market, if a system has rules that are sensitive to that change, a new optimization will cause parameters that better exploit the change and produce more profits to be chosen for use in actual trading.
However, this is only an advantage if the market changes have enough persistence beyond the latest study period. And this is the Achilles hee of WFO, that the parameters will be changed just in time to mis-perform during the next phase of the market. This is why WFO is not the Holy Grail: important decisions are made in doing WFO that leave it itself susceptible to over-curve-fitting. Among these decisions are how long a study period and application period to use, how many iterations are ideal in an optimization, , what makes the "best" measure of performance, what makes the "best" choice of the results in an optimization run, what period in time does one use as the start point of the first study period, and how much variance in approach is acceptable for different markets.
To lessen the chance of over-optimization and over curve-fitting, TradingVisions uses the same protocols for all systems and markets. Specifically:
1.) Exhaustive, rather than "genetic," optimization is used whenever possible. Genetic optimization--so called because it mimics the process of natural selection--uses data sampling, and this causes the walk-forward results to vary each time it's run, and to also vary from the exhaustive results. An exhaustive optimization uses all avaliable data. Therefore, to get a consistent reading, only exhaustive is used, at least at the end of the optimization analysis.
2.) Anchored walk-forward is used--each retired application period is added to all prior data as we move forward in time. I've found that for the TradingVisions systems, there is a slight performance edge to using anchored over running walk-forward. Anchored WFO also helps lessen the possibility of making parameter changes just in time to get out of phase with the market. The parameters change less dramatically and less frequently, and by optimizing over longer and longer study times, the best parameters will tend to be those that are able to adapt to a wider range of market permutations.
3.) A "run" is an out-of-sample application period. TradingVisions uses the number of runs that yields a period closest to 1 year. For systems that trade 30-200 times per year, this yields a statistically significant number of trades. A year is also a natural time demarcation. For most of the TradingVisions systems, this means 8 or 9 runs.
4.) The "best" performance is determined by net profit/maximum drawdown. I believe this is the safest and most important metric, since it takes both risk and reward into account. My studies have shown this to be the metric that is most likely to yield good out-of-sample performance.
5.) Subjective judgments are strictly minimized, although it is impossible to remove all subjectivty and curve-fitting from WFO. One important area where there may be some subjective decisions is in order to fit the realities of trading. What I mean by this, is that sometimes the "best" values may be ones that in practice a trader would be reluctant to trade or would prefer not to trade, if a reasonable alternative tests out almost as well. For example, for AXIOM ES a protective stop was tested from $300 to $1600. For the regular optimization test, the very best profit/max drawdown ratio of 13.92 is at $1500, and when the results from the regular optimization are tested in the TradeStation walk-forward module, the ratio of out-of-sample results to the max drawdown is 6.89. However, I also noticed that in the regular optimization results, the profit/drawdown ratio for a stop of $1100 is 13.89, virtually the same as at $1500. Since one reality of trading is a preference for lower stops, I then also tested the stop range of $300-$1100 in the walk-forward optimizer, and the profit/drawdown ratio is 8.63, considerably better (the profits are slightly lower and the max drawdown is lower). I therefore chose to go with the $300-$1100 optimization range, which means that when that data is run through the walk-forward optimizer, the maximum recommended stop loss value for any given period is $1100. I would have chosen this even if the resulting ratio had been a little lower, just on the principal that most traders psychologically prefer to trade with a lower stop loss.
Walk-forward optimization is the closest we can get to actual results prior to trading, and this is what makes it so powerful. Because the results are from out-of-sample, unoptimized data, it allows us to see how a strategy could have performed, had we been trading it in the past. This helps us to measure the system's robustness and to give us increased confidence when we start actual trading. WFO does not guarantee profits, but it is a step closer to accurately measuring the strength of a system.
Walk-Forward Optimization: TradingVisions' WFO Series
Translate This Page
In mid-2013TradingVisions released new versions of Delphi II, AXIOM II, and Sentinel that incorporate a "walk-forward optimization" protocol, as well as new Vista WFO Portfolios that utilize these versions. In December, 2015, TradingVisions latest system, Metrica WFO, joined the offerings.
But what is walk-forward optimization?
Historically, the typical way to achieve strategies that are robust and give a reasonable expectation of future success is to test all available data for a market, adopt the "best" rules and parameters, and trade the system in real-time. We can call this approach look-backward optimization.
An improved approach is to not just look at all available data for one market, but to also look at additional markets, and to adopt the rules and parameters that best fit all or most of them. This approach--what I'll call look-beside optimization--is one I have used since I began developing systems over a dozen years ago.
I have been a firm believer that over-curve-fitting is the bane of system development, and to avoid it, I have been very reluctant to re-optimize systems once released. It is just too easy to find something that works on a single market for a few years in backtesting and re-jigger it every time results are poor.
My approach has worked quite well, but in early 2007 I began intensively testing a way of judiciously using re-optimization on Delphi II that proved to be preferable. This involves the use of walk-forward optimization (WFO), and it helps to solve two critical shortcomings of traditional optimization: 1.) traditional hypothetical performance records are idealized (optimized), "in-sample" results (i.e. results that are generated by the same data used to formulate the system rules & parameters), and these results usually vary widely from real-time results, and 2.) traditional optimization has no articulated way to adapt to different markets or to changes within a market.
But what if we had a way to find out how a system performs without waiting for years and risking our money?
WFO optimizes a system over a set time period from the past--say three years--and applies the "best" parameters to a subsequent time period, say one year. The performance results of the three years study period are "in-sample," as defined above; in other words, the in-sample data was used to find the optimal values. These results are curve-fitted, idealized performance that is rarely matched in actual trading. In contrast, the performance on that one year after the study period is "out-of-sample," meaning that they are the result of trades that occur outside the period from which the rules and parameters were chosen. This is also called the "application period." The performance from the application period is much closer to what would have been achieved in actual trading, since the parameters were chosen prior to the trading, and we could theoretically have been trading during that time with that set of rules and parameters. In fact, if we use realistic slippage and commissions, we have what can be called "real-time" results, even though they are hypothetical in the sense that the trades were not actually made. These results are categorically more trustworthy than backtested results from a system's in-sample study period, and this is the most important advantage of WFO.
The "walk-forward" aspect of this approach comes in the next step. Let's say I decide that 3 years of data is sufficient to establish the rules and parameters of a system. If I used data from 1.1.2001 to 12.31.2003 as the first study period of the analysis, I would then determine my system rules and the ideal paramters from that 3 years. I could then run the system on 2004 data as my first application year, and the 2004 trading results are now my first year of the out-of-sample performance record. I now walk-forward a year and use the data from 1.1.2002 to 12.31.04 as my next study period, and I apply the "best" rules & parameters from this period to 2005. These 2005 out-of-sample results are added to my 2004 out-of-sample performance record. I continue to walk-forward to 1.1.2003-12.31.05, apply the best rulse & parameters to 2006, and I now have a 3-year hypothetical out-of-sample track record that should be reasonably close to what I could have achieved in actual trading. This methodology is calling "rolling" or "running" walk-forward, as opposed to "anchored" walk-forward. The anchored approach does not drop the earlier "study" periods of data, which has the effect of making each study period increasingly longer. This method, which is the one used by TradingVisions, usually leads to increasing stabilization of the parameter choices (more on this below).
There are several additional benefits to WFO.
In the past, I have worked very hard to have little or no difference in rules/parameters for different markets trading the same system. Optimizing to each market
using only in-sample data is very risky because there is no logical point at which to stop doing it. The almost inevitable result is over-curve-fitting and trading losses.
WFO allows for using different parameters for different markets because the backtested out-of-sample performance results for each market provide the validation of the methodology. When only in-sample data is used, the validation is based upon a circular relationship between the study data and the rules/parameters adopted: the results justify the rules/parameters, but the rules/parameters have been justified by the results. WFO takes us out of this pernicious loop and puts our expectations on a realistic foundation.