Futures trading systems and commodity trading bear a high degree of risk. People can and do lose money. Hypothetical results have many inherent limitations. Past performance does not guarantee future results. Hypothetical results have many inherent limitations. Please read the disclosures & disclaimers page.
TradingVisions Systems, Inc. | Sedona, AZ | 928-554-4052
Translate This Page
Ideal versus Real Results
One of the most significant issues in system trading is "real" versus "ideal" performance numbers, so I'd like to provide a few informal definitions and discuss this important facet of automated trading, as well as provide some data.
Computers are powerful tools that allow us to analyze myriad data. When we state trading rules in analysis software like TradeStation and run tests on historical data, we are "backtesting." With computers we can run thousands of backtests, finding the needle of a sharp idea in a haystack of numbers. Anyone can look at a chart, spot a pattern consisting of a few examples, and see where they should have entered and exited, but the computer allows us to find hundreds of examples of the same conditions to see whether trading that pattern is consistent enough to be profitable over time. A computer further allows us to refine our ideas through "optimization," which is a test of a numerical idea, running through a series of values to find the optimal one(s). Backtested, optimized results are idealized because they give us the "best" fit to the curves of past data, or they are "curve-fitted." The data used as our testing period is called the sample data, and to distinguish it from other data, it is called "in-sample" data. Results that are from past data are also called "in-sample," meaning that they are derived from the tested data sample.
Unfortunately, although backtesting is a powerful tool, devising a system that shows profitable backtesting does not provide assurance of profits in a trading account. This is because markets are constantly evolving and tend to mutate. As we must say, "Past performance does not guarantee future results." We therefore should not place too much credence in backtested, in-sample, optimized, curve-fitted, idealized results. Instead, we look to results that come from "out-of-sample" data. This data is any that resides outside the original test data used. It can be from an earlier period of time, a later period of time, or from a different market (e.g. the e-mini Russell, instead of the e-mini S&P). The results from out-of-sample data have much more import because they tell us how our system trades in real markets, as opposed to the ideal one of our test sample. When those results are from data after the end of our study data, it is "real-time" results (some people use "real-time" to mean results from actual trades placed in an account, but I prefer to use it as any trades generated after the end period of in-sample data). The results are "real" in the sense that the trades could have been taken in real time, since the system already existed, and they were not results from the "ideal" time period over which the system was developed.
"Hypothetical" has two primary meanings in the context of trading. Less strictly it means results from the in-sample data set. More strictly it means any results that were not actual trades in an account. Unfortunately, this latter category masks an important distinction in hypothetical results. If the results are from the in-sample period, then they are idealized results. But if they are results from the out-of-sample period that may or may not have been actually traded in an account, then they have much more significance because they could have been real trades.
In judging a system, the real-time, out-of-sample, unoptimized results should carry more weight than the in-sample results, if there are a significant number of trades. We want to know how the system performed in less-than-ideal conditions.
The TradingVisions systems--particularly the WFO series--have a significant amount of out-of-sample, unoptimized performance, which in this case means performance that either occurred after the release of the system rules and parameters or performance based on out-of-sample data. If, for example, a system was released July 15, 2004, and it has not been substantively changed since then, the period after 7/15/04 is considered to be out-of-sample and real-time. Or if an optimization was run over the period 1/1/2000 to 12/31/2003, then any results from after that end date would be considered out-of-sample.
TradingVisions System Release Dates & Out-of-sample Periods