Backtesting biases

There are many biases that can make simulated returns unrealistic and unlikely to be representative of the future performance of a strategy. We've gone through great lengths to eliminate them, or give you the ability to avoid biases with historical simulations. In fact, it's one of our core goals -- we are ruthless in squashing them whenever we find them.

Survivorship Bias

Survivorship bias happens when you include only those companies that still exist today in backtests. Most lists of energy stocks won't include Enron, for example, and your portfolio will see the meteoric rise of Amazon but not the bankruptcy of Pets.com. We always include dead companies in backtests.

Look-ahead Bias

Look-ahead bias happens when you test using financial data before the market has seen that information. This happens, for example, when you start using December year-end figures in December, even though they weren't released until February or March. We use effective dates from our data provider whenever possible. If missing, we fallback to SEC filing dates with some lag to account for processing delays. As a last resort we lag financial data by 30 days for interim data and 45 days for annual data.

Congruence bias

Congruence bias is the tendency of people to over-rely on testing their initial hypothesis (the most congruent one) while neglecting to test alternative hypotheses. That is, people rarely try experiments that could disprove their initial belief, but rather try to repeat their initial results. We encourage our users to stress-test their strategies whenever possible.

Curve Fitting

Curve fitting is the subconscious process of making small changes to a strategy to achieve higher and higher returns. It's very hard to avoid even for the most seasoned designer. For this bias we can only give you tools and tips to combat it. You can, for example, rerun the strategy excluding all the stocks that were used in a previous run. A serious degradation in performance is a sign of curve fitting. We also have more advanced tools like rolling simulations and rank performance tests.

Market Impact

The data history that you use does not include your trades. So simulations do not really forecast the price you’re most likely to get when you trade. Since trading and pricing go hand in hand, neglecting market impact causes a bias that will influence your backtesting result. To address this bias we always anticipate that when you trade, the prices will be against you. In every simulation you can use variable slippage as a transaction cost. You can also set the slippage to zero, of course, but we strongly advise against that.