The Probability of Backtest Overfitting
Read the paperopens papers.ssrn.com in a new tab
What they found
The authors propose combinatorially symmetric cross-validation as a way to estimate how likely it is that a strategy selected as the best in-sample will underperform the median out of sample. The method splits the data into many blocks, forms all combinations of in-sample and out-of-sample sets, selects the best strategy configuration in each in-sample set, and records how it ranks out of sample. The fraction of combinations where the in-sample winner is below median out of sample is the probability of backtest overfitting. They show that common optimization procedures produce high overfitting probabilities.
What you can use
- You can directly estimate the probability that your parameter optimization picked a lucky configuration by re-running the selection on many data splits.
- If the in-sample winner is usually mediocre on the held-out data, your optimization is fitting noise regardless of how good the final backtest looks.
- The method is a practical procedure any systematic trader can implement with a modest amount of code.
Caveats
Computationally intensive and sensitive to how blocks are formed; it evaluates a selection procedure rather than a single strategy. SSRN version linked.
Tags: backtesting, overfitting, cross-validation, parameter-optimization
Summaries are our own reading of the paper, not the authors' words. Educational only, not advice. Discuss it in Book Club.