A Reality Check for Data Snooping
Read the paperopens doi.org in a new tab
What they found
White provided the first formal statistical test for whether the best of many forecasting models or trading rules is genuinely better than a benchmark, after accounting for the search. The reality check uses a bootstrap of the whole set of models to build the distribution of the best model's performance under the null hypothesis that none of them has any edge, and compares the observed best against that distribution. It applies to any situation where a researcher or trader picks the winner from a large set of candidates.
What you can use
- If you tried 100 strategies and the best has a t-statistic of 2.5, that is roughly what luck alone would produce; the reality check quantifies this.
- The test needs the full set of rules you tried, including the failures; only reporting the winners makes it impossible to assess.
- Keep a log of everything you tested; without it you cannot judge your own results honestly.
Caveats
Econometric theory; heavy mathematics. The test is conservative when many irrelevant models are included (see Hansen's SPA test).
Tags: backtesting, data-snooping, statistics, bootstrap
Summaries are our own reading of the paper, not the authors' words. Educational only, not advice. Discuss it in Book Club.