It worked on SPY 2010 to 2021
Lesson 22 · about 10 min
The most common backtest posted online is a long-only strategy on a US equity index fund, tested over the decade after the 2008 crisis. It shows a rising equity curve, a good Sharpe, and a modest drawdown. Almost none of these strategies have an edge. This lesson explains why that particular test proves so little, and what to do instead.
One instrument, one regime
From March 2009 to the end of 2021, the S&P 500 rose roughly fivefold with only two drawdowns beyond 20% (late 2018 and early 2020), both recovered within months. Any rule that was long most of the time made money. Any rule that bought dips made money, because every dip recovered. Any rule that used a trend filter made money and looked clever in 2020 by avoiding part of the March crash.
The strategy did not have to be good. It had to be long.
| Strategy on SPY, 2010 to 2021 | Annual return | Max drawdown | Looks like |
|---|---|---|---|
| Buy and hold | ~14% | −34% | The benchmark |
| Buy any 3-day pullback, hold 10 days | ~12% | −22% | "Mean reversion edge" |
| Long above 200-day MA, cash below | ~11% | −19% | "Trend filter edge" |
| Long on Monday, flat by Friday | ~6% | −18% | "Weekly seasonality" |
| Random long entries, 10-day hold, 50% of the time | ~7% | −17% | Noise |
The random strategy in the last row made 7% a year. Any strategy on this instrument in this period has to be judged against that, not against zero.
Test against the right benchmark
For a long-biased strategy on an instrument that rose, the question is not "did it make money" but "did it make more, per unit of risk, than simply being long with the same average exposure". Compute the strategy's average time in market, and compare with buy-and-hold scaled to that exposure.
| Metric | Strategy | Buy and hold at same exposure (55%) |
|---|---|---|
| Annual return | 9.1% | 7.7% |
| Volatility | 9.8% | 10.4% |
| Sharpe | 0.93 | 0.74 |
| Max drawdown | −14% | −19% |
Now there is a real, if modest, comparison: the strategy beat its exposure-matched benchmark by a little. That "little" is the entire claim, and it needs the sample-size and multiple-testing checks applied to the difference, not to the raw return.
Test across instruments
A mechanism that is real should appear in more than one place. If "buy 3-day pullbacks" works because of short-term over-reaction, it should show up, in some form, in other liquid indices, in other large-cap stocks, and possibly in other asset classes. If it only works on one instrument, the likelier explanation is that the instrument's particular path happened to suit it.
| Instrument | 2010 to 2021 avg R | Comment |
|---|---|---|
| SPY | +0.21 | The original |
| QQQ | +0.18 | Similar; highly correlated with SPY |
| EFA (developed ex-US) | +0.03 | Flat |
| EEM (emerging) | −0.05 | Negative |
| TLT (long bonds) | +0.02 | Flat |
| GLD (gold) | −0.02 | Flat |
The strategy works on US equities and nothing else. Since QQQ and SPY are not independent tests, the honest count is one instrument, and the honest description is "long US equities with a pullback timing rule during a bull market".
Test across periods
Extend the data backward and forward. Does the rule survive 2000 to 2003, 2007 to 2009, and 2022?
| Period | Regime | Strategy avg R | Buy-and-hold outcome |
|---|---|---|---|
| 2000 to 2002 | Bear | −0.12 | −40% |
| 2003 to 2007 | Bull | +0.15 | +80% |
| 2008 to 2009 | Crash, recovery | −0.20 then +0.35 | −50% then +60% |
| 2010 to 2021 | Bull | +0.21 | +400% |
| 2022 | Bear | −0.09 | −18% |
This is a strategy that makes money in bull markets and loses in bear markets. It has beta, and a small amount of timing. Whether that is worth trading depends on whether you can tolerate it losing in exactly the environments where losing hurts most.
Key idea: A backtest on one instrument in one regime tells you how the strategy did in that regime. To learn whether the strategy has an edge, test it against an exposure-matched benchmark, across uncorrelated instruments, and across periods that include the regimes you fear.
The general version
Replace "SPY 2010 to 2021" with any single-instrument, single-regime test and the argument is the same:
- Crypto strategies tested only on 2020 to 2021.
- Short-volatility strategies tested only on 2012 to 2017.
- Forex carry tested only on 2003 to 2007.
- Any momentum strategy tested only on the years after a crash.
In each case the strategy is mostly a bet on the regime. That can be a fine bet, but it is a different claim from "this rule has an edge", and it should be sized and described as what it is.
What "enough" looks like
A minimum honest test of a directional strategy covers at least one full cycle (a bull, a bear, and a sideways period), at least two instruments that are not highly correlated, and is judged against an exposure-matched benchmark. If your data does not cover a full cycle, say so, and treat the strategy as regime-conditional until it does.
Try it: Take a long-biased strategy you have tested. Compute its average exposure and compare it with buy-and-hold scaled to that exposure. Then run it on the worst two-year period your data contains. Write down both results next to the headline number.
Recap
- A long-biased rule tested on a decade-long bull market mostly measures the bull market.
- Benchmark against buy-and-hold scaled to the strategy's average exposure, not against zero.
- Test across uncorrelated instruments; correlated ones are not independent tests.
- Test across regimes including bears and crashes; a rule that only wins in bulls is a beta bet.
- Describe regime-conditional strategies as such, and size them accordingly.
See it drawn
Original diagrams for the ideas on this page. Illustrative, not real market data.