In-sample and out-of-sample
Lesson 2 · about 10 min
If you build a strategy on a set of data and then measure it on the same data, you are marking your own homework. The strategy has already seen every bar. Any parameter you tuned, any filter you added because "it removed those three bad trades in March", was chosen with the answers in front of you. The fix is old and simple: split the data before you start.
The split
- In-sample (IS): the data you are allowed to look at, tune on, and change rules against.
- Out-of-sample (OOS): data you do not touch until the rules are frozen. You run the finished system on it once, and record what happens.
A common split is 60 to 70% in-sample and 30 to 40% out-of-sample, with the out-of-sample section being the most recent data. Recent data is the harder test, because markets drift and the recent period is closest to the conditions you will actually trade.
| Data available | In-sample | Out-of-sample |
|---|---|---|
| 2012 to 2024 (13 years daily) | 2012 to 2020 | 2021 to 2024 |
| 2019 to 2024 (6 years 15-minute) | 2019 to 2022 | 2023 to 2024 |
| 18 months of 5-minute crypto | first 12 months | last 6 months |
What "once" means
The out-of-sample test can be run once per strategy version. If it fails and you go back, change a rule, and run it again, the out-of-sample data is no longer out-of-sample. You have looked at it. Every extra pass turns a little more of it into in-sample data, and after five or six rounds you are back to marking your own homework with extra steps.
This is why the hypothesis discipline from the previous lesson matters. If you decided in advance what a failure looks like, you can accept the failure and move on to the next idea instead of nudging the rules until the number turns green.
Key idea: Out-of-sample data is only out-of-sample until you look at it. Tune on the in-sample set, freeze the rules, run the out-of-sample test once, and treat the result as the answer.
Reading the comparison
The interesting number is not the out-of-sample result on its own; it is how it compares to the in-sample result.
| Metric | In-sample | Out-of-sample | Reading |
|---|---|---|---|
| Trades | 240 | 105 | Reasonable, OOS is ~40% of the period |
| Win rate | 48% | 44% | Small decay, normal |
| Avg R | +0.31 | +0.22 | Decayed by about a third; expected |
| Profit factor | 1.65 | 1.38 | Still above 1.2; acceptable |
| Max drawdown | 11R | 14R | Deeper OOS; plan for worse live |
A drop of 20 to 40% in the headline metrics from in-sample to out-of-sample is normal, because the in-sample figures are inflated by the tuning you did on them. What you are looking for is a system that is still clearly positive, with the same character: similar win rate, similar holding time, similar trade frequency. A system that is +0.31R in-sample and +0.02R out-of-sample was fitted to noise. A system that is +0.31R in-sample and +0.55R out-of-sample is not "even better than expected"; it is a hint that the out-of-sample period was unusually favourable, and you should look at what regime it covered.
Three ways to split
- Chronological. Oldest data is in-sample, newest is out-of-sample. Simplest and most honest, because it mimics the passage of time.
- Cross-market. Tune on one instrument or basket, test on another that shares the mechanism. Useful for stocks, where you can hold back half the universe.
- Rolling (walk-forward). Repeated chronological splits stepping through time. Module 6 covers this in detail.
Cross-market splits carry a trap: if the held-back instruments are highly correlated with the in-sample ones (two index futures, two large-cap tech stocks), the test is much weaker than it looks.
Small data
With 18 months of intraday crypto data, a 12/6 split leaves you with a short out-of-sample window, and a short window can be dominated by one trend. The honest answer is that you cannot conclude much from six months. Report it anyway, note the regime it covered, and treat the strategy as "not yet disproved" rather than "validated".
Try it: Take any strategy you have already tested end to end. Re-run it on only the last 30% of the data and compare the numbers to the full-period result. Write down the percentage decay in average R and profit factor. That decay is a rough measure of how much the full-period figure was flattered by hindsight.
Recap
- Split the data before tuning: in-sample to build, out-of-sample to check.
- Out-of-sample is usually the most recent 30 to 40%, and is run once per frozen rule set.
- Expect a 20 to 40% decay in metrics from IS to OOS; a collapse to zero means overfit.
- Compare the character of the results, not just the headline number.
- With short data, an out-of-sample pass is "not disproved", not "proven".
See it drawn
Original diagrams for the ideas on this page. Illustrative, not real market data.