Parameter sensitivity and degrees of freedom
Lesson 19 · about 11 min
Overfitting is what happens when a strategy learns the noise in its test data instead of the signal. The symptoms are a beautiful backtest and a flat or losing live result. The cause is almost always the same: too many adjustable numbers, tuned too precisely, on too little data. This lesson gives you two tools for catching it before you trade: a sensitivity table and a count of degrees of freedom.
Every parameter is a chance to fit noise
A moving average length, an RSI threshold, an ATR multiple for the stop, a time-of-day filter, a minimum volume: each is a dial. Turn any dial and the trade list changes. If you turn each dial until the backtest is best, you have selected, out of all the possible trade lists, the one that happened to line up best with the past. Some of that alignment is a real edge. Some is luck. The more dials, the more of it is luck.
| Parameters | Values tried per parameter | Combinations tested |
|---|---|---|
| 2 | 10 | 100 |
| 3 | 10 | 1,000 |
| 5 | 10 | 100,000 |
| 8 | 10 | 100,000,000 |
An optimiser that tries 100,000 combinations on 200 trades will find one that looks extraordinary, with certainty, whether or not any edge exists.
The sensitivity table
The best single defence is to look at how the result changes as each parameter moves. Take the two most important parameters and tabulate the expectancy across a grid.
Below: a breakout strategy, expectancy in R, varying the lookback length (rows) and the ATR stop multiple (columns).
| Lookback \ Stop | 1.0 ATR | 1.5 ATR | 2.0 ATR | 2.5 ATR | 3.0 ATR |
|---|---|---|---|---|---|
| 10 | −0.05 | +0.02 | +0.06 | +0.08 | +0.07 |
| 15 | +0.01 | +0.09 | +0.14 | +0.15 | +0.12 |
| 20 | +0.04 | +0.13 | +0.19 | +0.20 | +0.17 |
| 25 | +0.05 | +0.14 | +0.21 | +0.21 | +0.18 |
| 30 | +0.03 | +0.11 | +0.17 | +0.18 | +0.15 |
| 40 | +0.01 | +0.06 | +0.10 | +0.11 | +0.09 |
This is what a healthy surface looks like. There is a broad plateau around lookback 20 to 30 and stop 2.0 to 3.0 where everything is between +0.17R and +0.21R. Move any parameter one step and the result barely changes. A strategy on this plateau is likely to survive live, because the live market will effectively shift the parameters a little and the plateau absorbs it.
Now an unhealthy one:
| Lookback \ Stop | 1.0 ATR | 1.5 ATR | 2.0 ATR | 2.5 ATR | 3.0 ATR |
|---|---|---|---|---|---|
| 10 | −0.08 | −0.03 | +0.01 | −0.02 | −0.05 |
| 15 | −0.04 | +0.02 | +0.04 | +0.03 | −0.01 |
| 20 | −0.02 | +0.05 | +0.38 | +0.06 | +0.01 |
| 25 | −0.03 | +0.03 | +0.05 | +0.04 | 0.00 |
| 30 | −0.06 | −0.01 | +0.02 | +0.01 | −0.03 |
Lookback 20 with a 2.0 ATR stop shows +0.38R and every neighbour is near zero. This is a spike, not an edge. Something about that exact combination happened to catch a few trades right. Live, it will behave like its neighbours.
Key idea: A real edge is a plateau; an overfit one is a spike. Pick parameters from the middle of a plateau, never from the peak, and treat any result that collapses when a parameter moves one step as noise.
Choosing from the plateau
Once you have the surface, choose the parameter set at the centre of the widest flat region, even if it is not the maximum. In the healthy table, lookback 25 and stop 2.5 ATR sits at +0.21R with all eight neighbours between +0.17 and +0.21. The "best" cell, at the same value, happens to be on the plateau; had the best cell been an isolated +0.29 at lookback 40, stop 1.5, the correct choice would still be the plateau centre.
Degrees of freedom
The second tool is a count. List every number in the strategy that could have been set differently: parameters, thresholds, filter values, time windows, and also the choices that do not look like numbers (which moving average type, which order type, which exit precedence). Each is a degree of freedom.
Then compare with the trade count. A rough working rule: you want at least 30 to 50 trades per degree of freedom, and more if the trades are short-term and correlated.
| Strategy | Degrees of freedom | Trades | Trades per DoF | Reading |
|---|---|---|---|---|
| Simple breakout | 3 (lookback, stop, time stop) | 220 | 73 | Adequate |
| Breakout with filters | 7 (adds regime MA, ATR floor, volume min, day-of-week) | 140 | 20 | Underpowered; the filters were likely fitted |
| Pattern from Module 3 | 9 | 60 | 7 | Not testable at this size |
The pattern-based strategy is not disqualified by this; it means it needs far more data, across more instruments, before any result can be trusted.
Reducing degrees of freedom
- Remove any filter that does not have a mechanism you can state. "No trades on Fridays" without a reason is a fitted parameter.
- Tie parameters together. If the stop is 2 ATR and the target is 6 ATR, that is two parameters; if the target is defined as 3 × the stop, it is one plus a ratio you fix by policy.
- Use round numbers. A 20-bar lookback is a hypothesis; a 23-bar lookback is an optimisation result.
- Fix anything that can be fixed by convention (order types, precedence rules, session) rather than testing it.
Try it: List every adjustable choice in your strategy, including the ones that do not look like parameters. Count them. Divide your trade count by that number. Then build a 5 × 5 sensitivity table for the two parameters you are least sure of and describe the surface in one word: plateau or spike.
Recap
- Each parameter is a chance to fit noise; combinations grow multiplicatively.
- A sensitivity table shows whether the result is a plateau (robust) or a spike (fitted).
- Choose parameters from the centre of the plateau, not the peak.
- Count degrees of freedom, including non-numeric choices; aim for 30 to 50 trades per DoF.
- Remove filters without a mechanism, tie parameters together, and prefer round numbers.
See it drawn
Original diagrams for the ideas on this page. Illustrative, not real market data.