76 terms

Backtesting method

Anchored walk-forward: A walk-forward where the fitting window always starts at the beginning of history and only its end moves forward.
Bayesian optimisation: A search method that builds a probabilistic model of the objective surface and picks the next parameter set most likely to be informative.
Bonferroni correction: Divide your significance threshold by the number of tests you ran. Crude, conservative, and better than pretending you only ran one.
Borrow cost: The fee paid to borrow shares in order to short them. Cheap and stable for large caps, punitive and unpredictable for the names shorts most want.
Combinatorial purged cross-validation: A validation scheme that tests many different combinations of held-out blocks, producing a distribution of backtest results instead of a single path.
Commission model: The explicit fee schedule applied in the backtest: per share, per contract, percentage of notional, or tiered, plus exchange and regulatory fees.
Cross-validation: Splitting data into several folds and repeatedly training on some while testing on the rest, so every observation gets used for testing once.
Curve fitting: Shaping a strategy's rules around the particular history you tested on, so it describes the past precisely and predicts the future not at all.
Data leakage: Any path by which information about the test period reaches the model during training, from full-sample scaling to overlapping labels.
Data snooping: The bias that arises when you test many ideas on one dataset and report only the winner, whose performance is inflated by luck.
Deflated Sharpe ratio: A Sharpe ratio adjusted downward for the number of trials run, the sample length, and the non-normality of returns.
Degrees of freedom: The number of independent choices you made while fitting a strategy; more choices mean less trustworthy results from the same data.
Embargo: A gap of dropped observations immediately after a test fold, preventing serial correlation from leaking test information into later training data.
Equity curve: The running account value over the life of a strategy. The most-shown and least-informative single chart in trading, unless you know what to look for.
Event-driven backtest: A backtest that replays market events in order through the same components that will run live: data handler, strategy, order manager, execution simulator, portfolio.
Expanding window: A window that starts at the first observation and grows with each new data point, so every estimate uses all history to date.
False discovery rate: The expected share of your accepted discoveries that are wrong. A gentler target than eliminating all false positives, and usually the right one for strategy search.
Fill assumption: What your backtest believes about whether an order executed, at what price, and in what size. Usually the most optimistic part of the whole model.
Filter: A secondary condition that blocks trades from a primary signal, intended to remove the signals that historically lost.
Financing cost: The interest paid on leverage and the interest earned on cash. At non-zero rates it is a first-order term that many backtests silently set to zero.
Grid search: Testing every combination of parameter values on a predefined grid; exhaustive, simple, and exponentially expensive as parameters multiply.
Holdout set: A block of data locked away at the start of research and opened only once, at the end, as a final sanity check.
Hyperparameter: A setting that governs how a model is fitted rather than being learned from the data: tree depth, penalty strength, learning rate, number of neighbours.
In-sample: The slice of history you used to build and tune a strategy; results on it are always flattering and prove nothing on their own.
Incubation: Running a finished strategy on live data without money, or with token size, for a set period before funding it. The only truly clean out-of-sample test available.
Label: What a supervised model is trained to predict: the future return, its sign, or a category such as target-hit versus stop-hit.
Look-ahead bias: Using information in a backtest that would not have been available at the moment the decision is made, which silently inflates results.
Minimum backtest length: The shortest history over which a claimed Sharpe ratio could be distinguished from the best of your search, roughly scaling with the number of trials.
Missing data: Gaps in a series, from halts, holidays, vendor outages, or instruments that did not exist yet. How you fill them changes results more than most people expect.
Multiple testing: Running many statistical tests at once, which makes false positives near-certain unless you adjust your significance threshold.
Next-bar execution: The convention that a signal computed on a bar can only be filled on the following bar, usually at its open. The safe default for bar-based backtests.
Null hypothesis: The boring explanation a test tries to rule out: that your strategy has no edge and the results you saw are what luck produces.
Objective function: The single number an optimisation tries to maximise or minimise; the choice quietly decides what kind of strategy you end up with.
OHLCV bar: Open, high, low, close and volume aggregated over an interval. The standard research format, and a lossy summary of what happened.
Out-of-sample: Data the strategy was never fitted on, used once to estimate how it will behave on data it has never seen.
Overfitting (curve fitting): Tuning a strategy so closely to past data that it captures the noise as well as the pattern, and therefore fails on new data.
P-hacking: Adjusting the data, sample, or test until a result crosses the significance threshold, then presenting it as if the analysis was planned.
p-value: The probability of seeing a result at least as extreme as yours if the null hypothesis were true. Small means surprising, not necessarily true or profitable.
Parameter: A number in a trading rule that you chose rather than derived, such as a lookback length, a threshold, or a stop distance.
Parameter optimisation: Searching parameter space for the settings that maximise some objective; useful for mapping the landscape, dangerous as a way to choose settings.
Parameter plateau: A broad region of parameter values that all produce similar, decent results; the shape you want instead of a single sharp peak.
Parameter sensitivity: How much a strategy's results change when you nudge its inputs; a robust strategy degrades gracefully, a fitted one collapses.
Permutation test: A significance test that scrambles the link between signal and outcome many times to see how often chance alone produces your result.
Point-in-time data: Data stored as it was known on each historical date, including the wrong first estimates, rather than as it looks after later corrections.
Probability of backtest overfitting: An estimate of how often the strategy you would have chosen in sample turns out to be below median out of sample.
Purged cross-validation: Cross-validation that deletes training observations whose labels overlap in time with the test set, removing a common source of leakage.
Random search: Sampling parameter combinations at random rather than on a grid; usually finds good regions faster when most parameters barely matter.
Reality check test: A statistical test that asks whether the best strategy out of many tested beats a benchmark by more than the search itself would produce by chance.
Regime filter: A market-wide condition that switches a strategy on or off, such as trading trend rules only when volatility is below a threshold.
Resampling (data): Converting data from one frequency to another, such as building hourly bars from minute bars. Simple in principle and full of edge cases in practice.
Research log: A written record of every idea tested, with parameters, results and the decision taken. It is the only way to know how many tests your final result survived.
Restatement bias: Using the final, corrected value of a data series in a backtest when only the original, later-revised value was available at the time.
Robustness: The property of a strategy whose results survive small changes to parameters, data, start dates, instruments, and cost assumptions.
Rolling regression: Re-fitting a regression over a moving window so coefficients can change over time, at the cost of noisier estimates.
Rolling Sharpe ratio: The Sharpe ratio computed over a moving window, so you can see whether performance was consistent or came from one lucky stretch.
Rolling window: A fixed-length span of recent data that slides forward one step at a time, used for fitting parameters or computing statistics.
Sandbox environment: A broker-provided test environment that accepts orders without real money. Essential for testing plumbing and misleading about fills.
Selection bias: Drawing conclusions from a sample that was not chosen independently of the outcome you are measuring.
Signal lag: The deliberate delay between the data a signal uses and the moment the resulting order can execute. Getting it wrong is the most common way to fake a backtest.
Significance level: The p-value threshold you commit to before testing, conventionally 0.05, below which you are willing to reject the null hypothesis.
Slippage model: The rule your backtest uses to convert a theoretical price into a realistic one. It can be a fixed number of ticks, a fraction of the spread, or a function of size and volatility.
Statistical power: The chance a test detects an effect that is genuinely there. Low power means your research mostly produces silence and flukes.
Stress test: Re-running a strategy under deliberately hostile assumptions to find what breaks it before the market does.
Synthetic data: Artificially generated price series used to test a strategy on markets that never happened, including deliberately hostile ones.
Timestamp alignment: Making sure every dataset in a backtest agrees on what time it is, including timezones, daylight saving, exchange sessions, and whether a stamp is an event or an arrival time.
Trade order randomisation: Shuffling the sequence of your historical trades to see how differently the equity curve could have unfolded with the same trades.
Train, validation, test split: Three separate data blocks: one to fit the model, one to choose between models, and one used once to estimate real performance.
Transaction cost modelling: Accounting for every cost of trading in the backtest: commissions, spread, slippage, market impact, borrow, financing and taxes. Usually the difference between a great strategy and a bad one.
Triple barrier method: Labelling each observation by which of three barriers is hit first: a profit target, a stop level, or a time limit. It labels trades the way a trader experiences them.
Turnover: How much of the portfolio is replaced over a period, usually annualised. It multiplies every per-trade cost and is the fastest way to convert an edge into fees.
Type I error: Concluding a strategy has an edge when it does not. The expensive error in trading, because you fund it.
Type II error: Discarding a strategy that really does have an edge, usually because the sample was too small to show it.
Universe construction: Deciding which instruments a strategy is allowed to trade on each historical date, using only information available on that date.
Variance ratio test: A test of the random-walk hypothesis that compares the variance of long-horizon returns to that of short-horizon returns scaled up.
Vectorised backtest: A backtest computed as array arithmetic over the whole history at once: signals times forward returns. Fast, easy to write, and easy to make wrong.
Walk-forward analysis: Repeatedly fit parameters on a block of history, trade the block immediately after it, then roll both windows forward and stitch the untouched results together.

Back to the full dictionary.