Skip to content
GetProfitable
Search
Dictionary

Regularisation

Penalising model complexity during fitting so coefficients stay small and the model cannot chase noise. The main defence against overfitting in a fitted model.

Ridge (L2) shrinks all coefficients toward zero proportionally and handles correlated features gracefully. Lasso (L1) can drive coefficients exactly to zero, performing feature selection. Elastic net blends the two and is a sensible default when features are numerous and related.

The strength of the penalty is a hyperparameter and must be chosen on validation data, never on training data, where more flexibility always looks better. On financial data the chosen penalty is usually far stronger than practitioners from other fields expect.

Regularisation limits variance; it cannot manufacture signal. If the honest out-of-sample performance is near zero at every penalty level, the answer is that the features do not predict the label, and no amount of tuning changes that.

Related: bias-variance-tradeoff, overfitting, hyperparameter, multicollinearity

Educational only, not advice. Spotted an error? Post in Site Feedback.