Skip to content
GetProfitable
Search
Dictionary

Feature importance

A ranking of which inputs the model relies on. Useful for sanity-checking and dangerous when read as a statement about cause.

Tree-based importance counts how much each feature reduced error at splits, which biases toward high-cardinality and correlated features. Permutation importance, shuffling one feature and measuring the drop in out-of-sample score, is slower and far more honest.

The diagnostic value is high. If the top feature is something that should not matter, such as a row index, an instrument identifier, or a variable that encodes the date, you have found data-leakage before it cost you money.

What importance does not tell you is direction, stability, or causation. A feature can be important in one fold and irrelevant in the next, which is itself a useful signal that the model is fitting noise.

Related: feature-engineering, data-leakage, multicollinearity, overfitting

Educational only, not advice. Spotted an error? Post in Site Feedback.