Two conventions exist and they are not interchangeable. Excess over the risk-free-rate measures compensation for taking risk at all; excess over a benchmark measures whether active decisions beat the passive alternative. A record quoting one while implying the other is being slippery.
Worked: a portfolio returns 14%, bills pay 5%, the relevant index returns 11%. Excess over cash is 9 points - the reward for bearing risk. Excess over the index is 3 points - the reward for your work. The second number is the one that justifies the effort, and it is usually much smaller than the first.
Compounding matters here too. Excess returns should be computed geometrically for multi-period comparison - subtracting annual percentages is an approximation that drifts badly over long records.
Related: risk-free-rate, benchmark, information-ratio, jensens-alpha