modified-duration assumes fixed cash flows. That breaks for a callable-bond, a putable-bond or a mortgage-backed-security, where the cash flows themselves change when rates move. Effective duration solves this by brute force: shift the curve up and down, revalue with a model, and measure the price difference.
The formula is (P_down - P_up) / (2 x P_0 x shift). For a callable bond the result is lower than modified duration, and it can even shrink as rates fall, which is the signature of negative-convexity.
Example: shift rates 25 bp each way. Price up-shift 98.10, down-shift 100.70, current 99.40. Effective duration = (100.70 - 98.10) / (2 x 99.40 x 0.0025) = 5.23.
Related: modified-duration, negative-convexity