I wouldn’t care that much either, it’s more the collateral damage you can create as OP outlined. My father would suffer a lot if I didn’t see him on that day for example
After my dad's death, I found my mother would become far more emotional around her birthday. These occasions, special days, are reminders of a void. Something missing that will never be again. In her case, her husband saying Happy Birthday first thing in the morning.