Sadly this is my approach, too. If it's important they can leave a voicemail message and I'll call back at my convenience.

I always thought the expectation of interrupting whatever you're doing at a few seconds notice was incredibly rude anyway, even more so now we have so many other ways of communicating.