My biggest gripe with the Big Sur update is that it's now incredibly slow to swipe between desktops. It used to be responsive and now I need to wait 5-6 seconds before anything happens.
It’s still instantaneous normally, even on old systems like the 2012 MacBook Air I still use. I’d look in Activity Monitor / Console to see if it’s something like being memory-constrained and the switch is forcing a lot of background activity.
