I'm not a fan of the label 'neoliberalism' (outside of IR theory where it's well-defined and different) and think it's often a marker for poor quality partisan analysis, but the fact is it's simply prepending a 'neo' in front of the well established and broad classical liberal school of thought to distinguish it from the colloquial US use of 'liberal' to mean 'left wing'. And most of the people referred to wouldn't argue with the idea their ideas are in the ideological tradition of Adam Smith, even as they take great issue with some of the nonsense written about their motives and ideals and argue a label so broad that the entire EU political project and most of its opponents falls into it isn't particularly useful for understanding politics or economics.

That's quite different from referring to the other side of a debate on sexuality as 'Cultural Marxism', despite Marx having had zero interest in sexuality, many self-proclaimed Marxist states having conservative policies on it and few people on that side of the debate having any interest in Marx. At best, you could argue right wingers describing their culture war opponents as Marxists are trying to invoke the spirit of Joe McCarthy rather than Lyndon LaRouche's ridiculous Frankfurt School conspiracy theory and literal Nazi origins of the term.

My feeling is that "neoliberalism" is a term with a greater degree of academic formality behind it, as well as consensus in terms of definition. I don't think it was originally pejorative, although it has come to be used as an insult, in the same way "Boomer" has. The term "Cultural Marxists" is probably pejorative to those it targets, although I've seen its use as a shorthand for a collection of ideas the right finds disagreeable, rather than a pejorative. It's a term that feels self-describing and immediately obvious to those who use it, but it does not have an academic foundation like "neoliberal" does. And its definition has varied over time, which is the cause of the current lack of consensus on what it refers to. As a result, each political side I think uses the definition that most favors their own ideological bias.