In common American parlance, "nazi" is becoming a slur for any racist/sexist/*ist/*phobe, and doesn't refer specifically to those with interest or sympathy for/with Nazi Germany.
That may be how some people use it, but that doesn't mean we should put up with that usage. The Nazis slaughtered millions of people because of their ethnicity/religion/sexual orientation/what have you. To use that word to describe an immature group of friends who say racist things because they think it's edgy waters down the meaning of a very powerful word in a way that severely harms public discourse.
If we're going to use the phrase "literally a neo-Nazi", I want to see swastikas or tattoos or something other than garden-variety hooliganism.