I'm not a mathematician, but it's my understanding from what I've read that part of it is becoming used to groping around in the dark, sometimes getting stuck, and accepting that as normal. Solving the really tough puzzles means getting stuck a lot.
This might explain the relative lack of attention towards the user experience of non-mathematicians?
I find it amusing that you call it "user experience", a very computery/hackery term for mathematics. There are so many computer-like expectations for mathematics that I see on HN. :-)
Anyway, yes. Every mathematician I know acknowledges that frustration is the natural state of affairs. If you're not frustrated that's because you haven't been doing enough mathematics yet. There's always a bigger problem, a new concept to master, a new way to look at an old idea.
Yes, to extend the metaphor, writers often pride themselves in offering a good "user experience." Bad writing does not flow smoothly and is unnecessarily difficult to read.
Good games try to provide an optimal learning experience, providing just enough challenge to be interesting without players getting stuck. Play-testing is vital; if your players commonly get stuck in ways you didn't intend, it's a bug.
There's a lot to be said for designing a learning experience to flow smoothly. We can admire the work that goes into making that happen. (It then seems strange that, by contrast, writers of math books often don't seem to be playing the same game.)
One thing a well-designed experience doesn't do, though, is prepare you for being stuck and overcoming difficulty when you're not on an artificially smoothed path.
> It then seems strange that, by contrast, writers of math books don't seem to be playing the same game.
Oh, but they are playing that game. It's just a very difficult game. Everyone wants easy math books and lots of people are trying to write easy math books (for example, my buddy Ivan and his No Bullshit guides: https://minireference.com/ ). It's just a very difficult game, and very few have come close to success. When they do succeed, it's usually only for one kind of audience and not another. For example, Spivak's Calculus is widely admired in the mathematical community for its presentation, but I wouldn't be surprised if HN derided it for being stuffy, too mathematical, elitist, and full of itself.
It's kind of sobering to realize that even the pro's have this problem.