I don't think the hindrance when starting to learn graphics programming is the math or the lack of ideas of what to build. The biggest problem is that the most known APIs (DirectX, OpenGL and especially DirectX 12 and Vulkan) are so badly designed and so poorly documented that it deters anyone trying to start doing anything. All these API are like magic invocation you have to make the calls in just the right order or it doesn't work at all. There is no real specification of what goes underneath and it can't be because the actual implementation is written by different vendors (Nvidia or AMD) which only slightly abides to the specification.