I would not call this "destructive." Solving problems is fun in its own right, and things would start making sense in their interconnection, regardless of whether they can be applied to sciences or not.
It is destructive to people who do not find abstract math problems inherently fun, which is (I would guess) the great majority of people. They come to the conclusion that they hate math or that they are bad at math, even though they would have grasped it better and enjoyed it had it been presented to them differently.
If your goal is teach math to the bulk of people, including those who will not go on to be mathematicians, it makes sense to tie math to something that the bulk of people can relate to. And most people DO enjoy thinking about physical objects and physical space (because we ourselves are physical beings who evolved to interact with our physical environment), so this is a really great starting point for introductory math for the average person.
If your goal is to only teach the subset of the population who prefer highly abstract puzzles, and to alienate all others, then our current methods are working fine I guess. But I don't think this is a good goal for general math education.
