I did some work on data visualization for the astrophysics department when I was in college. I started to work with the simulation code, but found that the math was sprinkled everywhere, which made it really difficult for me to make structural changes without risking the integrity of the program.
One of the most elusive skills for self-taught programmers is how to structure code properly. A good architecture would allow domain experts and non-expert programmers to coexist, but that would require throwing away a lot of existing spaghetti code written by domain experts, which is not going to be a popular decision.
I'm a programmer who is studying physics in college, and a couple years back I had a similar experience with simulation code as you did. I didn't have any issues with the math—the program I was working on didn't have anything more conceptually advanced than multivariable calculus—but I did struggle significantly to understand the physics behind the simulation.
It didn't help that most programs use, for example, the variable 'rho' for density instead of just writing out 'density'.
On the other hand, reading game physics libraries (written by programmers, not physicists) can be just as bad. There are physics hacks all over ("it's not stable, so let's throw in an arbitrary constant") and there's code repetition where the programmer doesn't understand that two concepts are closely related.
This is why I think, rather than every data analysis/visualization program being written from scratch with it's own custom UI and I/O, formatting etc, these programs should be written as modules to a data workflow program like RapidMiner that handles all that for you.
I'm lucky to be in a lab with enough funding for this. This is rarely the case. In medical science it's easy to get large grants and pay programmers in something like biology the programmer is usually that guy that learned perl since he's already getting paid to do science.
Grad students probably wrote the majority of the tools used in my lab and it shows because when they leave the knowledge of the different issues and bugs in the software that they had goes with them. Years later they resurface and no one has any idea of the thought process of the original author.
It's quite annoying. We have such a project right now that is a basic piece of software we use for all our research. There is no current funding for someone to sit there and clean up the code. Most funding agencies want new work not maintenance work to be done with their money. There just isn't any incentives anywhere for this.
This happens already with mechanical and electrical engineers in my field (space physics). There are challenges, though: the engineers are paid less than they would earn in industry, and long-term employment is contingent on scientists winning new grants which call for their skills. So, the low value for expertise (in academia) is largely driven by the grant funding system.
I could see a role for staff computer scientists in areas of research where computation plays a particularly large role, but for typical grad-student data munging, the cost/benefit ratio is likely far too high.
Even when money allows for this, it's not always possible to segregate the work. I.e. a scientist may fully understand the mathematics of his work, but not understand that translating the math directly into code will cause serious performance issues. A programmer should understand the performance issues, but without a deep and intuitive understanding of the mathematics, including what assumptions are implicit therein, will often have difficulty making the translation.
In many cases, projects with sufficient technical depth require detailed domain knowledge that can be acquired quicker and cheaper by hiring scientists or engineers with that knowledge and then teaching them to code (or code better), rather than segregating the work and dealing with the problems that result from the communication gap.
It's usually outside of the scope of what you can pay for using a grant, and grants are currently the "standard" way to fund science. In this particular case also the permanently part won't get along with with grants being given for specific goals and also for fixed periods of time.
