As for the integrated stack problem, I can wholeheartedly recommend the Enthought Python Distribution [1]. I've used it on both Windows and Mac. It includes all the important libraries for linear algebra, matrix computation, and visualization (SciPy, NumPy, matplotlib, etc.). So it is a great replacement for Matlab, which probably falls short of any programming language known to mankind.
That would have been my answer a year ago as well, however we've (continuum analytics) put out our own distribution now which gives you much more - all for free.
How does this distribution play with the outside world? Will I be able to install other third party libraries into the distribution? Does it have its own 'installation procedure'?
So, we ship anaconda with our own package management system, called conda. Conda is open source and was created because for scientific python, we need to manage versions of non-python libraries (blas/atlas/mkl,libhdf5, etc..). But you don't need to use conda if you don't want to. Conda is just what we use to install stuff. You can use pip on top of Anaconda if you would like, and we even have functions to turn whatever you did to your anaconda environment into a conda package if you wanted to.
[1] http://www.enthought.com/