Hmmm ... reinforces my counter-intuition. The stack of pennies might explain why the bottom of the jar would explode, but not the sides, area not below the stack of pennies.
My intuition (wrong here) is that the extra surface not beneath the stack of pennies (your analogy) would in fact distribute the pressure and therefore represent a lower PSI on all sides of the jar.
I guess I translated "pennies" into "little bags of water". The little bag of water at the bottom only gets pressurized from the little bags of water above it.
And so the bottom bag's "pushing" outward from compression would be affected only by those above it.
My intuition (wrong here) is that the extra surface not beneath the stack of pennies (your analogy) would in fact distribute the pressure and therefore represent a lower PSI on all sides of the jar.