It would be cool if there was a way to penalize high number and high ratio of rejected patents from a single company. OTOH I don't see a way that allows genuine mistakes from small shops while being immune from big bad corps acting via shell companies.
That is an excellent idea. (To clarify, you're saying a company would need to have both a high number and a high ratio rejected to be penalized.)
More generally, it seems logical to have the size of the penalty increase at a quadratic or even exponential rate, based on the number of rejected patents within a certain trailing time period[1], and scaled by the rejected/submissions ratio. So every rejection would carry a penalty, but the more that were rejected (both in total and as a percentage), the larger the penalty would be, at an increasing rate.
It might also be an idea to do this with individual claims as well as, or even instead of, entire patents, to also deal with the overly broad claims problem.
[1] Or even for all time, but with a weighting factor based on recency. Wouldn't want to get overly complex though.