You're reading way too much into my analogy. The point is just that lifting up someone who's been pushed down does not also require you to lift up someone who's doing just fine in order to be "fair." This applies whether the disadvantage is due to outside bullying (as in the case of feminism and affirmative action) or an actual disability (it's not "unfair" to make special allowances for an autistic office worker, for example).