Because your distinction is wrong.
A drive-by -1 is in no way different from a drive-by +1.
Both are equally useful. Both indicate how many people agree or disagree. Neither explains why someone agrees or disagrees.
If the explanation for +1 is in the post you are upvoting, then the explanation for -1 is just as well in the post you are downvoting.
Because your distinction is wrong.
A drive-by -1 is in no way different from a drive-by +1.
Both are equally useful. Both indicate how many people agree or disagree. Neither explains why someone agrees or disagrees.