"I have a VOIP business account, and I can set any caller ID I want (including on SMS). And this is validated at the carrier level."

There is an easy fix to this and I believe Twilio does this ...

You allow senders to announce CID for any number they have validated with your service (in this case, Twilio) ... so you prove to Twilio that you control that number and they they let you set that number as CID.

Seems like a very simple and elegant solution ... and allows for the use-cases that you are alluding to.