>>>. The real problem is not with estimation but with the management denial.
^^ This is the truth. All estimation processes in any company are just ceremony. Management already has a deadline fixed. They will make you agree to their deadlines, even though whatever process stipulated by them says otherwise.
The real problem is saying "yes we can do it all in that timeline" instead of saying "That's fine, but we won't get to everything. We will work with you to prioritise the most important things to get done in that time."
I know it's painful, but just hold that line. It's not lying or negotiating or padding; in fact it may be the only true statement you can say.
If a civil engineer were building a bridge, and couldn't do it safely in the budget and timeline, they'd say so, and the project manager would nod and diligently write that down. We have a more subservient relationship with project managers in software because we choose to.
You have to get them to take that perspective. What happens is "we want all this stuff how long is it going to take" and then they fight that number down while not answering any clarifying questions. So you have to sumo wrestle a bunch of assholes twice your weight into even framing the context correctly before the conversation you recommend can even take place.
> They will make you agree to their deadlines, even though whatever process stipulated by them says otherwise.
I'm not sure how. I mean, if I know it takes X time to do something, what would make me to commit to Y<<X time when I know it will fail and will be made my fault? If they want to record it as a deadline, so be it, but if it's an impossible deadline I'd make every effort to be on record that I said it from the start that it's impossible.
Not EVERY company. Just the ones that don't understand how estimation works... and unfortunately in software, that's still a very very common case. In more established kinds of engineering it works differently... which gives me hope that eventually our industry will understand too.
^^ This is the truth. All estimation processes in any company are just ceremony. Management already has a deadline fixed. They will make you agree to their deadlines, even though whatever process stipulated by them says otherwise.