OP here. I recall a similar passage in Thinking Fast and Slow by Daniel Kahneman. He was part of a committee to rewrite textbooks and everyone estimated it to be part of a normal distribution instead of a power law. He lamented how even trained psychologists were bad at estimating completion times in spite of being aware of the data.
I think this holds for most human endeavors that have intellectual property as the end result - software projects, books, doctoral thesis, etc.
Two questions -
1- I would really like to understand why?
2- I have always thought lean is the answer to the above in a startup context but really curious of hear of others.
Not just intellectual endeavours, the planning fallacy shows up everywhere.
I have a book called Industrial Megaprojects by Murrow[0] which gives a lot of examples of financially disastrous multi-billion dollar projects. His conclusion? It's not really the doing of the project that was wrong, it was that the projects shouldn't have been done in the first place. The estimates and preliminary investigations were underdeveloped and this typically leads to overoptimism.
Bent Flyvbjerg has also done a lot of work on megaprojects[1] and his basic conclusion is that well-estimated projects don't get built, because almost no megaproject is ever viable or cost-effective in itself. So there's a survivor bias towards "bad" projects, ones that are poorly planned in the first instance, magnifying the effect.
In terms of software, agile/lean has the advantage of limiting commitment. It's easier to terminate something that's cost very little and not gotten far than to terminate something that's several years and millions of dollars into going nowhere. The former can be dismaying and annoying. But by the time serious time and money have been spent, there's a sunk-cost fallacy and often, personal pride or status of powerful individuals is involved.
My guess is it is a statistical artefact. If you have a project and you pick a random time in that project then you expect to pick the midpoint.
So, if the only information we have is that the project has gone on for 2 years we expect that to be the midpoint and that the project will continue for 2 years.
In particular if the project manager has lost control of the project (eg, not scheduled in a contingency, missed requirements, etc) then there is no reason to believe that anyone knows what % of the project is done. So assume 50% because that is the Most Likely Estimate. And be surprised at how often that is the right guess in my cynical and not inconsequential experience :p.
If the project manager is in control (usually evidenced by people opining that the project will finish early) then expect the project to finish exactly on time when something unexpected goes wrong.
I think this holds for most human endeavors that have intellectual property as the end result - software projects, books, doctoral thesis, etc.
Two questions - 1- I would really like to understand why? 2- I have always thought lean is the answer to the above in a startup context but really curious of hear of others.