The larger the project the more possible variables there are that can affect it. It's impossible to predict all of them.

There is a really good podcast on why large projects fail....

https://omegataupodcast.net/181-why-megaprojects-fail-and-wh...

It talks about sources of bias in estimates, and why that bias is so prevelant.

For public projects, there are usually rules forcing agencies to contract the lowest bidder. So companies will bid unrealistically low, then after a while make new claims. Theoretically they are in breach of contract, and the agency could get out of the contract - but at that point finding another contractor would add even more cost and much more time to the project.