DNS is a fragile system to use within a DC to do service/endpoint discovery. That's because DNS tends to be a single point of failure.

Though the DNS system can inherently be resilient, within a DC most people only operate a single DNS server because a hierarchical domain scheme and DNS setup within DC is too cumbersome and is much less reactive to end point changes. Eg: Changing a service endpoint in an emergency takes way too long.

A single DNS server means, still there are propagation delays(local caches) and a single point of failure at critical moments(when there is a thundering herd)

That's a pretty weak argument. It's like calling a databases a single point of failure because the people who set them up tend to be too lazy to setup HA.

DNS is surprisingly tricky as a service discovery tool. A lot of clients are poorly behaved, and will cache values for too long (or forever). It'd also be another dependency critical for site functionality.

Heh, the number of times DNS has bitten me in production... My main objections to DNS are:

1) You have no control over clients, there is absolutely no rule that says clients have to respect TTLs (I'm looking at you Java)

2) We are talking about hundreds of HAProxy load balancers, thousands of clients and hundreds of backend services which are moving around all the time. I just honestly didn't want to deal with DNS propagation limiting my flexibility.

3) At least at Yelp, we don't really have particularly nice control apis for DNS. This is sort of specific to Yelp, but it was a factor.

Don't muck with the DNS record. Use an "A record" associated with a secondary/floating IP; then simply move the IP associated with the A record to the new machine and issue a gratuitous ARP.

You can avoid setting arbitrarily low--but not low enough--TTL, etc.