The truth is gRPC, like kubernetes, was built with decades of lessons of RPC framework inside a container oriented distributed environment; and more importantly, gRPC is the blessed framework inside Google as well, meaning it's qualified to power the largest and most complex distributed systems in the world (I think it'd safe to omit 'one of' here), which in comparison is not the case for kubernetes.
Addition: Borg and kubernetes are designed with similar goals but different emphasis. They are like complementing twins had different personalities. For this I recommend Min Cai's Kubecon'18 presentation about peleton [1], the slide is titled "comparison of cluster manager architecture".
Wait, I don’t get it. Kubernetes:Borg::gRPC:Stubby. Google uses gRPC internally to the same extent that they use Kubernetes internally, i.e. hardly at all.
This analogy is very misleading. Kubernetes is probably never going to run any real workload internally at Google, but gRPC powers all external APIs of Google Cloud, and increasing other Google properties (e.g. ads, assistant), used by mobile apps like Duo and Allo, and have some big internal service use cases. The reason Stubby still dominates internally is simply taking lots of time to migrate to gRPC that might be hard to justify, but I do see gRPC being used very widely internally at Google; it’s simply a matter of time. I don’t see that happening to Kubernetes; it’s a joke when compared to Borg.
That's not true. GCE uses Borg very differently than normal Google internal systems, which you can imagine is quite natural as they are reserving different customers. GCS and other system, in turn, also differ wildly than GCE. when you talk about GCP as a whole, it becomes impossible to summarize in a few statement, and I doubt there is anyone on earth who is capable to describe it coherently even without time constraint.
What I said (GCP runs on Borg) is absolutely and technically correct, affirmed by your own comment, which highlights the power and flexibility of Borg. The point being no one[1] at Google relies on Kubernetes for raw cluster management capabilities at scale. They might use it for other things that can make deployment more friendly in some scenarios. (This doesn’t make Kubernetes a bad system by any means, just quite different and not a substitute for Borg whereas gRPC is a direct substitute for Stubby). This debate is better argued in your own eng-misc@ and not on a public forum.
[1]: no-one that we care. At Google this is obviously always incorrect. There’s always that someone who uses weird things like mongoDB and AWS.
And there's no reason why a small project should not. But nobody is going to move, say, indexing to GCP. And when it comes to power laws the big things are big and the small things are not.
This sounds like a No True Scotsman argument to me, that if something runs on GCP instead of Borg, it isn't "real". Also throw in shades of moving goalposts.
Indexing doesn't run on GCP primarily because it's legacy (as in, the first product Google ever did) and thus long predates GCP itself.
It’s neither of those fallacies. The fallacy is to suppose that if you know several people using technology X then it must be quite popular. We see this all the time on HN where people suppose that, say, Erlang is quite popular because there are dozens of companies, each with five engineers, using it. But then we ignore that there are five companies with a hundred thousand engineers each that do everything in c++. It’s the same with these other things. It’s quite like that K8s satisfies the requirements of 80% of the projects at Google and it’s also quite likely that all of them put together consume 1% of the production resources, so it leads to the question of whether it’s even capable of solving a really large problem, as mehrdada argues elsewhere in this thread.
It's not a Google product obviously, but Snapchat runs on GCP. That's quite big. Is that not a "real" product? Admittedly they're on App Engine, a much older product than Kubernetes, but I suspect they'd be able to run on Kubernetes, and perhaps that's what they would choose if they were to build from scratch right now.
Indexing has been rewritten many times over. Even if you removed all the dependencies on Bigtable and co., I think indexing would be the last to move, for quite practical reasons, due to its sheer size and design. The parent poster picked probably the worst workload to migrate to public GCP. Gmail, YouTube and search serving are easier in comparison.
This may be true today (although even on this metric I’d estimate Kubernetes to be off by some orders of magnitude, bordering zero). My point is there’s a path and plan forward for gRPC adoption and it’s a matter of transition to a new system (which can, admittedly, be very long). For Kubernetes, I don’t think there is a credible path for replacing Borg.
I used to work on one of Borg teams at Google and now run Kubernetes platform. Nearly every Kubernetes component (node, master, networking) will melt down at fraction of Borg scale. It's not even close.
Borg is cirra 2003, pb/stubby was before that. Gfs probably was similar in timing as stubby. And many other cluster level foundations. In the end, Borg is the true corner stone that ties everything together and completes the Google infrastructure puzzle (or the modern global scale cluster computing).
Addition: Borg and kubernetes are designed with similar goals but different emphasis. They are like complementing twins had different personalities. For this I recommend Min Cai's Kubecon'18 presentation about peleton [1], the slide is titled "comparison of cluster manager architecture".
[1] https://kccna18.sched.com/event/GrTx/peloton-a-unified-sched...