Powering cost-efficient AI inference at scale with Cloud TPU v5e on GKE

To the end user, everything at Google has become shit or not changed at all. I am sure GCS is great but most people don't use it. What they do see is search becoming more ads than content and search not even including any relevant search terms in "here are your 1000000 results". I actually go to yandex and bing if I want to search for a specific search term.

Let's not talk about friends losing all access to their accounts, forever.

I switched a whole firm off GCS when their troubleshooting by 'senior' techs was just copy and pasted boilerplate too many times. The way I set things up was cloud agnostic at the time. Not something that is possible at scale.

Can anyone else be trusted? Probably not.

Can you trust someone on the crack-pipe full of advertising dollars?

My guess is that google ran the numbers and isn’t willing to lose money like openai is, so they have been slower to develop things as cool as openai, leading to a perception that they can’t do it. Hence the lack of excitement.

But I think that’s unlikely, and once it’s not money-losing, google will be right there. Which, as you say, they might have an edge on due to hardware.

> Why are more people not excited by what is happening at Google? What am I missing?

I'd love contra-indicators. I'm not plugged in to this community nearly as muc has I am the rest of tech. But the general scuttlebutt I got was that access to TPU4 was quite competitive. It didn't seem to be like a cloud resource where you could just go use it: it took work and effort and time and many other people could butt in line ahead of you all the time to get TPUv4. Apologies for relaying hearsay, but if this is far better tech, it feels like Google is going to have to solve the scale-out concerns exponentially harder if TPUv5e is to actually* have an effect on most practitioners. It's my hope that these negative concerns were overstated; it's my hope that the situation is better. Does anyone have any idea what we could look at to suss how much of a concern this is?

I think you've also casually stumbled into another issue. To quote you:

> but they will own the whole stack.

This feels like the typical slant against Google, and I think often this is a disservice.

I personally attribute much better to Google. To me, Google has been one of the few companies that still has some of the intertwingularity in their ethos, that still has some Cluetrain Manifesto blood in their veins where they grok that the way to be a winning player is to bring a ton of selfless. But we keep seeing spin that makes Google look egocentric & self-biased & self-dealing, in so many ways. Any bad Google anywhere outshadows the rest of the good Google; Google is a massive goliath powered by the vastest ad market on the planet, but where-as most companies constantly focus on mining value of their ecosystem & imho Google is one of the rare few companies that groks that the existential stakes of creating open healthy accessible ecosystems. Imho they have almost never stacked the deck to be overlords (fuck you very very much MV3, you self-dealing app-store owners how casually rescind so many dimensions of freedom so easily for yourselves) but instead espoused a multi-polar world, one made up of many independent empowered factions. If anything, I think much of the scorn laid at Chrome's feet, for example, actually reflects on the rest of the world's inability to be as good, to serve as even remotely adequate patrons to the web: governments everywhere should be funding web development, web specs, web standards. There's just so few players & that's not Google's fault. The bread & butter of open information society is all too moldy & stale in most areas, too few players. Still, I've seen Google persist, try. I find it admirable. I see this in many ways, but the faults & criticisms are always such drastic lightning rods for anger.

No anger here yet, but: there's hardly a better example of Google trying to play well than today's Keras 3.0 announcement. 'Hello, we are Google, we've optimized the ever loving hell out of our existing machine learning platform to target a whole bunch of other environments, make them 2D-grid data & model parallel across a variety of frameworks targeting by far the widest range of hardwares. And so now, if you target Keras, any of these other platforms will work intercompatibly very well.' Once again, Google steps in the ring to carry water for the world, to make practioner's lives much easier. Sure, this enables TPUv5e to be easily targetted, via a variety of different code-styles; that is self interested. But it also enables a degree of portability out of TPU & across systems that seems magical, that is extremely late bound, that permits permits permits. I'm harping on just a small bit you said, apologies, but in contrast to your 'own the whole stack', it feels like Google is the only player trying to make sure no one owns the stack, and it seems like they're doing great at it. TPUv5e happens to be one option these actions bolster. https://keras.io/keras_3/ https://news.ycombinator.com/item?id=38446353

Thank you! Regarding performance, TPU v5e which was benchmarked in the MLPerf results and showed impressive performance/dollar, see https://cloud.google.com/blog/products/compute/performance-p... which has more details. Combined with k8s & GKE, TPU v5e workloads can leverage auto-scaling, for example setting up autoscaling based on traffic, so workloads can scale down rapidly when not in use, increasing cost efficiency.

At this point I can’t see why anyone would build their business on any Google product aside from Google Docs.

My vague read from before was that TPU4 has been out for a long long time & was basically near impossible to get time in unless you could bid super high.

Not my field & this could be super wrong. But my skepticism that a TPU5e launch was going to help anyone but some niche well connected other big players has been very high. I'm interested from an academic sense, and I love work like the Keras 3.0 release that makes me much more universal/portable, such that hardware like this or others can burn f-ing metal. But I don't have much trust this will actually be available at a reasonable price/with any kind of availability. https://news.ycombinator.com/item?id=38446353 https://keras.io/keras_3/

I'd love to be wrong & have this be actually available at a good price to change what is possible for people, & still interesting. Your call out is right. There's no lightning rod here, nothing to be hyoer-critical of, and few step outside the comfort zone of CUDA have anywhere near enough idea how to assess. This is an unknown on the radar for most.

My take as a GCP employee: Amazon has really dominated the startup world, but google cloud seems to really be hitting its stride with big enterprise tech companies.

Between its support for K8s, networking stack/infra, and TPUs/ML support we have a lot of huge well known tech companies knocking down our door.

But none of those things really matter as a differentiator for the average startup, scale up, or small business. So all in all, i’m not surprised the majority of HN engineers just aren’t excited by GCP.

I have no knowledge of Cloud’s long term strategy, but it wouldn’t surprise me if by leaning into its success in big tech, they are neglecting the startup scene in terms of marketing or building out new features.

They lost their way or HN readers are out of touch?

Google, structurally, sucks at anything but sucking down Ad money.

A good LLM allows one to basically bypass the internet for many use cases and drastically reduces ad time. As cool as LLM capabilities are the thing that delights me most about them is I can get high quality (YMMV) answers from the internet without all the godforsaken ad bullshit and wonky websites made only as mazes to keep users in ad gardens.

Google is stuck in one hell of an innovators dilemma, one that I can't really see them getting unstuck from.

As far as this particular product, it's touting cost efficiency, but that's not really dominating many decisions yet. There is an all-in sprint to get models trained and take their piece of the pie before the moats get too deep and AI hardware everywhere is back-ordered 12+ months.

Google is no longer a tech heavy company. The only “interesting” things coming out of there are high level executives assigning engineers on random projects against their will, return to office orders, sarcastic offers to relocate workers into company owned hotels so they can “sleep longer” in the morning, and many other mediocre “achievements” of mba type “leaders”.

Just by looking at their core hook, search, you notice a clear reduction in quality. Their core products, ads, are also rife with dubious practices - even more so than in the past. I doubt these are due to technical issues but rather a result of poor product management. Lesson is that no technical company should be led by non technical people. They all turn to shit when that happens.