I Didn't Need Kubernetes, and You Probably Don't Either

Those "necessary" add-ons and sidecars are out of control, but its the people problem. I'm part of the infra team and we manage just couple of k8s clusters, but those are quite big and have very high traffic load. The k8s + terraform code is simple, with no hacks, reliable and easy to upgrade. Our devs love it, we love it too and all of this makes my job pleasant and as little stressful as possible.

But we recently hired a staff engineer to the team (now the most senior) and the guy just cannot rest still. "Oh we need a service mesh because we need visibility! I've been using it on my previous job and its the best thing ever." Even though we have all the visibility/metrics that we need and never needed more than that. Then its "we need a different ingress controller, X is crap Y surely is much better!" etc.

So its not inexperienced engineers wanting newest hotness because they have no idea how to solve stuff with the tools they have, its sometimes senior engineers trying to justify their salary, "seniority" by buying into complexity as they try to make themselves irreplaceable.

I would buy the book. Just translate all "new language" concepts into well known concepts from networking and system administration. It would be best seller.

If I would only have a penny for each time I wasted hours trying to figure out what something in "modern IT" is, just to figure out that I already knew what it is, but it was well hidden under layers of newspeak...

>expert blindness at work.

>It's not that complicated if you limit yourself to the core stuff.

Isn't this the core problem with a lot of technologies. There's a right way to use it, but most ways are wrong. An expert will not look left and right anymore, but to anyone entering the technology with fresh eyes it's a field with abundance of landmines to navigate around.

It's simply bad UX and documentation. It could probably be better. But now it's too late to change everything because you'd annoy all the experts.

>There's way too much vendor garbage surrounding the ecosystem

Azure has been especially bad in this regard. Poorly documented in all respects, too many confusing UI menus that have similar or same names and do different things. If you use Azure Kubernetes the wrapper makes it much harder to learn the "core essentials". It's better to run minkube and get to know k8s first. Even then a lot of the Azure stuff remains confusing.

> Admittedly, I've been using Kubernetes since the early days and I manage an Infra team

I think this is where the big difference is. If you're leading a team and introduced all good practices from the start, then the k8s and Terraform or whatever config files can never get so very complicated that a Gordian knot isn't created.

Perhaps k8s is nice and easy to use - many of the commands certainly are, in my experience.

Developers have, over years and decades, learned how to navigate code and hop from definition to definition, climbing the tree and learning the language they're operating in, and most of the languages follow similar-enough patterns that they can crawl around.

Configuring a k8s cluster has absolutely none of that knowledge built up; and, reading something that has rough practices is not a good place to learn what it should look like.

This, all the sidecars. Use kubernetes to run your app like you would without it, take advantage of the flexibility, avoid the extra complexity. Service discovery sidecars? Why not just use the out of the box dns features?

Agreed. The best thing we did back when we ran k8s clusters, was moving a few stateful services to dedicated VMs and keep the clusters for stateless services (the bulk) only. Running k8s for stateless services was an absolute bliss.

At that time stateful services were somewhat harder to operate on k8s because statefulness (and all that it encapsulates) was kinda full of bugs. That may certainly have changed over the last few years. Maybe we just did it wrong. In any case if you focused on the core parts of k8s that were mature back then, k8s was (and is) a good thing.

So you don't run any databases in those thousands of clusters?

To your point, and I have not used k8s I just started to research it when my former company was thinking about shoehorning cassandra into k8s...

But there was dogma around not allowing access to VM command exec via kubectl, while I basically needed it in the basic mode for certain one-off diagnosis needs and nodetool stuff...

And yes, some of the floated stuff was "use sidecars" which also seemed to architect complexity for dogma's sake.

> It's not that complicated if you limit yourself to the core stuff

Completely agree. I use Kubernetes (basically just Deployments and CronJobs) because it makes deployments simple, reliable and standard, for a relatively low cost (assuming that I use a managed Kubernetes like GKE where I don’t need to care at all about the Kubernetes engine). Using Kubernetes as a developer is really not that hard, and it gives you no vendor lock-in in practice (every cloud provider has a Kubernetes offer) and easy replication.

It’s not the only solution, not the simplest either, but it’s a good one. And if you already know Kubernetes, it doesn’t cost much to just use it.

Being nitpicky about Python specifically, Python not necessarily single-threaded; gunicorn gets you a multiprocess HTTP/WSGI server if you configure it to. asyncio and gevent have made it easier to do things actually-concurrently. Judicious use of generator functions lets you stream results back instead of blocking I/O for big chunks. And we have threads. Yeah, the Global Interpreter Lock is still hanging around, it's not the fastest language, but there are ways to produce a Docker image of an API written in Python which can handle thousands of concurrent HTTP requests and actively use all available CPU cores to do intensive computation.

It really, really wants a higher-level abstraction layer over the top of it. I can see how you'd build something heroku-like with it, but exposing app developers to it just seems cruel.

Run a node process on each core?

Speaking of Kubernetes performance: I had a need for fast scale-out for a bulk testing exercise. The gist of it was that I had to run Selenium tests with six different browsers against something like 13,000 sites in a hurry for a state government. I tried Kubernetes, because there's a distributed Selenium runner for it that can spin up different browsers in individual pods, even running Windows and Linux at the same time! Very cool.

Except...

Performance was woeful. It took forever to spin up the pods, but even once things had warmed up everything just ran in slow motion. Data was slow to collect (single-digit kilobits!), and I even saw a few timeout failures within the cluster.

I gave up and simply provisioned a 120 vCPU / 600 GB memory cloud server with spot pricing for $2/hour and ran everything locally with scripts. I ended up scanning a decent chunk of my country's internet in 15 minutes. I was genuinely worried that I'd get put on some sort of "list" for "attacking" government sites. I even randomized the read order to avoid hammering any one site too hard.

Kubernetes sounds "fun to tinker with", but it's actually a productivity vampire that sucks up engineer time.

It's the Factorio of cloud hosting.

These comments:

    > He didn’t believe it was possible to have this kind of performance in a “legacy monolithic” (his words) application.

    > I think it’s so heavily ingrained in most cloud/web developers that this is the only option that they will not even entertain the thought that it can be done another way.

I mostly agree with you, but at least using Docker is something one should be doing even if one is on bare metal.

Pure bare metal IME only leads to people ssh'ing to hotfix something and forgetting to deploy it. Exclusively using Docker images prevents that. Also, it makes firewall management much, much easier as you can control containers' network connectivity (including egress) each on their own, on a bare-metal setup it involves loads of work with network namespaces and fighting the OS-provided systemd unit files.

What do you think its niche is?

The real benefits of LLMs is to give me a good summary of these documents. Still, using these abstractions is probably not worth it.

K8s is absolutely not Google-scale. Not even close. The scheduler is not built for it.

Terraform has the same problem as Kubernetes sidecars with terraform providers trying to do magic for you. If you stick to the cloud platform provider it is actually much nicer than using the CLI.

Although my experience is with AWS, I find the terraform AWS provider docs better documentation than the official AWS docs for different options. If they don't answer any question I have right away they at least point me where to look for answers in the mess that is AWS docs.

This was a good read! I have similar thoughts especially about IaC vs a bash script. Definitely clear pros and cons to both, but I’m wondering how you handle infrastructure drift with imperative bash scripts?

I mean hopefully no one is logging into Azure to fuck with settings but I’m sure we’ve all worked with that one team that doesn’t give a flying fuck about good practices.

Or say you wish to now scale up a VM, how does your bash script deal with that?

Do you copy past the old script, pass new flags to the Azure CLI, and then run that, then delete the old infrastructure somehow?

I’m curious because I think I’d like to try your approach.

This for me is THE reason for using container management. Without containers, you end up with hundreds of VMs. Then, when the time comes that you have to upgrade to a new OS, you have to go through the dance, for every service:

- set up new VMs

- deploy software on new VMs

- have the team responsible give their ok

It takes forever, and in my experience, often never completes because some snowflake exists somewhere, or something needs a lib that doesn't exist on the new OS. VMs decouple the OS from the hardware, but you should still decouple the service from the OS. So that means containers. But then managing hundreds of containers still sucks.

With container management, I just

- add x new nodes to cluster

- drain x old nodes and delete them

Even as a K8s hater, this is a pretty salient point.

If you are serious about minimizing ops work, you can make sure people are deploying things in very simple ways, and in that world you are looking at _very easy_ deployment strategies relative to having to wire up VMs over and over again.

Just feels like lots of devs will take whatever random configs they find online and throw them over the fence, so now you just have a big tangled mess for your CRUD app.

I can imagine. Do you have complete automation setup around maintaining the cluster?

We are now on-prem using “pet” clusters with namespace as a service automated on it. This causes all kinds of issues with different workloads with different performance characteristics and requirements. They also share ingress and egress nodes so impact on those has a large blast radius. This leads to more rules and requirements.

Having dedicated and managed clusters where everyone can determine their sizing and granularity of workloads to deploy to which cluster is paradise compared to that.

> than provisioning a bunch of new VMs each time a team need some services deployed.

Back in the old days before cloud providers this was the only option. I started my career in early 2010s and got the tailend of this, it was not fun.

I remember my IT department refusing to set up git for us (we were using SVN before) so we just asked a VM and set up a git repo in there ourselves to host our code.

> For most small setups, the cost of running an empty kubernetes cluster (managed) is typically higher than setting up a db, a couple of vms and a loadbalancer, which goes a long way for running a simple service.

Not really, the cost of an empty EKS cluster is the management fee of $0.1/hour, or roughly the price of a small EC2 instance.

GCP has $74 free credit for Zonal cluster, so you effectively have the first cluster for free.

And even $70 is cheap, considering that a cluster should be shared by all the services from all the teams in the same environment, bar very few exceptions.

That's around the cost of a single VM (cheapest 8GB ram I found quickly).

Azure has a free tier with control plane completely free (but no SLA) - great deal for test clusters and testing infra.

If you are that worried about costs, then public cloud may not be for you at all, or you should look at ECS/App containers or serverless.

> just use a PaaS like Google App Engine, Amazon Elastic Beanstalk, Digital Ocean App Platform, or Heroku.

This is the right way for web most of the time, but most places will choose k8s anyway. It’s perplexing until you come to terms with the dirty secret of resume driven development, which is that it’s not just junior engs but lots of seniors too and some management that’s all conspiring to basically defraud business owners. I think the unspoken agreement is that Hard work sucks, but easy work that helps you learn no transferable skills might be worse. The way you evaluate this tradeoff predictably depends how close you are to retirement age. Still, since engineers are often disrespected/discarded by business owners and have no job security, oaths of office, professional guilds, or fiduciary responsibility.. it’s no wonder things are pretty mercenary out there.

Pipelines are as important as web these days but of course there are many options for pipelines as a service also.

K8s is the obviously correct choice for teams that really must build new kinds of platforms that have many diverse kinds of components, or have lots of components with unique requirements for coupling (like say “scale this thing based on that other thing”, but where you’d have real perf penalties for leaving the k8s ecosystem to parse events or whatever).

The often mentioned concern about platform lock in is going to happen to you no matter what, and switching clouds completely rarely happens anyway. If you do switch, it will be hard and time consuming no matter what.

To be fair, k8s also enables brand new architectural possibilities that may or may not be beautiful. But it’s engineering, not art, and beautiful is not the same as cheap, easy, maintainable, etc.

PaaS get such a bad rap from devs in my experience, even though they would solve so many problems. They'd rather keep their k8s clusters scaled to max traffic and spend their nights dealing with odd networking and configuration issues than just throw their app on Cloud Run and call it a day.

The same argument can be made about Borg. Someone at Google needs to know about things like Juniper switches and hard drive bays. Someone needs to repair or replace defective compute nodes and power switches. Before they had software load balancing, someone had to manage the hardware load balancers.

But the idea of Borg is that all of that's abstracted away for the typical developer. It's the same with k8s. The infrastructure team in your org needs to understand the implementation details, but really only a few.

You can also configure load balancers, IAM groups & policy etc as k8s CRDs. So all that stuff can be in one place in the code base alongside the rest of your infrastructure. So in that sense it does abstract those concepts. You still need to know something about them, but you don't have to configure them programmatically yourself since k8s will do that.

You can self host Kubernetes on "dumb" VMs from Hetzner or OVH.

K8s was designed around deployment on premise on bare metal hardware.

The cloud extensions were always just a convenience.

Is nomad still around?

how does it work? how many nodes? what do you use for consensus, request routing, etc...?

>Just as a carpenter doesn't use a hammer to saw, we as a profession should strive to use the right tool for the right job. (Albeit it's less clear than for the carpenter, granted)

The problem is that most devs don’t view themselves as carpenters. They view themselves as hammer capenters or saw carpenters etc…

It’s not entirely their fault, some of the tools are so complex that you really need to devote most of your time to 1 of them.

I realize that this kind of tool specialization is sometimes required, but I that it’s overused by at the very least an order of magnitude.

The vast majority of companies that are running k8s, react, kafka etc… with a team of 40+, would be better off running rails (or similar) on heroku (or similar), or a VPS, or a couple servers in the basement. Most of these companies could easily replace their enormous teams of hammer carpenters and saw carpenters with 3-4 carpenters.

But devs have their own gravity. The more devs you have the faster you draw in new ones, so it’s unclear to me if a setup like the above is sustainable long term outside of very specific circumstances.

But if it were simpler there wouldn’t be nearly many jobs, so I really shouldn’t complain. And it’s not like every other department isn’t also bloated.

Along those lines, I am building https://github.com/claceio/clace for teams to deploy internal tools. It provides a Cloud Run type interface to run containers, including scaling down to zero. It implements an application server than runs containerized apps.

Since HTMX was mentioned, Clace also makes it easy to build Hypermedia driven apps.

> Just as a carpenter doesn't use a hammer to saw, we as a profession should strive to use the right tool for the right job

I think this is a gross misunderstanding of the complexity of tools available to carpenters. Use a saw. Sure, electric, hand powered? Bandsaw, chop saw, jigsaw, scrollsaw? What about using CAD to control the saw?

> Suddenly there is no build step anymore

How do you handle making sure the JS you write works on all the browsers you want to support? Likewise for CSS: do you use something like autoprefixer? Or do you just memorize all the vendor prefixes?

It's actually kinda hilarious how RSC (React Server Components) is pretty much going back to what PHP was but yeah proves your point as hype moves on people begin to realize why certain things were good vs not

where does tailwind stand on this? you can use it without a build step but it's strongly recommended in production

[dead]

Why would you want to run it anywhere. People mostly select an OS and just update that. It may be great when distributing applications for others to host, but not when it’s the only strategy. I have to reverse engineer dockerfiles when the developer wouldn’t provide a proper documentation.

But how much this costs extra? Sounds like you are using cloud-provided k8s.

That's an interesting set of requirements though. If that is indeed your set of requirements then perhaps Kubernetes is a good choice.

But the set seems somewhat arbitrary. Can you reduce it further? What if you don't require 2 cloud providers? What if you don't need zero-downtime?

Indeed given that you have 4 machines (2 instances, x 2 providers) could a human manage this? Is Kubernetes overkill?

I ask this merely to wonder. Naturally if you are rolling out hundreds of machines you should, and no doubt by then you have significant revenue (and thus able to pay for dedicated staff) , but where is the cross-over?

Because to be honest most startups don't have enough traction to need 2 servers, never mind 4, never mind 100.

I get the aspiration to be large. I get the need to spend that VC cash. But I wonder if Devops is often just premature and that focus would be better spent getting paying customers.

"Imagine you are in a rubber raft, you are surrounded by sharks, and the raft just sprung a massive leak - what do you do?". The answer, of course, is to stop imagining.

Most people on the "just use bash scripts and duct tape" side of things assume that you really don't need these features, that your customers are ok with downtime and generally that the project that you are working on is just your personal cat photo catalog anyway and don't need such features. So, stop pretending that you need anything at all and get a job at the local grocery store.

The bottom line is there are use cases, that involve real customers, with real money that do need to scale, do need uptime guarantees, do require diverse deployment environments, etc.

I think you are proving the point; there are very, very few applications that need to run on two cloud providers. If you do, sure, use Kubernetes if that makes your job easier. For the other 99% of applications, it’s overkill.

Apart from that requirement, all of this is very doable with EC2 instances behind an ALB, each running nginx as a reverse proxy to an application server with hot restarting (e.g. Puma) launched with a systemd unit.

My personal goto-solution for those requirements -- well 1 cloud provider, I'll follow up on that in a second -- would be using ECS or an equivalent service. I see the OP was a critic of Docker as well, but for me, ECS hits a sweet spot. I know the compute is at a premium, but at least in my use-cases, it's so far been a sensible trade.

About the 2 cloud providers bit. Is that a common thing? I get wanting migrate away from one for another, but having a need for running on more than 1 cloud simultaneously just seems alien to me.

Do you know of actual (not hypothetical) cases, where you could "flip a switch" and run the exact same Kubernetes setups on 2 different cloud providers?

I've worked at tiny startups before. Tiny startups don't need zero-downtime-deployment. They don't have enough traffic to need load balancing. Especially when you are running locally, you don't need any of these.

Why does a startup need zero-downtime-deployment? Who cares if your site is down for 5 seconds? (This is how long it takes to restart my Django instance after updates).

We’ve been deploying software like this for a long ass time before kubernetes.

There’s shitloads of solutions.

It’s like minutes of clicking in a ui of any cloud provider to do any of that. So doing it multiple times is a non issue.

Or automate it with like 30 lines of bash. Or chef. Or puppet. Or salt. Or ansible. Or terraform. Or or or or or.

Kubernetes brings in a lot of nonsense that isn’t worth the tradeoff for most software.

If you feel it makes your life better, then great!

But there’s way simpler solutions that work for most things

Having done both: running a small Kubernetes cluster is simpler than managing a bunch of systemd files.

I wish Docker Swarm would get more attention. It could be the perfect Kubernetes lightweight alternative. Instead it seems like it could get deprecated any day now.

This assumes your Postgres has a an infinite amount of memory which would also be very expensive. I think you'd probably want to assume each connection takes ~10-20MB of memory.

GCR assumes it's workload is HTTP, or a "job" (container that exits once it's task has completed). It scales on request volume and CPU, and the load balancer is integrated into the service. It's not obvious to me how you'd even run a "raw" TCP service like pgbouncer on it.

I’m not an expert, but from what I understand, the standard set up is like:

4x(Web processes) -> 1x(pgbouncer) -> database

This ensures that the pgbouncer instance is effectively multiplexing all the connections across your whole fleet.

In each individual web process, you can have another shared connection pool.

This is how we set it up

Nice, I didn't know about this, it wasn't available last time I checked.

With that said... there are so many limitations on that list, that seriously, I can't imagine it would really be so much easier than Kubernetes.

I've been running apps on Kubernetes clusters for the past 6 years and the only thing that really comes to mind that was a breaking change was when the ingress class resource type was introduced. Everything else has been incremental. Maybe I'm forgetting something.

What's wrong with recommending a managed cluster? I wouldn't use one but it is certainly an option for teams that don't want to spin up a cluster from scratch, although it comes with its own set of tradeoffs.

My project at the moment is definitely easier thanks to Kubernetes as pods are spun up dynamically and I've migrated to a different cloud provider and since migrated to a mix of dedicated servers and autoscaled VMs, all of which was easy due to the common deployment target rather than building on top of a cloud provider specific service.

This is obvious FUD from a throwaway account. 1.x Kubernetes breakage has rarely affected me. I’m a team of one and k8s has added a lot of value by allowing me to build and run reliable, auto-managed applications that I’m confident I can transfer across clouds.

That's pretty much it. I think the main issue nowadays is that companies think full stack engineering means OG(FE BE DB) + CICD + Infra + security compliance + SRE.

If a team of 5-10 SWEs have to do all of that while only graded on feature releases, k8s would massively suck.

I also agree that experienced platform/infra engineers tend to whine less about k8s.

Nah the difference between managing k8 and the system it was based on is VASTLY different. K8 is much harder than it needs to be because there wasn't tooling for a long time to manage it well. Going from google internal to K8 is incredibly painful.

I think this is only true if the original k8s cluster you're operating against was written by an expert and laid out as such.

If you're entering into k8s land with someone else's very complicated mess across hundreds of files, you're going to be in for a bad time.

A big problem, I feel, is that if you don't have an expert design the k8s system from the start, it's just going to be a horrible time; and, many people, when they're asked to set up a k8s setup for their startup or whatever, aren't already experts, so the thing that's produced is not maintainable.

And then everyone is cursed.

You typically have it configured with a cap on instances for cost reasons.

Even if you don't, if you have other services (like a database) downstream, you might not want it to scale infinitely as then you're simply DoS'ing the DB instead of the Cloud Run service.

Backpressure is really important for the resiliency of any distributed system, IMO.