Exo: Run your own AI cluster at home with everyday devices

Repo maintainer here. It supports any device tinygrad does, which is a lot. We didn’t expect it to blow up so soon - the repo is still experimental. Internally we’ve mostly been testing on MacBooks and Mac Minis, and that’s where dev is happening. The swift implementation is outdated and currently broken, since Python has been changing so fast (over 20 commits in the last day). On my ToDo is CI/CD pipeline with integration tests for different device and network configurations, so we don’t randomly break stuff for certain devices.

We’re moving fast to get it stable and usable. The goal is for this to be as simple as running Dropbox. Bear with us :)

One of the maintainers has a video demo on his twitter claiming iOS, android and Linux. Some of the code is not released and I wish they were advertising that properly.

The README says they plan to add llama.cpp support which should cover a lot of targets, also they have tinygrad already integrated I think.

We didn't expect this to blow up so quickly. A lot of work needs to be done on getting different setups working. I have made an issue here: https://github.com/exo-explore/exo/issues/11

Yes, so you would have a vector about 8k values long to be transferred on each token generated.

You could do that easily with any modern network.

Yes, that’s how it works (pipeline parallelism)

I'd rather my CPU to be idle and not consome much power

This might not work for use cases where you need low latency, but for longer winded processing it would be amazing if possible.

For example, if I have a few servers, laptop (connected to power) as well as a desktop PC and they’re all connected to a fast local network, it’d be great to distribute the task of rendering a video or working with archive files across all of them.

This exists: https://aihorde.net/

I haven’t tried it, and not the norm, but I agree it should be more common. We have a global supercomputer with higher latency, but still a supercomputer.

Maybe you want to conduct experiments that the cloud API doesn't allow for.

Perhaps you'd like to plug it into a toolchain that runs faster than API calls can be passed over the network? -- eventually your edge hardware is going to be able to infer a lot faster than the 50ms+ per call to the cloud.

Maybe you would like to prevent the monopolists from gaining sole control of what may be the most impactful technology of the century.

Or perhaps you don't want to share your data with Microsoft & Other Evils (formerly known as dont be evil).

You might just like to work offline. Whole towns go offline, sometimes for days, just because of bad weather. Nevermind war and infrastructure crises.

Or possibly you don't like that The Cloud model has a fervent, unshakeable belief in the propaganda of its masters. Maybe that propaganda will change one day, and not in your favor. Maybe you'd like to avoid that.

There are many more reasons in the possibility space than my limited imagination allows for.

> Sure, I can run some slow Llama3 models on my home network, but why bother when it is so cheap or free to run it on a cloud service?

Obvious answer: because it's not free, and it's not cheap.

If you're playing with a UI library, lets say, QT... would you:

a) install the community version and play with ($0)

b) buy a professional license to play with (3460 €/Year)

Which one do you pick?

Well, the same goes. It turns out, renting a server large enough to run big (useful, > 8B) models is actually quite expensive. The per-api-call costs of real models (like GPT4) adds up very quickly once you're doing non-trivial work.

If you're just messing around with the tech, why would you pay $$$$ just to piss around with it and see what you can do?

Why would you not use a free version running on your old PC / mac / whatever you have lying around?

> I used to be excited about running models locally

That's an easy position to be one once you've already done it and figured out, yes, I really want the pro plan to build my $StartUP App.

If you prefer to pay for an online service and you can afford it, absolutely go for it; but isn't this an enabler for a lot of people to play and explore the tech for $0?

Isn't having more people who understand this stuff and can make meaningful (non-hype) decisions about when and where to use it good?

Isn't it nice that if meta released some 400B llama 4 model, most people can play with it, not just the ones with the $7000 mac studio? ...and keep building the open source ecosystem?

Isn't that great?

I think it's great.

Even if you don't want to play, I do.

We are running smaller models with software we wrote (self plug alert: https://github.com/singulatron/singulatron) with great success. There are obvious mistakes these models make (such as the one in our repo image - haha) sometimes but they can also be surprisingly versatile in areas you don't expect them to be, like coding.

Our demo site uses two NVIDIA GeForce RTX 3090 and our whole team is hammering it all day. The only problem is occasionally high GPU temperature.

I don't think the picture is as bleak as you paint. I actually expect Moore's Law and better AI architectures to bring on a self-hosted AI revolution in the next few years.

I have found many similarities between home AI and home astronomy. The equipment needed to get really good performance is far beyond that available to the home user, however intellectually satisfying results can be had at home as a hobby. But certainly not professional results.

Why bother running models locally? Privacy, for once, or censorship resistance.

For my advanced spell-checking use-case[^1], local LLMs are, sadly, not state-of-the-art. But their $0 price-point is excellent to analyze lots of sentences and catch the most obvious issues. With some clever hacking, the most difficult cases can be handled by GPT4o and Claude. I'm glad there is a wide variety of options.

[^1] Hey! If you know of spell-checking-tuned LLM models, I'm all ears (eyes).

I'm not sure why you have resigned?

If you don't care about running it locally, just spend it online. Everything is good.

But you can run it locally already. Is it cheap? No. Are we still in the beginning? yes. We are still in a phase were this is a pure luxury and just getting into it by buying a 4090, is still relativly cheap in my opinion.

Why running it locally you ask? I personally think running anythingllm and similiar frameworks on your own local data is interesting.

But im pretty sure in a few years you will be able to buy cheaper ml chips for running models locally fast and cheap.

Btw. aat least i don't know a online service which is uncensored, has a lot of loras as choice and is cost effective. For just playing around with LLMs for sure there are plenty of services.

https://x.com/karpathy/status/1814038096218083497

LLMs will start shrinking massively in size soon, without any loss in performance.

> Cloud cannot be beaten on compute / price

Sorry, I can't let misinformation like that slide.

Cloud cost/benefit ratio is not good in many circumstances.

For hobbyists it works well because you run your job for very brief periods and renting is much cheaper than buying in those cases. Similarly, if your business usage is so low as to be effectively run once per day then cloud has major benefits.

However, if you are doing any kind of work that consumes more than 8hrs of computer time in a day, cloud is going to start being much more expensive.

The exact cost/benefit depends on the SKU and I'm mostly talking about CPU/Memory/Storage- for managed services like databases it's significantly worse, and I'm comparing to rented servers not self-hosting at home, which is significantly cheaper still.

Local hardware has downsides (availability, inflexibility), but it's faster and cheaper in almost all real workload scenarios where the compute would otherwise be completely idle/turned off >90% of the time.

You can beat gpt4/claude in terms of price/performance for most things by a mile using fine tuned models running in a colo. Those extra parameters give the chatbots the ability to understand malformed input and to provide off the cuff answers about almost anything, but small models can be just as smart about limited domains.

Repo author here. This sounds interesting. Could you elaborate on the benefits of Bumblebee / Axon?

That’s where we want to get eventually. There’s a lot of work that needs to be done but I’m confident we’ll get there. Give us 3 months and it’ll be as simple as running Dropbox.

I suspect that most of the devices you'd expect to find in your consumer cluster are too small/slow to saturate the link.

Edit: it's also a matter of scale. You probably have a small number of small/slow devices in a consumer network versus a lot of large/fast devices in your enterprise cluster.

I haven't looked into exactly what this project is doing, but here's my understanding:

Inference across O(N) pre-trained hidden layers isn't exactly an "embarrassingly parallel" problem, but it is an "embarrassingly pipeline-able" problem (in the CPU sense of "pipelining.") Each device can keep just one or a few layers hot in their own VRAM; and also only needs to send and receive one small embedding (<1MB) vector per timestep — which is so trivial that it's easily achievable in realtime even if all the devices are on wi-fi, talking to the same router, in your "noisy" apartment where 100 other neighbours are on the same bands.

(To put it another way: running a single inference job, has more forgiving realtime latency+throughput requirements than game streaming!)

Assuming that you have a model that's too big for any of your home machines to individually hold; and that all you care about is performance for single-concurrent-request inference on that model — then in theory, you just need one GPU of one node of your homespun Beowulf GPU cluster to have enough VRAM to keep the single largest layer of your model always-hot; and then other smaller devices can handle keeping the smaller layers always-hot. And the result should be faster than "overloading" that same model on that single largest-VRAM device and having some layers spill to CPU, or worse yet, having the GPU have to swap layers in and out repeatedly with each inference step.

(Also, if you're wondering, in the case where a single machine/node has multiple GPUs — or a GPU+VRAM and also a CPU+RAM! — you can treat this as no different than if these were multiple independent nodes, that just-so-happen to have a very efficient pipeline communication channel between them. As the VRAM+computation cost of running inference far outweighs the communication overhead of forward propagation during inference, a home-network inference-pipelining cluster scheduler like this project, would still likely "schedule" the model's layers purely in consideration of the properties of the individual GPU+VRAM (or CPU+RAM), rather than bothering to care about placement.)

---

That being said, AFAIK training is "pipeline parallelizable" exactly as inference is. And people training models do do this — but almost always only across multiple top-of-the-line GPUs in one machine; not across multiple machines.

When you think about what pipelining achieves for training — all you get is either:

1. the ability to use a bunch of small-aggregate-VRAM nodes to achieve the aggregate training capacity of fewer, larger-aggregate-VRAM nodes — but with more power consumption = higher OpEx; and where also, if you scale this to O(N), then you're dumping a quadratic amount of layer-propagation data (which is now both forward-prop and backprop data, and backprop data is bigger!) over what would likely be a shared network just to make this work. (If it's not a shared network — i.e. if it's Infiniband/other RDMA — then why did you spend all that CapEx for your network and not on your GPUs!?)

2. the ability to pipeline a bunch of large-aggregate-VRAM nodes together to train a model that will then never be able to be deployed onto any single node in existence, but can instead only exist as a "pipelined inference model" that hogs O(log N) nodes of your cluster at a time for any inference run. Which makes cluster scheduling hell (if you aren't just permanently wiring the scheduler to treat O(log N)-node groups as single "hyper-nodes"); makes it so that you'll never be able to practically open-source the model in a way anybody but other bigcorps could ever run it (if that's something you care about); and very likely means you're cutting the concurrent-inference-request-serving capacity of your huge expensive GPU cluster by O(log N)... which the product team that allowed that cluster to be budgeted is really not gonna like.

That being said, I imagine at some point one of these proprietary "Inference-as-a-Service" models has been trained at a layer size that puts it into pipelined-inference-only territory, temporarily. Doing so would be the ML engineer's equivalent to the CPU engineer's "we have no fundamentally clever advance, so this quarter we'll just crank up the clock frequency and deal with the higher TDP." (Heck, maybe GPT-4o is one of these.)

---

What people with GPU clusters want, is 1. for the output of the process to be a model that runs on a single (perhaps multi-GPU) node; and 2. for the process itself to be mostly-shared-nothing with as little cross-node communication burden as possible (such that it's just a question of building highly internally communication-efficient nodes, not so much highly-communication-efficient clusters.)

And both of those goals are achieved by sizing models so that they fit within a single node; continuously fanning out streams of training data to those nodes; and then periodically fanning back in model-weights (or model-weight deltas) in an AllReduce operation, to merge the learning of O(N) independently-training nodes to become the new baseline for those nodes.

(If you'll note, this architecture doesn't put any latency requirements on the network, only some monstrous throughput requirements [at the fan-in step] — which makes it a lot easier to design for.)

So I just tried with 2x macbook pros (M2 64GB & M3 128GB) and it was exactly the same speed as with just 1 macbook pro (M2 64GB) Not exactly a common setup but at least it's something

I'm sure someone will show their benchmarks in a couple years!

Do you mean with Apple Intelligence? You can already query models you host from Apple using exo or even just local on-device inference.