Mesh LLM Pools Idle GPUs Into a Distributed AI Supercomputer

iroh's Mesh LLM lets teams pool existing GPUs across machines into one OpenAI-compatible API, splitting models too large for any single box.

When people picture running a large language model, they picture a data center — racks of GPUs that belong to someone else, a metered API, and a bill that grows every month. Mesh LLM takes a different approach: it pools the GPUs and memory you already have, across as many machines as you want, and exposes the whole thing as a single OpenAI-compatible API.

Built on iroh, the peer-to-peer networking library from n0-computer, Mesh LLM handles AI inference three ways: locally on a single machine, routed to a peer that already has the model loaded, or split across several machines as a pipeline — internally called "Skippy."

The split mode partitions a model by layer ranges: layers 0 to 15 on one node, 16 to 31 on the next, and so on. Activations flow from stage to stage over authenticated QUIC connections, so several modest machines can run a model none of them could hold alone. The catalog ships with 40+ models, from half-a-billion-parameter models that fit on a laptop to 235B mixture-of-experts giants.

There is no central server. Every node boots an iroh endpoint identified by a public key, and iroh handles NAT traversal, hole-punching, and relay fallback. A lightweight 18 MB install points any standard OpenAI client at localhost:9337/v1 and stops caring where the work actually happens.

The project is open source and accepting public mesh contributions. A mobile app built on iroh's Swift SDK is on the roadmap, with plans to speak ACP, the emerging agent standard, so other clients can join the mesh too.

Sources: iroh Blog, Mesh LLM GitHub, Hacker News Discussion