A Self-Hosted, Distributed AI Pipeline

The itch

Every time I experimented with hosted AI, two things nagged at me. There was a meter running on every request, and my data had to travel to someone else’s servers to get an answer back. I wanted to find out how much heavy work, large language models, translation, and media rendering, I could run on machines I already owned, with nothing leaving my network.

The move

The obvious move was to buy one powerful machine and run everything on it. I went the other way, with the four very different machines I had to hand: a Raspberry Pi 5, a Mac Mini M4, a Mac Mini M4 Pro, and a desktop with a big GPU. Rather than force the work onto one box, I looked at what each was actually good at. The Pi 5 sits at the edge as a sensor node, always on and sipping a couple of watts. The M4 Mini became the workhorse for services and data: the database, the message broker, the APIs, and the dashboards. The M4 Pro Mini, with far more memory, earned the role of brain, running the language models on its own silicon. The desktop, with its GPU, wakes only when there is heavy rendering to do.

The second decision mattered more: a strict rule that no stage talks to another directly. Everything moves through a database and a message broker. It felt like extra ceremony at the time. It is also the single reason the system shrugs off a machine going offline mid-job instead of falling over.

The first time a node died mid-run and the job simply carried on, the ceremony paid for itself.

How it works

Eight stages, spread across the machines on a private Tailscale mesh. The M4 Mini runs PostgreSQL, the broker, and the job workers that hand work out, alongside the APIs and dashboards. The M4 Pro Mini runs the language models through Ollama. The desktop wakes on demand for GPU-accelerated rendering, and the Pi 5 feeds in readings from its sensors at the edge. Because state lives in the database and the queue rather than in any one process, every step is independently retryable, and the pipeline keeps moving even if a node drops.

What I learned

• Splitting the work by what each machine is good at beat buying one big box, on cost and on resilience.
• Message-passing over direct calls is what turned four separate machines into one system.
• Local models are capable for real workloads, with no per-token cost and no data leaving home.
• This lab is now the standing proof behind the on-premise AI work we sell: the same pattern, hardened.