Why we run AI on our own machines

The default assumption in 2026 is that “adding AI” means sending your data to someone else’s servers and paying by the token. For a lot of products that is a reasonable trade. For a lot of others it quietly becomes a liability, and nobody notices until it does.

We build AI that runs on hardware you control. Not out of nostalgia, and not because it is always the right call. We do it because for a specific and growing set of problems, it is simply the better engineering decision. Here is the actual reasoning.

What on-premise AI means now

Five years ago, running a capable model yourself meant a research budget. That is no longer true. Open-weight models like Llama, Gemma, and their fine-tunes are good enough for most production tasks that are not frontier reasoning. Runtimes like Ollama and llama.cpp make serving them a matter of one command. Quantization (running a model at 4-bit instead of 16-bit) cuts the memory footprint by roughly four times with a small quality cost, which means a 27-billion-parameter model fits comfortably on a single prosumer GPU or a Mac with enough unified memory.

Concretely: a mid-range machine can run a 7 to 13 billion parameter model for classification, extraction, and drafting at interactive speed, and a well-specced one can run a 27B-plus model for genuinely hard work. That is not a lab setup. That is a desk.

What three problems does it actually solve?

Your data stays in the building. The moment a customer record, a contract, or a medical note leaves your network for an external API, you have created a compliance surface and a trust question you now have to answer forever. On-premise inference removes the question entirely. Nothing leaves. For regulated work, or for clients who simply do not want their data used to train someone else’s next model, this is not a feature. It is the requirement.

Costs stop being a variable you cannot control. Per-token pricing is fine until volume grows, and then it is a tax that scales with your success. Local inference has a fixed, knowable cost: the hardware and the electricity. We run a four-machine lab whose active nodes idle in the single-to-low-double-digit watts. Past a certain usage, owning the compute is cheaper, and the cost curve is flat instead of climbing.

You stop depending on a vendor’s roadmap. APIs deprecate models, change pricing, rate limit you at the worst moment, and occasionally go down. A model running on your hardware does none of that. It behaves the same today as it did six months ago, because it is the same weights on the same silicon. That reproducibility is worth a great deal when you are building something you intend to keep.

What are the honest trade-offs?

On-premise is not free, and pretending otherwise would be exactly the content-mill nonsense we try to avoid.

Frontier hosted models are still ahead on the hardest reasoning, the longest context, and the most obscure knowledge. If your product lives or dies on that top slice of capability, an API is the right tool, and you should use it.

You also take on the hardware. Something has to run, stay patched, and be monitored. That is real operational work, though far less than people fear once it is set up properly.

And smaller local models reward good engineering. A 7B model with clean retrieval, tight prompts, and a narrow task will outperform a giant model pointed at a vague one. The work moves from “pay for a bigger model” to “design the system well,” which happens to be the work we prefer anyway.

When is it the right call?

A simple test. On-premise AI tends to win when at least one of these is true:

• The data is sensitive, regulated, or contractually cannot leave your environment.
• The volume is high enough that per-token cost is a real line item.
• You need the behaviour to be stable and reproducible over time.
• The task is well-defined (extraction, classification, search, translation, drafting, routing) rather than open-ended frontier reasoning.

If none of those hold, use an API and move on. If one or more do, running it yourself is usually the calmer, cheaper, more defensible path, and it is the one we know how to build.

What it takes in practice

The setup is less exotic than it sounds: a GPU box or an Apple Silicon machine sized to the model, a runtime like Ollama, a small queue so requests do not collide, and models quantized to fit the hardware with headroom. Add retrieval over your own documents when the task needs knowledge, and a thin API in front so the rest of your stack does not care where the model lives.

That is the whole shape of it. The hard part is not the AI. The hard part is the same as it always is: building the system around it well, so it is fast, reliable, and something your team can own.

Sovereignty should be the default, not a paid upgrade. That is the bet we are making, and so far the hardware keeps agreeing with us.

If you are weighing this call for a workflow of your own, that is what the Sovereign AI Pilot is for: two to three weeks, one real workflow, and a clear answer, including “do not do this” if that is the honest one.

Frequently asked questions

What does on-premise AI actually mean today?

Running capable open-weight models like Llama or Gemma yourself, on hardware you control, served with runtimes such as Ollama or llama.cpp and quantized to fit a single prosumer GPU or a Mac with enough unified memory. A mid-range machine handles classification, extraction and drafting at interactive speed.

When is running AI on your own machines the right call?

When at least one of these holds: the data is sensitive, regulated or contractually cannot leave your environment; the volume makes per-token cost a real line item; you need behaviour that stays stable and reproducible over time; or the task is well defined rather than open-ended frontier reasoning.

What are the trade-offs versus a hosted API?

Frontier hosted models are still ahead on the hardest reasoning, the longest context and the most obscure knowledge. You also take on the hardware, which has to run, stay patched and be monitored. And smaller local models reward good engineering rather than simply paying for a bigger model.

What hardware do you need to run a model locally?

A GPU box or an Apple Silicon machine sized to the model, a runtime like Ollama, a small queue so requests do not collide, and models quantized to fit the hardware with headroom. Add retrieval over your own documents when the task needs knowledge, and a thin API in front so the rest of your stack does not care where the model lives.