How to Run an LLM Locally (and When It Makes Business Sense)
You can run a useful language model locally today by picking a small open-weight model, serving it with llama.cpp, MLX, vLLM, Ollama or LM Studio, and testing it on your own task first. It makes business sense when privacy, offline use or per-token cost dominate, and much less sense when you need frontier reasoning.
Key takeaways
- Local inference means the model runs on hardware you or your user controls, not on a hosted API.
- The case is privacy, zero per-token cost and offline operation; the cost is capability.
- Open-weight models trail the frontier by 12 points on Artificial Analysis Index v5.
- A 2B text model can fit in about 1 GB at 4-bit and still handle classification, extraction and routing.
- Benchmark the candidate model on your own data and read the licence before you ship.
You can run a usable language model locally today. The work is to pick a small open-weight model, serve it with a runtime such as llama.cpp, MLX, vLLM, Ollama or LM Studio, and test it on your own task before you commit. Local inference makes business sense when privacy, offline use or per-token cost dominate, and much less sense when your product depends on frontier reasoning.
TL;DR
- Local inference means the model runs on hardware you or your user controls.
- The upside is privacy, no per-token bill and no network dependency.
- The downside is capability: open-weight models are 12 points behind on Artificial Analysis Index v5.
- A 2B text model can fit in about 1 GB at 4-bit with a 256K context window.
- Pick the task, benchmark on your data, check the licence, then ship with a fallback.
Disclosure: ZeroRaised publishes this analysis and is a launch site for bootstrapped products. Facts come from the linked sources as checked on 29 Oct 2026; ZeroRaised has no commercial relationship with the companies named.
Local inference is running a language model on hardware that you control, or on the user's device, instead of sending every request to a hosted API.
Why run an LLM locally?#
Three reasons hold up.
Privacy. Data can stay on the user's device. Opera's WebAI, for example, lets websites and extensions use AI running locally in the browser, and the company says page data can stay on the user's device (Opera). If you handle medical notes, legal documents or customer records, that is a simpler story to tell a buyer than a data processing agreement.
Zero per-token cost. Local inference has no meter. Liquid AI says of its models: "Since tokens are free, dozens of agents can run continuously in the background" (Liquid AI). Opera makes the same point from the browser side: "No per-request cloud costs, and no AI infrastructure to run" (Opera).
Offline and always available. A local model does not wait on a network round trip. For an agent watching a folder, log file or queue, that is a real architectural difference.
What is the honest trade-off?#
Capability. Artificial Analysis introduced Intelligence Index v5 on 27 Oct 2026 and reported that the open-weights gap widened from 3 points to 12 (Artificial Analysis).
The numbers are blunt. Claude Opus 5.5 at max effort led at launch with 58, while Kimi K3.1 Pro, the top open-weights model, scored 46, and its weights were not yet released (Artificial Analysis). Among models with released weights, DeepSeek V4.1 Flash 1017 and GLM-5.3 led at 32, below Claude Haiku 5.5 at 36 (Artificial Analysis).
If your product sells general reasoning, local is the wrong tool this month; if it sells a workflow around a narrow task, the gap matters less. The open-weight versus closed model comparison covers how to think about that split.
Which tools should you use?#
Liquid lists native support for llama.cpp, MLX and vLLM, and thanks Ollama, LM Studio, Hugging Face and Unsloth in its builder ecosystem (Liquid AI). In practice:
- Ollama or LM Studio for a first trial on a laptop.
- llama.cpp when you want to control quantisation and run on modest hardware.
- MLX on Apple silicon, where it is built for the platform.
- vLLM when you are serving a GPU box and need throughput.
WebAI has limits worth knowing. It is available on Windows PC with 64-bit Intel or AMD chips and on Apple silicon Macs, but not on Intel Macs, Linux or Windows on Arm, and speed depends on the user's device (Opera).
How small can a useful model be?#
Smaller than most founders assume. The LFM3 text model fits in about 1 GB at 4-bit precision and has a 256K context window, and every model in the family can be fine-tuned on a single GPU (Liquid AI).
A fine-tuned LFM3-VL-2B reached 67% accuracy spotting wind turbine blade defects after about 3,000 public photos and a few hours on one A100, where the base model and GPT-6 Luna found defects in fewer than 10% of images out of the box (Liquid AI). Separately, MacPaw reported 64% task success for LFM3-1.7B against 48% for Qwen3.5-2B across 65 everyday macOS agent tasks, with 58% fewer tokens per task and fewer stuck attempts (Liquid AI).
Those are vendor and partner numbers, not your result. The LFM3 licence guide covers the commercial terms.
How do you check the licence?#
The LFM Open License v1.0 is based on Apache 2.0, allows free commercial use below $10M annual revenue, has no copyleft and requires attribution (Liquid AI). Above the threshold you must buy a commercial licence, and the licence terminates automatically on non-compliance (Liquid AI).
Three questions to ask of any model you plan to ship: what happens above a revenue line, must you publish modifications, and what attribution do you have to keep? The licence shapes your margin as much as the hardware does.
How do you run an LLM locally, step by step?#
- Pick one task. Classification, extraction, routing or a short decision. Write down the pass mark before you test anything.
- Collect real examples. A few hundred to a few thousand items from your own product, with the correct answers.
- Choose the smallest model that could work. Start in the 2B range and move up only when the data says you must.
- Run it with the right runtime. Ollama or LM Studio to try it, llama.cpp or MLX to tune it, vLLM to serve it.
- Benchmark on your data, not a leaderboard. Compare the local model with the hosted model you use now, on accuracy, latency and cost per finished task. The output evaluation guide explains how to set that up.
- Check the licence and ship with a fallback. Route the hard cases to a hosted model and keep the local path for the volume.
When does local AI make business sense?#
It makes sense when the task is narrow, the volume is high and the cost of a wrong answer is low or reviewable. It makes less sense when the task is open-ended, when quality is the product, or when your users run old hardware.
Compare honestly. Artificial Analysis found that cost per task varies by more than 13x among the ten highest-scoring models, from $0.96 for GPT-6.1 Terra at max to $13.04 for Claude Fable 5.1 at max (Artificial Analysis). At small volume, a cheap hosted model may beat the engineering time of a local stack. The LLM cost guide has the trade-offs.
Where ZeroRaised fits#
ZeroRaised is a launch and discovery site for products built without institutional venture capital or accelerator equity. If you have shipped something with a local model, or you are about to, submit your product. The pricing page explains the current options, and nothing here promises traffic or results.
If you want more practical analysis, join the ZeroRaised newsletter. It is a plain email.
Sources#
Frequently asked questions
Can I run an LLM locally without a GPU?
Often, yes, but slowly. Small models run on CPUs, and Liquid says every LFM3 model can be fine-tuned on a single GPU, which is a different job from inference. Measure speed on the hardware your users actually have before promising anything.
Which local inference tools should I start with?
Liquid names llama.cpp, MLX and vLLM as natively supported runtimes, and credits Ollama and LM Studio in its builder ecosystem. Use Ollama or LM Studio for a quick trial, llama.cpp or MLX on a laptop, and vLLM when you are serving a GPU.
Is a local model as good as a frontier API model?
No, not at the top end. Artificial Analysis reported that the open-weights gap widened from 3 points to 12 on its Intelligence Index v5. For narrow tasks the gap matters less, which is why you benchmark on your own data.
Do I have to pay to use an open-weight model commercially?
It depends on the licence. The LFM Open License v1.0 is free for commercial use below $10M annual revenue, with no copyleft, but requires a commercial licence above that line. Check the terms for the model you choose.
Can local AI run inside a browser?
Yes, in limited form. Opera's WebAI preview lets websites and extensions call local models through Prompt, Decision and Semantic Embeddings APIs, with no API key and no local server. It is opt-in and available on Windows 64-bit and Apple silicon Macs.