Task guide · open weights
Best local open-weight AI: decide what local actually means
Open weights, self-hosting, private inference, and a local workstation are different claims. This guide separates them before you buy hardware.
What actually matters
- Check the licence, weights availability, quantisation, serving stack, and hardware footprint together.
- A model being open-weight does not mean it fits a laptop or permits every commercial use.
- Compare total cost of ownership: GPUs, power, engineering, monitoring, and upgrade cycles.
- Use a hosted API first when the model is datacentre-scale and your reason for going local is untested.
- Coding agents need long context and tool-call reliability — raise Ollama num_ctx (often 16–32k+) and expect small 7B scaffolds to fail loops that frontier models hide.
- Match the API “door”: Claude Code wants Anthropic Messages–compatible endpoints; OpenCode-class tools want OpenAI /v1.
- Start local agents scaffold-first: map one repository, make one documentation or test edit, and inspect tool calls before attempting a multi-file build.
- Treat VRAM as a floor for the chosen quantisation and context, not a marketing maximum; use the /gpus hub to compare usable memory before buying.
The shortlist
- DeepSeek V4-Flash for a low-cost open option with a hosted API and MIT-licensed weights.
- Kimi K3 when long context and open weights matter more than a small hardware footprint.
- Small, specialised releases such as LFM2.5 when on-device latency is the actual goal.
- Aider or Continue for true offline assistive edits; treat full agent loops on local models as scaffold-first experiments, not drop-in GPT replacements.
Evidence-backed entries
A sensible test workflow
- 01Write down the privacy and deployment requirement in one sentence.
- 02Estimate memory, throughput, quantisation quality, and serving complexity before downloading weights.
- 03Run a representative prompt set locally and against the hosted alternative.
- 04Before blaming the model, verify context length, endpoint compatibility, and whether telemetry retries are hammering a local server.
- 05Repeat the scaffold-first task at the target context length and concurrency before trusting a local agent with broad edits.
Common mistakes
- Calling an API-only preview open source because weights were promised.
- Ignoring the licence or model-specific commercial conditions.
- Buying hardware before measuring workload throughput and concurrency.
- Leaving Ollama at a ~4k default context and concluding local agents are useless.
- Pointing Claude Code at an OpenAI /v1 proxy (or OpenCode at an Anthropic-only door) and chasing auth ghosts.
Popular-AI shortlist snapshot: August 6, 2026. This guide is a starting framework, not a permanent ranking. Model prices, access, policies, and capabilities move quickly. Check the provider before committing spend or sending sensitive data.