Buyer guide · agentic harnesses
Local LLM coding setup: Ollama, LM Studio, llama.cpp, and MLX
Pick a local inference stack and wire it to an agent harness without the usual traps — wrong ports, tiny default context, bad quants, and mismatched API prefixes that make agents look broken when the model is fine.
Who this is for: Developers wiring local inference (Ollama, LM Studio, llama.cpp, MLX) to coding agents like Cline, Continue, Aider, or OpenCode — especially on Apple Silicon or a single-GPU workstation with privacy or burst-cap requirements.
What actually matters
- Default ports differ by runtime: Ollama listens on 11434, LM Studio on 1234, llama.cpp server commonly on 8080 — point each harness at the correct host:port.
- Agent workflows need 16k–32k context; Ollama’s small default can silently drop files unless you set OLLAMA_CONTEXT_LENGTH or Modelfile num_ctx.
- For Aider with Ollama, use the ollama_chat/model prefix — not bare ollama/ — per official Aider docs.
- Tool-calling quality degrades below Q4_K_M quantisation; lower quants often break JSON and edit schemas.
- OpenAI-compatible clients need a /v1 base URL (e.g. http://localhost:11434/v1 for Ollama), not a bare host root.
- MLX on Apple Silicon is a strong path for local coding models when CUDA hardware is unavailable.
- Keep a cloud escalation path for ambiguous refactors and long-horizon tasks local models stall on.
ACE recommendation
Start with Ollama or MLX on a practice repo, set context before the first agent task, stay at Q4_K_M or better, match each harness’s API prefix, and keep a cloud escalation path for tasks local models stall on.
Useful comparisons
How to run this decision
- 01Install one runtime (Ollama, LM Studio, llama.cpp, or MLX) and pull a coding-oriented model at Q4_K_M or better.
- 02Set context before the first agent task: export OLLAMA_CONTEXT_LENGTH=32768 or equivalent in your Modelfile/harness override.
- 03Verify the endpoint with curl against the /v1/models or chat completion path your harness expects.
- 04Configure the harness: Cline/Kilo base URL with /v1 suffix; Aider with ollama_chat/model; OpenCode with OpenAI-compatible provider settings.
- 05Run a small practice-repo task (add test, fix lint) and confirm the agent sees full file context in the prompt.
- 06Document when to escalate to cloud (large refactors, unfamiliar frameworks) vs stay local (secrets, offline, burst cap).
Common pitfalls
- Pointing OpenCode or Cline at http://localhost:11434 without the /v1 prefix — requests fail with opaque errors.
- Using bare ollama/model in Aider instead of ollama_chat/model — chat formatting breaks.
- Running agents on Q2/Q3 quants and blaming the harness when tool calls return malformed JSON.
- Assuming 4k default context is enough — multi-file agent prompts truncate silently.
- Buying GPU hardware before checking whether MLX on an existing Mac covers your model size.
Ask before you buy
- Which runtime fits your hardware: Ollama (11434), LM Studio (1234), llama.cpp server (~8080), or MLX on Apple Silicon?
- Do you need an editor agent loop (Cline/Kilo), assistive chat (Continue), or terminal git workflow (Aider/OpenCode)?
- Is context long enough for agents (16k–32k via OLLAMA_CONTEXT_LENGTH, Modelfile num_ctx, or harness override)? Ollama’s small default can silently discard files.
- For Aider with Ollama, are you using ollama_chat/model — not bare ollama/ — per official docs?
- Is the quant at least Q4_K_M for tool-calling? Lower quants often break JSON and edit schemas.
- Does OpenCode or another client point at an OpenAI-compatible /v1 base URL, not a bare host?
- When does local win (privacy, offline, burst cap) vs when cloud wins (ambiguous refactors, long horizon)?
- Does your GPU or Mac fit the model size you pulled? See GPU inference and local GPU sizing before buying hardware.
More buyer guides
Related decisions
Other harness buyer guides that answer the next question after this one.
Buyer guide
Editor vs terminal: which coding assistant workflow fits?
Choose the surface your team will actually use before comparing model quality.
Buyer guide
How to roll out an agentic harness without losing review discipline
A staged rollout pattern for teams that want speed without turning generated patches into unowned production risk.
Buyer guide
Agentic harness pricing: seats, tokens, and overage
Compare subscriptions and usage-based plans by the work they enable, not the number printed on the landing page.
Buyer guide
Coding-assistant myths in 2026: what the product pages blur
A myth-versus-reality check for coding assistants whose pricing, editor surfaces, privacy paths, and autonomy are easy to collapse into one misleading label.
Buyer guide
Enterprise AI coding assistant procurement checklist
A compact procurement block for security, legal, IT, and finance teams evaluating coding assistants beyond the demo and the seat price.
Buyer guide
Migrating from Roo Code to Kilo Code
Roo Code’s discontinuation pushed many teams toward Kilo’s OpenCode-lineage stack. This guide covers Marketplace install, credential migration, project rules, and the VS Code vs JetBrains paths without assuming settings auto-port.