Buyer guide · agentic harnesses
Local LLM coding setup: Ollama, LM Studio, llama.cpp, and MLX
Pick a local inference stack and wire it to an agent harness without the usual traps — wrong ports, misconfigured context, untested quantisation, and mismatched API prefixes that make agents look broken when the model is fine.
Who this is for: Developers wiring local inference (Ollama, LM Studio, llama.cpp, MLX) to coding agents like Cline, Continue, Aider, or OpenCode — especially on Apple Silicon or a single-GPU workstation with privacy or burst-cap requirements.
What actually matters
- Default ports differ by runtime: Ollama listens on 11434, LM Studio on 1234, llama.cpp server commonly on 8080 — point each harness at the correct host:port.
- For agent workflows, current Ollama guidance recommends at least 64K context when hardware permits; effective context depends on runtime settings and the model/harness combination.
- For Aider with Ollama, use the ollama_chat/model prefix — not bare ollama/ — per official Aider docs.
- Quantisation can change structured-output and tool-call reliability; there is no universal Q4_K_M threshold that guarantees correct tool calls.
- OpenAI-compatible clients need a /v1 base URL (e.g. http://localhost:11434/v1 for Ollama), not a bare host root.
- MLX on Apple Silicon is a strong path for local coding models when CUDA hardware is unavailable.
- Keep a cloud escalation path for ambiguous refactors and long-horizon tasks local models stall on.
ACE recommendation
Start with Ollama or MLX on a practice repo, verify effective context and structured tool calls for the chosen model/quant/runtime, match each harness’s API prefix, and keep a cloud escalation path for tasks local models stall on.
Useful comparisons
How to run this decision
- 01Install one runtime (Ollama, LM Studio, llama.cpp, or MLX) and choose a coding model/quantisation that fits the target hardware; then verify structured output and tool-call behavior on that exact combination.
- 02For an Ollama coding agent, start from its current 64K guidance where memory permits. Configure OLLAMA_CONTEXT_LENGTH or num_ctx and verify the allocated value with `ollama ps` before the first task.
- 03Verify the endpoint with curl against the /v1/models or chat completion path your harness expects.
- 04Configure the harness: Cline/Kilo base URL with /v1 suffix; Aider with ollama_chat/model; OpenCode with OpenAI-compatible provider settings.
- 05Run a small practice-repo task (add test, fix lint) and confirm the agent sees full file context in the prompt.
- 06Document when to escalate to cloud (large refactors, unfamiliar frameworks) vs stay local (secrets, offline, burst cap).
Common pitfalls
- Pointing OpenCode or Cline at http://localhost:11434 without the /v1 prefix — requests fail with opaque errors.
- Using bare ollama/model in Aider instead of ollama_chat/model — chat formatting breaks.
- Assuming a quantisation grade alone predicts tool-call reliability; test the exact model, quant, runtime, and harness.
- Assuming a nominal context setting is enough — check the context Ollama allocated and whether the harness truncated the prompt.
- Buying GPU hardware before checking whether MLX on an existing Mac covers your model size.
Ask before you buy
- Which runtime fits your hardware: Ollama (11434), LM Studio (1234), llama.cpp server (~8080), or MLX on Apple Silicon?
- Do you need editor or terminal agent workflows (Cline, Kilo, Continue), or Aider/OpenCode's terminal workflow? Check the exact client's supported capabilities.
- For Ollama, does the effective context meet the agent's needs? Current Ollama documentation recommends at least 64K for coding/agent tasks where memory allows; verify allocation with `ollama ps` and check harness truncation.
- For Aider with Ollama, are you using ollama_chat/model — not bare ollama/ — per official docs?
- Have structured output and tool calls been tested on this exact model, quantisation, runtime, and harness? No quantisation grade alone guarantees them.
- Does OpenCode or another client point at an OpenAI-compatible /v1 base URL, not a bare host?
- When does local win (privacy, offline, burst cap) vs when cloud wins (ambiguous refactors, long horizon)?
- Does your GPU or Mac fit the model size you pulled? See GPU inference and local GPU sizing before buying hardware.
More buyer guides
Related decisions
Other harness buyer guides that answer the next question after this one.
Buyer guide
Editor vs terminal: which coding assistant workflow fits?
Choose the surface your team will actually use before comparing model quality.
Buyer guide
How to roll out an agentic harness without losing review discipline
A staged rollout pattern for teams that want speed without turning generated patches into unowned production risk.
Buyer guide
Agentic harness pricing: seats, tokens, and overage
Compare subscriptions and usage-based plans by the work they enable, not the number printed on the landing page.
Buyer guide
Coding-assistant myths in 2026: what the product pages blur
A myth-versus-reality check for coding assistants whose pricing, editor surfaces, privacy paths, and autonomy are easy to collapse into one misleading label.
Buyer guide
Enterprise AI coding assistant procurement checklist
A compact procurement block for security, legal, IT, and finance teams evaluating coding assistants beyond the demo and the seat price.
Buyer guide
Migrating from Roo Code to Kilo Code
Roo Code’s extension shut down on May 15, 2026. Kilo Code is one alternative with its own IDE/CLI surfaces; this guide covers installation, credential setup, project rules, and platform differences without assuming settings auto-port.