Overview
The best GPU for local AI is usually the one with enough usable memory for the models you will actually run, not the card that wins a synthetic raster leaderboard. This guide separates capacity-first builds from gaming-first cards before you overspend.
Start with whether the model fits
Local inference fails as memory planning first. At Q4, a real Q4_K_M file costs about 0.6GB per billion parameters (roughly 4.85 bits per weight, not a flat 4), plus headroom for context and the OS. A 70B model is a ~42.5GB file and needs about 48GB with a modest context — which is why a 24GB card is a mid-size workhorse, not a 70B free pass.
Quantization and context length change the answer more than brand loyalty. A 7B–8B chat model at Q4 fits an 8GB card, and a 13B–14B model wants 12–16GB once a long prompt is loaded. A 32B–34B coding model wants 20GB+ before you start fighting swap. Multi-modal or long-context workloads eat VRAM that parameter-count tables never show. Write down the largest model you actually intend to keep resident, then size the card to that — not to a leaderboard you will never run.
Capacity tiers that match real workloads
Treat these as planning buckets, not product rankings:
- 12–16GB — small and mid chat models, image tools that stay under the limit, light coding assistants. Fine for experimenting; painful if every useful model spills to CPU RAM.
- 24GB — the practical sweet spot for many builders: mid-size open weights, batch jobs, and a second model occasionally in memory. Still the wrong expectation for comfortable single-GPU 70B.
- 32GB — the RTX 5090 and the new 32GB prosumer cards (Intel Arc Pro B70, AMD Radeon AI PRO R9700): room for current 27B–32B models with long context, still not a 70B host.
- 48GB+ discrete or 64GB+ unified memory — where 70B-class and long-context work stop being a science project. A 32GB card is short of the ~42.5GB Q4 weights alone; 48GB discrete or 64GB unified is a tight fit, and 72GB+ discrete or 96GB+ unified is comfortable. Apple Silicon or a Strix Halo / DGX Spark-class unified-memory box is often cheaper capacity than chasing multi-GPU desktops.
A 24GB or 32GB card can beat a faster 16GB gaming SKU for local AI specifically — but check used prices first: in September 2026 a used RTX 4090 ($2,499) cost more than the RTX 5090's $1,999 launch MSRP, and a used RTX 3090 ($1,000–1,350) about as much as a new 32GB Arc Pro B70 ($949). Do not buy a 16GB “faster” raster champion if your workload is “does the model fit.”
Then pick the ecosystem you can live with
CUDA remains the path most guides, Docker images, and tutorials assume. NVIDIA wins on “will this random repo run on day one,” which matters if you are not already fluent in ROCm debugging.
AMD and Intel often win on price per gigabyte — the 32GB Arc Pro B70 ($949) and Radeon AI PRO R9700 ($1,299 MSRP) undercut NVIDIA's 32GB card — with thinner day-one tooling. That trade is acceptable when you already know the stack you will run and have verified ROCm or Vulkan paths for it. It is a bad surprise when your first three GitHub READMEs only list CUDA.
Apple unified memory trades discrete gaming leadership for large-model capacity inside a Mac. Buy that path when the Mac form factor and memory size are the product; do not buy it as a substitute for a Windows gaming PC that also “does some AI.”
Power, case, and cooling are not footnotes
Local AI cards that sit at high utilization for hours stress the PSU and thermals harder than a gaming session that spikes and recovers. Check connector count, case clearance, and whether the card will throttle under sustained load in your chassis. A card that fits the VRAM math but browns out the power supply is not a bargain.
Laptop GPUs with the same marketing name usually have less usable memory bandwidth and worse sustained clocks than the desktop twin. If the machine must be portable, size for laptop reality — not desktop charts.
What to judge before spending
- What is the largest model (parameters + quantization + context) you will keep loaded day to day?
- Is day-one CUDA compatibility worth more than raw GB per pound?
- Will this machine also game, or is it an inference box?
- New vs used: is software support still active for the generation you are buying?
- Desktop discrete, workstation, or Mac unified memory — which form factor you will actually keep on the desk?
Run the finder, then verify on the tables
Use the GPU finder for a shortlist keyed to your constraints, then confirm fit on the GPU memory tables and Mac capacity notes before you buy. Street prices move weekly; treat launch MSRP as an anchor only. If two cards both clear the VRAM bar, prefer the one whose software path you already use — capacity that sits idle behind broken drivers is still a failed purchase.