What Memory Bandwidth Actually Buys You: LLM Inference Hardware in 2026

PrefillDecodeRoofline, HardwareTierMatrix, MemoryBudgetBreakdown, SparkTopologyDiagram, InferenceTcoComparison, HardwareNativeStack, } from "@/components/blog/local-llm-hardware";

Cloud APIs are the correct way to start. They stay the correct way for bursty work, frontier-only quality floors, and anything you have not decided to own yet. They stop being cheap the moment volume is steady, data cannot leave the boundary, or you are paying for idle capacity you already decided you need every day.

Buying silicon does not automatically fix that. The common failure mode in 2026 is confusing fits with feels fast. A 70B model that loads into 128GB unified memory and crawls at a few tokens per second is not a serving win. It is a capacity demo.

This post is about inference — chat, agents, RAG, batch eval. Training gets one contrast, then we move on. The math below is what I use when someone asks whether to buy a Mac, a Spark, a PRO 6000 workstation, or a Lenovo 8-GPU node.

Scope: inference, not training

Training is compute-bound and fabric-hungry. Gradients sync across the cluster every step. That is why InfiniBand fat-trees and NVLink domains show up in training BOMs.

Inference is different. Weights are mostly static. Prefill is matrix–matrix work (FLOPS). Decode at low batch is memory-bandwidth work (bytes per token). If the model fits in one chassis, you do not need a training interconnect to answer chat. You need enough fast memory, enough bandwidth, and a runtime that does not waste either.

If you are sizing InfiniBand for a decode problem, stop. Re-read the next two sections first.

Prefill vs decode

Prefill turns the prompt into KV cache. It is usually compute-bound. Time-to-first-token tracks FLOPS, kernel quality, and how long the prompt is.

Decode emits tokens one by one. At batch size one, the accelerator streams roughly the active weight footprint from memory every token. That is why a card with fewer FLOPS but much higher memory bandwidth can feel faster for interactive chat than a FLOPS monster sitting on slow DRAM.

A useful first-order decode ceiling (batch ≈ 1, dense, short context):

tok/s ≈ η × (memory_bandwidth / bytes_streamed_per_token)

η is efficiency — call it 0.5–0.8 in real runtimes. MoE helps here: you must reside total parameters, but you only stream active experts per token. A 671B MoE with 37B active streams ~22 GB per token at Q4; a dense 70B streams ~42 GB. The model ten times larger on disk can decode faster, as long as every expert stays in fast memory.

Those decode numbers are planning estimates, not lab benches. Measure your stack.

The math behind VRAM sizing

Compute is rarely the first wall. Memory capacity is. You need room for weights, KV cache, activations, and the framework’s bad habits.

Weights

W ≈ P × bytes_per_param × 1.2

P is parameter count. bytes_per_param is roughly 0.5 for Q4, 1.0 for Q8, 2.0 for FP16. The 1.2 factor is a planning buffer for runtime overhead — not a law of physics.

For MoE, P for capacity is total parameters. Decode streams closer to active parameters. Do not size CapEx on the marketing “active only” figure and then wonder why the model will not load.

KV cache (GQA-style)

KV ≈ batch × seq_len × 2 × layers × kv_heads × head_dim × bytes_per_elem

Concurrent users (batch) times context (seq_len) is what ambushes serving boxes. Run the numbers on a 70B GQA shape (80 layers, 8 KV heads, 128 head dim): ~320 KB per token at FP16. Eight users at 32K context is ~84 GB of KV — twice the Q4 weights.

KV cache (MLA — DeepSeek / Kimi lineage)

Multi-head Latent Attention caches a low-rank latent plus a small RoPE slice instead of full multi-head K/V. For a DeepSeek-V3-class model that works out to roughly 70 KB per token — versus ~320 KB for the 70B GQA shape above, on a model nearly ten times larger. On MLA architectures, long context is cheap relative to weight residency. Verify which attention mechanism you are buying hardware for before you optimize for the wrong cliff.

Ladder A — Consumer / desk / lab

These are the machines people actually buy for local work in mid-2026. CapEx figures are planning anchors.

Apple note. The M3 Ultra 512GB Studio that anchored earlier guides is no longer orderable — the DRAM shortage reshuffled Apple's lineup, and as of mid-2026 the M5 Max 128GB MacBook Pro is the highest-memory Mac you can buy. Apple's published figure is 614 GB/s of unified memory bandwidth — but only on the 40-core-GPU M5 Max; the binned 32-core part runs 460 GB/s, so configure carefully. The 128GB/2TB inference config lands around $5.2k; maxing storage and display pushes it to $7,349. The per-GPU-core Neural Accelerators mainly help prefill, and decode still rides that bus. A Mac Studio M5 refresh is rumored for later this year — treat it as a rumor until it ships.

Spark note. Spark is NVIDIA’s GB10 personal AI supercomputer (ex–Project DIGITS): 128GB coherent unified memory at 273 GB/s, with a ConnectX-7 NIC onboard. Marketing petaFLOPS do not repeal 273 GB/s. Use it for the CUDA/vLLM/TRT-LLM path and models that fit the bandwidth envelope. Cluster it for capacity, not because eight boxes automatically mean eight times the chat speed.

Spark networking: cables are the product

Two Sparks want one certified QSFP DAC on the same physical port on each box. A second cable does not magically double throughput — the CX-7 is fed by aggregated PCIe Gen5 x4 paths with a real ceiling near 200 Gbps aggregate.

For three-plus nodes you need a switch. NVIDIA's Sync Cluster Assistant covers supported 2–4 node layouts; eight-node fabrics are community/manual territory. Practical checklist:

• QSFP112 DAC, Ethernet-mode parts (NVIDIA's docs cite Amphenol NJAAKK-N911 and Luxshare LMTQF022-SD-R)
• MTU 9000 on the fabric
• NCCL_IB_MERGE_NICS=1 and correct HCA lists so both logical halves of the port participate
• Management traffic on 10GbE, not the CX-7
• Expect the “link shows 200G but iperf is ~13 Gbps” failure mode when firmware, power state, or cable class is wrong (widely reported in the Spark community; symptoms vary by firmware revision)

Federate models that fit on one node. Only shard when residency forces it. Idle CX-7 ports are not free tokens; they are unused CapEx.

Ladder B — Datacenter / production

When concurrency, SLA, and frontier open-weight residency matter, you leave the desk. I use Lenovo ThinkSystem as the OEM anchor so CapEx rows are comparable (LP2368 TCO guide). Dell and Supermicro ship peer SKUs — especially on AMD.

Planning CapEx anchors from Lenovo’s 2026 TCO note (not quotes): ~$400k for 8× H200, ~$550k for 8× B200, ~$785k for 8× B300. An 8× RTX PRO 6000 Blackwell node runs roughly $135k–$200k depending on chassis and integrator — a third or less of the SXM nodes. That gap is the whole point of the value tier: same 768GB pool, slower memory, PCIe instead of NVLink.

PCIe vs NVLink. Eight PRO 6000s pool memory over PCIe. That is fine for many steady inference loads. Heavy tensor parallel under tight latency SLAs prefers an NVLink domain (H200/B200 SXM). Same VRAM number, different sync tax.

AMD. MI300X/MI350X compete on memory pool and $/GB. The honest tax is CUDA ecosystem gravity — ROCm + vLLM/SGLang is real in 2026, but your team’s muscle memory still matters. Pick AMD when the memory math and procurement story win, not as a protest vote.

Hardware-native optimization

Buying the card and running a stock Hugging Face generate loop can leave half or more of the silicon idle. The runtime has to match the hardware.

Practical stack map:

Myths worth killing: more nodes ≠ more tok/s; Q8 is not always better on bandwidth-bound boxes; Apple Neural Accelerators are not the Neural Engine; hybrid offload is not “having a 70B GPU.”

SOTA APIs vs rented GPUs vs on-prem

Three options. Same two volume anchors so the crossover is visible.

• Team / lab: ~20M input + 5M output tokens/day
• Sustained production: ~200M input + 50M output tokens/day

Option A — SOTA cloud APIs

At the sustained 250M tok/day mix, frontier APIs land near ~$640k/year at list rates and scale linearly. Two honest qualifiers before anyone quotes that number back at me. First, almost nobody at that volume pays list: prompt caching (agent and RAG traffic is heavily repetitive), batch APIs, and committed-use discounts routinely take 30–70% off. Second, the API is buying you a frontier model; every owned row below serves open weights. If your use case genuinely needs frontier quality, the comparison ends here.

Option B — Rented GPUs

You own the serving stack (vLLM, SGLang, TRT-LLM). The hyperscaler owns the silicon. Capacity Blocks / reservations change the annual number a lot — DiscreteStack’s reserved B200 math lands far below on-demand. Refresh rates the week you buy; this market moves.

Option C — Own silicon

What the sticker price leaves out. Anyone who has run a P&L on this will push back on bare CapEx rows, and they should. Fully loaded, an owned datacenter node also carries:

• Power and cooling. An 8× H200 node draws ~10kW; 8× B200 closer to 14kW. At $0.10–0.15/kWh running 24/7 that is $9–18k/year in electricity alone, before cooling overhead (PUE) if it is your facility.
• Space. Colo runs roughly $150–250/kW/month all-in, so a 10kW node is ~$18–30k/year of rack space — often more than the electricity.
• Support and spares. OEM support contracts, warranty uplift, a spare NIC and drives. Budget low single-digit percent of CapEx per year.
• People. Someone owns the vLLM upgrades, the CUDA driver matrix, the on-call. Even a fractional SRE (0.25–0.5 FTE) can rival the amortized hardware cost on a single node. This is the line item that quietly decides most build-vs-buy debates.
• Depreciation horizon. I use 3-year straight-line because that is the conservative planning convention. Stretch to 4–5 years (defensible — H100s are still earning in 2026) and the annual number drops 25–40%. Resale value is real but I would not bank on it.

Owned boxes have near-flat marginal token cost once filled. Empty boxes are just depreciation. The chart below folds power and colo into the owned rows; staffing is excluded everywhere because it varies too much to fake a number.

Net/net: at lab volume, APIs or a desk box usually win. At sustained production volume, an owned HBM or value Blackwell node beats frontier list pricing on cash — provided utilization is real, open-weight quality clears your bar, and your team can actually run the stack. Weaken any of those three assumptions and the crossover moves right, sometimes past the horizon. Rent sits in the middle for people who need HBM without a CapEx conversation yet.

For the portfolio framing (what to own vs rent as a strategy, not a SKU), see From Renting Tokens to Owning AI Assets and Running Your Own LLM OS.

Decision matrix

Close

The hardware decision is not “Mac vs NVIDIA vs AMD.” It is which constraint you are buying your way out of: capacity, bandwidth, concurrency, or boundary.

Desk machines made serious local inference normal. Datacenter nodes made it operational. The math did not change — weights plus KV still have to fit, and decode still has to drink from a fast enough memory bus. Get those two right, match the runtime to the silicon, and the CapEx conversation gets honest.

If you want the control-plane and portfolio half of this, start with rent-vs-own. If you want the enterprise stack taxonomy, start with the LLM OS modes series. This post is the silicon half.