9 min read

llama.cpp KV cache VRAM with Devstral 16 GB: from 0.33 to 19.3 tok/s

The machine was working: llama.cpp running Devstral with 16 GB of VRAM. The model loaded without error. The first generation came back correct in Portuguese.

And it was completely useless.

In this post I describe the hardware we built to run local LLM, explain why a model that fits on the SSD can idle the GPU at zero, show how the KV cache math decided which configuration stayed in production, and why one of the two available models has no structural solution on this hardware.

Measured symptom: 0.33 tok/s

On September 17, 2026, on the first afternoon of testing, the server loaded the Qwen3-30B-A3B model with a context of 8,192 tokens. The response came in Portuguese, correct, after many long seconds.

I measured: 0.33 tok/s generation. More than three seconds per token. A 400-token response would take 20 minutes.

The GPU monitor read approximately 0% utilization. The process was alive, the model loaded, and the card resting while the CPU waited.

The wrong hypothesis

The initial assumption was simple: a 12.9 GB model on a disk, GPU with 16 GB of VRAM. The math checked out. With the entire file on the card, the GPU should dominate generation.

I was wrong. What I did not account for was the KV cache.

Root cause: the KV cache math

Every language model with multi-head attention stores, for each token in context, the K and V vectors of every layer. Qwen3-30B-A3B has 40 transformer layers, 8 KV heads per layer, with internal dimension 128. In fp16 (two bytes per value), the cache for a single context position uses:

40 layers x 8 heads x 128 dimensions x 2 (K and V) x 2 bytes = 163,840 bytes per token.

With 8,192 tokens of context, the KV cache occupies 1.31 GB. The model weights already used 12.9 GB of the 16,311 MiB available. The sum exceeds the limit by roughly 400 MB. The NVIDIA driver spilled the overflow to system RAM with no error message.

The problem is not the model size. It is the KV cache.

The VRAM bandwidth of an RTX 5060 Ti is approximately 448 GB/s. DDR4 at 3200 MT/s in dual channel achieves approximately 40 GB/s at best. When the KV cache lands in RAM, the GPU fetches every attention vector through the system memory interface for each generated token. The direct result: GPU at approximately 0% and 0.33 tok/s generation, bottlenecked entirely by system RAM.

The data path nobody mentioned: PCIe

There is a second bottleneck that does not appear in model specifications: the PCIe interface.

The server uses PCIe 4.0 x8, which provides approximately 16 GB/s of bidirectional bandwidth. When the KV cache is in RAM and the GPU needs to access it at every attention layer, the path is: GPU requests data through the PCIe interface, the CPU fetches from RAM, returns through the same interface. With 448 GB/s of internal bandwidth and 16 GB/s of ingress, the GPU is limited by the doorway, not by the processing itself.

The llama.cpp log records the number of tensor splits when the model does not fit entirely on the GPU:

graph splits (tensors moved to host): N

With KV in VRAM: graph splits: 0. With KV in RAM: the value rises with context size. This field is the most direct indicator that the bottleneck is data transfer, not computation.

The fix: ctx 4096

Reducing context to 4,096 tokens brought the KV cache to 655 MB. Together with 12.9 GB of model weights, the total landed at 13.6 GB, below the 16,311 MiB available. The server allocated 16,002 MiB, leaving 309 MB of headroom.

The startup log confirmed:

INFO [server_context_init_load_model] llama_new_context_with_model:
  KV self size = 640.00 MiB, K (f16): 320.00 MiB, V (f16): 320.00 MiB
  graph splits (tensors moved to host): 0

No tensors moved to host. All computation happened on the GPU.

The proof: 19.3 tok/s

Same hardware, same model, same prompt, only the context changed from 8,192 to 4,096:

37.2 tok/s prompt processing and 19.3 tok/s generation. GPU at 99%.

The difference between 0.33 tok/s and 19.3 tok/s came entirely from fitting the KV cache inside VRAM.

Calculating KV for any model

The formula is straightforward. For any multi-head attention model:

bytes_per_token = n_layers x n_kv_heads x head_dim x 2 (K and V) x bytes_per_value

In fp16: bytes_per_value = 2. In q8_0: bytes_per_value = 1. In q4_0: bytes_per_value = 0.5.

The result, multiplied by the desired maximum context, is what the KV cache will occupy. Add that to the model weights to determine whether everything fits in VRAM.

For Devstral Small 2 24B (40 layers, 8 KV heads, dimension 128), in fp16, the calculation is identical: 160 KB per token. In q8_0, half that: 80 KB per token.

The structural dead end: Devstral on this hardware

Devstral Small 2 24B has weights of 14.6 GB in Q4_K_M. Available VRAM is 16,311 MiB (17.1 GB). That leaves 1.5 GB for the KV cache.

At 160 KB per token in fp16 with 1.5 GB available, the maximum context that fits in VRAM is approximately 9,375 tokens. This ignores CUDA context memory, activation buffers, and server overhead: in practice, nothing runs at that context without spilling to RAM.

The solution that works: KV quantized at q8_0, which reduces the cache to 80 KB per token. With ctx 12,288 and q8_0, the KV occupies 983 MB, and the total (weights plus KV) sits at approximately 15.6 GB, within the 16,311 MiB limit. That is exactly what the C3P configuration does, confirmed by tests on September 18, 2026: 15,097 MiB allocated, 26.3 tok/s with two queued calls.

Devstral only works on this hardware with quantized KV. That is not a configuration limitation, it is a memory calculation.

What the machine is

Hardware context: Ryzen 7 5700X (8 cores, 16 threads), 64 GB DDR4 at 3200 MT/s in dual channel, WD Green SN3000 1 TB SSD, Windows 10 Pro. RTX 5060 Ti 16 GB (16,311 MiB usable), driver 616.92, 180 W power limit. PCIe 4.0 x8 interface (approximately 16 GB/s bidirectional).

The llama.cpp server runs on build 11024. A PowerShell script reads a config file at boot and restarts the process within 5 seconds if it crashes. Access is through a private VPN.

The machine was barely one day old when these tests began: Windows installed on 09/16 at 20:49, working directory created on 09/17 at 09:50, first llama.cpp download on 09/17 at 10:04.

What it cost

Half a day of tests to discover that the configuration failure was not the model weights; it was the 160 KB of cache per context token. The llama.cpp documentation describes the parameters but does not work through the arithmetic of what happens when the total exceeds VRAM.

The second cost was understanding that RAM bandwidth, not just VRAM size, sets generation speed when any part of the KV cache escapes to system memory. The difference from 448 GB/s to 40 GB/s is not marginal: it is an 11x factor in peak sequential read throughput, and it shows up as CPU wait time on the GPU monitor.

The models on disk: Devstral Small 2 24B Q4_K_M (13.3 GB on disk, 14.6 GB after load) and Qwen3-30B-A3B UD-Q3_K_XL (12.9 GB). With the KV math for ctx 4096, Qwen3 fits with room. Devstral only fits with q8_0 KV quantization. That trade-off was the start of testing the configurations that followed.

Next post: three simultaneous requests that all returned HTTP 500 at the same moment, and how we went from 5.8 to 25.7 tok/s by changing only how the server distributes KV cache across slots.

Subscribe to the newsletter to follow the series.

Notes
[1] Measurement on 09/17/2026, supervisor.log and direct llama-server output. Hardware described from system readings on 09/18/2026.
[2] KV calculation: Qwen3-30B-A3B and Devstral Small 2 24B, both with 40 layers, 8 KV heads, dimension 128, fp16 (2 bytes). Model structure confirmed on 09/17/2026.
[3] Bandwidth: RTX 5060 Ti (~448 GB/s), DDR4 3200 MT/s dual channel (~40 GB/s), PCIe 4.0 x8 (~16 GB/s). Approximate figures from public specifications; no bandwidth benchmarks were run on this machine.
[4] C3P configuration: ctx 12288, KV q8_0, flash-attn on, 1 slot. 15,097 MiB allocated, 25.7 tok/s single call, 26.3 tok/s two queued calls. Measured 09/18/2026.