8 min read

llama-server HTTP 500 context size has been exceeded: shared KV buffer and the fix

I sent three concurrent requests to the llama-server. All three failed together, 246 seconds after starting. None returned a partial result. The process did not crash, the model did not hang. The server kept responding to new requests as if nothing had happened.

The error was "Context size has been exceeded". And the root cause was in none of the three requests individually.

Measured symptom: 0/3 with HTTP 500 at 246 s

Earlier tests measured the server with isolated calls. The next step was concurrency: three parallel requests, representing a real workload.

The production prompt I had estimated at approximately 3,000 tokens was measured: 4,362 tokens actual (with an expected response of 1,738 tokens). With that real number, three simultaneous calls would consume roughly 13,000 tokens of context at the same time.

Result: 3 out of 3 returned HTTP 500 “Context size has been exceeded” after 246 seconds.[1] An isolated call with the same configuration measured 5.84 tok/s. Concurrent calls: zero deliveries.

The wrong hypothesis

The first read was “the server ran out of memory”. The second was “the model is too slow for three concurrent calls”.

Both were wrong. The process kept running. The GPU did not saturate. The problem was not the hardware.

Root cause: the shared KV buffer

llama-server, without explicit parallelism configuration, opens 4 context slots. When the --no-kv-unified flag is not set, the server creates a single KV buffer shared across all slots. The total size of that buffer in the default configuration was 16,384 tokens.

With three requests of 4,362 prompt tokens each, plus tokens being generated in response, the shared buffer was exhausted before any of the three finished. The server cannot partially free space to let one request finish before another. When the buffer overflows, all in-flight requests receive the same error at the same moment.

It is like three people writing on the same fixed-size notepad. It does not matter that each requested a separate section: if the total exceeds the pad, nobody finishes.

Why continuous batching does not fix this

The llama-server has continuous batching enabled by default: the server can process tokens from multiple slots in parallel within a single inference step, making use of the GPU for several requests at once. This improves utilization when calls have different sizes.

What continuous batching does not resolve is the physical limit of the KV buffer. Intelligent token batching does not allocate more memory: it distributes computation better within what has already been allocated. If the total in-flight tokens exceed the buffer, the batching only engages after the exhaustion has already happened.

The practical distinction: continuous batching is a throughput optimization when space exists. The unified KV configuration is the allocation policy for that space. The two settings are independent.

Three configurations, three outcomes

I tested three approaches with the same hardware and the same model (Devstral Small 2 24B):

Original config: 4 automatic slots, 16,384-token unified KV, KV in RAM.
Isolated call: 5.84 tok/s. Three concurrent: 0/3 (HTTP 500 at 246 s).

Config C2: --parallel 3 --no-kv-unified, 9,216 tokens per slot (3 x 3,072).
Each slot has its own fixed context buffer, no sharing. Isolated call: 7.93 tok/s. Three concurrent: 3/3 OK, 2.15 tok/s each, 615 s total.[1]

Config C3P: 1 slot, ctx 12288, KV quantized q8_0 on GPU (no --no-kv-offload).
With KV on VRAM at reduced precision, 1 slot of 12,288 tokens uses 983 MB instead of the 1.97 GB fp16 would require. Single call: 25.7 tok/s, complete in 72 s. Two queued: 2/2 OK, 26.3 tok/s. VRAM allocated: 15,097 of 16,311 MiB.[1]

The command that summarizes the difference:

# original config (fails with 2+ long concurrent calls)
llama-server --model devstral.gguf --ctx-size 16384

# C3P (current production)
llama-server --model devstral.gguf --ctx-size 12288 --slots 1 \
  -ctk q8_0 -ctv q8_0

How to size ctx to prevent overflow

The direct lesson is that the KV buffer size must be calculated from the actual size of calls, not from estimates.

The process: measure the real production prompt before configuring the server. In this workload, the initial estimate was 3,000 tokens; the actual measurement returned 4,362. That 45% difference would have left any buffer sized from the estimate undersized from the start.

With the real size measured, the correct sizing for N slots is:

ctx_per_slot = p95_of_prompt_plus_response
ctx_total = ctx_per_slot x N_slots  (unified KV)
ctx_total = ctx_per_slot             (per-slot KV, --no-kv-unified)

For a server with variable load, using p95 instead of p50 as the baseline is the difference between a buffer that sometimes overflows and one that does not under the vast majority of conditions.

The memory math

Devstral Small 2 24B has 40 layers, 8 KV heads, dimension 128. In fp16, the cache uses 160 KB per token. In q8_0, half: 80 KB. For a context of 12,288 tokens: 80 KB x 12,288 = 983 MB in q8_0, versus the 1.97 GB fp16 would require.

That near-gigabyte difference allowed the KV cache to fit in VRAM alongside the model weights. And KV in VRAM means the GPU does not have to fetch attention vectors from RAM for each token.

The proof: from 5.84 to 25.7 tok/s

The difference between the original config and C3P:

Original config, isolated call: 5.84 tok/s (KV in RAM, zero concurrency headroom).
C3P, isolated call: 25.7 tok/s (quantized KV on GPU, 1 slot, queue).

The honest caveat: the difference from 5.84 to 7.93 tok/s (original vs C2) might reflect another GPU consumer during measurement, not the config change alone. The jump from 7.93 to 25.7 tok/s comes from moving KV from RAM to GPU, which is the structural difference.

What C3P sacrificed: true concurrency. With 1 slot, calls are serialized. The second call waits for the first to finish. For workloads where individual response latency matters more than simultaneous throughput, this is a real trade-off. For batch workloads where completion without loss is what matters, serialization at high speed beats slow concurrency.

What it cost

The production prompt estimated at approximately 3,000 tokens had 4,362. The cost of using the estimate instead of measuring: discovering the concurrency bug only when three real calls failed simultaneously.

The second lesson: the error "Context size has been exceeded" does not report which request caused the overflow. With a shared buffer, all requests receive the same error. Without knowing the buffer was shared, the obvious reading is “one of the calls was too long”, not “two short calls summed to exhaust the common pool”.

Measuring the actual prompt size before testing concurrency would have avoided the cycle of wrong hypotheses. The measurement takes one command:

llama-server --model model.gguf --tokenize --tokens-per-line 0 < prompt.txt | wc -l

Next post: the model that writes 16 em-dashes per response, how to identify which BPE vocabulary tokens contain the character, and how to ban them at the logit level.

Subscribe to the newsletter to follow the series.

Notes
[1] Measurements on 09/18/2026, raw data in bench_results.jsonl and lab_run.log. Configurations tested on the same hardware (RTX 5060 Ti 16 GB), same model (Devstral Small 2 24B Q4_K_M), same real production prompt (4,362 tokens). The caveat on 5.84 vs 7.93 tok/s is documented in the source report.
[2] C3P configuration: ctx 12288, KV q8_0, flash-attn on, 1 slot. VRAM 15,097 of 16,311 MiB. Measurement 09/18/2026 21:03-21:05.