8 min de leitura

fastembed onnxruntime memory leak: 737 MB/min until OOM, and the patch that never ran

Two fastembed onnxruntime memory leak incidents, same root cause. The first brought the company down for three hours in July. The second revealed that the fix I had installed was a placebo with a reassuring log: the code was there, confirmation messages were printed, and the memory arena kept growing without a ceiling.

Here is how the ONNX Runtime BFCArena silently consumed memory until the OOM killer fired, why the first patch did nothing despite printing “patched” in the logs, and which measurement rule proves the correct fix actually worked.

Symptom measured

On July 16, 2026, at 6 PM, the AI agent orchestration service started leaking memory. The rate peaked at 737 MB per minute.[1]

The service is the AI orchestration pipeline: it receives dispatch requests, runs agents, and manages the state of each active run. The failure point was the semantic search endpoint, which performs search over the company knowledge base. Each call to that endpoint triggered the fastembed library to generate a query vector for comparison against the indexed facts.

In under 20 minutes from the first alerts, the Linux OOM killer fired. The pipeline process died. All agents with open tasks lost their connection to the dispatcher. The dispatch system went dark: with the process gone, no agent could return a result, update state, or create child tasks. The company stopped.[1]

The wrong hypothesis

The operating hypothesis was straightforward and wrong: the startup log said "[brain_server] onnxruntime patched: BFCArena bounded", so the arena was disabled. The server had restarted cleanly and stayed stable in the initial tests.

The hypothesis was wrong because the patch checked if so is None before creating a SessionOptions with the arena disabled. fastembed never passes sess_options=None: it delivers its own SessionOptions with the arena enabled. The condition was never true, the _Bounded instance was created, the log appeared, and the fastembed SessionOptions reached the runtime unchanged. The arena kept growing normally.[3]

Root cause

The onnxruntime library uses an internal allocation arena called BFCArena (Best-Fit with Coalescing). Its default behavior has two aspects that, combined, are fatal in long-running processes:

First, the arena grows in powers of two as demand increases: 256 MB, 512 MB, 1 GB, 2 GB. Each jump is irreversible.

Second, the arena never returns memory to the operating system. If an inference needs 1 GB of arena, that gigabyte stays reserved for the life of the process, regardless of how little subsequent inferences need.

fastembed is a high-level wrapper over onnxruntime. Internally it creates an InferenceSession while passing in its own SessionOptions. That SessionOptions from fastembed has BFCArena enabled by default, which is the runtime default behavior.

With continuous calls to the semantic search endpoint, the arena grew in exponential steps. Each call that required a slightly larger block forced a new power-of-two jump. The previous block stayed reserved. The next jump reserved twice as much. In under 20 minutes: 12 GB exhausted.[1]

The detail that destroyed the first fix: fastembed always passes its own SessionOptions to InferenceSession. It never leaves the parameter absent.[3]

The patch that logged “patched” and never ran

With BFCArena identified as the culprit, I applied a monkey-patch to onnxruntime before importing fastembed. The idea was to replace InferenceSession with a subclass that forced a SessionOptions with the arena disabled:

import onnxruntime as _ort

_orig = _ort.InferenceSession
class _Bounded(_orig):
    def __init__(self, *args, **kwargs):
        so = kwargs.get("sess_options")
        if so is None:   # <<< the bug was here
            so = _ort.SessionOptions()
        so.enable_cpu_mem_arena = False
        so.enable_mem_pattern = False
        kwargs["sess_options"] = so
        super().__init__(*args, **kwargs)
_ort.InferenceSession = _Bounded

The startup log confirmed: "[brain_server] onnxruntime patched: BFCArena bounded". The server restarted cleanly. The service stayed stable in initial tests.

Seven days later, the leak returned.[2]

I measured VmRSS while running embeddings with no external load: 902 MB growing to 2.57 GB in 5 minutes.[3] The patch was installed, logging “patched”, and the arena had never been disabled in any real inference. Over that period, the service accumulated 308 restarts.[3]

The reason was if so is None. fastembed never passes sess_options=None: it delivers its own SessionOptions with the arena enabled. The patch intercepted the call, saw that so was not None, and skipped the disable logic entirely. The _Bounded instance was created, but the fastembed SessionOptions reached the runtime unchanged.[3]

The correct fix removes the conditional and forces the arena off regardless of what the caller passed:

class _Bounded(_orig):
    def __init__(self, *args, **kwargs):
        so = kwargs.get("sess_options")
        if so is None:
            so = _ort.SessionOptions()
        # force ALWAYS, ignoring what fastembed passed
        so.enable_cpu_mem_arena = False
        so.enable_mem_pattern = False
        so.intra_op_num_threads = 1
        so.inter_op_num_threads = 1
        kwargs["sess_options"] = so
        super().__init__(*args, **kwargs)

Alongside the corrected patch, I activated a kill-switch via environment variable: PEI_DISABLE_LOCAL_EMBED=1 set in the systemd service. With the kill-switch active, the recall endpoint disables local embedding and degrades gracefully to pure FTS5 search. We have 316,000 facts indexed in FTS5 and textual search is sufficient for normal operations while remote embedding is unavailable.[4]

The measurement rule that proves it

The signal is the process VmRSS. Without measuring this at runtime, the patch “works” on paper and no test detects the failure. The log "patched" is not a signal: it confirms that a function ran, not that the intended effect was produced.

Acceptance rule: run at least 200 real embeddings and measure VmRSS at three spaced points in time. Acceptance criterion: the delta between the midpoint and the final point must be below 100 MB. Growing RSS with the patch installed means the condition is not executing.[5]

for i in 1 2 3; do
  grep VmRSS /proc/$(pgrep -f brain_server)/status
  sleep 30
done

With the wrong patch: linear growth across all three points, from 902 MB to 2.57 GB.

With the correct patch: stabilization between the second and third points, delta below 80 MB.

The question that closes any proof: “if the patch were broken, would my test have caught it?” With VmRSS at three points and 200 real embeddings, the answer is yes. With only the “patched” log, the answer is no.

What it cost

The first incident took the service down for 3 hours on July 16 (6 PM to 9 PM).[1] Agents with open tasks lost dispatcher connectivity. Some tasks required manual intervention to restore state.

The second episode, seven days later, revealed the fix had been installed and inoperative.[2] During that period, every semantic search call ran with an unbounded arena. The service was silently accumulating OOM risk. Only the VmRSS measurement surfaced the problem: 902 MB growing to 2.57 GB in 5 minutes with no external load, with 308 restarts accumulated.[3]

The lesson now in the company documentation: confirmation log is not proof of effect. The proof is the RSS number measured before and after, with real load, at spaced points in time. A number that grows with the patch installed is a condition that is not executing.

Today, any import of fastembed or onnxruntime in a shared process requires the monkey-patch with the corrected condition plus VmRSS verification as a deploy prerequisite. Without that rule, the code looks safe and behaves like a time bomb waiting for the next traffic spike.

Subscribe to the newsletter to receive the next posts in this series with logs and numbers included, no hype.

More posts on AI agent engineering on this blog.