6 min read

The patch that logged “patched” and never ran

6 min read

Two incidents. Same root cause. The first one took down the company for three hours in July. The second revealed that the installed fix was a placebo with a reassuring log: the code was there, it logged a confirmation, and the memory arena kept growing without a ceiling.

Here I’ll tell you how ONNX Runtime’s BFCArena silently grew until it crashed everything with an OOM, why the first patch didn’t work despite printing “patched” in the logs, and what measurement proves the correct fix worked.

Measured Symptom

On July 16, 2026, at 18:00, the AI agent orchestration service started leaking memory. The rate reached 737 MB per minute.

The service is the pipeline running on port 4090: it receives dispatch requests, executes the agents, and manages the state of each ongoing run. The point of failure was the memory_recall_v2 route, responsible for semantic search in the company’s fact index. Each call to this route triggered the fastembed library to generate the query vector and compare it with the index.

In less than 20 minutes from the first alerts, the kernel activated the OOM killer. The pipeline process died. All agents with open tasks lost their connection to the dispatcher. The dispatch system went blind: without the process, no agent could return results, update its state, or create child tasks. The company came to a halt.

The Wrong Hypothesis

The working hypothesis was clear and wrong: the startup log said "[brain_server] onnxruntime patched: BFCArena bounded", so the arena was disabled. The server had rebooted cleanly and remained stable during initial tests.

The hypothesis was wrong because the patch checked if so is None before creating a SessionOptions with a disabled arena. The fastembed never passes sess_options=None: it provides its own SessionOptions with the arena enabled. The condition was never true, the _Bounded was instantiated, the log appeared, and the SessionOptions from fastembed reached the runtime intact. The arena grew normally.

Root Cause

The onnxruntime uses an internal allocation arena called BFCArena (Best-Fit with Coalescing). The default behavior has two aspects that, when combined, are fatal for long-running processes:

First, the arena grows in powers of two as demand increases: 256 MB, 512 MB, 1 GB, 2 GB. Each jump is irreversible.

Second, the arena never returns what it allocated to the operating system. If an inference needs 1 GB of arena, that gigabyte remains reserved for the process for the rest of its life, regardless of how many other inferences need less.

With continuous calls to memory_recall_v2, the arena grew in exponential steps. Each call that needed a larger block forced a new power-of-two jump. The previous block remained reserved. The next jump reserved double the amount. In less than 20 minutes, 12 GB were exhausted.

The patch that logged “patched” and never ran

Once BFCArena was identified as the culprit, I applied a monkey-patch to onnxruntime before importing fastembed. The idea was to replace InferenceSession with a subclass that would force SessionOptions with a disabled arena:

import onnxruntime as _ort
_orig = _ort.InferenceSession
class _Bounded(_orig):
    def __init__(self, *args, **kwargs):
        so = kwargs.get("sess_options")
        if so is None:   # o bug estava aqui
            so = _ort.SessionOptions()
        so.enable_cpu_mem_arena = False
        so.enable_mem_pattern = False
        kwargs["sess_options"] = so
        super().__init__(*args, **kwargs)
_ort.InferenceSession = _Bounded

The startup log confirmed it: "[brain_server] onnxruntime patched: BFCArena bounded". The server rebooted cleanly. The service was stable during initial tests.

Seven days later, the leak returned.

I measured the VmRSS of the process running embeddings without any external load: 902 MB growing to 2.57 GB in 5 minutes. The patch was installed, it logged “patched”, and the arena had never been disabled in any real inference. Over this period, the service accumulated 308 restarts.

The reason was the if so is None condition. The fastembed never passes sess_options=None: it provides its own SessionOptions with the arena enabled. The patch would intercept the call, see that so was not None, and skip the disabling logic completely.

The Correct Fix

The correct fix removes the condition and forces the arena to be disabled regardless of what the caller passed:

class _Bounded(_orig):
    def __init__(self, *args, **kwargs):
        so = kwargs.get("sess_options")
        if so is None:
            so = _ort.SessionOptions()
        # força SEMPRE, ignorando o que o fastembed passou
        so.enable_cpu_mem_arena = False
        so.enable_mem_pattern = False
        so.intra_op_num_threads = 1
        so.inter_op_num_threads = 1
        kwargs["sess_options"] = so
        super().__init__(*args, **kwargs)

Along with the corrected patch, I activated a kill-switch via an environment variable: PEI_DISABLE_LOCAL_EMBED=1 configured in the systemd service. With the kill-switch active, the recall route disables local embedding and degrades to a pure FTS5 search. We have 316 thousand facts indexed in FTS5, and the text search is sufficient for normal operation while the remote embedding is unavailable.

The monkey-patch was chosen over an isolated subprocess because the subprocess would add initialization latency per worker: unacceptable for synchronous lookups with a response SLA. The kill-switch (PEI_DISABLE_LOCAL_EMBED=1) is plan B: when local embedding is disabled, the system degrades to pure FTS5 without loss of functionality for the 316 thousand indexed facts.

The Proof

The signal is the process’s VmRSS. Without measuring this at runtime, the patch “works” on paper, and no test detects the failure. The "patched" log is not a signal: it’s text that confirms a function ran, not that the effect was produced.

Acceptance criteria: run at least 200 real embeddings and measure the VmRSS at three points in time. Approval criterion: the delta between the middle point and the final point must be less than 100 MB. A growing RSS with the patch installed means the condition is not executing.

for i in 1 2 3; do
  grep VmRSS /proc/$(pgrep -f brain_server)/status
  sleep 30
done

With the wrong patch: linear growth across the three points, from 902 MB to 2.57 GB.

With the correct patch: stabilization between the second and third points, delta below 80 MB.

The question that concludes any proof: “If the patch were broken, would my test have caught it?” With VmRSS at three points and 200 real embeddings, the answer is yes. With only the “patched” log, the answer is no.

References

ONNX Runtime: performance tuning and threads: official documentation on SessionOptions, including enable_cpu_mem_arena.
onnxruntime/bfc_arena.cc: implementation of BFCArena in the official repository.
qdrant/fastembed: library that instantiates InferenceSession with its own SessionOptions.
proc(5): Linux manual: reference for VmRSS in /proc/PID/status.
tensorflow/bfc_allocator.h: origin of the BFCAllocator pattern, from which ONNX Runtime derived its implementation.