MLX engine: weights are unwired ~2 s after each request on macOS 27, so the first request after idle pages them back in

What is the issue?

On the MLX engine, a loaded model's weights stay wired only while a request is running. About two seconds after each request macOS releases them, and they become ordinary pageable memory. On a machine under any memory pressure they are then compressed or swapped, and the next request waits for them to come back. This happens with OLLAMA_KEEP_ALIVE=-1 and with load_duration near zero, so the model looks loaded the whole time.

The runner means for the weights to stay wired. After the weights load, configureWiredMemory in mlxrunner/runner.go sets MLX's wired limit to the active memory. The limit is set, but on current macOS it does not keep the memory wired between requests.

Measurements

Mac mini M4 Pro, 64 GB, macOS 27.0 (26A428), Ollama 0.34.2 (MLX 0.32.2-61-gd9add9d), qwen3.6:35b-a3b-mxfp8 on the MLX engine, OLLAMA_KEEP_ALIVE=-1.

System-wide wired memory (vm_stat, "Pages wired down") around one /api/chat request:

The same request (native /api/chat, 57-token prompt, num_predict: 8, think: false), first after about four idle hours and then warm:

Timed more finely with MLX alone (M2, macOS 27.2, sampled every 0.25 s), the whole allocation is released at once, 2.1–2.4 s after the last GPU work, in each of three runs. A tiny GPU op every 1 s keeps it wired; every 3 s does not.

It is still the case on main as of 1 October: configureWiredMemory sets the limit once at load, and nothing renews residency afterwards. From 0.40.0-rc0, models run on MLX by default on Apple Silicon. So this now reaches default installs, not only people who chose an MLX model. #16030 may be related: under memory pressure, weights that are no longer wired are what macOS compresses.

Cause

The cause is in MLX's residency handling rather than in Ollama's code. MLX makes one standing requestResidency() when it creates a residency set (mlx/backend/metal/resident.cpp) and never renews it, and macOS collects idle residency about 2 s after the last GPU work. It reproduces with MLX alone, without Ollama: ml-explore/mlx#4609.

llama.cpp met the same thing and keeps its residency sets alive with a heartbeat (ggml-org/llama.cpp#17766, in ggml/src/ggml-metal/ggml-metal-device.m). The heartbeat requests residency for every set until three minutes after the last compute, and GGML_METAL_RESIDENCY_KEEP_ALIVE_S changes the duration. So on the same machine, the GGUF engine keeps weights wired between requests and the MLX engine does not.

Workaround

sudo sysctl iogpu.disable_wired_collector=1. With it set, wired memory stayed at 37.3–37.4 GiB for three minutes after a request, where before it fell back to 2 GiB within seconds. It helps a lot but does not finish the job:

• First request after 33 idle minutes: prompt evaluation took 431 ms against 26 ms warm. Before the setting, it was 3,044 ms.
• Residency still drifts: wired memory had fallen from 37.3 to 29.9 GiB over those 33 minutes. It can still be lost under memory pressure, and only the next request makes it resident again. A heartbeat that re-requests residency would recover this in the background, which the sysctl cannot.

It resets at reboot, and it applies to the whole machine: GPU memory that any process has wired is never reclaimed. That is acceptable on a dedicated inference machine, but it is not a general recommendation.

Suggested fix

The fix that belongs in MLX is llama.cpp's: call requestResidency() on each residency set about once a second, for as long as the memory should stay wired. I checked this against Metal directly on macOS 27. The test mirrors MLX's setup: one residency set with a standing request, attached to the command queue. Holding 4 GiB wired, then idle for 60 s:

Residency is per process: a second process running its own heartbeat did not keep the first one's memory wired. So the runner has to do this itself; a companion process cannot. Until MLX renews residency, the runner can keep its sets in use by running a trivial op on its GPU stream about once a second while a model is loaded and idle.

The repro in ml-explore/mlx#4609 sets the wired limit the way configureWiredMemory does and shows the same thing with MLX alone. A tiny GPU op every 1 s keeps the memory wired, and one every 3 s does not, so the interval has to stay inside the ~2 s window.

Tying the heartbeat's lifetime to the model's keep-alive would match what llama.cpp does with its three-minute default.

Separately, configureWiredMemory logs its result at debug level. At the default level the log says nothing about what was wired, and an info line with the limit would have made this quicker to find.

Relevant log output

level=INFO source=pipeline.go:101 msg=memory peak="43.35 GiB" held="43.27 GiB"

There is no wired-memory line at INFO.

OS

macOS 27.0

GPU

Apple

CPU

Apple

Ollama version

0.34.2