Baike.dev
Log in
Back to tool/Back to issues
#18396·ollama

Jetson Orin Nano 8GB: Gemma 4 E4B OOM with --load-mode dio, while identical configuration succeeds without DIO

Author: ch40sCreated Sep 11, 2026Updated Sep 16, 2026
Labelsbug

What is the issue?

Jetson Orin Nano 8GB: Gemma 4 E4B multimodal projector causes host OOM, despite successful CPU-projector configuration

Description

On an NVIDIA Jetson Orin Nano Super 8GB with unified CPU/GPU memory, Ollama 0.34.0 cannot load a Gemma 4 E4B multimodal model because the multimodal projector is offloaded to CUDA.

The underlying llama-server can successfully run the exact same model on the same hardware when the multimodal projector is kept on CPU using --no-mmproj-offload. However, Ollama does not add --no-mmproj-offload in this configuration, even after reducing the scheduler's reported GPU memory using OLLAMA_GPU_OVERHEAD.

The result is that Linux's OOM killer terminates llama-server.

Hardware

  • NVIDIA Jetson Orin Nano Super 8GB
  • 7,607 MiB physical unified memory
  • Compute capability: 8.7
  • CUDA / JetPack 6
  • Jetson Linux R36.5.2
  • Kernel variant: oot
  • Docker with NVIDIA runtime

Ollama

  • Ollama: 0.34.0
  • Docker image: ollama/ollama
  • OLLAMA_MAX_LOADED_MODELS=1
  • OLLAMA_NUM_PARALLEL=1
  • Context: 4096

Model

hf.co/HauhauCS/Gemma-4-E4B-Uncensored-HauhauCS-Aggressive:Q2_K_P

Model characteristics:

  • Architecture: Gemma 4
  • Parameters: ~7.52B
  • Quantization: Q2_K
  • Context length: 131072
  • Layers: 42
  • Embedding length: 2560
  • Separate multimodal projector
  • Projector size: ~945 MiB on disk
  • Ollama/llama-server estimated worst-case projector memory: 1156.32 MiB

Ollama failure

With the default configuration, Ollama starts llama-server with:

--model ... \
--mmproj ... \
-c 4096 \
-np 1 \
...

but does NOT include:

--no-mmproj-offload

The scheduler reports approximately:

system memory total="7.4 GiB"
gpu memory ... available="5.5 GiB" free="6.0 GiB"
estimated worst-case memory usage of mmproj is 1156.32 MiB

It then loads:

offloaded 43/43 layers to GPU

and the Linux kernel OOM killer terminates the process:

Out of memory: Killed process ... (llama-server)

Ollama reports:

Error: 500 Internal Server Error: llama-server process has terminated: signal: killed

Attempted OLLAMA_GPU_OVERHEAD workaround

I tested OLLAMA_GPU_OVERHEAD to see whether making the scheduler more conservative would cause Ollama to disable projector offload.

At 2 GiB:

gpu memory ... available="4.0 GiB" free="6.5 GiB" overhead="2.0 GiB"

Ollama still:

offloaded 43/43 layers to GPU

and did not add --no-mmproj-offload.

At 3 GiB:

gpu memory ... available="3.0 GiB" free="6.5 GiB" overhead="3.0 GiB"

Ollama again:

offloaded 43/43 layers to GPU

and did not add --no-mmproj-offload.

Both configurations ended with the Linux OOM killer terminating llama-server.

Important control experiment: llama-server works with CPU projector

I manually tested the same Ollama-bundled llama-server binary with the exact same model and CUDA libraries.

The following configuration works reliably:

--gpu-layers 43
--ctx-size 4096
--batch-size 512
--ubatch-size 512
--mmproj <projector>
--no-mmproj-offload

In other words:

43/43 Gemma layers on CUDA + multimodal projector on CPU works.

Observed results:

  • ~13.8 tokens/sec generation
  • GPU utilization up to ~99%
  • RAM ~7.4 GB
  • Stable inference
  • No Linux OOM kill

This strongly suggests that the issue is not that the model is fundamentally too large for the device. The problem is specifically the placement of the multimodal projector.

Expected behavior

On a shared/unified-memory NVIDIA Jetson Orin Nano, Ollama should ideally recognize that CPU and CUDA allocations consume the same physical memory pool and avoid GPU offloading of the multimodal projector when doing so would cause host memory exhaustion.

At minimum, I would expect one of the following:

  1. Ollama automatically selects:
--no-mmproj-offload

while retaining the Gemma layers on CUDA; or

  1. Ollama exposes a model/runtime option allowing the projector to remain on CPU; or

  2. Ollama's scheduler correctly accounts for the projector's unified-memory impact before deciding that the model fits.

Relevant log excerpt

time=2026-09-11T15:47:25.459-07:00 level=INFO source=sched.go:613 msg="system memory" total="7.4 GiB" free="7.1 GiB" free_swap="11.4 GiB"

time=2026-09-11T15:47:25.459-07:00 level=INFO source=sched.go:620 msg="gpu memory" id=0 library=CUDA available="3.0 GiB" free="6.5 GiB" minimum="457.0 MiB" overhead="3.0 GiB"

srv load_model: [mtmd] estimated worst-case memory usage of mmproj is 1156.32 MiB

common_memory_breakdown_print:
| - CUDA0 (Orin) | 7607 = 5594 + (2216 = 2006 + 104 + 106) + -203 |

load_tensors: offloading output layer to GPU
load_tensors: offloading 41 repeating layers to GPU
load_tensors: offloaded 43/43 layers to GPU

time=2026-09-11T15:47:39.018-07:00 level=INFO source=sched.go:641 msg="Load failed" ... error="llama-server process has terminated: signal: killed"

Additional context

There are existing Gemma 4 / Jetson issues and discussions around memory usage and multimodal projector offloading.

This appears particularly relevant to Jetson Orin Nano because it uses unified memory rather than a conventional discrete GPU with dedicated VRAM.

I am happy to provide additional diagnostics, including:

  • full Ollama debug logs
  • tegrastats output during model loading
  • ollama show --verbose
  • Jetson/JetPack information
  • Docker configuration
  • kernel OOM logs
  • results from the successful --no-mmproj-offload control experiment

Question

Is this expected behavior for Jetson/unified-memory CUDA devices, or should Ollama's scheduler automatically select CPU projector offload in this situation?

If automatic selection is not currently supported for Jetson Orin Nano, would exposing --no-mmproj-offload as an Ollama model/runtime option be feasible?

Relevant log output

time=2026-09-11T15:47:25.459-07:00 level=INFO source=sched.go:613 msg="system memory" total="7.4 GiB" free="7.1 GiB" free_swap="11.4 GiB"

time=2026-09-11T15:47:25.459-07:00 level=INFO source=sched.go:620 msg="gpu memory" id=0 library=CUDA available="3.0 GiB" free="6.5 GiB" minimum="457.0 MiB" overhead="3.0 GiB"

srv load_model: [mtmd] estimated worst-case memory usage of mmproj is 1156.32 MiB

common_memory_breakdown_print:
| - CUDA0 (Orin) | 7607 = 5594 + (2216 = 2006 + 104 + 106) + -203 |

load_tensors: offloading output layer to GPU
load_tensors: offloading 41 repeating layers to GPU
load_tensors: offloaded 43/43 layers to GPU

time=2026-09-11T15:47:39.018-07:00 level=INFO source=sched.go:641 msg="Load failed" ... error="llama-server process has terminated: signal: killed"

OS

Linux

GPU

Nvidia

CPU

Other

Ollama version

0.34.0

Source: ollama/ollama

View original on GitHubView discussion on GitHub