Baike.dev
Log in
Back to tool/Back to issues
#12994·ipex-llm

llama_load_model_from_file: failed to load model with SYCL/Level Zero on Intel Arc B580 GPU

Author: duhmojoCreated Mar 23, 2025Updated Dec 31, 2025
Labelsuser issue

Describe the bug When running the Ollama server with an Intel Arc GPU (B580) using SYCL/Level Zero, the model fails to load with the error:

llama_load_model_from_file: failed to load model
panic: unable to load model: /root/.ollama/models/blobs/sha256-04778965089b91318ad61d0995b7e44fad4b9a9f4e049d7be90932bf8812e828

The logs indicate that the model metadata is loaded successfully, but the actual model tensors fail to load with an Input/output error. GPU offloading is attempted, but the process ultimately crashes.

llama_load_model_from_file: using device SYCL0 (Intel(R) Graphics [0xe20b]) - 11605 MiB free
llama_model_loader: loaded meta data with 20 key-value pairs and 325 tensors from /root/.ollama/models/blobs/sha256-04778965089b91318ad61d0995b7e44fad4b9a9f4e049d7be90932bf8812e828 (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
...
llm_load_tensors: offloading 32 repeating layers to GPU
llm_load_tensors: offloading output layer to GPU
llm_load_tensors: offloaded 33/33 layers to GPU
llm_load_tensors:        SYCL0 model buffer size =  1456.19 MiB
llm_load_tensors:    SYCL_Host model buffer size =    70.31 MiB
time=2025-03-23T10:12:40.886+08:00 level=INFO source=server.go:605 msg="waiting for server to become available" status="llm server loading model"
llama_model_load: error loading model: read error: Input/output error
llama_load_model_from_file: failed to load model
panic: unable to load model: /root/.ollama/models/blobs/sha256-04778965089b91318ad61d0995b7e44fad4b9a9f4e049d7be90932bf8812e828

How to reproduce I'm basically following this guide: https://syslynx.net/llm-intel-b580-linux/

My container run props aren't special:

bash
podman run  -itd \
            --device=/dev/dri\
            --name=intel-llm \
            -v /srv/llm/ollama/models:/root/.ollama/models \
            -e no_proxy=localhost,127.0.0.1 \
            -e OLLAMA_HOST=0.0.0.0 \
            -e DEVICE=Arc \
            -e HOSTNAME=intel-llm \
            -e OLLAMA_NUM_GPU=999 \
            -e ZES_ENABLE_SYSMAN=1 \
            -e SYCL_DEVICE_FILTER=level_zero:gpu \
            --network ml_shared \
            --restart=always \
            docker.io/intelanalytics/ipex-llm-inference-cpp-xpu:latest \
            sh -c 'mkdir -p /llm/ollama && cd /llm/ollama && init-ollama && exec ./ollama serve'

I've trying to use open-webui as in the guide but it results in a 500, with the above logs in the intel-llm.

I tried running a model manually, but I must be doing something wrong. Maybe the default pathing is incorrect for ollama to be run like this:

$ podman exec -it intel-llm ollama/ollama list
ggml_sycl_init: found 1 SYCL devices:
NAME                                      ID              SIZE      MODIFIED
phi:latest                                e2fd6321a5fe    1.6 GB    55 minutes ago

$ podman exec -it intel-llm ollama/ollama run phi:latest "test" --verbose
ggml_sycl_init: found 1 SYCL devices:
Error: llama runner process has terminated: error loading model: read error: Input/output error
llama_load_model_from_file: failed to load model

# just to confirm the container envs passed in:
$ podman exec -it intel-llm env
...
SYCL_DEVICE_FILTER=level_zero:gpu
SYCL_CACHE_PERSISTENT=1
OLLAMA_NUM_GPU=999
DEVICE=Arc
ZES_ENABLE_SYSMAN=1
...

Environment information

My environment is an Ubuntu 24.10 VM with the 6.13.7 kernel and raw PCI B580 GPU pass through.

Host OS: Proxmox 8.3.5 w/ 6.11 kernel and Intel driver blacklisted GPU: Intel Arc B580 12GB

Container Image: docker.io/intelanalytics/ipex-llm-inference-cpp-xpu:latest Ollama Version: 0.5.4-ipexllm-20250320 Environment Variables: ZES_ENABLE_SYSMAN=1 SYCL_DEVICE_FILTER=level_zero:gpu OLLAMA_NUM_GPU=999

The model metadata is loaded successfully, but the actual model tensors fail to load with an Input/output error. GPU offloading is attempted, but the process crashes before completing. The issue persists even after re-downloading the model and verifying the model files.

$ ls -al /dev/dri/
crw-rw----+  1 root video  226,   0 Mar 22 19:03 card0
crw-rw----+  1 root video  226,   1 Mar 22 19:03 card1
crw-rw----+  1 root render 226, 128 Mar 22 19:03 renderD128

# uname -a
Linux 7aab3474ddb9 6.13.7-061307-generic #202503131244 SMP PREEMPT_DYNAMIC Fri Mar 14 02:34:39 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

# Ubuntu VM:
$ cat /etc/os-release
PRETTY_NAME="Ubuntu 24.10"
NAME="Ubuntu"
VERSION_ID="24.10"
VERSION="24.10 (Oracular Oriole)"

# CONTAINER: docker.io/intelanalytics/ipex-llm-inference-cpp-xpu:latest
# cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.5 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.5 LTS (Jammy Jellyfish)"

# sycl-ls
[level_zero:gpu][level_zero:0] Intel(R) oneAPI Unified Runtime over Level-Zero, Intel(R) Graphics [0xe20b] 20.1.0 [1.6.32224.500000]
[opencl:cpu][opencl:0] Intel(R) OpenCL, 13th Gen Intel(R) Core(TM) i5-13400 OpenCL 3.0 (Build 0) [2024.18.12.0.05_160000]
[opencl:gpu][opencl:1] Intel(R) OpenCL Graphics, Intel(R) Graphics [0xe20b] OpenCL 3.0 NEO  [24.52.32224.5]

To confirm, the Intel(R) oneAPI Unified Runtime over Level-Zero, Intel(R) Graphics [0xe20b] lines up with the lspci listed Intel Corporation Battlemage G21 [Intel Graphics] [8086:e20b] 

$ sudo lspci -vnn | grep -i vga -A 12
[sudo] password for user:
00:02.0 VGA compatible controller [0300]: Device [1234:1111] (rev 02) (prog-if 00 [VGA controller])
...
00:10.0 VGA compatible controller [0300]: Intel Corporation Battlemage G21 [Intel Graphics] [8086:**e20b**] (prog-if 00 [VGA controller])
	Subsystem: Intel Corporation Device [8086:1100]
	Physical Slot: 16
	Flags: bus master, fast devsel, latency 0, IRQ 38
	Memory at 7000000000 (64-bit, non-prefetchable) [size=16M]
	Memory at 7400000000 (64-bit, prefetchable) [size=16G]
	Expansion ROM at 000c0000 [disabled] [size=128K]
	Capabilities: [40] Vendor Specific Information: Len=0c <?>
	Capabilities: [70] Express Endpoint, IntMsgNum 0
	Capabilities: [ac] MSI: Enable+ Count=1/1 Maskable+ 64bit+
	Capabilities: [d0] Power Management version 3
	Kernel driver in use: xe
	Kernel modules: xe

As requested I tried running the nv-check.sh in the container, but some things are missing from it. 

root@7aab3474ddb9:~# ./env-check.sh
-----------------------------------------------------------------
PYTHON_VERSION=3.11.11
-----------------------------------------------------------------
/usr/local/lib/python3.11/dist-packages/transformers/utils/generic.py:441: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
  _torch_pytree._register_pytree_node(
transformers=4.36.2
-----------------------------------------------------------------
torch=2.2.0+cu121
-----------------------------------------------------------------
ipex-llm Version: 2.2.0b20250320
-----------------------------------------------------------------
IPEX is not installed.
-----------------------------------------------------------------
CPU Information:
Architecture:                         x86_64
CPU op-mode(s):                       32-bit, 64-bit
Address sizes:                        39 bits physical, 48 bits virtual
Byte Order:                           Little Endian
CPU(s):                               6
On-line CPU(s) list:                  0-5
Vendor ID:                            GenuineIntel
Model name:                           13th Gen Intel(R) Core(TM) i5-13400
CPU family:                           6
Model:                                183
Thread(s) per core:                   1
Core(s) per socket:                   6
Socket(s):                            1
Stepping:                             1
BogoMIPS:                             4992.00
Flags:                                fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology cpuid tsc_known_freq pni pclmulqdq vmx ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch cpuid_fault ssbd ibrs ibpb stibp ibrs_enhanced tpr_shadow flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves avx_vnni arat vnmi umip pku ospke waitpkg gfni vaes vpclmulqdq rdpid movdiri movdir64b fsrm md_clear serialize flush_l1d arch_capabilities
Virtualization:                       VT-x
-----------------------------------------------------------------
Total CPU Memory: 15.117 GB
Memory Type: sudo: dmidecode: command not found
-----------------------------------------------------------------
Operating System:
Ubuntu 22.04.5 LTS \n \l

-----------------------------------------------------------------
Linux 7aab3474ddb9 6.13.7-061307-generic #202503131244 SMP PREEMPT_DYNAMIC Fri Mar 14 02:34:39 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
-----------------------------------------------------------------
./env-check.sh: line 148: xpu-smi: command not found
-----------------------------------------------------------------
./env-check.sh: line 154: clinfo: command not found
-----------------------------------------------------------------
Driver related package version:
ii  intel-level-zero-gpu                             1.6.32224.5                             amd64        Intel(R) Graphics Compute Runtime for oneAPI Level Zero.
ii  intel-level-zero-gpu-legacy1                     1.3.30872.22                            amd64        Intel(R) Graphics Compute Runtime for oneAPI Level Zero.
ii  level-zero-devel                                 1.20.2                                  amd64        oneAPI Level Zero
-----------------------------------------------------------------
igpu not detected
-----------------------------------------------------------------
xpu-smi is not installed. Please install xpu-smi according to README.md

Additional context Add any other context about the problem here.

Source: intel/ipex-llm

View original on GitHubView discussion on GitHub