[Unsloth Studio] Prebuilt ROCm llama.cpp binary crashes with "out of memory" (hipStreamCreateWithFlags); source build of the same commit works fine

Prebuilt ROCm binary crashes with "out of memory" on hipStreamCreateWithFlags; source build of the same exact commit works fine

Summary

The prebuilt linux/rocm release asset for tag b11160-mix-a6922cc crashes on load with a ROCm "out of memory" error at the very first hipStreamCreateWithFlags call. Building llama-server from source at the exact same commit (119fa9dc43cd4d63e89a7b5d11f9378b84c52822), with a standard Nix-packaged ROCm 7.2.3 toolchain, works correctly and produces real generation output at ~121 tok/s. This points to something specific to how the prebuilt asset bundles/links its ROCm runtime, not a hardware or driver problem.

Environment

• OS: NixOS 26.05, kernel 7.2.7 (amdgpu driver in-tree, no dmesg/kfd errors at any point)
• GPU: AMD Radeon 680M (Rembrandt iGPU), reports as gfx1035. Not on AMD's or llama.cpp's officially validated gfx list, so HSA_OVERRIDE_GFX_VERSION=10.3.0 is set to present as gfx1030 (RDNA2 desktop) — without this the runtime refuses to initialize at all, which is expected/unrelated to this bug.
• rocminfo (from a Nix-packaged ROCm 7.2.3 SDK) correctly enumerates the device as gfx1030, 12 CUs, reports a 13.6GB GLOBAL/COARSE_GRAINED memory pool.
• ulimit -l (memlock): tested at both the OS default (8MB, soft+hard) and after raising it to 8GB via security.pam.loginLimits + full reboot. No difference in behavior — ruled out as the cause.
• No amdgpu/kfd-related messages in journalctl -k at the time of the crash — the failure appears to be purely userspace (inside the bundled HIP/HSA runtime), not a kernel/driver rejection.

Steps to reproduce (prebuilt asset — crashes)

1. Install via unsloth studio update (this pulls unslothai/llama.cpp@b11160-mix-a6922cc, which for this platform resolves to a prebuilt binary bundling its own libamdhip64.so.7, libhsa-runtime64.so.1, libhipblas.so.3, etc. directly alongside the binary).
2. Run:
   bash复制

   ./llama-server -m <any GGUF, tested with bartowski/Qwen2.5-0.5B-Instruct-GGUF Q4_K_M> \
     --port 38999 -ngl 99

3. Observe:
   复制

   ggml/src/ggml-cuda/ggml-cuda.cu:134: ROCm error
   ROCm error: out of memory
     current device: 0, in function stream at ggml/src/ggml-hip/../ggml-cuda/common.cuh:1561
     hipStreamCreateWithFlags(&streams[device][stream], 0x01)

   Crash is immediate, before any real model-memory allocation. -ngl value doesn't matter; happens the same with -ngl 1.

Steps to reproduce the fix (source build — works)

git clone --branch b11160-mix-a6922cc https://github.com/unslothai/llama.cpp.git src
# ROCm 7.2.3 toolchain from nixpkgs (rocmPackages: clr, hipcc, rocm-cmake, rocminfo,
# rocm-comgr, rocm-device-libs, rocm-runtime, hipblas(+common/lt), rocblas, rocsolver)
cmake -B build -S src -DGGML_HIP=ON -DAMDGPU_TARGETS=gfx1030 -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_C_COMPILER=gcc -DCMAKE_CXX_COMPILER=g++ \
  -DCMAKE_HIP_COMPILER=<rocm-toolchain>/bin/clang
cmake --build build -j16 --target llama-server llama-quantize

(Note: had to force gcc/g++ for host C/C++ — using the ROCm-vendored clang for host code too hits an unrelated mm_malloc.h/glibc posix_memalign noexcept conflict on this glibc version, orthogonal to the GPU issue.)

Running the resulting binary with the exact same flags Studio uses (--parallel 4 --flash-attn on --no-context-shift -c 32768 -ngl -1 --fit off --metrics --kv-unified --jinja --spec-default --load-mode none) against the same model works correctly:

model loaded
listening on http://127.0.0.1:38997

and a real /v1/chat/completions request returns a valid completion at 121.6 tok/s (predicted_per_second in the response timings).

What this suggests

Since it's the same source commit, the difference must be in how the prebuilt release asset was built/linked (possibly a different exact ROCm runtime version than the 7.2.3 SDK used here, different AMDGPU_TARGETS set baked into the prebuilt multi-arch "mix" bundle, or a build/link flag difference specific to the release pipeline). Given the "mix" naming in the release tag, this may be specific to how the multi-architecture prebuilt bundle is assembled for gfx1030/RDNA2 targets — possibly not exercised in that pipeline's own testing since gfx1035 (and the override-to-gfx1030 path) is an unofficial combination to begin with, but worth checking whether genuine gfx1030 hardware (RX 6600/6700/etc.) hits the same prebuilt-vs-source discrepancy, since that would point to a real packaging bug rather than something override-specific.

What I'm not certain about

• Whether this reproduces on officially-supported gfx1030 hardware (I only have the override case to test).
• The exact ROCm runtime version bundled in the prebuilt asset (I didn't extract/inspect it beyond ldd showing local libamdhip64.so.7 / libhsa-runtime64.so.1 files bundled next to the binary, vs. our source build linking against the Nix-packaged ROCm 7.2.3 versions of the same).

Happy to run any additional diagnostics against the prebuilt asset if useful — I already have both binaries side by side for comparison.