Vulkan compute output corrupted on Imagination/PowerVR with opt.lightmode=1 blob recycling — allocator bookkeeping proven correct by internal tracing during a corrupted inference; barriers and submit boundaries ruled out

error log | 日志或报错信息 | ログ

No error is reported anywhere — that is part of the problem. load_param, load_model and every extract() return 0, no validation/driver message, but the extracted output of a RIFE v4.7 frame-interpolation net is corrupted ("melted"): frames mathematically unrelated to the two input frames (our automated oracle: a blended frame farther from BOTH endpoints than they are from each other — impossible for any convex weighting of the inputs; ratios 2.9x–5.4x the endpoint separation).

The same net, same weights, same inputs:

• ncnn CPU path: bit-clean, always.
• ncnn Vulkan on ARM Mali / Qualcomm Adreno (6 devices in the field): clean.
• ncnn Vulkan on Imagination PowerVR (3 GPU generations, incl. DXT-48 / Pixel 10 Pro, vendorID 0x1010): corrupted, reproducibly.
• Same device, opt.lightmode=0: always clean (2/2) — at ~1.8x time and ~18x peak memory, as expected with no blob recycling.

So: PowerVR + lightmode=1 blob recycling is the triggering combination.

context | 编译/运行环境 | バックグラウンド

• ncnn tag 20260526, static lib, NCNN_VULKAN=ON, built with the release workflow's own CMake options (also reproduced with the official ncnn-20260526-android-vulkan.zip prebuilt — identical corruption), NDK 27.2.12479018, arm64-v8a, Android app.
• Device: Google Pixel 10 Pro (Tensor G5), GPU Imagination PowerVR DXT-48, VkPhysicalDeviceProperties::vendorID = 0x1010, Android 17 (google/blazer/blazer:17/CP2A.260805.005/15828068, ro.hardware.vulkan=powervr). Also observed on two older PowerVR generations on real-user devices.
• Net: RIFE v4.7 (583 layers / 752 blobs), buffer storage mode (use_image_storage off), fp32 (use_fp16_* off), elempack=1 on every dispatch (measured), 1920x1088 inputs, one custom layer (rife.Warp, warp/grid-sample — exonerated below). rough_score() on this device = 23 → pending-dispatch threshold 1M, which auto-flushes ~9-10 submit_and_wait() per extraction (measured; relevant to what is ruled out below).

how to reproduce | 复现步骤 | 再現方法

1. Any PowerVR-Vulkan Android device (vendorID 0x1010). Build a deep net whose blob lifetimes force VkBlobAllocator sub-buffer recycling (hundreds of transient blobs), opt.lightmode=1, buffer storage, fp32.
2. Run the same input through the CPU path and the Vulkan path; diff.
3. Vulkan output is corrupted; CPU is the reference. opt.lightmode=0 on the same device is clean, which isolates the trigger to blob recycling.

We can share our full deterministic repro (app + clip + forced inputs) privately if useful; we have not yet reduced it to a standalone ~60-line .param chain — happy to do so if a maintainer wants to pursue this. What a maintainer without a PowerVR device would most need from us is runs of instrumented builds — we can run any patch on the affected device and return traces.

more | 其他 | その他

We spent several sessions trying to localize this INSIDE ncnn before concluding the defect is most likely in the PowerVR driver's handling of a legal-per-spec pattern ncnn's allocator produces. Evidence, each item established by an on-device experiment (not argument):

1. Allocator bookkeeping is CORRECT — proven by internal tracing DURING two corrupted inferences (the strongest artifact here; traces + patch in the evidence gist below). We built ncnn from source at the pin with instrumentation inside VkBlobAllocator::fastMalloc/fastFree (patch in the evidence gist): a shadow interval set of live hand-outs that flags any hand-out overlapping a live range, plus free-list invariant validation (self-overlap, free-vs-live overlap) after every mutation, plus per-release VkBufferMemory::refcount logging at net.cpp's lightmode release sites. Result over two renders (~15,700 hand-outs, ~440 recycled ranges and ~570 refcount>1 releases per extraction): zero anomalies — no overlap-with-live, no free-list corruption, no double-free — in BOTH runs, and both runs' outputs were corrupted (allocdiag_cut173_melted.log and allocdiag_cut64_melted.log in the gist). Every recycled range was handed out only after its final refcount-0 free. So the earlier suspicion that nested sub-buffer reuse (allocator.cpp budget merging) was aliasing live data is REFUTED.

2. It is not the "erased hazard state" / missing-barrier issue either. allocator.cpp:723-724 stamps recycled hand-outs access_flags=0 / stage_flags=TOP_OF_PIPE (empty first sync scope for the first barrier_readwrite). We tested a VkBlobAllocator subclass stamping conservative flags (SHADER_READ|SHADER_WRITE / COMPUTE_SHADER) on every allocation, proven active by logging: output bit-identical to baseline (identical corruption ratios to 3 significant figures). command.cpp:2193-2194 already writes those values after first use, so steady-state barriers are real — and making the first-use barrier real too changes nothing.

3. It is not a synchronization-timing race ncnn could fix with more submits. Explicit submit_and_wait(); reset(); inserted at 16 points per extraction (after each Warp) still produces corrupted output (bounded "ghosting" variant). Note the device also auto-flushes ~9-10 submit_and_wait() per extraction anyway (rough_score cadence, net.cpp:250-272) — corruption survives real fence-waited submission boundaries.

4. Layer-level suspects exonerated. The custom Warp layer's GPU-vs-CPU self-test matches to 2.4e-06; forcing ONLY Warp to CPU (ncnn's own download/upload path) is clean — but that inserts full submit+wait+copy, i.e. it removes the recycling exposure, not a Warp bug. use_packing_layout=false corrupts bit-identically (pack conversions exonerated). fp16 fully off.

5. Corruption has race-like signatures. A "delay reuse by K allocations" allocator experiment changed WHICH content was corrupted and made output nondeterministic across runs (from ghosting on one input to full-frame noise on another) — the signature of re-rolling a scheduling race, not of a functional bug. The corruption itself is deterministic per (build, input) — reproducible on demand.

Summary: with lightmode recycling ON, PowerVR intermittently mis-executes a dispatch stream in which sub-ranges of one VkBuffer are freed and re-bound as storage buffers across dispatches — while ncnn's bookkeeping, barriers, and even hard submission boundaries are demonstrably in order. lightmode=0 (no recycling) is the only clean configuration at GPU speed. We currently ship with ncnn-Vulkan denied on vendorID 0x1010 as a mitigation.

Possible prior art: #5990 ("NCNN has different results based on CPU vs vulkan") — intermittent corrupt Vulkan output, fp16 ruled out by the reporter, closed unresolved; consistent with this class.

We are filing a parallel report with Imagination. From ncnn's side, the actionable options we see: (a) a documented vendor workaround (e.g. treat vendorID 0x1010 as lightmode-unsafe or add an option to disable sub-buffer recycling without retaining every blob), or (b) help reducing this to a standalone repro the driver team can run. We're glad to run instrumented builds on the affected device.

---

Evidence (public gist): https://gist.github.com/di2g/a3788246430458f8bad13c9308f18adf — instrumentation patch, both full allocator traces (each from a corrupted inference, zero anomalies), trace line-format key, and app-log excerpts proving the GPU path executed.