[错误][CUDA] 具有前导零扩展的张量编译成功,但生成的内核在第一次 StreamSync 时会出现 cudaErrorIllegalAddress 错误
作者: lackjava-creator创建于 2026年9月6日更新于 2026年9月6日
标签type: bugneeds-triage
Environment
- TVM version:
main @ 48242ec33403f2b6e4fac6e763ca7a683fb9d5df(2026-09-03 21:20:55 -0400, reports0.26.dev0); also present on release0.26.0,0.19.0and0.12.0. - Build/install: source build (
/home/lxx/tvm-main),USE_CUDA=ON, nvcc 12.2, LLVM 15 - OS/Python: Ubuntu 24.04.3 LTS, x86_64 (Xeon E5-2698 v4), Python 3.10
- GPU/driver: Tesla V100-DGXS-32GB, driver 535.309.01,
CUDA_VISIBLE_DEVICES=0, sm_70 - Target:
cuda.opt_level3 and 0 both fail.llvmis clean. - Every run is one case per fresh process —
cudaErrorIllegalAddressis sticky for the life of a CUDA context. - Pipeline coverage (
relax.get_pipeline("default_build")performs no operator fusion, so all three are reported separately). Verified 2026-09-06 onmain @ 48242ec, CUDA, one pipeline per fresh process (acudaErrorIllegalAddresspoisons the context for the life of the process):get_pipeline("default_build")(no FuseOps/FuseTIR) — fails, 2/2 fresh processesget_pipeline("zero")(fuses) — fails, fresh process- forced
FuseOps+FuseTIRappended to the legalize/fold sequence — fails, fresh process
Minimal reproducer
Single Neg node with input X : float32[0]. model.onnx + feed.npz + run.py attached (a MaxPool variant, model_maxpool_alt.onnx + feed_maxpool_alt.npz, is included to show the trigger is the shape, not the operator).
import numpy as np, onnx, tvm
from tvm import relax
from tvm.relax.frontend.onnx import from_onnx
model = onnx.load("model.onnx"); feed = dict(np.load("feed.npz")) # X: float32[0]
mod = from_onnx(model, shape_dict={k: list(v.shape) for k, v in feed.items()}, keep_params_in_input=False)
order = [i.name for i in model.graph.input]
with tvm.transform.PassContext(opt_level=3):
ex = tvm.compile(relax.get_pipeline("default_build")(mod), target="cuda") # succeeds
dev = tvm.cuda(0)
out = relax.VirtualMachine(ex, dev)["main"](*[tvm.runtime.tensor(np.ascontiguousarray(feed[n]), dev) for n in order])
print(np.asarray(out.numpy())) # <-- faults hereExpected vs actual
- Expected: an empty
float32tensor of shape(0,).numpy.negative(numpy.zeros((0,), numpy.float32))is an empty array; onnxruntime returns the same; the identical program on TVM'sllvmtarget returns the same. - Actual:
tvm.compilesucceeds and the VM call succeeds; the failure surfaces at the first synchronisation:
CUDA Runtime Error: cudaErrorIllegalAddress (700)
cuda_device_api.cc:279 CUDADeviceAPI::StreamSync
<- runtime/tensor.cc:104 Tensor::CopyToBytesAfter this the whole CUDA context is poisoned; every later allocation in the same process fails.
- Because the fault is asynchronous, a validator that never copies the empty result back to the host will report a pass. (This is why an earlier bisect of ours wrongly recorded a "(0.12, 0.19] regression" — the 0.12 run never …
内容来源: apache/tvm